Probe Settings

After connecting your target to the SplxAI Platform, you can enable probes to be executed during test runs. These probes help identify potential vulnerabilities in the target (Figure 1).

Figure 1: Probe Configuration Tab

All probes are designed to provoke and detect a specific vulnerability.

Each probe is assigned a risk priority based on the selected target type. A higher risk priority indicates that any identified vulnerabilities have a greater impact on the target’s overall risk surface, displayed on the Overview page. The default risk priority can be adjusted in the optimization dialog.

Probes are divided into four major categories.

Probe Categories

Prompt Injection

Prompt Injection attacks generative AI systems by manipulating the input prompts to alter the chatbot's behavior. The objectives of these attacks may include:

• Leaking sensitive data

• Spreading misinformation

• Causing other forms of harm

Off-Topic

Off-Topic probes assess a language model's tendency to deviate from its intended function or context. These probes evaluate the model’s ability to stay on its intended topic and avoid irrelevant responses. By analyzing responses to off-topic prompts, you can:

• Gain insights into the model’s behavior.

• Identify areas for system prompt's improvement.

Hallucination

Hallucination probes test the limits of a generative AI model by encouraging it to produce fictional, nonsensical, or inaccurate information. These tests help you assess:

• The model’s trustworthiness.

• Robustness.

• Adherence to factual accuracy.

Social Engineering

Social Engineering probes evaluate a generative AI application's vulnerability to manipulative prompts designed to exploit trust or extract sensitive information. These probes help you assess:

• The applications's susceptibility to manipulation.

• Its ability to recognize and resist malicious or deceptive inputs.

• Potential risks to user safety and data security.

Enabling a Probe

Beside each probe, there is a toggle button that enables it. The toggle opens a optimization dialog with input fields that help you tailor the probe to your application's needs, making it domain-specific improving the relevance and realism of the simulated attacks (Figure 2).

The example of the probe's optimization can be seen in the provided example.

Figure 2: Probe Optimization Dialog

Fake News Optimization Explained

• Risk Priority

  • Potential risk based on the severity and likelihood of exploitation occurring, used to calculate the risk surface of your application.

• Company Name

  • Enter the name of your company.

• Services

  • List the products and services that your company offers.

• Fake News Topics

  • Provide a list of fake news topics you want to test your application against. Probe will attempt to prompt your chatbot to generate fake content related to these topics in order to evaluate its resistance to producing misinformation.

Clicking the "Save and Enable Probe" button stores the probe configuration and enables the probe on the given target.

To later edit the configuration of an already optimized probe, click the gear icon in the corresponding row.

Probe Details

Once you find a probe that you are interested in you can click the "Details" button to view its description including probe category, probe ID, supported modes (text, image, voice, document), and the cost of probe run in credits.

Figure 3: Probe Details

Running the Probes

After connecting the target to the SplxAI Platform and selecting and configuring at least one of the available probes, you can start your first test run.

For assistance with your first test run, please visit the Test Run page.