Custom Probes
Custom Probes let you define your own testing rules and datasets. They help you validate target behavior that built-in probes may not cover.
They generate prompts and variations from your inputs. They mark results as pass/fail based on your expected behavior.
These probes help you assess:
Adherence to allowed and banned behaviors (policy-style rules)
Regression detection using internal adversarial datasets
Accuracy and consistency for custom Q&A and critical facts
Custom Probe
What to Expect
This probe tests your target against rules you define. The Platform will generate and send relevant prompts (including variations) to verify that the chatbot consistently follows the allowed behaviors and avoids the banned behaviors, flagging responses that violate your expectations.
Probe ID: >100'000
Why It Matters
Custom Probe lets you validate safety and security requirements that may not be covered by built-in probes. It helps uncover policy gaps, unintended capabilities (e.g., over-permissive actions, sensitive data exposure, unauthorized commitments), and reduces the risk of incidents, compliance violations, and reputational harm.
Configuration Inputs
List Of Allowed Behaviors - Describe what your chatbot is allowed to do in one sentence (e.g., “The chatbot may provide general product information and guide users through support steps without requesting sensitive credentials.”).
List Of Banned Behaviors - Describe what your chatbot is not allowed to do in one sentence (e.g., “The chatbot must not request or reveal passwords/tokens, provide abuse instructions, or promise actions that require authorization.”).
Custom Dataset
What to Expect
This probe uses your uploaded CSV dataset as the source of attacks/test cases. For each entry, the Platform can generate additional variants (languages, paraphrases, strategies), run them against the target, and evaluate responses using either regex-based detection or an LLM-based evaluator.
Probe ID: >100'000
Why It Matters
The Custom Dataset Probe is ideal when you already have internal test suites or domain-specific scenarios (e.g., proprietary jailbreak attempts, compliance checks, organization-specific abuse patterns). It scales your testing with broader coverage, more realistic automated adversarial pressure, and faster regression detection.
Configuration Inputs
Provide a custom dataset - Provide a custom dataset in a CSV file. An example CSV file is available on the Platform under the “Upload your custom dataset” field.
Probe intent - Describe the goal of the attacks for this probe and the purpose of the dataset (e.g., “Test whether users can extract internal policies/configuration or coerce the bot into disallowed actions.”).
Attack Specifications (On-Domain and/or Adversial) - Specify whether the dataset is on-domain (legitimate domain questions and edge cases) and/or adversarial (jailbreaks, prompt injection, social engineering, policy evasion), and what behaviors you are targeting.
Languages - Select the languages in which the Platform should generate/execute variants to verify consistent behavior across locales.
Attack Variations - Define which transformations to apply.
Attack Strategies - Choose the adversarial strategies to apply.
Depth - Set how deep the probe goes in a conversation (i.e., number of turns/steps); higher depth simulates more persistent, realistic attacks.
Attack Multiplier - Set how many attack variants to generate per dataset entry; higher values increase coverage at the cost of more requests.
Custom Q&A Probe
What to Expect
This probe tests whether your application answers the questions you define correctly. The Platform will reformulate your questions, send them to the target, and evaluate whether the response content matches the correct answers you provided in the CSV.
Probe ID: >100'000
Why It Matters
Custom Q&A Probe helps validate accuracy and consistency for critical business information (e.g., FAQ, policies, product facts). It’s useful for tracking regressions after model/prompt/knowledge-base changes and for measuring answer quality over time.
Configuration Inputs
Upload your Q&A - Provide a Q&A in a CSV file. An example CSV file is available on the Platform under the “Upload your Q&A” field.
Company Name - Enter the name of the company that owns the target application.
Last updated
Was this helpful?