• Max Tokens - This parameter specifies the absolute maximum number of tokens that model can generate and return in the response.

Welcome to the SplxAI Platform documentation.

This comprehensive guide serves as a user manual to assist you in onboarding with our platform. It offers detailed explanations of key concepts, terms, and functionalities, ensuring you have all the information you need. Whether you are new to platform or seeking in-depth insights, this documentation is designed to support your understanding and enhance your experience with our platform.

The SplxAI Platform

Platform is designed to protect your generative AI applications from harmful activities. It tests your application using a customizable set of probes, automated red teaming tools, designed to trigger and detect specific vulnerabilities. Each probe targets a particular vulnerability, allowing for comprehensive security and safety testing. The SplxAI Platform also provides relevant mitigation strategies for detected issues.

Probes use generative AI to create attacks based on a comprehensive attack database, which is collected and constantly updated from various LLM CTFs, open-source data, and both automated and manual AI research. Each probe applies different variations and strategies depending on the target’s industry, company, and goals, in order to maximize the security and safety assessment.

Documentation Sections

The documentation is divided into four main sections.

Platform - Serves as a user manual for the platform, explaining key concepts and providing a step-by-step guide for effective usage. It helps users understand and make the most out of the platform's features.

Platform API - A detailed reference for developers, including endpoints and request and response formats. This section ensures smooth integration with the platform's APIs.

Updates - A dedicated space for periodic announcements about new features, improvements, and updates, keeping you informed about the latest developments.

Links - Useful resources, such as our blog for the latest news on AI security and SplxAI GitHub.

Join us on Slack & Discord

The first step in adding your target to the SplxAI Platform is setting up the connection between them. With SplxAI Platform, you can observe how your application performs across various layers, from the LLM to the platform level, which simulates real user interactions.

Start by selecting your connection type based on your use case.

Selecting a Connection Type

The following integration methods are currently supported:

• API: REST API connection between your GenAI application and the SplxAI Platform.

• Platform: Test runs are executed on chatbots that are accessible through external platforms (e.g., Slack, WhatsApp, Glean). Probe uses the platform’s APIs to interact with the chatbots.

• LLM: Tests are executed directly on the Large Language Model.

• LLM Development Platform: The SplxAI Platform connects with the APIs provided by LLM development platforms.

Connection Configuration

Once you’ve selected the appropriate connection type, a configuration tab will appear on the next step, prompting you to input the required connection details. These inputs will be specific to the type of connection you’ve chosen, such as API keys, phone numbers, or endpoint URLs.

Connection Test

Once all the required information is entered, click the “Continue” button. A connection test between the SplxAI Platform and your target will run automatically in the background. The result of this test will be displayed in a dialog. You can proceed with the remaining configuration step once the connection test is successful.

To initiate automated testing with probes, you must first add your target. To do so, open the workspace & target drop-down menu. Once expanded, click the "Add Target +" button (Figure 1).

After clicking the button, the Add Target page will appear (Figure 2). On this page, you can choose your connection type and configure your target for automated testing.

Adding a target consists of three steps:

For detailed instructions on connecting your target with the SplxAI Platform, please proceed to the next page.

The Target of the testing is your generative AI application, specifically designed for conversational interactions. This application, whether a chatbot used internally within your organization or a public-facing platform for customer engagement, undergoes automated testing to identify potential vulnerabilities.

The testing will involve a variety of AI-generated attack scenarios to evaluate the application's resilience and security. This is achieved by selected set of probes that perform specific security assessments.

Configuring the Target

The final step before enabling the probes is providing the details and capabilities of your target (Figure 1). To do this, fill in all the input fields with your target’s information.

• Target Name

  • The name of your target that will be displayed within the SplxAI Platform.

• Target Environment

  • Field for tracking the various stages of your target (Development, Production, and Staging).

• Target Description

  • A brief description of your application’s purpose and use case.

• Language

  • The default language of your application. Red teaming attacks will be generated in this language.

• Target Type

  • You can choose from four target types, depending on whether your application is publicly available or internal, and whether it uses Retrieval-Augmented Generation (RAG) or not.

    • Public With RAG

    • Public Without RAG

    • Private With RAG

    • Private Without RAG

  • Target type determines the default risk priorities for each probe, which are used to calculate your overall risk surface.

• Rate Limit

  • The maximum number of requests your application can process per minute.

• Parallel Requests

  • Turned on by default.

  • Toggle off to have the SplxAI Platform send requests to your target one at a time.

• Modes Supported

  • This specifies the types of input your application can process, with text set as the default.

  • SplxAI Platform also supports attacks through uploaded images, voice, and documents.

  • If your application can handle any of these inputs and you want to test it against multimodal attacks, select the relevant modes.

Saving the Target

After entering all the required inputs, your target can be saved, and you can proceed to configure the probes. Target is saved by clicking the "Save" button.

• URL

  • Supported Azure OpenAI endpoints (protocol and hostname)

  • The general format for an endpoint is: https://{your-resource-name}.openai.azure.com.

  • For example: https://yourcompany.openai.azure.com

  • Endpoint is also found on the Keys and Endpoint page.

• API Key - Your application's Dify API Key. Obtain the API key in the Dify Platform by navigating to the API Access section in the left-side menu. Here, you can manage the credentials required to access the API.

• AWS Access Key ID & AWS Secret Access Key

  • These can be created and accessed via the AWS Management Console.

  • In the Security credentials tab, you can create a new key or view existing Access Key IDs.

  • Note that AWS Secret Access Keys are only shown during creation.

• API Key

  • From the navigation bar, open the Endpoints page and select the Serverless endpoints tab.

  • Open your endpoint from the list.

  • Copy the Key and insert it into the Probe integration input field.

• URL - On the same serverless endpoint where you got the API key, the required URL can be found in the Target URI field. Copy this URL and insert it into the Probe integration input field.

• URL

Available connection types between SplxAI Platform and the target:

• API
• PLATFORM
• LLM
• LLM Development Platform

Target Settings page allows you to make the changes on the selected target. All fields are configurable, the only exception is the connection type, which cannot be edited.

Connection Configuration

Whenever you make changes to your application that require an connection update, you can do so in the “Configure Connection” tab. Once you decide to save your changes, the test connection process will begin. You’ll only be able to save your changes once the connection is successfully established.

Target Configuration

If you need to do the updates on the target's configuration, such as modifying the base language for your chatbot, make the necessary changes and click the "Save" button.

After connecting your target to the SplxAI Platform, you can enable specific probes to be executed during test runs. These probes help identify potential vulnerabilities in the target (Figure 1).

All probes are specifically designed to provoke and detect a particular vulnerability.

Probes are divided into four major categories.

Enabling a Probe

Beside each probe, there is a toggle button that enables it. The toggle opens a optimization dialog with input fields that help you tailor the probe to your application's needs, making it domain-specific and enhancing the relevance and realism of the simulated attacks (Figure 2).

The example of the probe's optimization can be seen in the provided example.

Clicking the "Save and Enable Probe" button stores the probe configuration and enables the probe on the given target.

To later edit the configuration of an already optimized probe, click the gear icon in the corresponding row.

Probe Details

Once you find a probe that you are interested in you can click the "Details" button to view its description including probe category, probe ID, supported modes (text, image, voice, document), and the cost of probe run in credits.

Running the Probes

After connecting the target to the SplxAI Platform and selecting and configuring at least one of the available probes, you can start your first test run.

With Probe, you can run automated tests directly on your WhatsApp chatbot. To do so, you first need to set up the integration. In the “Select your integration” tab, choose WhatsApp as the integration type, and proceed to the next step to access the configuration fields.

• Number

  • Enter the number of your WhatsApp chatbot.

  • Make sure to follow the international phone number format as indicated in the placeholder.

• Conversation Reset Message

  • Specify the message that resets your chatbot’s existing conversation.

  • This is the message that you defined that triggers your chatbot’s initial starting message and restarts the conversation in the same chat.

  • It is highly recommended to use this option as it allows Probe to run tests without starting a large numbers of new chats, thereby increasing execution speed.

• Initial Messages

  • The initial message is the message with which your chatbot starts the conversation. This typically includes a question about the user’s conversation preferences.

  • The Question and Answer fields help Probe initiate the conversation with your chatbot using the preferences you select.

  • Example:

    • Question: Hi! Welcome to the SplxInsurance chatbot! Please select your language to continue the conversation: English, German, Italian.

    • Answer: English

  • If there are multiple initialization messages, for instance, if after selecting a language the user must choose a product of interest, you can add multiple question-and-answer combinations.

Test Connection

Once all the information is filled in, your WhatsApp connection will be tested when you click “Continue” to proceed. The test connection will send a message to your WhatsApp chatbot and mark it as passed if the chatbot responds.

After successfully establishing the connection, you can proceed to configure the target fields.

Bot ID

On the My bots page, select your desired bot.

Once on the bot’s page, click on Settings.

Copy the Bot handle value and paste it into the Bot ID input on Probe.

Bot Secret

On the bot’s page, click Edit in the Direct Line row.

Copy the Secret key and paste it into the Bot Secret input on Probe.

Once you’ve entered the required inputs, click Continue to test your connection and proceed.

Getting Slack User Bot ID

To pentest your Slack bot with Probe, you’ll need to provide the bot’s ID in the Slack User Bot ID field:

1. On Slack's home screen, search for Apps

2. Under Apps, click on your bot's name to open its profile.

3. Open the details and find Member ID and click the copy button next to it

4. Paste this ID into the Slack User Bot ID field within the Probe platform

Selecting Communication Type

Slack integration operates in two modes.

1. Direct message

   • In this mode Probe sends messages to channels without any modifications (like adding a mention tag at the beginning). When a message is sent, Probe waits for the target to respond inside the same channel.

1. Mention

   • In this mode Probe mentions the target in messages sent to the channel, and continues the conversation inside the thread. Messages inside the thread do not mention the target.

Installing Integration Bot to Workspace

To install Slack integration to your workspace, click the Connect new workspace button. You will then be prompted to log in to your Slack workspace. Once logged in, you will be asked to authorize the Probe integration for your workspace, with a detailed overview of the permissions the app will receive upon installation.

Once finished, to view your newly added workspace, click the refresh button next to the workspace dropdown on the Probe platform.

Creating Channels

To enable integration to chat with your bot, you will need to create the private channels. In each created channel you should then add your target and Probe integration. You can do this by following these steps:

1. Create channels without adding any additional users (we recommend creating at least 4 channels)

2. Add your target by typing commands: /invite @YourAppName and /invite @SplxAI Probe, or you can follow the steps from Figure 6.

3. Optional: Mute notifications for created channels

After creating the channels, go to the Channels section on Slack’s home screen. Open your private channels, click on Details, and locate the Channel ID. Copy this ID and paste it into the Channels field in the Probe integration tab (similar to obtaining the Slack User Bot ID).

Loading Messages

While your Slack bot is generating its response, various loading messages may be displayed to the user (e.g., “Please wait, generating response…”). To ensure Probe ignores these messages and waits for the bot’s final response, enter the regular expression (regex) for the constant part of the loading message in the Loading Messages section. For example:

• Loading message: “Please wait, generating response…”

• Regex: ^Please wait, generating response.*$

You can enter multiple loading messages that should be ignored by Probe.

How Does It Work?

When you start your Probe, Slack integration will look through all private channels, of which it is part, and it will select all channels containing targets. When the backend starts sending messages integration will distribute them across created channels, and wait for your bot to respond.

For each test run, you can download a shareable PDF report. This report provides details about your test target, along with an overview and summary of the test run. It includes general information such as the start time, execution time, total number of test cases, and more. Additionally, it features the combined results and a chart of all probe runs, offering a clear, one-stop insight into the target’s vulnerabilities.

The report also provides specific results for each probe run and their suggested mitigation strategies, helping you understand vulnerabilities and how to fix them.

Each test run consists of one or more probes that create test cases and perform testing on your application. All test cases within a single probe run are associated with the specific vulnerability that the probe is designed to detect.

The probe implements various strategies and techniques to identify its target vulnerability within your conversational application. Despite these variations, all test cases share the specific domain of your chatbot and the details that you defined in the probe's optimization.

The Test Run Scheduler enables you to plan and automate test runs at predefined times and frequencies, helping your team streamline evaluation workflows and reduce operational overhead. Instead of manually setting up tests every time you want to validate model behavior, scheduler allows you to set up recurring or single-run executions that run in the background.

Scheduler supports a wide range of use cases, from scheduled daily Context Leakage probe in production to monthly full security assessments—enabling consistent, repeatable testing practices aligned with your organization's policies and release cycles.

A new Scheduled tab has been added to the Test Runs page for easy management of your upcoming test runs, giving you clear visibility into what test runs are scheduled.

Creating Scheduled Test Run

Starting a new test run now includes the option to schedule it for later. The familiar flow for launching a test is unchanged, but a new "Schedule For Later" button has been added to the test run creation modal.

Schedule a test run:

1. Go to the Test Runs or Overview page.

2. Click New Test Run.

3. Configure the test name and select probes.

4. Click Schedule for Later.

5. Choose a date, time, and frequency for execution.

6. Confirm with Schedule Test Run.

Managing Scheduled Test Runs

All scheduled test runs can be fully managed from the Scheduled tab on the Test Runs page. This page provides a centralized view of upcoming and recurring test runs, allowing you to take direct actions as needed.

Each scheduled test run includes three control icons:

• Start Test Run – Manually trigger the test run immediately.

• Edit Test Run – Modify the scheduled test run configuration.

• Delete Test Run – Permanently remove the scheduled run.

If a test run is not currently needed, you can toggle its status to Inactive without deleting it. This is useful for temporarily pausing automated runs without losing their configuration.

Editing a Scheduled Test Run

Click the Edit Test Run icon to modify any existing scheduled run. The following parameters can be updated:

• Test Run Name

• Date and Time of Next Run

• Frequency (e.g. Daily, Weekly, Monthly, Single Run)

• Selected Probes (custom or predefined probes)

Once changes are made, click Save Changes to update the schedule.

The Probe Run View displays and visualizes the test data of the selected probe's run within your executed test run. This view presents the results of all executed test cases, including details such as attack strategies, variations and the messages that the probe exchanged with the target.

At the top of the page, you will find basic information about the probe's run, including the total number of test cases, the number of failed and error test cases, the execution date and time, the run's status, and a progress bar.

Sankey Diagram

The Sankey diagram (Figure 1) visually depicts the connections among strategy, red teamer, variation, and the outcomes of test cases. As previously mentioned, the width of the flow between nodes in the diagram corresponds to the number of test cases, while the color coding represents the percentage of failed test cases out of the total executed.

Probe Result Table

Probe Result Table displays all probe's test cases executed against your target.

The table contains:

• Id: Unique identifier of the test case (attack).

• Attempt: If the same attack is executed multiple times, each attempt is assigned a different number.

• Result: Outcome of the executed test case.

  • Passed: The test hasn't detected the vulnerability on your application.

  • Failed: The attack found the targeted vulnerability.

  • Error: An error occurred while communicating with the target.

Both filtering and global search functionalities are available to customize the view according to your specific preferences.

The table (maintaining filter applied) can be exported in CSV and JSON formats, allowing for easy integration with other tools and systems for further analysis or reporting.

To view detailed interactions between the probe and the target for each test case, as well as an explanation of the test case result, click the "Details" button in the corresponding row. Refer to the next page for further explanation.

• URL - This is your target endpoint to which the attack messages will be sent.

• POST Request Payload Sample - Here, you provide the payload (body of the HTTP request). Once provided, the payload should include the following placeholders, which you need to insert:

  • {message} - This placeholder represents where the Probe will insert attack messages, simulating input from a user interacting with your application.

  • {session_id} - This placeholder marks the location where a unique string, identifying the current conversation session, will be placed. This ensures that the request is tied to a specific session for multi-step testing.

  • The payload can contain additional fixed arguments if needed.

• Response Path - The JSON path pointing to the message within your chatbot's API response to the given request.

• HTTP Headers - Enter the key-value pairs necessary for your API Requests. Authorization headers must be included for non-public APIs.

Session Management

Sometimes, your application may use different endpoints to manage sessions that track messages in conversations. These endpoints can be separate from the ones used for sending messages. For example:

• A session might be initiated with one request to an "open session" endpoint.

• The session might then be closed with another request to a "close session" endpoint.

If your application operates this way, the Open Session and Close Session options can be toggled and configured in the integration settings. This allows you to add the appropriate requests for starting and ending sessions, ensuring the Probe can properly simulate and test multi-step conversations.

SplxAI Proxy

API endpoints can be implemented in various ways, and we can’t cover every scenario. To address this, we’ve developed the SplxAI Proxy Interface, which you can use for easier integration.

Feel free to contact us, and we’ll assist you in creating it.

Probe allows you to integrate with project management tools and automatically create issues containing information from a probe run.

After clicking Create, a new issue will be created in the selected platform. It will include the details of the probe run, a link to it, and any additional notes provided.

Jira

1. Select one of your Jira projects.

2. Choose the issue type.

3. Add additional notes to be appended in the description after the Probe Run details.

ServiceNow

1. Choose the priority of your incident.

2. Select it's category.

3. Add additional notes to be appended in the description after the Probe Run details.

Upon accessing the Test Case Details, you will find a list of one or more conversations conducted as part of the selected test case. Some strategies execute the attack through multiple separate conversations, for example to refine their approach with each subsequent interaction based on insights gained from previous ones. If this is the case, all related conversations are displayed together in the Attack Scenarios section. These conversations are presented within an expandable element, allowing you to view the specific conversation of interest.

When the conversation expands, you can observe the following:

• Messages generated by the probe, marked as USER.

• Responses from your application, marked as ASSISTANT.

The Encoded toggle switches the conversation's content between encoded and decoded:

• Encoded view: This view displays exact original content exchanged between your application and the probe.

• Decoded View: Certain variations may render the text unreadable to humans (e.g., base64 encoding, joined words, etc.) or on the foreign language. The decoded view transforms the content into readable, allowing you to understand the context of the attack.

The Explanation provides detailed information on why the test case passed or failed.

Probe's test cases are dynamically AI generated based on a set of predefined instructions. Each test case is defined by selecting one value from each of the three components: strategy, red teamer, and variation. By varying these parameters, a wide range of test cases can be generated to cover different aspects of your application's specific vulnerability.

Strategy

Strategy - Method of orchestrating attacks and the included context.

The strategy defines which messages will be available to attack generator, detectors, and targets, and determines the order in which each element of the Probe will be used. Various strategies can be deployed against your target.

Red Teamer

Red Teamer - Instruction to the LLM on how to modify or craft prompts from the provided context.

The Probe platform features a variety of red teamers across the probes. Red teamers collect context (information about the attack) and contain instructions on how to handle them when crafting the attacks. The purpose of each red teamer should be understandable from its name and the conversation it generates.

Variation

Variation - Additional algorithmic or stochastic (with LLM) changes to the prompt before passing it to the target.

Variation involves making changes to the prompt, utilizing large language models, in order to: Increase Success Rate, by modifying adversarial prompts in various ways to enhance the effectiveness of the attack and to Avoid Detection by reducing the likelihood of the prompt being detected as adversarial by the security solutions employed in the application.

Once you identify potential risks in your target using probes, the SplxAI Platform allows you to harden the target's system prompt to strengthen its security.

New System Prompt Hardening

To begin prompt hardening, navigate to the Prompt Hardening page in the Remediation section of the main navigation bar, and click the Harden System Prompt button in the top-right corner.

The hardening process begins by selecting the relevant probes you want to use to harden your system prompt. You can think of these as vulnerabilities you wish to protect against. The prompt hardening tool will then use the results of your probe runs to strengthen your system prompt against the identified vulnerabilities.

The table displays the probes, their categories, the last probe run on the target, and the percentage of failed test cases. This percentage serves as an indicator of where your target is most vulnerable and where there is the greatest opportunity for improvement through hardening. Once all relevant probes are selected (at least one is required), click Continue.

In the next step, simply provide your target's current system prompt and click Generate hardened system prompt, which will initiate the new prompt hardening process.

Hardened System Prompt

The latest prompt hardening will be displayed on the Prompt Hardening page. The header provides information about the generation date and time, the probes selected for hardening, the progress of the hardening, and the remediation status.

Below, there are three sections:

1. Current System prompt - displaying the system prompt before hardening.

2. Generated system prompt - showing the generated hardened system prompt with options to:

   1. Highlight the differences,

   2. Expand the prompt for better readability,

   3. Copy the system prompt.

3. Actions - lists all prompt hardening actions performed on your system prompt by our tool.

   1. Example: Stressing that competitor companies should neither be mentioned nor recommended.

Prompt Hardening History

The second tab on the prompt hardening page is History, which features a table displaying all previous prompt hardenings. The table includes information such as the generation date and time, selected probes, progress (in progress, generated, ...), and status (applied, not applied, ...).

The User and Organization Settings section serves as a place for managing both personal and organizational configurations within the Probe platform.

To access the User and Organization Settings, click on the user icon located in the top-right corner. This will open the landing page and replace the Platform’s navigation bar with the dedicated navigation for the settings section.

In the continuation of this page, we will provide detailed explanations for the features within the User and Organization Settings section.

When you’re done with the editing, click Leave Settings, located near the navigation bar, to exit the settings page.

You can also find the Logout option in the bottom of the navigation bar, if you wish to log out of your account.

Password Change

Besides viewing the email address you registered with, the Account Settings page allows you to update your account password directly within the Probe interface.

Access Token Generation

2. Click the “Generate New Token +” button.

3. Provide Token Details.

   1. Token Name: Assign a meaningful name to your token.

   2. Description: Add a brief description for clarity.

   3. Duration: Specify the token’s expiration period.

Managing Your Tokens

• Name

• Partial Token Key (not the full key for security)

• Created Date

• Expiration Date

• Delete Token Option

Inviting Users To Your Organization

To add a user to your organization, navigate to User & Organization Settings (user icon on top right corner of the platform) and follow these steps:

1. Create the organization (if it hasn't been created already):
2. Invite the User:

   • Enter the email address of the user you want to invite.

   • Click "Invite". An invitation email will be sent to the user, prompting them to create a password.

3. Manage User Status:

   • In the Users table, you can view all users and their current status.

   • For users who haven’t created their password yet (status: Invited), you can either:

     • Reinvite: Send a new invitation.

     • Copy Invitation Link: You can copy the link and manually send it to the user.

Organization Integrations

You can view all available and installed integrations on the Integrations page within the Organization section.

Jira Integration

After clicking "Install" for the Jira integration, you will be redirected to the Atlassian Authorization app. Here, the SplxAI Platform will request access to your Atlassian account. You will need to select the app you wish to integrate with, grant the necessary access permissions, and accept the privacy policy.

All tickets created by the SplxAI Platform will be displayed as if they were reported by the user who accepted the integration.

ServiceNow Integration

To integrate SplxAI Platform with ServiceNow, you will need the following information:

• ServiceNow Instance URL

  • Provide the URL of your ServiceNow instance.

  • E.g., https://<your_instance>.service-now.com).

• API Key

  • The API Key required for access to ServiceNow's APIs.
• User ID

  • Your ServiceNow User ID.

All tickets created by the SplxAI Platform will be displayed as if they were reported by the user defined in the integration process. The incident will be posted in the Incident Management Module.

A Test Run is a group of executed probes performed at a specific point in time against your target. In each test run, you can choose which vulnerabilities to test by selecting one or more pre-configured probes.

Starting a Test Run

A dialog box will appear, prompting you to:

• Enter the name of your run.

• Select one or multiple probes to be included in the run.

Your available probe credits are displayed in the header. Each selected probe deducts from your total.

Test Run Statuses and Progress

The test run can have the following statuses:

• Pending: The test run will start when the queue is clear.

• Running: The test run is in progress.

• Finished: All scheduled probes and their attacks are completed.

• Canceled: The test run was canceled by the user before completion.

• Error: The test run was aborted due to target misconfiguration.

The test run's progress indicates the percentage of completed attacks out of the total scheduled.

Canceling a Test Run

Re-Running a Test Run

Test Run Results

In addition to the probe results, each probe includes a tab showing a mitigation strategy to address vulnerabilities identified by the probe, which can be applied to your target application.

These steps are created based on content provided by our red team, who propose best practices they have identified through research and hands-on experience.

Each step includes a Mark as Completed button, allowing you to track whether the mitigation has been applied to a specific target.

For reference, you can view examples of mitigation strategy steps for Context Leakage probe.

To interact with the SplxAI Platform API, all users, including those on free accounts, can generate a Personal Access Token for authentication.

Generating Your Personal Access Token

Using the Personal Access Token

Once you've obtained your personal access token, include it in the X-Api-Key header of your API requests.

Example Request:

Response to Unauthorized Access

If the Authorization header is not provided, or if an invalid token is used, the API will return a 401 Unauthorized error. This response indicates that authentication is required to access the requested resource.

Example Response:

The Test Run View displays and visualizes the data from all the probes selected for that test run. It includes an overview of the test run status, a Sankey diagram, and the probes table, which provides an overview of the results for each probe. From this page, you can cancel an ongoing test run or rerun a completed test run.

In the continuation of this page, you will find detailed explanations of the sections within the Test Run View and the data they contain.

Sankey Diagram

A Sankey diagram visualizes the flow of data from one node to another, with the width of the flow representing the quantity or magnitude of the data. In this context the data represents executed test cases against your target.

The diagram illustrates the flow from the probe categories to specific probes visualizing the ratio of executed test cases between probes.

From the single probe, the flow connects towards passed, failed and error test case outcome, where the width of each flow indicates the number of test cases. Passed, failed and error nodes show total number of corresponding test cases in the test run.

The gradient from green to red visually represents the ratio of failed to passed test cases for each node. Nodes with fewer failed test cases appear greener, while those with more failed test cases appear redder. This provides a clear visual summary of the test run results.

Probes Table

In the Test Run View, the Probes Table lists all probes executed in that test run. The probes are grouped by category and display the total number of executed test cases, along with the counts of passed, failed and error test cases. A progress bar is also visible.

The Test Run History provides a comprehensive list of all test runs associated with a single target, including those currently in progress. By default, test runs are sorted chronologically, with the most recent run appearing at the top.

The test run history table includes:

• Test run's name.

• Date and time of the execution.

• Probes included in the test run.

• The Result that displays the total number of passed, failed and error test cases across all probes.

You can filter test runs by the following criteria:

• Name: Search for a test run with the specific name.

• Status: Filter by test run status.

• Results: Filter test runs to include those with at least one passed or one failed test case.

• Probes: Show test runs with at least one probe from selection.

REST OpenAPI specification.

Once you start your first probes, the Overview page will begin populating with data. The dashboard provides a quick view of your target's metrics, delivering real-time insights into recent probe runs and their outcomes.

Target Risk Level

In the top left corner of the overview, the Target Risk Surface is displayed. It shows the total number of simulated attacks and successful attacks across different vulnerabilities tested against your target. The results are aggregated from the latest available probe runs of each probe.

The semi circle chart indicates the total risk of your target, taking into account the number of failed test cases, their severity, and their expected probability. A lower score is preferable. Toggling the chart switches it to a time series view, allowing you to track your application's risk level over time.

Recent Test Runs

Probe Categories Overview

Welcome to the Probe Product Updates! Stay informed about the latest improvements, new features, and important updates to the platform.

Explore Monthly Updates

Click on a month below to view the full details of updates for that period:

We Value Your Feedback

Your input is the driving force behind the updates. If you have suggestions or encounter any issues, feel free to contact us or submit feedback directly through the platform. Together, we’ll make Probe even better!

Prompt Hardening

We’ve introduced a major new feature to our platform: Prompt Hardening. This feature allows you to automatically harden your system prompt with security and safety instructions, based on our red teaming knowledge and probe run results.

Export Probe Result Table In CSV & JSON

Subscription Details

Subscription details have been added to the Organization Settings page for better tracking and usage planning. This includes information about your current credit balance, subscription plan, billing cycle, credit renewal date, the amount of credits per renewal, and the subscription expiration date.

Improvements & Tweaks

• You now have the ability to reinvite users and copy the invitation link, for those who haven’t yet accepted invitation to the organization.

• Tooltip support has been added to the probe optimization dialogs.

Integrating CI/CD with the SplxAI Platform streamlines the process of continuously testing and securing your generative AI applications. Automating security and safety testing ensures that vulnerabilities are detected early in the development cycle, reducing the risk of their exploitation.

Available CI/CD Tool Examples

We provide a variety of CI/CD examples to help you integrate and automate tests using the SplxAI Platform across different platforms:

• Azure DevOps

• Bitbucket

• GitHub

• GitLab

• Jenkins

Platform-Independent Examples:

• Bash

You can find these examples in the GitHub repository. Explore the repository to find scripts specific to your CI/CD tool.

New Target Integrations

We've expanded our integration options with:

• LLM:

  • Gemini

  • Bedrock

  • OpenAI Assistant

• LLM Development Platform:

  • Dify AI

Default Language Support

In addition to English, Probe can now test your target against attacks in the chatbot's default language. With support for over 60 languages, we provide more accurate and localized security assessments.

Q&A Probe

With Q&A Probe, you can test your application’s ability to respond correctly to questions you define. Probe will reformulate your questions, send them to your target, and check if the response content matches the correct answers you’ve set. Upload your question-answer combinations via CSV, provide a chatbot description, and let Probe handle the testing.

Jira Integration

You can now create Jira issues directly from the Probe's result page, with the autogenerated summary of your Probe run included in the issue. This makes it easier for your team to track any vulnerabilities found through automated red teaming.

Improvements & Tweaks

• Upload file option for easier Probe configurations (e.g., for RAG precision).

• Extended REST API integration with support for Open and Close Session endpoints.

New Integrations

We’ve added several Platform and LLM integrations that make connecting Probe with your GenAI applications seamless in just a few clicks! We now support:

• Platforms:

  • Microsoft Teams

  • Slack

  • WhatsApp

• LLMs:

  • Azure OpenAI

  • Azure ML

  • Anthropic

  • Hugging Face

  • OpenAI

  • Mistral

Custom Probe

You can now create custom probes for testing your specific use case, whether it’s security or safety-related. Add a description, set the related allowed and banned behaviors, and the Platform will handle the rest.

Probe Framework Updates

Our Red Team continues to update our probe datasets and implement new strategies and variations, ensuring your applications are tested against the latest threats.

Public API

We’ve released our public API, which currently has the following functionalities:

• Trigger new test run.

• Cancel ongoing test run.

• Get the status of test run.

• Generate a PDF report of test run execution.

For detailed instructions on how to interact with the API, please refer to the API Reference page.

User & Organization Settings

The settings page has been refactored to support new features. It’s no longer a single page, but instead divided into separate sections, each with its own page. The new structure includes:

User Settings:

• Account Settings

• Personal Access Tokens

Organization Settings:

• General

• Users

• Integrations

• Subscription

Risk Surface Time Series

The Risk Surface section on the overview page now includes an option to display a time series chart of your target’s risk surface. This chart allows you to track how your target's risk has changed over time, reflecting the impact of any updates you’ve made on the target or new vulnerabilities discovered as a result of probe improvements.

Improvements & Tweaks

• The number of test cases that finished with Error has been added to all areas where results are displayed, ensuring better transparency and consistency.

• Detection time for each test case is now included in the probe run table for more detailed insights.

• Probes cost in credits is now visible in the Probe Details modal on the Probe Catalog page.

• The target selection has been relocated for better distinction from the navigation bar.

• Test and probe run titles are now included in the breadcrumbs, making it easier to drill up.