June 2025

Model Benchmarks

Our new Model Benchmarks feature helps you choose the most secure and reliable model for your specific use case. We continuously stress-test leading open-source and commercial models using thousands of attack simulations, then provide an intuitive interface to rank and compare them based on your needs.

You can learn more about this feature on the Model Benchmarks documentation page, and in our latest blog post.

Two-Tiered Role-Based Access Control (RBAC)

We’ve introduced a two-tiered RBAC system consisting of Organization-Level Roles and Workspace-Level Roles. Each tier manages a distinct set of permissions:

• Organization-Level Roles govern platform-wide settings and administrative operations.

• Workspace-Level Roles allow control within individual workspaces, enabling teams to customize access based on project-specific needs.

This structure ensures clear separation of responsibilities and more flexible permission management.

Custom Datasets

The Custom Dataset Probe allows users to go beyond SplxAI’s built-in probes by uploading and testing with their own datasets. It’s ideal for teams with specialized testing needs or existing test suites. Detection can be performed using either regex or an LLM.

Users start by uploading their custom datasets. Within the platform, they can then extend their attacks, for example, by selecting from predefined languages, variations, and attack strategies managed by SplxAI’s internal red team. This adds an additional layer of depth and flexibility to their tests, enabling broader coverage and more realistic, automated validation of chatbot behavior.

Multimodal Attack via Documents

Support for multimodal attacks using uploaded documents has been added to all relevant probes across the platform. This allows users to evaluate whether their application is vulnerable to scenarios where a document serves as an attack vector, expanding test coverage beyond text-only, image and audio inputs.

These attacks can be enabled at the target level in the Target Configuration page.

Improvements & Tweaks

• Response in Failed Test Connection When a test connection fails, the response from your target is now displayed to help with debugging.

• Tooltips & Placeholders Tooltips and placeholders have been added to probe optimization dialogs to improve clarity and usability.

• Select All Probes A "Select all probes" checkbox is now available in the start new test run modal for quicker probe selection.

• Filter on Actions Taken Users can now filter probe run results based on actions taken, such as accepted risk or status changes. for easier tracking and review.