Test Run Scheduler

The Test Run Scheduler enables you to plan and automate test runs at predefined times and frequencies, helping your team streamline evaluation workflows and reduce operational overhead. Instead of manually setting up tests every time you want to validate model behavior, scheduler allows you to set up recurring or single-run executions that run in the background.

Scheduler supports a wide range of use cases, from scheduled daily Context Leakage probe in production to monthly full security assessments—enabling consistent, repeatable testing practices aligned with your organization's policies and release cycles.

A new Scheduled tab has been added to the Test Runs page for easy management of your upcoming test runs, giving you clear visibility into what test runs are scheduled.

Figure 1: Test Runs page - Scheduler tab

Creating Scheduled Test Run

Starting a new test run now includes the option to schedule it for later. The familiar flow for launching a test is unchanged, but a new "Schedule For Later" button has been added to the test run creation modal.

Schedule a test run:

1. Go to the Test Runs or Overview page.

2. Click New Test Run.

3. Configure the test name and select probes.

4. Click Schedule for Later.

5. Choose a date, time, and frequency for execution.

6. Confirm with Schedule Test Run.

Figure 2: Creating new Scheduled Test Run

Managing Scheduled Test Runs

All scheduled test runs can be fully managed from the Scheduled tab on the Test Runs page. This page provides a centralized view of upcoming and recurring test runs, allowing you to take direct actions as needed.

Each scheduled test run includes three control icons:

• Start Test Run – Manually trigger the test run immediately.

• Edit Test Run – Modify the scheduled test run configuration.

• Delete Test Run – Permanently remove the scheduled run.

If a test run is not currently needed, you can toggle its status to Inactive without deleting it. This is useful for temporarily pausing automated runs without losing their configuration.

Editing a Scheduled Test Run

Click the Edit Test Run icon to modify any existing scheduled run. The following parameters can be updated:

• Test Run Name

• Date and Time of Next Run

• Frequency (e.g. Daily, Weekly, Monthly, Single Run)

• Selected Probes (custom or predefined probes)

Once changes are made, click Save Changes to update the schedule.

Figure 3: Edit Scheduled Test Run