Regrada Documentation

Cases

A case is a YAML file (default: regrada/cases/**/*.yml) containing a prompt (chat messages or structured input) plus optional assertions.

Assertions vs Policies

• Case assertions (assert: in a case file) mark individual runs as pass/fail and feed metrics like pass_rate.
• Policies (policies: in regrada.yml) decide what counts as a warning or error in CI.

To fail CI on failed assertions, add an assertions policy with severity: error.

Baselines

A baseline is a stored snapshot (golden output + aggregate metrics) used for regression checks.

Regrada stores baselines under the snapshot directory (default: .regrada/snapshots/), keyed by:

• Case ID
• Provider + model
• Sampling params (temperature / top_p / max tokens / stop)
• System prompt content

Changing any of these produces a different baseline key and requires regenerating the snapshot.

regrada init

Creates regrada.yml, an example case, and runtime directories.

Use --non-interactive for a fast mock-provider setup, or the default interactive flow if you want help choosing provider and CI defaults.

Flags: --path, --force, --non-interactive

regrada record

Starts an HTTP proxy to capture LLM traffic. Defaults to forward proxy with HTTPS MITM. When a subcommand is provided, the proxy exits when that process finishes.

For forward proxy mode, run regrada ca init and regrada ca install once on the machine first.

Traces written to .regrada/traces/ (JSONL). Sessions written to .regrada/sessions/. Auto-detects provider by host (OpenAI, Anthropic, Azure, Bedrock).

regrada accept

Converts traces from the latest (or specified) session into cases and baselines.

regrada baseline

Runs all discovered cases once and writes baseline snapshots to the local snapshot directory.

Use this before regrada test in local mode, or when refreshing the snapshots you commit on your baseline branch.

Supports --output text|json so scripts can consume the written case IDs and snapshot directory directly.

regrada test / regrada check

Runs cases, diffs against baselines, evaluates policies, and writes reports. regrada check is an alias.

Baseline behavior comes from baseline.mode: local compares against local snapshots, while git compares against the configured git ref (default: origin/main).

Flag --concurrency N controls parallel case execution (default: 1).

regrada ca

Manages the local Root CA required for forward-proxy HTTPS interception.

regrada migrate

Compares two model targets across the same test suite so you can judge a model migration before you flip traffic.

Outputs Markdown by default, JSON when requested, and exits non-zero if regressions are detected.

regrada fuzz

Generates adversarial mutations for your cases and measures whether the model still behaves acceptably under those variants.

Returns exit code 2 when any case falls below the robustness threshold. Use Markdown for human review or JSON for automation.

Case Discovery

Defaults (can be overridden under cases:):

• Roots: ["regrada/cases"]
• Include globs: ["**/*.yml", "**/*.yaml"]
• Exclude globs: ["**/README.*"]

Execution Mode

execution:
  mode: replay  # replay (default) or live

• replay — compare against baselines (requires snapshots to exist)
• live — run without baselines; enforce invariants and checks only

Case Defaults

cases:
  defaults:
    runs: 3
    timeout_ms: 30000
    concurrency: 8

Baseline Modes

Git baseline config (recommended for CI):

baseline:
  mode: git
  git:
    ref: origin/main
    snapshot_dir: .regrada/snapshots

Reports

Enable JUnit output for CI:

report:
  format: [summary, markdown, junit]
  junit:
    path: .regrada/junit.xml

CI Behavior

By default, Regrada fails on any severity: error violation. To also fail on warnings:

ci:
  fail_on:
    - severity: error
    - severity: warn

OpenAI

providers:
  default: openai
  openai:
    model: gpt-4o-mini
    api_key_env: OPENAI_API_KEY  # default

Credential resolution: api_key_env → api_key → OPENAI_API_KEY

Anthropic

providers:
  default: anthropic
  anthropic:
    model: claude-3-5-sonnet-20241022
    api_key_env: ANTHROPIC_API_KEY  # default

Credential resolution: api_key_env → api_key → ANTHROPIC_API_KEY

Azure OpenAI

providers:
  default: azure_openai
  azure_openai:
    endpoint: https://my-resource.openai.azure.com
    deployment: gpt-4o-mini
    api_version: 2024-02-15-preview  # default
    api_key_env: AZURE_OPENAI_API_KEY

The deployment field is required. The endpoint URL is also resolved from AZURE_OPENAI_ENDPOINT if not set inline.

Calls {endpoint}/openai/deployments/{deployment}/chat/completions?api-version={api_version}

AWS Bedrock

providers:
  default: bedrock
  bedrock:
    model_id: anthropic.claude-3-5-sonnet-20241022-v2:0
    region: us-east-1
    # Optional: explicit credentials
    access_key_env: AWS_ACCESS_KEY_ID
    secret_key_env: AWS_SECRET_ACCESS_KEY

Uses the Bedrock Converse API. If explicit credentials are not set, falls back to the AWS SDK default credential chain (instance profile, environment, shared credentials file). Region resolves from region → AWS_REGION → AWS_DEFAULT_REGION.

Mock

providers:
  default: mock

Returns a fixed "mock response" string. Useful for wiring up tests without real API calls.

Setup

1. Create an API key in the dashboard under Settings → API Keys
2. Add the key to your environment (e.g., REGRADA_API_KEY)
3. Enable backend upload in regrada.yml

project:
  name: my-project       # required: the dashboard project to sync to

backend:
  enabled: true
  api_key_env: REGRADA_API_KEY
  upload:
    traces: true         # upload traces captured during recording
    test_results: true   # upload test run results after regrada test

What Gets Synced

API Endpoints

The CLI communicates with https://api.regrada.com (override with REGRADA_API_URL):

All requests use Authorization: Bearer <api_key>. Trace uploads during recording are non-blocking (async, background goroutine).

"config not found"

Create regrada.yml by running regrada init or pass --config to specify a different path.

Exit code 4 / baseline missing

Run regrada baseline on your baseline ref and commit snapshots. Ensure CI fetches baseline.git.ref with fetch-depth: 0.

Provider auth errors

• OpenAI: set OPENAI_API_KEY or configure providers.openai.api_key
• Anthropic: set ANTHROPIC_API_KEY or configure providers.anthropic.api_key
• Azure OpenAI: set AZURE_OPENAI_API_KEY and AZURE_OPENAI_ENDPOINT
• Bedrock: ensure AWS_REGION is set and valid credentials are available via the AWS credential chain

Recording HTTPS fails

Run regrada ca init + regrada ca install, and confirm capture.proxy.allow_hosts includes your provider host (e.g., api.openai.com, api.anthropic.com).

Dashboard upload fails

Verify REGRADA_API_KEY is set and that project.name is configured in regrada.yml. Check that backend.enabled: true and the relevant upload flags are set.