Tests for AI agents that actually work

AgentSpec is a testing framework for non-deterministic AI behavior. Write tests in YAML, catch regressions before they ship, and see exactly what changed.

Get Started → Features

Assertions designed for AI output

Non-deterministic matching

contains_any, contains_all, regex — assertions that handle the variability of AI responses without flaky tests.

Behavior metadata

Assert on tool_called, max_latency_ms, max_tokens — verify not just what the agent said, but what it did.

Semantic similarity

Check if responses are semantically on-topic, even when the exact words differ. No API calls needed.

JSON path assertions

Extract and verify specific fields from JSON agent outputs. Perfect for structured API responses.

CI/CD native

Exit codes, JUnit XML, JSON output. GitHub Action included. Drop into any CI pipeline.

Local-first

No cloud, no account, no telemetry. Tests run on your machine, in your CI, on your terms.

Write tests in YAML, not code

# agentspec.yaml
name: "support agent"
tests:
  - name: "handles expired token"
    input: "my token expired"
    expect:
      contains: "refresh token"
      not_contains: "I don't know"
      max_latency_ms: 5000

  - name: "routes billing question"
    input: "I want a refund"
    expect:
      contains_any: ["billing", "refund", "support"]
      tool_called: "transfer_to_billing"

$ agentspec run ── AgentSpec ─ Test Results ────────────── ✓ support agent > handles expired token (12ms) ✓ support agent > routes billing question (11ms) ✓ support agent > handles greeting (12ms) ✓ support agent > fallback is helpful (11ms) ── 4 passed, 0 failed (4 total) ──

Why not just use Jest?

Jest / Vitest

• Exact string matching

• No tool_called assertion

• No latency/token gates

• No semantic similarity

• No AI behavior diffs

• Built for deterministic code

AgentSpec

• Non-deterministic matching

• tool_called, max_latency_ms, max_tokens

• Semantic similarity built-in

• JSON path extraction

• Behavior diff reports (soon)

• Built for AI agents

Get started in 30 seconds

$ npm install -g agentspec $ agentspec init → Created agentspec.yaml $ agentspec run ✓ All tests passed

GitHub → npm