The problem
Static evals don't catch how agents actually break
Most agent evals I'd used before this were just a fixed set of single-turn prompts checked against an expected answer. That catches static failures fine. But that's not usually how things break in production. Real breakdowns happen mid-conversation, when a user pushes back, goes off script, or straight up tries to manipulate the agent. A pass/fail row in a spreadsheet doesn't show you the exact turn where things went sideways.
So Gauntlet runs full adversarial conversations instead, against a system prompt or a live endpoint. Angry customers, confused first-timers, prompt-injection attempts. It scores the session the moment it ends, so I can go watch the actual turn where the agent broke instead of squinting at a number.
Architecture
The moving pieces
A FastAPI backend sits in the middle. The CLI and the Next.js frontend both talk to it the same way, one over a terminal, one over SSE in the browser. It reads and writes suite runs and embeddings in Postgres (with pgvector for the regression matching), uses Redis for job state, and calls out to Groq or Claude for the judge pass.
How it works
Two passes instead of one LLM call
Every conversation gets checked two ways. A regex pass runs first against a deny-list of hard-fail patterns, things like prompt leaks or an agent waiving a fee it had no authority to waive. I didn't want anything that deterministic riding on an LLM's mood. Then an LLM judge (Groq by default, Claude if it's configured) grades the same conversation at temperature 0.1 against a 10-check rubric split into three buckets: Outcome, Safety, and Composure.
Everything streams over Server-Sent Events, so the frontend shows turns arriving live instead of a spinner. There's also a CLI (npx @trygauntlet/cli) that wraps the same engine for local runs and CI, so a release can get gated on a score threshold through a GitHub Action.
How it works
Telling a new failure from an old one with different words
The harder problem is figuring out whether a failure in a new suite run is actually new, or just the same bug worded differently. Every finding gets embedded (a 384-dim BAAI/bge-small-en-v1.5 vector via fastembed) and stored in Postgres with pgvector. When I diff two suite runs, new failures get matched against the baseline's fixed failures for the same persona and check, using an HNSW-indexed cosine-distance search so it's an actual database query and not a Python loop over every vector. A rephrased version of an old failure gets marked as still there instead of counted as a brand-new regression.
What broke
Two bugs that actually shaped the design
Bug 1 — judge output got cut off mid-JSON
The Claude judge was capped at 1500 tokens. That was fine until a persona like the Policy Prober generated enough checks and hard fails to blow past it. The response got cut off mid-object, failed to parse, and quietly dropped that persona from the comparison instead of raising something I'd notice.
# The concierge rubric emits many checks + hard fails; 1500 truncated
# the JSON mid-object on longer evaluations (e.g. Policy Prober),
# surfacing as an "Expecting ',' delimiter" parse error and dropping
# that persona from the comparison.
max_tokens=2500,Bumping the cap to 2500 fixed the immediate bug. The real lesson was that a parse failure needs to show up as a finding, not vanish silently.
Bug 2 — pgvector isn't on by default on Azure
I deployed to Azure Database for PostgreSQL assuming CREATE EXTENSION vector would just work, the way it does locally. It doesn't. Azure has to allowlist pgvector in the server's parameters first.
try:
await conn.execute(text("CREATE EXTENSION IF NOT EXISTS vector"))
vector_enabled = True
except Exception:
pass # pgvector not allowlisted yet — enable it in Server ParametersNow the extension gets created defensively, and if it's not available yet, semantic regression matching just falls back to exact matching instead of taking the whole app down.
Design decision
A safety regression can't get cancelled out by an unrelated win
Averaging all ten checks into one score can hide something important. A more talkative, more accommodating agent can complete more bookings and nudge the Outcome score up enough to offset a new safety failure in the blended average, even though the safety problem actually got worse. So the regression diff splits findings into safety (the Safety category plus hard fails) and competence (Outcome, Composure) and reports them separately. A new jailbreak or a data leak can't get quietly netted out by an unrelated improvement somewhere else in the score.
One thing I haven't gone back to fix: the cosine-similarity cutoff for matching a rephrased failure to an old one is a flat 0.8. I picked that number by hand, not from tuning it against real suite history. It works fine so far, but it's still a guess.