why evals?

You changed a prompt, swapped a model, or added a tool, and now something broke. You don't know what, you don't know when, and your users find out before you do. That is the problem evals solve.

Your agent does not behave like normal software

Regular code is deterministic: same input, same output, so a test suite can check it once and trust it. An AI agent is not. The same question can get a different answer depending on the model, the prompt, the retrieved context, or just sampling luck. "It felt better when I tried it" is not evidence; it's an anecdote.

An eval is a repeatable question

An evaluation is a small dataset of inputs with known-good answers, a way to run your agent on them, and a scorer that judges the outputs. Because the inputs never change, every run is comparable: today's agent against last week's, this model against that one, your new prompt against the old one.

Think unit tests, but for behavior. You wouldn't ship an API without tests; the same instinct applies to anything with an LLM in the loop.

What evals buy you

Regression safety. Change the prompt, the model, the tools, then run the evals. If a score drops, you know before your users do, and the failing samples tell you exactly where.

Decisions with numbers. Which model is worth the cost? Did the fancy new solver actually help? An eval turns opinions into a table you can act on.

Evidence for shipping. "Score went from 61% to 74% on our support dataset" is a release note you can stand behind, and a bar you can gate CI on.

Start small, start now

You don't need a benchmark. Ten real inputs from your actual users, a scorer that checks the important thing, and a habit of running it before you ship. That's an eval program. Grow the dataset from real failures as you go.

That's what Evaris is for: author the evals with Inspect AI, publish every run, and compare over time. Start with the quickstart; it takes about fifteen minutes.