Support agent eval

A walkthrough of the reference example. It evaluates a LangChain support/refund agent against a small dataset, scored by an LLM-as-judge, and publishes the result to Evaris. Use this as a template for evaluating your own agent.

What's in the example

agents/support_agent.py     # the agent under test
evals/evals.py              # wraps the agent as an inspect solver + defines the task
evals/samples.jsonl         # the dataset

The agent

agents/support_agent.py builds a LangChain create_agent with four tools:

• lookup_order(order_id): order status, item, price, delivery age, condition.
• refund_policy(condition, delivered_days_ago): policy guidance for the condition and how long ago delivery happened.
• estimate_refund(price_usd, refund_type): refund amount for full/partial/none.
• create_escalation_ticket(order_id, reason): mock human-support ticket.

The agent is created with a system prompt instructing it to use tools before deciding and to return one concise sentence. The module exposes a single entry point:

def run_agent(question: str) -> str:
    agent = create_support_agent()
    result = agent.invoke({"messages": [{"role": "user", "content": question}]})
    return str(result["messages"][-1].content).strip()

The model and base URL come from env (LANGCHAIN_AGENT_MODEL, OPENAI_BASE_URL). See OpenAI-compatible providers.

Wrapping it as an inspect solver

Inspect needs a solver: something that takes a sample's input and writes an output. evals/evals.py does this with a @solver-decorated factory:

from inspect_ai.model import ModelOutput
from inspect_ai.solver import TaskState, Generate, solver
from agents.support_agent import run_agent

@solver
def langchain_agent_solver():
    async def solve(state: TaskState, generate: Generate) -> TaskState:
        del generate
        answer = run_agent(str(state.input))
        state.output = ModelOutput.from_content(model="langchain-agent", content=answer)
        return state
    return solve

The inner solve reads the sample's input, calls the agent, and writes the reply into state.output via ModelOutput.from_content. That is the entire bridge between LangChain and Inspect.

The task

@task
def support_refund_smoke():
    return Task(
        dataset=json_dataset(str(SAMPLES_PATH)),
        solver=langchain_agent_solver(),
        scorer=model_graded_qa(),
    )

json_dataset reads samples.jsonl (each line: {"input": "...", "target": "..."}). model_graded_qa() is Inspect's built-in LLM-as-judge: it asks the grader model whether the solver's output matches target, producing a score.

The dataset

{"input":"Customer asks for a refund for order A100. What should we tell them?","target":"full refund"}
{"input":"Customer wants refund for order B200 after trying the product. What is the decision?","target":"escalate"}
{"input":"Customer asks to refund order C300 because it has not arrived. What should the agent do?","target":"tracking"}

Each line is one sample. target is the rubric the judge scores against.

Running

With your provider env vars set (see Quickstart §3):

inspect eval evals/evals.py \
  --model "$INSPECT_EVAL_MODEL" \
  --log-dir logs \
  --limit 5

This writes a timestamped .eval file under logs/.

Publishing

evaris publish logs --project <your-project-id>

See Quickstart §4.

Adapting to your agent

1. Replace agents/support_agent.py with your agent. Keep a single run_agent(question: str) -> str entry point if you want to reuse the solver unchanged.
2. Replace evals/samples.jsonl with your inputs + targets.
3. If your task needs a different scorer (deterministic match, multi-rubric), swap model_graded_qa(); see the Inspect scorers docs.
4. Run and publish.

For a solver that does not use LangChain, see Custom solver.