Evaluation Harness (evals)¶
LLM-as-a-judge evaluation of the AI failure analyzer’s diagnosis quality.
Runs FailureAnalyzer (Qwen3.8 27B via
Groq) over a curated golden dataset, then scores each diagnosis with a separate
judge model (openai/gpt-oss-120b by default — a different model family, so
no self-preference). Run with poetry run python -m evals.
Dataset¶
Golden-dataset loader for the failure-analyzer evaluation.
Reads the curated YAML dataset of TMDB failure contexts (one or more per diagnosis category) and validates each case has the structure the runner and judge expect. The dataset is the ground truth the judge scores correctness against, so a malformed case is a hard error rather than a silent skip.
- evals.dataset.load_dataset(path: str | Path) list[dict][source]¶
Load and validate the golden eval dataset.
- Parameters:
path – Path to the YAML dataset (a mapping with a top-level
caseslist).- Returns:
The list of case dicts, each with
id,failure_context, andexpectedkeys. Example return:[ { "id": "timeout_slow_response", "failure_context": { "test_name": "test_update_list_description[update_list0]", "api_url": "https://api.themoviedb.org/4/list/8284605", "status_code": 201, "error_message": "AssertionError: Response time 17.7s exceeded expected max 15s", }, "expected": { "expected_category": "timeout", "expected_root_cause": "The call succeeded (HTTP 201) but took ~17.7s, over the 15s threshold.", }, }, ]
- Raises:
ValueError – If the file is malformed or any case is missing a required field, has a duplicate
id, or names a category outsideVALID_CATEGORIES. The message names the offending case.
Judge¶
LLM-as-a-judge for the AI failure analyzer.
Sends a JSON payload (the failure context, the diagnosis under test, and — for offline evaluation — the expected reference) to a judge model on Groq and gets back a strict per-dimension verdict on whether the diagnosis is correct, grounded, complete, and actionable.
Two modes share one judge call:
judge_case()(offline eval) — scores all four dimensions. Correctness is scored against the case’sexpectedreference; groundedness, completeness, and actionability are judged fromfailure_contextalone.judge_live()(in-pytest, on real failures) — scores groundedness, completeness, and actionability (every dimension except correctness), because a real failure has no reference answer to score correctness against.
The judge runs on Groq (same provider as the analyzer it evaluates) but a different model family — GPT-OSS judging Qwen — so there is no self-preference bias.
- evals.judge._call_judge(system_prompt: str, output_schema: dict, payload: dict, model: str, api_key: str | None) dict | None[source]¶
Make one judge call and return the parsed, validated verdict.
Sends
system_promptplus the JSON-serializedpayloadto the judge model on Groq with strict structured outputs. If the account/model rejects strictjson_schema, it retries once with the looserjson_objectmode the analyzer itself uses. Any API error, unparseable response, or output missing a required key returnsNone(the caller treats that as a judge error rather than crashing).- Parameters:
system_prompt – The judge instructions (full or live variant).
output_schema – The strict JSON Schema for the expected output.
payload – The judge input object to serialize into the user message.
model – Groq model id for the judge.
api_key – Groq API key; falls back to the client’s env-based default.
- Returns:
The parsed judge output, or
Noneon failure. Example return:{ "groundedness": {"verdict": "pass", "reasoning": "...", "issues": []}, "completeness": {"verdict": "fail", "reasoning": "...", "issues": ["omits the 404 status code"]}, "actionability": {"verdict": "pass", "reasoning": "...", "issues": []}, "overall_verdict": "fail", }
- evals.judge._output_schema(dimensions: tuple[str, ...]) dict[source]¶
Build the strict judge output schema for a given set of dimensions.
- Parameters:
dimensions – The dimension names the judge must return (e.g.
("correctness", "groundedness", "completeness")).- Returns:
A JSON Schema requiring one
_DIMENSION_SCHEMAblock per dimension plus anoverall_verdictstring. Example return:{ "type": "object", "additionalProperties": False, "required": ["groundedness", "overall_verdict"], "properties": { "groundedness": { # one _DIMENSION_SCHEMA block per dimension "type": "object", "required": ["verdict", "reasoning", "issues"], "properties": {...}, }, "overall_verdict": {"type": "string", "enum": ["pass", "fail"]}, }, }
- evals.judge.judge_case(payload: dict, model: str = 'openai/gpt-oss-120b', api_key: str | None = None) dict | None[source]¶
Score one diagnosis on all four dimensions against its expected reference.
Used by the offline eval, where each case carries an
expectedblock.- Parameters:
payload – The judge input object (
case_id,failure_context,diagnosis,expected).model – Groq model id for the judge. Defaults to DEFAULT_JUDGE_MODEL.
api_key – Groq API key; falls back to the client’s env-based default.
- Returns:
The parsed judge output (per-dimension
{verdict, reasoning, issues}for correctness/groundedness/completeness/actionability plusoverall_verdict), orNoneif the call or parse fails. Example return:{ "correctness": {"verdict": "pass", "reasoning": "...", "issues": []}, "groundedness": {"verdict": "pass", "reasoning": "...", "issues": []}, "completeness": {"verdict": "pass", "reasoning": "...", "issues": []}, "actionability": {"verdict": "pass", "reasoning": "...", "issues": []}, "overall_verdict": "pass", }
- evals.judge.judge_live(failure_context: dict, diagnosis: dict, model: str = 'openai/gpt-oss-120b', api_key: str | None = None) dict | None[source]¶
Score a live failure’s diagnosis on groundedness, completeness, and actionability.
Used in-pytest on live failures, which have no reference answer — so correctness is deliberately not judged. The diagnosis is cleaned with
sanitize_diagnosis()before being shown to the judge.- Parameters:
failure_context – The context the analyzer was given for this failure.
diagnosis – The raw diagnosis the analyzer produced.
model – Groq model id for the judge. Defaults to DEFAULT_JUDGE_MODEL.
api_key – Groq API key; falls back to the client’s env-based default.
- Returns:
The parsed judge output (
groundedness,completeness,actionability,overall_verdict), orNoneif the call or parse fails. Example return:{ "groundedness": {"verdict": "pass", "reasoning": "...", "issues": []}, "completeness": {"verdict": "pass", "reasoning": "...", "issues": []}, "actionability": {"verdict": "fail", "reasoning": "...", "issues": ["suggested_fix is too vague"]}, "overall_verdict": "fail", }
- evals.judge.sanitize_diagnosis(diagnosis: dict) dict[source]¶
Clean an analyzer diagnosis before it is shown to the judge.
The analyzer (
FailureAnalyzer.analyze) injects two bookkeeping keys into its output —test_nameandmodel. We drop them so the judge sees only the model’s actual answer and is never biased by knowing which model produced it. We also coerceconfidencefrom a numeric string (the analyzer sometimes returns"90") to an int for a consistent type; a value we can’t parse is left untouched because the judge prompt explicitly tolerates a non-int confidence.- Parameters:
diagnosis – The raw diagnosis dict from the analyzer.
- Returns:
A new dict safe to embed in a judge payload and a report (
test_name/modeldropped,confidencecoerced to int). Example return:{ "root_cause": "Stale movie id 550 returned HTTP 404", "category": "data_issue", "suggested_fix": "Refresh the movie id used in test_data.yaml", "confidence": 90, "explanation": "The request to /movie/550 returned 404 Not Found...", "evidence": ["status_code 404", "error message 'could not be found'"], }
CLI Runner¶
CLI runner for the failure-analyzer evaluation.
Runs the AI failure analyzer (Qwen3.8 27B via Groq) live on each case in the golden dataset, then judges every produced diagnosis with the GPT-OSS judge on four dimensions (correctness, groundedness, completeness, actionability). Prints a per-case table plus aggregate metrics and writes a JSON report.
Both the analyzer under test and the judge call Groq, so only GROQ_API_KEY
is required.
Usage:
poetry run python -m evals
poetry run python -m evals --judge-model openai/gpt-oss-120b
- evals.__main__._build_payload(case: dict, diagnosis: dict) dict[source]¶
Assemble the single JSON object sent to the judge for one case.
The payload bundles the three things the judge needs: the
failure_contextthe analyzer saw, thediagnosisit produced (already cleaned), and theexpectedreference answer the dataset author wrote.The
failure_contextis fed to Scout to produce the whole diagnosis, and that one diagnosis is then judged on all four dimensions —failure_contextis not tied to any single dimension. What sets correctness apart is the judge, not the input: correctness compares Scout’sdiagnosis(itscategoryandroot_cause) againstexpected— the known-true answer a human authored for this case — so it needs that reference to grade against. Groundedness, completeness, and actionability need no reference: they judge the diagnosis against thefailure_contextevidence itself, asking whether it is internally sound, not whether it is right.This is why correctness is an offline-only dimension.
expectedexists only in the curated dataset; it is hand-written by someone who already knew the true cause, and is not derivable fromfailure_context(if it were, we would not need Scout). A live failure has no such answer key — sojudge_livedrops correctness and scores only the other three. This is the exact shapejudge.judge_casedocuments andJUDGE_SYSTEM_PROMPTdescribes.How it differs from the live path
┌──────────────────────┬───────────────────────────────────────────────┬─────────────────────────────┐ │ │ _build_payload (offline) │ judge_live’s inline payload │ ├──────────────────────┼───────────────────────────────────────────────┼─────────────────────────────┤ │ keys │ case_id, failure_context, diagnosis, expected │ failure_context, diagnosis │ ├──────────────────────┼───────────────────────────────────────────────┼─────────────────────────────┤ │ expected? │ yes (enables correctness) │ no (no reference) │ ├──────────────────────┼───────────────────────────────────────────────┼─────────────────────────────┤ │ sanitizes diagnosis? │ no — caller already did │ yes — inline │ └──────────────────────┴───────────────────────────────────────────────┴─────────────────────────────┘
- Parameters:
case – A dataset case (
id,failure_context,expected) fromevals.dataset.load_dataset()diagnosis – The cleaned diagnosis for that case.
- Returns:
The judge input object.
- evals.__main__._diagnose_cases(cases: list[dict]) list[dict][source]¶
Run the analyzer once per dataset case and collect the diagnoses.
Uses a single fresh analyzer instance (see
_make_scout()) for the whole pass. For each case it callsanalyze(failure_context)and records the produced diagnosis.analyzereturnsNonewhen the analyzer’s own LLM call fails; that is captured asscout_failureso the case is carried through the report (and excluded from quality metrics) rather than crashing the run.Exercise the analyzer (Qwen3.8 27B) on known inputs. Each dataset case carries a realistic
failure_context; this function feeds that context to a freshFailureAnalyzerand captures what the analyzer model says. This is the only place in the eval where the system-under-test actually runs.Purpose: Take the golden dataset and produce one diagnosis per case from the live analyzer — the raw material the judge will later grade.
- Parameters:
cases – The validated dataset cases.
- Returns:
One record per case, each with
case_id,expected_category(lifted out for the table/metrics), the cleaneddiagnosis(orNone), and ascout_failureflag.
- evals.__main__._judge_records(records: list[dict], cases_by_id: dict, model: str, api_key: str) None[source]¶
Score every diagnosed record with the judge, mutating each record in place.
For each record it: (1) skips analyzer failures, marking them as not passing; (2) computes
category_matchdirectly in code (exact string compare of the diagnosed category vs. the expected one) — this is deterministic and does not rely on the judge; (3) callsjudge.judge_case()and stores the raw judge output underjudge; (4) recomputesoverall_passhere as “all four dimensions passed” rather than trusting the judge’s ownoverall_verdict, so the headline pass/fail is always consistent with the per-dimension verdicts. A judge that errors or returns malformed JSON is recorded asjudge_error.- Parameters:
records – Records from
_diagnose_cases().cases_by_id – The dataset keyed by case id, to rebuild each judge payload.
model – Judge model id.
api_key – Groq API key for the judge.
- evals.__main__._make_scout()[source]¶
Build a fresh, force-enabled analyzer instance.
Mirrors how
tests/conftest.pyforce-enables analysis under--failure-analysis(setsenabledand re-reads the key), but on a new instance rather than the module singleton — so the eval never appends to the singleton’s_resultsor writesai_analysis/, and works regardless of theAI_ANALYSIS_ENABLEDenv var.- Returns:
A ready-to-use
FailureAnalyzer.
- evals.__main__._print_table(records: list[dict], summary: dict) None[source]¶
Print the per-case table and the aggregate summary to the console.
One row per case showing expected→diagnosed category (with a ✓/✗ match mark) and the four per-dimension verdicts, followed by the counts and pass-rate percentages from
_summarize().- Parameters:
records – The fully judged records.
summary – The summary dict from
_summarize().
- evals.__main__._summarize(records: list[dict]) dict[source]¶
Roll the per-case records up into the report’s aggregate
summaryblock.Every number here is just a count or a count/total fraction:
scout_failures/judge_errors— how many cases the analyzer or the judge could not produce output for. These are surfaced separately and excluded from the quality fractions below, so an infrastructure hiccup doesn’t masquerade as a quality regression.category_accuracy— of the cases the analyzer actually diagnosed, the fraction whose category matched the expected one. Computed in code, not by the judge.pass_rates— of the judged cases, the fraction the judge passed on each dimension (correctness/groundedness/completeness/actionability) plusoverall(passed all four). This is the headline quality score: one number per dimension that you can compare across runs to see if a prompt/model change made the analyzer better or worse.
- Parameters:
records – The fully judged records.
- Returns:
The
summarydict embedded in the report and printed to console.
- evals.__main__.main(argv=None) int[source]¶
Entry point: diagnose every dataset case, judge each diagnosis, report.
The orchestrator that Loads the dataset, runs the analyzer over it, judges the results, writes the JSON report, and prints the summary table.
The shape of it: It does no analysis or grading itself, it just sequences the named stages and handles the I/O boundary (args in, env check, file out, exit code out). Three things stand out as intentional: the early fail-fast on the missing key (exit 2 before any work), the single shared records list threaded through produce→grade→aggregate, and the three distinct exit codes that let a caller distinguish can’t-run / failed / passed
- Parameters:
argv – Optional argument list (defaults to
sys.argv).- Returns:
Process exit code —
0if every judged case passed all four dimensions,1if any failed,2ifGROQ_API_KEYis missing.