Evaluation Harness (evals)

LLM-as-a-judge evaluation of the AI failure analyzer’s diagnosis quality. Runs FailureAnalyzer (Qwen3.8 27B via Groq) over a curated golden dataset, then scores each diagnosis with a separate judge model (openai/gpt-oss-120b by default — a different model family, so no self-preference). Run with poetry run python -m evals.

Dataset

Golden-dataset loader for the failure-analyzer evaluation.

Reads the curated YAML dataset of TMDB failure contexts (one or more per diagnosis category) and validates each case has the structure the runner and judge expect. The dataset is the ground truth the judge scores correctness against, so a malformed case is a hard error rather than a silent skip.

evals.dataset.load_dataset(path: str | Path) → list[dict][source]

Load and validate the golden eval dataset.

Parameters:

path – Path to the YAML dataset (a mapping with a top-level cases list).

Returns:

The list of case dicts, each with id, failure_context, and expected keys. Example return:

[
    {
        "id": "timeout_slow_response",
        "failure_context": {
            "test_name": "test_update_list_description[update_list0]",
            "api_url": "https://api.themoviedb.org/4/list/8284605",
            "status_code": 201,
            "error_message": "AssertionError: Response time 17.7s exceeded expected max 15s",
        },
        "expected": {
            "expected_category": "timeout",
            "expected_root_cause": "The call succeeded (HTTP 201) but took ~17.7s, over the 15s threshold.",
        },
    },
]

Raises:

ValueError – If the file is malformed or any case is missing a required field, has a duplicate id, or names a category outside VALID_CATEGORIES. The message names the offending case.

Judge

LLM-as-a-judge for the AI failure analyzer.

Sends a JSON payload (the failure context, the diagnosis under test, and — for offline evaluation — the expected reference) to a judge model on Groq and gets back a strict per-dimension verdict on whether the diagnosis is correct, grounded, complete, and actionable.

Two modes share one judge call:

  • judge_case() (offline eval) — scores all four dimensions. Correctness is scored against the case’s expected reference; groundedness, completeness, and actionability are judged from failure_context alone.

  • judge_live() (in-pytest, on real failures) — scores groundedness, completeness, and actionability (every dimension except correctness), because a real failure has no reference answer to score correctness against.

The judge runs on Groq (same provider as the analyzer it evaluates) but a different model family — GPT-OSS judging Qwen — so there is no self-preference bias.

evals.judge._call_judge(system_prompt: str, output_schema: dict, payload: dict, model: str, api_key: str | None) → dict | None[source]

Make one judge call and return the parsed, validated verdict.

Sends system_prompt plus the JSON-serialized payload to the judge model on Groq with strict structured outputs. If the account/model rejects strict json_schema, it retries once with the looser json_object mode the analyzer itself uses. Any API error, unparseable response, or output missing a required key returns None (the caller treats that as a judge error rather than crashing).

Parameters:
  • system_prompt – The judge instructions (full or live variant).

  • output_schema – The strict JSON Schema for the expected output.

  • payload – The judge input object to serialize into the user message.

  • model – Groq model id for the judge.

  • api_key – Groq API key; falls back to the client’s env-based default.

Returns:

The parsed judge output, or None on failure. Example return:

{
    "groundedness": {"verdict": "pass", "reasoning": "...", "issues": []},
    "completeness": {"verdict": "fail", "reasoning": "...",
                     "issues": ["omits the 404 status code"]},
    "actionability": {"verdict": "pass", "reasoning": "...", "issues": []},
    "overall_verdict": "fail",
}

evals.judge._output_schema(dimensions: tuple[str, ...]) → dict[source]

Build the strict judge output schema for a given set of dimensions.

Parameters:

dimensions – The dimension names the judge must return (e.g. ("correctness", "groundedness", "completeness")).

Returns:

A JSON Schema requiring one _DIMENSION_SCHEMA block per dimension plus an overall_verdict string. Example return:

{
    "type": "object",
    "additionalProperties": False,
    "required": ["groundedness", "overall_verdict"],
    "properties": {
        "groundedness": {  # one _DIMENSION_SCHEMA block per dimension
            "type": "object",
            "required": ["verdict", "reasoning", "issues"],
            "properties": {...},
        },
        "overall_verdict": {"type": "string", "enum": ["pass", "fail"]},
    },
}

evals.judge.judge_case(payload: dict, model: str = 'openai/gpt-oss-120b', api_key: str | None = None) → dict | None[source]

Score one diagnosis on all four dimensions against its expected reference.

Used by the offline eval, where each case carries an expected block.

Parameters:
  • payload – The judge input object (case_id, failure_context, diagnosis, expected).

  • model – Groq model id for the judge. Defaults to DEFAULT_JUDGE_MODEL.

  • api_key – Groq API key; falls back to the client’s env-based default.

Returns:

The parsed judge output (per-dimension {verdict, reasoning, issues} for correctness/groundedness/completeness/actionability plus overall_verdict), or None if the call or parse fails. Example return:

{
    "correctness": {"verdict": "pass", "reasoning": "...", "issues": []},
    "groundedness": {"verdict": "pass", "reasoning": "...", "issues": []},
    "completeness": {"verdict": "pass", "reasoning": "...", "issues": []},
    "actionability": {"verdict": "pass", "reasoning": "...", "issues": []},
    "overall_verdict": "pass",
}

evals.judge.judge_live(failure_context: dict, diagnosis: dict, model: str = 'openai/gpt-oss-120b', api_key: str | None = None) → dict | None[source]

Score a live failure’s diagnosis on groundedness, completeness, and actionability.

Used in-pytest on live failures, which have no reference answer — so correctness is deliberately not judged. The diagnosis is cleaned with sanitize_diagnosis() before being shown to the judge.

Parameters:
  • failure_context – The context the analyzer was given for this failure.

  • diagnosis – The raw diagnosis the analyzer produced.

  • model – Groq model id for the judge. Defaults to DEFAULT_JUDGE_MODEL.

  • api_key – Groq API key; falls back to the client’s env-based default.

Returns:

The parsed judge output (groundedness, completeness, actionability, overall_verdict), or None if the call or parse fails. Example return:

{
    "groundedness": {"verdict": "pass", "reasoning": "...", "issues": []},
    "completeness": {"verdict": "pass", "reasoning": "...", "issues": []},
    "actionability": {"verdict": "fail", "reasoning": "...",
                      "issues": ["suggested_fix is too vague"]},
    "overall_verdict": "fail",
}

evals.judge.sanitize_diagnosis(diagnosis: dict) → dict[source]

Clean an analyzer diagnosis before it is shown to the judge.

The analyzer (FailureAnalyzer.analyze) injects two bookkeeping keys into its output — test_name and model. We drop them so the judge sees only the model’s actual answer and is never biased by knowing which model produced it. We also coerce confidence from a numeric string (the analyzer sometimes returns "90") to an int for a consistent type; a value we can’t parse is left untouched because the judge prompt explicitly tolerates a non-int confidence.

Parameters:

diagnosis – The raw diagnosis dict from the analyzer.

Returns:

A new dict safe to embed in a judge payload and a report (test_name/model dropped, confidence coerced to int). Example return:

{
    "root_cause": "Stale movie id 550 returned HTTP 404",
    "category": "data_issue",
    "suggested_fix": "Refresh the movie id used in test_data.yaml",
    "confidence": 90,
    "explanation": "The request to /movie/550 returned 404 Not Found...",
    "evidence": ["status_code 404", "error message 'could not be found'"],
}

CLI Runner

CLI runner for the failure-analyzer evaluation.

Runs the AI failure analyzer (Qwen3.8 27B via Groq) live on each case in the golden dataset, then judges every produced diagnosis with the GPT-OSS judge on four dimensions (correctness, groundedness, completeness, actionability). Prints a per-case table plus aggregate metrics and writes a JSON report.

Both the analyzer under test and the judge call Groq, so only GROQ_API_KEY is required.

Usage:

poetry run python -m evals
poetry run python -m evals --judge-model openai/gpt-oss-120b
evals.__main__._build_payload(case: dict, diagnosis: dict) → dict[source]

Assemble the single JSON object sent to the judge for one case.

The payload bundles the three things the judge needs: the failure_context the analyzer saw, the diagnosis it produced (already cleaned), and the expected reference answer the dataset author wrote.

The failure_context is fed to Scout to produce the whole diagnosis, and that one diagnosis is then judged on all four dimensions — failure_context is not tied to any single dimension. What sets correctness apart is the judge, not the input: correctness compares Scout’s diagnosis (its category and root_cause) against expected — the known-true answer a human authored for this case — so it needs that reference to grade against. Groundedness, completeness, and actionability need no reference: they judge the diagnosis against the failure_context evidence itself, asking whether it is internally sound, not whether it is right.

This is why correctness is an offline-only dimension. expected exists only in the curated dataset; it is hand-written by someone who already knew the true cause, and is not derivable from failure_context (if it were, we would not need Scout). A live failure has no such answer key — so judge_live drops correctness and scores only the other three. This is the exact shape judge.judge_case documents and JUDGE_SYSTEM_PROMPT describes.

How it differs from the live path

┌──────────────────────┬───────────────────────────────────────────────┬─────────────────────────────┐ │ │ _build_payload (offline) │ judge_live’s inline payload │ ├──────────────────────┼───────────────────────────────────────────────┼─────────────────────────────┤ │ keys │ case_id, failure_context, diagnosis, expected │ failure_context, diagnosis │ ├──────────────────────┼───────────────────────────────────────────────┼─────────────────────────────┤ │ expected? │ yes (enables correctness) │ no (no reference) │ ├──────────────────────┼───────────────────────────────────────────────┼─────────────────────────────┤ │ sanitizes diagnosis? │ no — caller already did │ yes — inline │ └──────────────────────┴───────────────────────────────────────────────┴─────────────────────────────┘

Parameters:
  • case – A dataset case (id, failure_context, expected) from evals.dataset.load_dataset()

  • diagnosis – The cleaned diagnosis for that case.

Returns:

The judge input object.

evals.__main__._diagnose_cases(cases: list[dict]) → list[dict][source]

Run the analyzer once per dataset case and collect the diagnoses.

Uses a single fresh analyzer instance (see _make_scout()) for the whole pass. For each case it calls analyze(failure_context) and records the produced diagnosis. analyze returns None when the analyzer’s own LLM call fails; that is captured as scout_failure so the case is carried through the report (and excluded from quality metrics) rather than crashing the run.

Exercise the analyzer (Qwen3.8 27B) on known inputs. Each dataset case carries a realistic failure_context; this function feeds that context to a fresh FailureAnalyzer and captures what the analyzer model says. This is the only place in the eval where the system-under-test actually runs.

Purpose: Take the golden dataset and produce one diagnosis per case from the live analyzer — the raw material the judge will later grade.

Parameters:

cases – The validated dataset cases.

Returns:

One record per case, each with case_id, expected_category (lifted out for the table/metrics), the cleaned diagnosis (or None), and a scout_failure flag.

evals.__main__._judge_records(records: list[dict], cases_by_id: dict, model: str, api_key: str) → None[source]

Score every diagnosed record with the judge, mutating each record in place.

For each record it: (1) skips analyzer failures, marking them as not passing; (2) computes category_match directly in code (exact string compare of the diagnosed category vs. the expected one) — this is deterministic and does not rely on the judge; (3) calls judge.judge_case() and stores the raw judge output under judge; (4) recomputes overall_pass here as “all four dimensions passed” rather than trusting the judge’s own overall_verdict, so the headline pass/fail is always consistent with the per-dimension verdicts. A judge that errors or returns malformed JSON is recorded as judge_error.

Parameters:
  • records – Records from _diagnose_cases().

  • cases_by_id – The dataset keyed by case id, to rebuild each judge payload.

  • model – Judge model id.

  • api_key – Groq API key for the judge.

evals.__main__._make_scout()[source]

Build a fresh, force-enabled analyzer instance.

Mirrors how tests/conftest.py force-enables analysis under --failure-analysis (sets enabled and re-reads the key), but on a new instance rather than the module singleton — so the eval never appends to the singleton’s _results or writes ai_analysis/, and works regardless of the AI_ANALYSIS_ENABLED env var.

Returns:

A ready-to-use FailureAnalyzer.

evals.__main__._print_table(records: list[dict], summary: dict) → None[source]

Print the per-case table and the aggregate summary to the console.

One row per case showing expected→diagnosed category (with a ✓/✗ match mark) and the four per-dimension verdicts, followed by the counts and pass-rate percentages from _summarize().

Parameters:
  • records – The fully judged records.

  • summary – The summary dict from _summarize().

evals.__main__._summarize(records: list[dict]) → dict[source]

Roll the per-case records up into the report’s aggregate summary block.

Every number here is just a count or a count/total fraction:

  • scout_failures / judge_errors — how many cases the analyzer or the judge could not produce output for. These are surfaced separately and excluded from the quality fractions below, so an infrastructure hiccup doesn’t masquerade as a quality regression.

  • category_accuracy — of the cases the analyzer actually diagnosed, the fraction whose category matched the expected one. Computed in code, not by the judge.

  • pass_rates — of the judged cases, the fraction the judge passed on each dimension (correctness/groundedness/completeness/actionability) plus overall (passed all four). This is the headline quality score: one number per dimension that you can compare across runs to see if a prompt/model change made the analyzer better or worse.

Parameters:

records – The fully judged records.

Returns:

The summary dict embedded in the report and printed to console.

evals.__main__.main(argv=None) → int[source]

Entry point: diagnose every dataset case, judge each diagnosis, report.

The orchestrator that Loads the dataset, runs the analyzer over it, judges the results, writes the JSON report, and prints the summary table.

The shape of it: It does no analysis or grading itself, it just sequences the named stages and handles the I/O boundary (args in, env check, file out, exit code out). Three things stand out as intentional: the early fail-fast on the missing key (exit 2 before any work), the single shared records list threaded through produce→grade→aggregate, and the three distinct exit codes that let a caller distinguish can’t-run / failed / passed

Parameters:

argv – Optional argument list (defaults to sys.argv).

Returns:

Process exit code — 0 if every judged case passed all four dimensions, 1 if any failed, 2 if GROQ_API_KEY is missing.