Source code for evals.judge

"""
LLM-as-a-judge for the AI failure analyzer.

Sends a JSON payload (the failure context, the diagnosis under test, and —
for offline evaluation — the expected reference) to a judge model on Groq and
gets back a strict per-dimension verdict on whether the diagnosis is correct,
grounded, complete, and actionable.

Two modes share one judge call:

- :func:`judge_case` (offline eval) — scores all four dimensions. Correctness
  is scored against the case's ``expected`` reference; groundedness,
  completeness, and actionability are judged from ``failure_context`` alone.
- :func:`judge_live` (in-pytest, on real failures) — scores groundedness,
  completeness, and actionability (every dimension except correctness), because
  a real failure has no reference answer to score correctness against.

The judge runs on Groq (same provider as the analyzer it evaluates) but a
different model family — GPT-OSS judging Qwen — so there is no self-preference
bias.

.. module:: evals.judge
   :synopsis: GPT-OSS-as-a-judge wrapper over Groq.
   :no-index:
"""

import json

from loguru import logger

from telemetry import ledger

DEFAULT_JUDGE_MODEL = "openai/gpt-oss-120b"

JUDGE_SYSTEM_PROMPT = """\
You are a strict, impartial evaluator of AI-generated API test failure diagnoses. You will receive a JSON object with three parts:

- "failure_context": the raw test failure information (test name, file, API URL, status code, response body, error message, traceback) that was given to a diagnosis model. This is the ONLY source of truth about what happened.
- "diagnosis": the diagnosis produced by that model, with fields: root_cause, category, suggested_fix, confidence, explanation, evidence.
- "expected": the reference answer, with expected_category (one of: api_bug | test_bug | data_issue | timeout | auth_error | schema_mismatch | environment) and expected_root_cause (a short gist of the true root cause).

Evaluate the diagnosis on exactly four dimensions and give each a verdict of "pass" or "fail":

1. correctness — Does the diagnosis identify the right failure cause?
   - FAIL if diagnosis.category does not exactly equal expected.expected_category.
   - FAIL if diagnosis.root_cause is not semantically consistent with expected.expected_root_cause (it need not match the wording, but it must describe the same underlying cause).
   - FAIL if diagnosis.suggested_fix addresses a different problem than expected.expected_root_cause describes.
   - Otherwise PASS.

2. groundedness — Is every claim supported by failure_context?
   - Check every factual claim in diagnosis.root_cause, diagnosis.explanation, and every item in diagnosis.evidence.
   - FAIL if any claim asserts a status code, field name, value, timing, URL, error text, or other fact that does not appear in, and cannot be directly derived from, failure_context. Paraphrase is fine; invention is not.
   - FAIL if any evidence item does not correspond to something actually present in failure_context.
   - Do NOT use the expected reference for this dimension; a diagnosis can be wrong yet grounded, or correct yet hallucinated.
   - Otherwise PASS.

3. completeness — Does the diagnosis cover the decisive signals?
   - FAIL if it ignores a key signal present in failure_context (e.g. the HTTP status code, the central error message, a timing value, or the specific assertion that failed) that a competent analyst would cite.
   - FAIL if any required field (root_cause, category, suggested_fix, confidence, explanation, evidence) is missing, empty, or filled with boilerplate that adds no information. Treat confidence as populated if it is a number between 0 and 100, even if encoded as a string.
   - Otherwise PASS.

4. actionability — Is suggested_fix a concrete, specific next step a developer could act on directly?
   - FAIL if suggested_fix is missing or empty.
   - FAIL if it is generic boilerplate that would fit almost any failure (e.g. "investigate the issue", "check the API", "review the test and fix it").
   - FAIL if it only names a direction without a concrete action (e.g. "handle the timeout better" fails; "raise exp_max_elp_secs above the observed 17.7s in the update_list test data" passes).
   - Judge only specificity and executability — a fix can be actionable even if it targets the wrong problem; that is correctness's concern, not this dimension's.
   - Otherwise PASS.

For each dimension provide 1-3 sentences of reasoning and a list of specific issues found (exact hallucinated claims, the missed signal, the category mismatch, the vague fix, etc.). Use an empty list when there are no issues.

Set overall_verdict to "pass" only if all four dimensions pass, otherwise "fail".

Be strict and literal. Judge only what is written; do not give credit for what the diagnosis "probably meant". Respond only with the required JSON object.\
"""

LIVE_JUDGE_SYSTEM_PROMPT = """\
You are a strict, impartial evaluator of AI-generated API test failure diagnoses. You will receive a JSON object with two parts:

- "failure_context": the raw test failure information (test name, file, API URL, status code, response body, error message, traceback) that was given to a diagnosis model. This is the ONLY source of truth about what happened.
- "diagnosis": the diagnosis produced by that model, with fields: root_cause, category, suggested_fix, confidence, explanation, evidence.

There is no reference answer here, so do NOT judge whether the diagnosis is correct. Evaluate exactly these three dimensions and give each a verdict of "pass" or "fail":

1. groundedness — Is every claim supported by failure_context?
   - Check every factual claim in diagnosis.root_cause, diagnosis.explanation, and every item in diagnosis.evidence.
   - FAIL if any claim asserts a status code, field name, value, timing, URL, error text, or other fact that does not appear in, and cannot be directly derived from, failure_context. Paraphrase is fine; invention is not.
   - FAIL if any evidence item does not correspond to something actually present in failure_context.
   - Otherwise PASS.

2. completeness — Does the diagnosis cover the decisive signals?
   - FAIL if it ignores a key signal present in failure_context (e.g. the HTTP status code, the central error message, a timing value, or the specific assertion that failed) that a competent analyst would cite.
   - FAIL if any required field (root_cause, category, suggested_fix, confidence, explanation, evidence) is missing, empty, or filled with boilerplate that adds no information. Treat confidence as populated if it is a number between 0 and 100, even if encoded as a string.
   - Otherwise PASS.

3. actionability — Is suggested_fix a concrete, specific next step a developer could act on directly?
   - FAIL if suggested_fix is missing or empty.
   - FAIL if it is generic boilerplate that would fit almost any failure (e.g. "investigate the issue", "check the API", "review the test and fix it").
   - FAIL if it only names a direction without a concrete action (e.g. "handle the timeout better" fails; "increase the read timeout above 30s" passes).
   - Judge only whether the fix is concrete and specific, not whether it actually solves the underlying problem.
   - Otherwise PASS.

For each dimension provide 1-3 sentences of reasoning and a list of specific issues found (exact hallucinated claims, the missed signal, the vague fix, etc.). Use an empty list when there are no issues.

Set overall_verdict to "pass" only if all three dimensions pass, otherwise "fail".

Be strict and literal. Judge only what is written; do not give credit for what the diagnosis "probably meant". Respond only with the required JSON object.\
"""

_DIMENSION_SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "required": ["verdict", "reasoning", "issues"],
    "properties": {
        "verdict": {"type": "string", "enum": ["pass", "fail"]},
        "reasoning": {"type": "string"},
        "issues": {"type": "array", "items": {"type": "string"}},
    },
}


[docs] def _output_schema(dimensions: tuple[str, ...]) -> dict: """ Build the strict judge output schema for a given set of dimensions. :param dimensions: The dimension names the judge must return (e.g. ``("correctness", "groundedness", "completeness")``). :returns: A JSON Schema requiring one ``_DIMENSION_SCHEMA`` block per dimension plus an ``overall_verdict`` string. Example return:: { "type": "object", "additionalProperties": False, "required": ["groundedness", "overall_verdict"], "properties": { "groundedness": { # one _DIMENSION_SCHEMA block per dimension "type": "object", "required": ["verdict", "reasoning", "issues"], "properties": {...}, }, "overall_verdict": {"type": "string", "enum": ["pass", "fail"]}, }, } """ return { "type": "object", "additionalProperties": False, "required": [*dimensions, "overall_verdict"], "properties": { **{dim: _DIMENSION_SCHEMA for dim in dimensions}, "overall_verdict": {"type": "string", "enum": ["pass", "fail"]}, }, }
JUDGE_OUTPUT_SCHEMA = _output_schema(("correctness", "groundedness", "completeness", "actionability")) LIVE_JUDGE_OUTPUT_SCHEMA = _output_schema(("groundedness", "completeness", "actionability"))
[docs] def sanitize_diagnosis(diagnosis: dict) -> dict: """ Clean an analyzer diagnosis before it is shown to the judge. The analyzer (``FailureAnalyzer.analyze``) injects two bookkeeping keys into its output — ``test_name`` and ``model``. We drop them so the judge sees only the model's actual answer and is never biased by knowing which model produced it. We also coerce ``confidence`` from a numeric string (the analyzer sometimes returns ``"90"``) to an int for a consistent type; a value we can't parse is left untouched because the judge prompt explicitly tolerates a non-int confidence. :param diagnosis: The raw diagnosis dict from the analyzer. :returns: A new dict safe to embed in a judge payload and a report (``test_name``/``model`` dropped, ``confidence`` coerced to int). Example return:: { "root_cause": "Stale movie id 550 returned HTTP 404", "category": "data_issue", "suggested_fix": "Refresh the movie id used in test_data.yaml", "confidence": 90, "explanation": "The request to /movie/550 returned 404 Not Found...", "evidence": ["status_code 404", "error message 'could not be found'"], } """ # rebuilds the diagnosis keeping every key except test_name and model so the judge can't be biased by knowing which # model wrote the answer cleaned = {k: v for k, v in diagnosis.items() if k not in ("test_name", "model")} if "confidence" in cleaned: try: # Parsing to int() for consistency in report and judge payload cleaned["confidence"] = int(str(cleaned["confidence"]).strip()) except (ValueError, TypeError): pass # leave as-is; the judge tolerates a non-int confidence return cleaned # stripped-and-normalized copy, ready to embed in a judge payload
[docs] def _call_judge(system_prompt: str, output_schema: dict, payload: dict, model: str, api_key: str | None) -> dict | None: """ Make one judge call and return the parsed, validated verdict. Sends ``system_prompt`` plus the JSON-serialized ``payload`` to the judge model on Groq with strict structured outputs. If the account/model rejects strict ``json_schema``, it retries once with the looser ``json_object`` mode the analyzer itself uses. Any API error, unparseable response, or output missing a required key returns ``None`` (the caller treats that as a judge error rather than crashing). :param system_prompt: The judge instructions (full or live variant). :param output_schema: The strict JSON Schema for the expected output. :param payload: The judge input object to serialize into the user message. :param model: Groq model id for the judge. :param api_key: Groq API key; falls back to the client's env-based default. :returns: The parsed judge output, or ``None`` on failure. Example return:: { "groundedness": {"verdict": "pass", "reasoning": "...", "issues": []}, "completeness": {"verdict": "fail", "reasoning": "...", "issues": ["omits the 404 status code"]}, "actionability": {"verdict": "pass", "reasoning": "...", "issues": []}, "overall_verdict": "fail", } """ # Lazy import from groq import Groq client = Groq(api_key=api_key) logger.debug(f"Sending user payload {json.dumps(payload, indent=2)}") messages = [ {"role": "system", "content": system_prompt}, {"role": "user", "content": json.dumps(payload, indent=2)}, ] # temperature — near-deterministic; a grader should be stable, not creative. # reasoning_effort — GPT model is a reasoning model; low effort keeps it cheap/fast as judging is a constrained task. # max_completion_tokens — headroom for the reasoning tokens GPT emits before the JSON. kwargs = dict(model=model, messages=messages, temperature=0.1, reasoning_effort="low", max_completion_tokens=1500) try: response = client.chat.completions.create( **kwargs, response_format={ "type": "json_schema", "json_schema": {"name": "diagnosis_judgement", "strict": True, "schema": output_schema}, }, ) except Exception as e: # Some models/accounts don't accept strict json_schema; retry with the # looser json_object mode the analyzer itself uses. logger.warning(f"Judge json_schema call failed ({e}); retrying with json_object") try: response = client.chat.completions.create(**kwargs, response_format={"type": "json_object"}) except Exception as e2: logger.warning(f"Judge call failed: {e2}") return None # Salvage the free usage block (both create branches converge here) before # parsing — the judge tokens were spent regardless of the parse outcome. ledger.record( model=model, role="judge", usage=response.usage, test_name=payload.get("failure_context", {}).get("test_name", payload.get("case_id", "unknown")), ) try: result = json.loads(response.choices[0].message.content) except (json.JSONDecodeError, TypeError) as e: logger.warning(f"Judge returned unparseable JSON: {e}") return None # checks every key is present; if any is missing, log the keys for debugging and return none if not all(k in result for k in output_schema["required"]): logger.warning(f"Judge output missing required keys: {sorted(result)}") return None return result
[docs] def judge_case(payload: dict, model: str = DEFAULT_JUDGE_MODEL, api_key: str | None = None) -> dict | None: """ Score one diagnosis on all four dimensions against its expected reference. Used by the offline eval, where each case carries an ``expected`` block. :param payload: The judge input object (``case_id``, ``failure_context``, ``diagnosis``, ``expected``). :param model: Groq model id for the judge. Defaults to DEFAULT_JUDGE_MODEL. :param api_key: Groq API key; falls back to the client's env-based default. :returns: The parsed judge output (per-dimension ``{verdict, reasoning, issues}`` for correctness/groundedness/completeness/actionability plus ``overall_verdict``), or ``None`` if the call or parse fails. Example return:: { "correctness": {"verdict": "pass", "reasoning": "...", "issues": []}, "groundedness": {"verdict": "pass", "reasoning": "...", "issues": []}, "completeness": {"verdict": "pass", "reasoning": "...", "issues": []}, "actionability": {"verdict": "pass", "reasoning": "...", "issues": []}, "overall_verdict": "pass", } """ return _call_judge(JUDGE_SYSTEM_PROMPT, JUDGE_OUTPUT_SCHEMA, payload, model, api_key)
[docs] def judge_live(failure_context: dict, diagnosis: dict, model: str = DEFAULT_JUDGE_MODEL, api_key: str | None = None) -> dict | None: """ Score a live failure's diagnosis on groundedness, completeness, and actionability. Used in-pytest on live failures, which have no reference answer — so correctness is deliberately not judged. The diagnosis is cleaned with :func:`sanitize_diagnosis` before being shown to the judge. :param failure_context: The context the analyzer was given for this failure. :param diagnosis: The raw diagnosis the analyzer produced. :param model: Groq model id for the judge. Defaults to DEFAULT_JUDGE_MODEL. :param api_key: Groq API key; falls back to the client's env-based default. :returns: The parsed judge output (``groundedness``, ``completeness``, ``actionability``, ``overall_verdict``), or ``None`` if the call or parse fails. Example return:: { "groundedness": {"verdict": "pass", "reasoning": "...", "issues": []}, "completeness": {"verdict": "pass", "reasoning": "...", "issues": []}, "actionability": {"verdict": "fail", "reasoning": "...", "issues": ["suggested_fix is too vague"]}, "overall_verdict": "fail", } """ payload = {"failure_context": failure_context, "diagnosis": sanitize_diagnosis(diagnosis)} return _call_judge(LIVE_JUDGE_SYSTEM_PROMPT, LIVE_JUDGE_OUTPUT_SCHEMA, payload, model, api_key)