Failure Analyzer

AI-powered test failure analyzer using open-source LLMs via Groq.

Sends test failure context (error, test code, API response) to an LLM and returns a structured diagnosis with root cause, suggested fix, and failure classification.

class tests.helpers.failure_analyzer.FailureAnalyzer(model: str = None)[source]

Bases: object

Analyzes test failures using an open-source LLM via Groq’s free API.

Disabled by default. Enable by setting AI_ANALYSIS_ENABLED=true and GROQ_API_KEY in the .env file.

Parameters:

model – The LLM model identifier to use on Groq.

analyze(failure_context: dict) → dict | None[source]

Send failure context to the LLM and return structured diagnosis.

Parameters:

failure_context – Dictionary containing failure details: - test_name: Name of the failed test - test_file: Path to the test file - error_message: The assertion or exception message - traceback: Full traceback string - api_url: The API URL that was called (if available) - status_code: HTTP status code received (if available) - response_body: API response body snippet (if available)

Returns:

Parsed JSON diagnosis dict, or None if analysis is disabled/fails. Example return:

{
    "root_cause": "Movie id 0 is invalid, so TMDB returned 404.",
    "category": "data_issue",
    "suggested_fix": "Use a valid movie id from movie_ids.txt in the test data.",
    "confidence": 92,
    "explanation": "The request targeted /3/movie/0; TMDB rejects id 0 ...",
    "evidence": ["HTTP 404", "status_message: The resource you requested could not be found."],
    "test_name": "test_get_movie_details",
    "model": "qwen/qwen3.8-27b"
}

property client

Lazy-initialize the Groq client on first access.

Delays the from groq import Groq import until analysis is actually needed, avoiding import errors when the groq package is installed but never used.

Returns:

Configured groq.Groq client instance.

refine(failure_context: dict, prior_diag: dict, weaknesses: list[str]) → dict | None[source]

Produce an improved diagnosis given a critique of a prior one.

Called by refine_until_confident() when the independent judge flags a diagnosis. Re-prompts the model with the original failure context, the prior diagnosis, and the judge’s concrete weaknesses, asking it to address each one and re-examine the evidence. The improved diagnosis replaces the prior one in self._results (matched by test_name) so the saved report and terminal summary reflect the final version.

Parameters:
  • failure_context – Same dict passed to analyze().

  • prior_diag – The diagnosis being critiqued.

  • weaknesses – Concrete weaknesses from the judge to address.

Returns:

Improved diagnosis dict (same shape as analyze()’s return), or None if analysis is disabled or the call fails.

property results: list[dict]

All diagnoses accumulated this session (read-only view).

save_results(output_dir: str = 'ai_analysis')[source]

Write all accumulated analysis results to a JSON file.

Called once at the end of the test session by pytest_sessionfinish in conftest.py. Skips writing if no failures were analyzed. Creates the output directory if it doesn’t exist.

Parameters:

output_dir – Directory path to write the results file to. Defaults to ai_analysis/ relative to the project root.

Output file: <output_dir>/failure_analysis.json

tests.helpers.failure_analyzer.refine_until_confident(analyzer_instance, failure_context, diagnosis, judge_fn, *, max_iterations, confidence_target)[source]

Iteratively improve a diagnosis using an independent judge as the critic.

Agentic loop: judge the current diagnosis; if the judge passes and the self-reported confidence clears confidence_target, stop. Otherwise feed the judge’s failed-dimension issues back into FailureAnalyzer.refine() and re-judge, up to max_iterations times.

The judge is injected (judge_fn) so this module stays decoupled from evals.judge and the loop is unit-testable. A None judgement (judge API error) stops the loop — without a critic signal there is nothing to refine against.

Parameters:
  • analyzer_instance – The FailureAnalyzer that produced diagnosis.

  • failure_context – The failure context dict.

  • diagnosis – The initial diagnosis from FailureAnalyzer.analyze().

  • judge_fn – Callable (failure_context, diagnosis) -> judgement | None.

  • max_iterations – Maximum number of refine passes.

  • confidence_target – Confidence (0–100) the diagnosis must reach.

Returns:

Tuple of the final (diagnosis, judgement) — the diagnosis has the shape shown in FailureAnalyzer.analyze(), the judgement is the judge_fn output (or None if the judge errored). Example return:

(
    {"root_cause": "...", "category": "data_issue", "confidence": 92, ...},
    {
        "groundedness": {"verdict": "pass", "reasoning": "...", "issues": []},
        "completeness": {"verdict": "pass", "reasoning": "...", "issues": []},
        "actionability": {"verdict": "pass", "reasoning": "...", "issues": []},
        "overall_verdict": "pass",
    },
)