Failure Analyzer¶
AI-powered test failure analyzer using open-source LLMs via Groq.
Sends test failure context (error, test code, API response) to an LLM and returns a structured diagnosis with root cause, suggested fix, and failure classification.
- class tests.helpers.failure_analyzer.FailureAnalyzer(model: str = None)[source]¶
Bases:
objectAnalyzes test failures using an open-source LLM via Groq’s free API.
Disabled by default. Enable by setting AI_ANALYSIS_ENABLED=true and GROQ_API_KEY in the .env file.
- Parameters:
model – The LLM model identifier to use on Groq.
- analyze(failure_context: dict) dict | None[source]¶
Send failure context to the LLM and return structured diagnosis.
- Parameters:
failure_context – Dictionary containing failure details: - test_name: Name of the failed test - test_file: Path to the test file - error_message: The assertion or exception message - traceback: Full traceback string - api_url: The API URL that was called (if available) - status_code: HTTP status code received (if available) - response_body: API response body snippet (if available)
- Returns:
Parsed JSON diagnosis dict, or None if analysis is disabled/fails. Example return:
{ "root_cause": "Movie id 0 is invalid, so TMDB returned 404.", "category": "data_issue", "suggested_fix": "Use a valid movie id from movie_ids.txt in the test data.", "confidence": 92, "explanation": "The request targeted /3/movie/0; TMDB rejects id 0 ...", "evidence": ["HTTP 404", "status_message: The resource you requested could not be found."], "test_name": "test_get_movie_details", "model": "qwen/qwen3.8-27b" }
- property client¶
Lazy-initialize the Groq client on first access.
Delays the
from groq import Groqimport until analysis is actually needed, avoiding import errors when thegroqpackage is installed but never used.- Returns:
Configured
groq.Groqclient instance.
- refine(failure_context: dict, prior_diag: dict, weaknesses: list[str]) dict | None[source]¶
Produce an improved diagnosis given a critique of a prior one.
Called by
refine_until_confident()when the independent judge flags a diagnosis. Re-prompts the model with the original failure context, the prior diagnosis, and the judge’s concreteweaknesses, asking it to address each one and re-examine the evidence. The improved diagnosis replaces the prior one inself._results(matched bytest_name) so the saved report and terminal summary reflect the final version.
- property results: list[dict]¶
All diagnoses accumulated this session (read-only view).
- save_results(output_dir: str = 'ai_analysis')[source]¶
Write all accumulated analysis results to a JSON file.
Called once at the end of the test session by
pytest_sessionfinishinconftest.py. Skips writing if no failures were analyzed. Creates the output directory if it doesn’t exist.- Parameters:
output_dir – Directory path to write the results file to. Defaults to
ai_analysis/relative to the project root.
Output file:
<output_dir>/failure_analysis.json
- tests.helpers.failure_analyzer.refine_until_confident(analyzer_instance, failure_context, diagnosis, judge_fn, *, max_iterations, confidence_target)[source]¶
Iteratively improve a diagnosis using an independent judge as the critic.
Agentic loop: judge the current diagnosis; if the judge passes and the self-reported
confidenceclearsconfidence_target, stop. Otherwise feed the judge’s failed-dimension issues back intoFailureAnalyzer.refine()and re-judge, up tomax_iterationstimes.The judge is injected (
judge_fn) so this module stays decoupled fromevals.judgeand the loop is unit-testable. ANonejudgement (judge API error) stops the loop — without a critic signal there is nothing to refine against.- Parameters:
analyzer_instance – The
FailureAnalyzerthat produceddiagnosis.failure_context – The failure context dict.
diagnosis – The initial diagnosis from
FailureAnalyzer.analyze().judge_fn – Callable
(failure_context, diagnosis) -> judgement | None.max_iterations – Maximum number of refine passes.
confidence_target – Confidence (0–100) the diagnosis must reach.
- Returns:
Tuple of the final
(diagnosis, judgement)— the diagnosis has the shape shown inFailureAnalyzer.analyze(), the judgement is thejudge_fnoutput (orNoneif the judge errored). Example return:( {"root_cause": "...", "category": "data_issue", "confidence": 92, ...}, { "groundedness": {"verdict": "pass", "reasoning": "...", "issues": []}, "completeness": {"verdict": "pass", "reasoning": "...", "issues": []}, "actionability": {"verdict": "pass", "reasoning": "...", "issues": []}, "overall_verdict": "pass", }, )