Telemetry Module¶
Process-wide token/cost ledger for the AI failure-analysis pipeline.
Every Groq response carries a usage block (prompt/completion/total tokens)
that the analyzer and judge would otherwise discard. Both call sites record it
here via TokenLedger.record(); conftest.py drains the ledger at
session end, rolls it up into one run-level row (tokens + estimated cost +
quality), and appends it to ai_analysis/token_usage.jsonl for cross-run
trending.
Tokens are the durable, provider-independent signal; the dollar figure is a
derived estimate from a small, dated price table and is best-effort only — a
model missing from the table contributes 0 to the cost estimate but its tokens
are still counted. Recording is defensive: a malformed usage object can
never raise into a test run.
- class telemetry.TokenLedger[source]¶
Bases:
objectAccumulates per-call token usage for one pytest session.
A single process-wide instance (
ledger) is shared by the analyzer and the judge;conftest.pyreads it once at session end. Not thread-safe — the pipeline runs calls serially per test.- estimate_cost_usd() float[source]¶
Best-effort dollar estimate for all recorded calls.
cost = Σ (prompt_tokens × in_price + completion_tokens × out_price)from_PRICE_PER_MILLION. Calls whose model is not in the price table contribute 0.- Returns:
Estimated cost in USD, rounded to 6 decimals. Example return for 1000 prompt + 200 completion tokens on qwen/qwen3.8-27b (
1000/1e6 * 0.80 + 200/1e6 * 4.00):0.0016
- record(*, model: str, role: str, usage, test_name: str = 'unknown') None[source]¶
Record one LLM call’s token usage from a Groq
response.usageobject.Defensive by design: a
Noneor malformedusagecounts as zero tokens rather than raising, so telemetry can never break a test run.- Parameters:
model – The Groq model id that served the call.
role – Which pipeline step spent the tokens (
analyze|refine|judge).usage – The Groq
CompletionUsageobject (response.usage);prompt_tokens/completion_tokensare read off it.test_name – The failed test this call was diagnosing.
- property records: list[_Record]¶
All calls recorded this session (read-only view).
- summary() dict[source]¶
Roll the recorded calls up into a run-level totals dict.
- Returns:
{calls, prompt_tokens, completion_tokens, total_tokens, est_cost_usd}— the token/cost half of the JSONL trend row. Example:{"calls": 5, "prompt_tokens": 8421, "completion_tokens": 1290, "total_tokens": 9711, "est_cost_usd": 0.0021}