Telemetry Module

Process-wide token/cost ledger for the AI failure-analysis pipeline.

Every Groq response carries a usage block (prompt/completion/total tokens) that the analyzer and judge would otherwise discard. Both call sites record it here via TokenLedger.record(); conftest.py drains the ledger at session end, rolls it up into one run-level row (tokens + estimated cost + quality), and appends it to ai_analysis/token_usage.jsonl for cross-run trending.

Tokens are the durable, provider-independent signal; the dollar figure is a derived estimate from a small, dated price table and is best-effort only — a model missing from the table contributes 0 to the cost estimate but its tokens are still counted. Recording is defensive: a malformed usage object can never raise into a test run.

class telemetry.TokenLedger[source]

Bases: object

Accumulates per-call token usage for one pytest session.

A single process-wide instance (ledger) is shared by the analyzer and the judge; conftest.py reads it once at session end. Not thread-safe — the pipeline runs calls serially per test.

estimate_cost_usd() → float[source]

Best-effort dollar estimate for all recorded calls.

cost = Σ (prompt_tokens × in_price + completion_tokens × out_price) from _PRICE_PER_MILLION. Calls whose model is not in the price table contribute 0.

Returns:

Estimated cost in USD, rounded to 6 decimals. Example return for 1000 prompt + 200 completion tokens on qwen/qwen3.8-27b (1000/1e6 * 0.80 + 200/1e6 * 4.00):

0.0016

record(*, model: str, role: str, usage, test_name: str = 'unknown') → None[source]

Record one LLM call’s token usage from a Groq response.usage object.

Defensive by design: a None or malformed usage counts as zero tokens rather than raising, so telemetry can never break a test run.

Parameters:
  • model – The Groq model id that served the call.

  • role – Which pipeline step spent the tokens (analyze | refine | judge).

  • usage – The Groq CompletionUsage object (response.usage); prompt_tokens / completion_tokens are read off it.

  • test_name – The failed test this call was diagnosing.

property records: list[_Record]

All calls recorded this session (read-only view).

reset() → None[source]

Drop all recorded calls (used by tests to isolate runs).

summary() → dict[source]

Roll the recorded calls up into a run-level totals dict.

Returns:

{calls, prompt_tokens, completion_tokens, total_tokens, est_cost_usd} — the token/cost half of the JSONL trend row. Example:

{"calls": 5, "prompt_tokens": 8421, "completion_tokens": 1290,
 "total_tokens": 9711, "est_cost_usd": 0.0021}