Repository navigation
token_usage_v1 averages a turn with no reported usage as 0, so the overall understates tokens per turn #7734
Description
Describe the Bug:
token_usage_v1 reports n/a (None) for a turn whose model calls returned no usage metadata, but the overall value still counts that turn as 0 tokens. With one turn that used 1,000 tokens and one turn with no reported usage, the overall is 500, half of what was measured.
The efficiency metrics sample README says "n/a never means zero", and inference_call_count_v1 and invocation_duration_v1 both leave n/a turns out of their averages. token_usage_v1 is the only one that counts them as zero.
The cause is in _TokenUsageV1Evaluator._average (src/google/adk/evaluation/_efficiency_evaluators.py), which divides by every invocation:
values = [getattr(details, field) for details in all_details]
present = [value for value in values if value is not None]
averaged[field] = sum(present) / len(values) if present else NoneTreating a single missing count as zero is intended and tested (test_token_usage_averages_an_unreported_count_as_zero_for_that_turn: a turn that reports usage but no thoughts_token_count spent no reasoning tokens). This bug is a different case: the turn reported no usage at all, so its token count is unknown, not zero. This can happen with backends that return no usage metadata, and in live evaluation (#7321).
Steps to Reproduce:
pip install google-adk==2.13.0- Run the script under "Minimal Reproduction Code".
Expected Behavior:
A turn with no reported usage is left out of the average, the same way the other efficiency metrics handle n/a. The overall is 1000.0.
Observed Behavior:
token_usage_v1 per turn=[1000.0, None] overall=500.0
inference_call_count_v1 per turn=[1.0, 1.0] overall=1.0
Environment Details:
- ADK Library Version (pip show google-adk): 2.13.0 (also on main @ bf67397)
- Desktop OS: macOS 26.6.2
- Python Version (python -V): 3.12.7
Model Information:
- Are you using LiteLLM: N/A (no model call needed to reproduce)
- Which model is being used: N/A
Minimal Reproduction Code:
from google.genai import types
from google.adk.evaluation.eval_case import Invocation, InvocationEvent, InvocationEvents
from google.adk.evaluation.eval_metrics import EvalMetric
from google.adk.evaluation.metric_evaluator_registry import DEFAULT_METRIC_EVALUATOR_REGISTRY
user = types.Content(role="user", parts=[types.Part(text="q")])
reply = types.Content(role="model", parts=[types.Part(text="a")])
# Turn 1: one model call that reported 1,000 tokens.
reported = Invocation(
user_content=user,
intermediate_data=InvocationEvents(invocation_events=[
InvocationEvent(
author="agent", content=reply, model_version="m",
usage_metadata=types.GenerateContentResponseUsageMetadata(
prompt_token_count=900, candidates_token_count=100, total_token_count=1000,
),
)
]),
)
# Turn 2: one model call whose backend reported no usage metadata.
unreported = Invocation(
user_content=user,
intermediate_data=InvocationEvents(invocation_events=[
InvocationEvent(author="agent", content=reply, model_version="m")
]),
)
for name in ("token_usage_v1", "inference_call_count_v1"):
evaluator = DEFAULT_METRIC_EVALUATOR_REGISTRY.get_evaluator(EvalMetric(metric_name=name))
result = evaluator.evaluate_invocations([reported, unreported])
per_turn = [r.score for r in result.per_invocation_results]
print(f"{name:24} per turn={per_turn} overall={result.overall_score}")Additional Context:
Proposed fix: in _average, compute the shared denominator over the invocations that reported any usage (total_tokens is not None). Missing counts within a reporting turn still average as zero, and total == input + output still holds after averaging. I tried this locally: the repro gives 1000.0 and all 39 tests in tests/unittests/evaluation/test_efficiency_evaluators.py pass. I'd be happy to send a PR with a regression test.
How often has this issue occurred?:
- Always (100%)
Activity
fix: leave no-usage turns out of the token_usage_v1 average