Skip to content

token_usage_v1 averages a turn with no reported usage as 0, so the overall understates tokens per turn #7734

Closed

Description

@Fruiterer333

Describe the Bug:

token_usage_v1 reports n/a (None) for a turn whose model calls returned no usage metadata, but the overall value still counts that turn as 0 tokens. With one turn that used 1,000 tokens and one turn with no reported usage, the overall is 500, half of what was measured.

The efficiency metrics sample README says "n/a never means zero", and inference_call_count_v1 and invocation_duration_v1 both leave n/a turns out of their averages. token_usage_v1 is the only one that counts them as zero.

The cause is in _TokenUsageV1Evaluator._average (src/google/adk/evaluation/_efficiency_evaluators.py), which divides by every invocation:

values = [getattr(details, field) for details in all_details]
present = [value for value in values if value is not None]
averaged[field] = sum(present) / len(values) if present else None

Treating a single missing count as zero is intended and tested (test_token_usage_averages_an_unreported_count_as_zero_for_that_turn: a turn that reports usage but no thoughts_token_count spent no reasoning tokens). This bug is a different case: the turn reported no usage at all, so its token count is unknown, not zero. This can happen with backends that return no usage metadata, and in live evaluation (#7321).

Steps to Reproduce:

  1. pip install google-adk==2.13.0
  2. Run the script under "Minimal Reproduction Code".

Expected Behavior:

A turn with no reported usage is left out of the average, the same way the other efficiency metrics handle n/a. The overall is 1000.0.

Observed Behavior:

token_usage_v1           per turn=[1000.0, None]  overall=500.0
inference_call_count_v1  per turn=[1.0, 1.0]  overall=1.0

Environment Details:

  • ADK Library Version (pip show google-adk): 2.13.0 (also on main @ bf67397)
  • Desktop OS: macOS 26.6.2
  • Python Version (python -V): 3.12.7

Model Information:

  • Are you using LiteLLM: N/A (no model call needed to reproduce)
  • Which model is being used: N/A

Minimal Reproduction Code:

from google.genai import types

from google.adk.evaluation.eval_case import Invocation, InvocationEvent, InvocationEvents
from google.adk.evaluation.eval_metrics import EvalMetric
from google.adk.evaluation.metric_evaluator_registry import DEFAULT_METRIC_EVALUATOR_REGISTRY

user = types.Content(role="user", parts=[types.Part(text="q")])
reply = types.Content(role="model", parts=[types.Part(text="a")])

# Turn 1: one model call that reported 1,000 tokens.
reported = Invocation(
    user_content=user,
    intermediate_data=InvocationEvents(invocation_events=[
        InvocationEvent(
            author="agent", content=reply, model_version="m",
            usage_metadata=types.GenerateContentResponseUsageMetadata(
                prompt_token_count=900, candidates_token_count=100, total_token_count=1000,
            ),
        )
    ]),
)
# Turn 2: one model call whose backend reported no usage metadata.
unreported = Invocation(
    user_content=user,
    intermediate_data=InvocationEvents(invocation_events=[
        InvocationEvent(author="agent", content=reply, model_version="m")
    ]),
)

for name in ("token_usage_v1", "inference_call_count_v1"):
    evaluator = DEFAULT_METRIC_EVALUATOR_REGISTRY.get_evaluator(EvalMetric(metric_name=name))
    result = evaluator.evaluate_invocations([reported, unreported])
    per_turn = [r.score for r in result.per_invocation_results]
    print(f"{name:24} per turn={per_turn}  overall={result.overall_score}")

Additional Context:

Proposed fix: in _average, compute the shared denominator over the invocations that reported any usage (total_tokens is not None). Missing counts within a reporting turn still average as zero, and total == input + output still holds after averaging. I tried this locally: the repro gives 1000.0 and all 39 tests in tests/unittests/evaluation/test_efficiency_evaluators.py pass. I'd be happy to send a PR with a regression test.

How often has this issue occurred?:

  • Always (100%)

Activity

  1. added
    eval[Component] This issue is related to evaluation
    on Oct 26, 2026
  2. added a commit that references this issue on Oct 27, 2026
    5e52b21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

eval[Component] This issue is related to evaluation

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions