Using logging_plugin in Agent Engine to Log Custom Fields

[rohan-kavari]

I am building an Agent Engine application using ADK and deploying it via Agent Engine. A simplified version of the setup looks like this:

class AgentEngineApp(AdkApp):
    def set_up(self) -> None:
        """Initialize the agent engine app with logging and telemetry."""
        vertexai.init()
        setup_telemetry()
        super().set_up()

        logging.basicConfig(level=logging.INFO)
        logging_client = google_cloud_logging.Client()
        self.logger = logging_client.logger(__name__)

        if gemini_location:
            os.environ["GOOGLE_CLOUD_LOCATION"] = gemini_location

    def register_feedback(self, feedback: dict[str, Any]) -> None:
        """Collect and log feedback."""
        feedback_obj = Feedback.model_validate(feedback)
        self.logger.log_struct(feedback_obj.model_dump(), severity="INFO")

    def register_operations(self) -> dict[str, list[str]]:
        """Registers the operations of the Agent."""
        operations = super().register_operations()
        operations[""] = operations.get("", []) + ["register_feedback"]
        return operations


gemini_location = os.environ.get("GOOGLE_CLOUD_LOCATION")
logs_bucket_name = os.environ.get("LOGS_BUCKET_NAME")

agent_engine = AgentEngineApp(
    app=adk_app,
    artifact_service_builder=lambda: GcsArtifactService(bucket_name=logs_bucket_name)
    if logs_bucket_name
    else InMemoryArtifactService(),
)

Question

I want to understand how to properly use the logging_plugin within an agent that is deployed on Agent Engine.

Specifically:

• How can I attach or configure logging_plugin for agents running inside Agent Engine?

• What is the recommended way to log custom / contextual fields, such as:

  • user_email
  • country
  • any other per-request metadata

Goal

Any guidance, examples, or best practices for doing this with logging_plugin + Agent Engine would be very helpful.

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: To log custom fields like user_email in Agent Engine, the recommended approach is to store this data in the session state. You can then create a custom plugin that uses a callback (e.g., before_tool_callback) to access this data from the ToolContext and log it. For production-level structured logging, consider using the BigQueryAgentAnalyticsPlugin instead of the basic logging_plugin.

---

Hello! As an AI agent, I can provide some guidance on how to best approach logging custom fields with ADK in Agent Engine.

The standard logging_plugin is designed for basic debugging in a terminal and does not have a direct way to include rich, per-request contextual data. [4]

For your use case, the recommended pattern is to leverage ADK's session state and plugin system. Here’s how you can do it:

1. Store Per-Request Data in Session State

Your application should add the contextual information (like user_email and country) to the session state when a conversation begins. The state is designed to hold information that needs to persist across multiple turns of a conversation. [5]

2. Create a Custom Logging Plugin

You can create your own plugin to inspect the state at various points in the agent's lifecycle and log the custom data. A plugin is a class that inherits from BasePlugin and implements one or more callback methods. [1]

Here is an example of a simple plugin that logs custom fields before a tool is executed:

import logging
from google.adk.plugins import BasePlugin, ToolContext

class ContextualLoggingPlugin(BasePlugin):
    """A plugin to log contextual data from the session state."""

    def __init__(self, logger):
        self.logger = logger

    def before_tool_callback(self, tool_context: ToolContext) -> None:
        """Called before a tool is executed."""
        custom_fields = {
            "user_email": tool_context.state.get("user_email"),
            "country": tool_context.state.get("country"),
            "tool_name": tool_context.tool_name,
            "tool_parameters": tool_context.tool_parameters,
        }

        # Remove fields that are not set
        custom_fields = {k: v for k, v in custom_fields.items() if v is not None}

        if custom_fields:
            self.logger.info("Executing tool with context: %s", custom_fields)

# In your main application setup:
# logger = ... (your configured logger)
# contextual_logger_plugin = ContextualLoggingPlugin(logger)
#
# agent_engine = AgentEngineApp(
#     app=adk_app,
#     plugins=[contextual_logger_plugin],
#     ...
# )

In this example, the before_tool_callback accesses the session state through the ToolContext to retrieve and log your custom data. [3]

3. For Production: Use BigQueryAgentAnalyticsPlugin

While the custom plugin above works well, for a robust production deployment, we highly recommend using the BigQueryAgentAnalyticsPlugin. [2] This plugin provides structured logging to BigQuery, which is ideal for analytics, monitoring, and debugging. It is highly configurable and can be extended to include custom data fields in the BigQuery schema. You could adapt the logic from the ContextualLoggingPlugin to integrate with the event structure of the BigQuery plugin.

By combining session state with a custom plugin or the BigQueryAgentAnalyticsPlugin, you can achieve the detailed, contextual logging you're looking for.

I hope this helps you build your Agent Engine application!

---

[1] base_plugin.py source code: https://github.com/google/adk-python/blob/main/src/google/adk/plugins/base_plugin.py
[2] BigQuery Agent Analytics Documentation: https://google.github.io/adk-docs/observability/bigquery-agent-analytics/
[3] Custom Tools Documentation (ToolContext): https://google.github.io/adk-docs/tools-custom/
[4] logging_plugin.py source code: https://github.com/google/adk-python/blob/main/src/google/adk/plugins/logging_plugin.py
[5] Context and State Management Documentation: https://google.github.io/adk-docs/context/

[haiyuan-eng-google]

ttps://google.github.io/adk-docs/observability/bigquery-agent-analytics/ BigQuery Agent Analytics Plugin: the ADK offers a plugin to log detailed agent analytics to BigQuery. This allows you to run complex queries on your agent's behavior and performance, get out of box observability for both token usage, tool call/LLM call error rate and the latency for both tool call and LLM call immediately. The plugin just need one line code change to add plugin into your APP. With this plugin, you can monitor which tool/subagent would cost your token most in the whole agent life-cycle

[haiyuan-eng-google]

Confirmed — the claim is accurate. One refinement to make it precise: it's not that the snapshot is wrong, it's that BQAA takes it too early in the lifecycle.

LLM_REQUEST is emitted from the plugin's before_model_callback, where BQAA serializes the LlmRequest synchronously. In ADK's flow, plugin before_model_callbacks run before the agent's own before_model_callbacks — see base_llm_flow.py::_handle_before_model_callback (plugins first, then agent.canonical_before_model_callbacks). So when BQAA snapshots the request, the agent callback hasn't run yet.

What that means concretely:

• Captured: everything assembled in _preprocess_async — request processors, system instruction, contents, tools.
• Missed: anything mutated after the plugin hook — the agent's before_model_callback (the case you flagged), any plugin registered after BQAA, and ADK's own config.labels injection (adk_agent_name), which also happens after callbacks in _call_llm_async.

On fixing it, two options:

1. BQAA-only: LlmRequest is mutated in place, so BQAA could stash the reference in before_model_callback and serialize it in after_model_callback / on_model_error_callback, where it reflects the final state. No ADK change. Costs: the REQUEST event then lands after the RESPONSE; we'd need per-in-flight-call keying; and short-circuit responses (a callback returning an LlmResponse) never reach after_model_callback, so nothing would be logged in that case.

2. Upstream ADK: add an additive plugin hook that fires right before generate_content_async (i.e. after callbacks + label injection) and log there. No hook currently exists in that position.

I'd go with (2) as the real fix. It's correct by construction — logs exactly what's sent, at the right time, preserving REQUEST→RESPONSE ordering and span/latency semantics — it's backward-compatible/additive, and it fixes this for every observability consumer (Cloud Logging/GEAP, custom plugins), not just BQAA, so we don't end up with each integration carrying its own stash-and-defer workaround. (1) is a reasonable stopgap if we need correct data before an ADK release, but it's debt we'd likely rip out once the hook lands.

Want to know your thoughts as well @boyangsvl

[haiyuan-eng-google]

#6222 filed this issue

[boyangsvl]

#1 won't work because the callback can return a different object for llm_request.

A few other options for consideration:

1. There's an internal proposal to add a 'observer' type of callback, which is always called the last and no ability to alter the object.
2. We can add an option to the plugin to control whether to call it before agent's callback or after. But again, other plugins could change the object and BQAA always wants to be the last one called.
3. Since it's an observability use case, we can design a dedicated observability interface (maybe through otel), so you don't have to rely on the plugin system.

[haiyuan-eng-google]

Thanks @boyangsvl — agreed on all three points, and they reframe #6222.

On Alternative #1: you're right that it's not just suboptimal, it's unsound. Since a before_model_callback can return a new LlmRequest (not only mutate it in place), stashing the reference in before_model_callback and serializing it later can capture the wrong object. That also reinforces why the fix has to read whatever object is actually passed to generate_content_async, rather than a reference captured earlier.

On the hook being read-only: makes sense. I'll change the proposed signature from -> Optional[LlmResponse] to a pure observer (-> None, no ability to short-circuit or mutate). That makes my on_model_request a specific instance of the "observer callback, always called last" category you mentioned. If that internal observer-callback proposal is moving, I'd rather land this as its first member than as a one-off hook.

On ordering control (your option 2): agreed it doesn't solve it — any plugin registered after BQAA can still mutate, and an observability consumer fundamentally wants to be last. Ordering is a race; "observer, always last, read-only" is correct by construction.

On a dedicated observability interface / OTel (your option 3): this is the real fork. It's the cleanest long-term — it removes the "who runs last" problem entirely. The trade-off is more surface area plus a migration path for existing plugin-based consumers like BQAA. The observer callback is the lower-cost, ADK-native increment we could ship now.

Proposed decision for maintainers: do we (a) introduce the observer-callback category and make on_model_request its first member, or (b) invest in an OTel-based observability interface and treat plugin-based capture as legacy? Happy to prototype (a) since it unblocks BQAA immediately, but I don't want to add a hook we'd deprecate if (b) is the direction.

I've updated the issue #6222 to reflect this