diff --git a/docs/observability/logging.md b/docs/observability/logging.md index 9069e8033b..47470d2409 100644 --- a/docs/observability/logging.md +++ b/docs/observability/logging.md @@ -1,41 +1,69 @@ # Agent activity logging
- Supported in ADKPython v0.1.0TypeScript v0.2.0Go v0.1.0Java v0.1.0 + Supported in ADKPython v0.1.0Go v0.1.0
-Agent Development Kit (ADK) provides flexible and powerful logging capabilities to monitor agent behavior and debug issues effectively. Understanding how to configure and interpret these logs is crucial for monitoring agent behavior and debugging issues effectively. +Agent Development Kit (ADK) provides flexible and powerful logging capabilities +to monitor agent behavior and debug issues effectively. -## Logging Philosophy +## Logging philosophy -ADK's approach to logging is to provide detailed diagnostic information without being overly verbose by default. It is designed to be configured by the application developer, allowing you to tailor the log output to your specific needs, whether in a development or production environment. +ADK's approach to logging is to provide detailed diagnostic information without +being overly verbose by default. It is designed to be configured by the +application developer, allowing you to tailor the log output to your specific +needs, whether in a development or production environment. -- **Standard Library Integration:** ADK uses the standard logging facilities of the host language (e.g., Python's `logging` module, Go's `log` package). -- **Structured GenAI Logging:** ADK uses OpenTelemetry to log structured events for GenAI requests and responses, allowing for advanced monitoring and debugging in cloud environments. -- **User-Configured:** While ADK provides defaults and integration with its CLI tools, it is ultimately the responsibility of the application developer to configure logging to suit their specific environment. +- **Standard Library Integration:** ADK uses the standard logging facilities of + the host language (e.g., Python's `logging` module, Go's `log` package). +- **Structured GenAI Logging:** ADK uses OpenTelemetry to log structured events + for GenAI requests and responses, allowing for advanced monitoring and + debugging in cloud environments. +- **User-Configured:** While ADK provides defaults and integration with its CLI + tools, it is ultimately the responsibility of the application developer to + configure logging to suit their specific environment. ---- +## Logging schema -## Configuring Logging in Python +ADK emits logs using standard library facilities and structured GenAI events via +OpenTelemetry. -In Python, ADK uses the standard `logging` module. +### Structured GenAI logs -### Example Configuration +Structured GenAI logs emitted via OpenTelemetry follow the [Semantic Conventions +for +GenAI](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/gen-ai-events.md). -To enable detailed logging, including `DEBUG` level messages, add the following to the top of your script: +By default prompt content is elided in logs for security. You can enable prompt +logging using environment variables or programmatic configuration (see Setup +section below). -```python -import logging +### Log levels (Python) -logging.basicConfig( - level=logging.DEBUG, - format='%(asctime)s - %(levelname)s - %(name)s - %(message)s' -) -``` +The following table describes what is logged at different levels in Python when +using the standard logger: -### Configuring Logging with the ADK CLI (Python) +| Level | Description | Type of Information Logged | +| :--- | :--- | :--- | +| **`DEBUG`** | **Crucial for debugging.** The most verbose level for fine-grained diagnostic information. | | +| **`INFO`** | General information about the agent's lifecycle. | | +| **`WARNING`** | Indicates a potential issue or deprecated feature use. The agent continues to function, but attention may be required. | | +| **`ERROR`** | A serious error that prevented an operation from completing. | | -When running Python agents using the ADK's built-in web or API servers, you can easily control the log verbosity directly from the command line. The `adk web`, `adk api_server`, and `adk deploy cloud_run` commands all accept a `--log_level` option. +!!! note + It is recommended to use `INFO` or `WARNING` in production environments. + Only enable `DEBUG` when actively troubleshooting an issue, as `DEBUG` logs + can be very verbose and may contain sensitive information. + +## Logging setup + +### Logging in ADK Web + +When running agents using the ADK's `adk web`, `adk api_server`, `adk deploy +cloud_run` and `adk deploy gke` commands, you can control the log verbosity or +destination. + +#### Logging level To start the web server with `DEBUG` level logging, run: @@ -43,48 +71,123 @@ To start the web server with `DEBUG` level logging, run: adk web --log_level DEBUG path/to/your/agents_dir ``` -The available log levels for the `--log_level` option are: +The available log levels for the `--log_level` option are: `DEBUG`, `INFO` +(default), `WARNING`, `ERROR`, `CRITICAL`. -- `DEBUG` -- `INFO` (default) -- `WARNING` -- `ERROR` -- `CRITICAL` +#### Capture prompt content -### Log Levels (Python) +By default a prompt content is elided in logs for security. You can enable +prompt logging using the environment variable: -| Level | Description | Type of Information Logged | -| :--- | :--- | :--- | -| **`DEBUG`** | **Crucial for debugging.** The most verbose level for fine-grained diagnostic information. | | -| **`INFO`** | General information about the agent's lifecycle. | | -| **`WARNING`** | Indicates a potential issue or deprecated feature use. The agent continues to function, but attention may be required. | | -| **`ERROR`** | A serious error that prevented an operation from completing. | | +```bash +export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true +``` -**Note:** It is recommended to use `INFO` or `WARNING` in production environments. Only enable `DEBUG` when actively troubleshooting an issue, as `DEBUG` logs can be very verbose and may contain sensitive information. +!!! warning + The `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` setting logs the + full content of user prompts and agent responses. This is useful for + debugging but may capture sensitive data or PII. In production, set this to + false or ensure you have appropriate data handling policies in place. ---- +#### OTLP export -## Configuring Logging in Go +To export logs to an OTLP-compatible backend, set the standard OTel environment +variables: -In Go, ADK uses the standard `log` package for general events and OpenTelemetry for GenAI activity logging. +```bash +export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="http://your-collector:4318/v1/logs" +adk web path/to/your/agents_dir +``` -### OpenTelemetry Logging +!!! note + You can also set the general `OTEL_EXPORTER_OTLP_ENDPOINT` environment + variable if you would like to send metrics and traces to the same endpoint + in addition to logs. -ADK Go uses OpenTelemetry (OTel) to log GenAI requests and responses. By default, prompt content is elided in logs for security. You can enable prompt logging using environment variables or programmatic configuration. -#### Enabling Prompt Logging +#### GCP export setup -Set the following environment variable to `true` to include full prompts in your OTel logs: +You can enable GCP export using the `-otel_to_cloud` flag: ```bash -export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true +adk web -otel_to_cloud path/to/your/agents_dir +``` + +### Python programmatic setup + +In Python, ADK uses the standard `logging` module and OpenTelemetry for +structured GenAI logs. + +#### Logging level + +To enable detailed logging, including `DEBUG` level messages, add the following +to the top of your script: + +```python +import logging + +logging.basicConfig( + level=logging.DEBUG, + format='%(asctime)s - %(levelname)s - %(name)s - %(message)s' +) +``` + +#### Capture prompt content + +You can enable full prompt logging programmatically by setting an environment +variable: + +```python +import os + +os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "true" ``` -#### Programmatic Configuration +#### OTLP export -You can configure telemetry providers using the `google.golang.org/adk/telemetry` package. +To export logs to an OpenTelemetry Collector (or an OTLP-compatible backend) +programmatically: + +```python +from google.adk.telemetry.setup import maybe_set_otel_providers +import os + +os.environ["OTEL_EXPORTER_OTLP_LOGS_ENDPOINT"] = "http://your-collector:4318/v1/logs" +os.environ["OTEL_SERVICE_NAME"] = "your-adk-agent" +os.environ["OTEL_RESOURCE_ATTRIBUTES"] = "key1=value1,key2=value2" +maybe_set_otel_providers() +``` + +#### GCP export setup + +To export logs to Google Cloud Logging programmatically, use the OpenTelemetry +Google Cloud exporter. Here is an example in Python: + +```python +from google.adk.telemetry.google_cloud import get_gcp_exporters +from google.adk.telemetry.setup import maybe_set_otel_providers +import os + +gcp_exporters = get_gcp_exporters( + enable_cloud_logging = True, +) +os.environ["OTEL_SERVICE_NAME"] = "your-adk-agent" +os.environ["OTEL_RESOURCE_ATTRIBUTES"] = "key1=value1,key2=value2" +maybe_set_otel_providers([gcp_exporters]) +``` + +### Go programmatic setup + +In Go, ADK uses the `google.golang.org/adk/telemetry` package for OpenTelemetry +configuration and the standard `log` package for general events. + +#### Capture prompt content + +You can enable full prompt logging programmatically when initializing telemetry: ```go +package main + import ( "context" "google.golang.org/adk/telemetry" @@ -92,121 +195,111 @@ import ( func main() { ctx := context.Background() - - // Initialize telemetry with prompt content logging enabled tp, err := telemetry.New(ctx, telemetry.WithGenAICaptureMessageContent(true), - // Add other options like WithOtelToCloud(true) for GCP export ) if err != nil { // handle error } defer tp.Shutdown(ctx) - - // Register as global OTel providers tp.SetGlobalOtelProviders() - - // Your ADK agent code follows... } ``` -### General Logging +#### OTLP export -General events (like server startup or HTTP requests) are logged using the standard Go `log` package. These logs are written to `stderr` by default. +To export logs to an OTLP-compatible backend, configure the standard +OpenTelemetry environment variables (e.g., `OTEL_EXPORTER_OTLP_ENDPOINT` or +`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`). The ADK telemetry package will +automatically use these settings when initialized. -### Configuring Logging with the ADK Go Launcher +#### GCP export setup -When using the ADK Go `full.Launcher` or `prod.Launcher`, telemetry is automatically initialized. You can enable GCP export using the `-otel_to_cloud` flag: +To export logs to Google Cloud Logging, use the `WithOtelToCloud` option: -```bash -go run main.go web -otel_to_cloud a2a +```go +package main + +import ( + "context" + "google.golang.org/adk/telemetry" +) + +func main() { + ctx := context.Background() + tp, err := telemetry.New(ctx, + telemetry.WithOtelToCloud(true), + ) + if err != nil { + // handle error + } + defer tp.Shutdown(ctx) + tp.SetGlobalOtelProviders() +} ``` ---- +If using the Go launcher, you can also enable GCP export via the CLI flag: -## Reading and Understanding the Logs +```bash +go run main.go web -otel_to_cloud +``` + +General events (like server startup or HTTP requests) are logged using the +standard Go `log` package. These logs are written to `stderr` by default. -The structure of logs depends on your configuration. Structured GenAI logs emitted via OpenTelemetry follow the [Semantic Conventions for GenAI](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/gen-ai-events.md). +## Understanding log output -### Sample Python Log Entry +### Sample Python log entry ```text 2025-07-08 11:22:33,456 - DEBUG - google_adk.models.google_llm - LLM Request: contents { ... } ``` + | Log Segment | Format Specifier | Meaning | | ------------------------------- | ---------------- | ---------------------------------------------- | | `2025-07-08 11:22:33,456` | `%(asctime)s` | Timestamp | | `DEBUG` | `%(levelname)s` | Severity level | | `google_adk.models.google_llm` | `%(name)s` | Logger name (the module that produced the log) | | `LLM Request: contents { ... }` | `%(message)s` | The actual log message | -By reading the logger name, you can immediately pinpoint the source of the log and understand its context within the agent's architecture. - -### Debugging with Logs: A Practical Example (Python) - -**Scenario:** Your agent is not producing the expected output, and you suspect the prompt being sent to the LLM is incorrect. -**Steps:** - -1. **Enable DEBUG Logging:** In your `main.py`, set the logging level to `DEBUG` as shown in the configuration example. - ```python - logging.basicConfig( - level=logging.DEBUG, - format='%(asctime)s - %(levelname)s - %(name)s - %(message)s' - ) - ``` - -2. **Run Your Agent:** Execute your agent's task as you normally would. - -3. **Inspect the Logs:** Look through the console output for a message from the `google.adk.models.google_llm` logger that starts with `LLM Request:`. - - ```log - ... - 2025-07-10 15:26:13,778 - DEBUG - google_adk.google.adk.models.google_llm - Sending out request, model: gemini-flash-latest, backend: GoogleLLMVariant.GEMINI_API, stream: False - 2025-07-10 15:26:13,778 - DEBUG - google_adk.google.adk.models.google_llm - - LLM Request: - ----------------------------------------------------------- - System Instruction: - You roll dice and answer questions about the outcome of the dice rolls. - You can roll dice of different sizes. - You can use multiple tools in parallel by calling functions in parallel(in one request and in one round). - It is ok to discuss previous dice roles, and comment on the dice rolls. - When you are asked to roll a die, you must call the roll_die tool with the number of sides. Be sure to pass in an integer. Do not pass in a string. - You should never roll a die on your own. - When checking prime numbers, call the check_prime tool with a list of integers. Be sure to pass in a list of integers. You should never pass in a string. - You should not check prime numbers before calling the tool. - When you are asked to roll a die and check prime numbers, you should always make the following two function calls: - 1. You should first call the roll_die tool to get a roll. Wait for the function response before calling the check_prime tool. - 2. After you get the function response from roll_die tool, you should call the check_prime tool with the roll_die result. - 2.1 If user asks you to check primes based on previous rolls, make sure you include the previous rolls in the list. - 3. When you respond, you must include the roll_die result from step 1. - You should always perform the previous 3 steps when asking for a roll and checking prime numbers. - You should not rely on the previous history on prime results. - You are an agent. Your internal name is "hello_world_agent". - The description about you is "hello world agent that can roll a dice of 8 sides and check prime numbers." - ----------------------------------------------------------- - Contents: - {"parts":[{"text":"Roll a 6 sided dice"}],"role":"user"} - {"parts":[{"function_call":{"args":{"sides":6},"name":"roll_die"}}],"role":"model"} - {"parts":[{"function_response":{"name":"roll_die","response":{"result":2}}}],"role":"user"} - ----------------------------------------------------------- - Functions: - roll_die: {'sides': {'type': }} - check_prime: {'nums': {'items': {'type': }, 'type': }} - ----------------------------------------------------------- - 2025-07-10 15:26:13,779 - INFO - google_genai.models - AFC is enabled with max remote calls: 10. - 2025-07-10 15:26:14,309 - INFO - google_adk.google.adk.models.google_llm - - LLM Response: - ----------------------------------------------------------- - Text: - I have rolled a 6 sided die, and the result is 2. - ... - ``` - -4. **Analyze the Prompt:** By examining the `System Instruction`, `contents`, `functions` sections of the logged request, you can verify: - - Is the system instruction correct? - - Is the conversation history (`user` and `model` turns) accurate? - - Is the most recent user query included? - - Are the correct tools being provided to the model? - - Are the tools correctly called by the model? - - How long it takes for the model to respond? - -This detailed output allows you to diagnose a wide range of issues, from incorrect prompt engineering to problems with tool definitions, directly from the log files. + +By reading the logger name, you can immediately pinpoint the source of the log +and understand its context within the agent's architecture. + +### Debugging example + +After enabling `DEBUG` logging (see [Logging level](#logging-level) above), run +your agent and look for messages from the `google.adk.models.google_llm` logger. +The output shows the full LLM request and response: + +```text +2025-07-10 15:26:13,778 - DEBUG - google_adk.google.adk.models.google_llm - +LLM Request: +----------------------------------------------------------- +System Instruction: + You roll dice and answer questions about the outcome of the dice rolls. + ... +----------------------------------------------------------- +Contents: +{"parts":[{"text":"Roll a 6 sided dice"}],"role":"user"} +{"parts":[{"function_call":{"args":{"sides":6},"name":"roll_die"}}],"role":"model"} +{"parts":[{"function_response":{"name":"roll_die","response":{"result":2}}}],"role":"user"} +----------------------------------------------------------- +Functions: +roll_die: {'sides': {'type': }} +check_prime: {'nums': {'items': {'type': }, 'type': }} +----------------------------------------------------------- +2025-07-10 15:26:14,309 - INFO - google_adk.google.adk.models.google_llm - +LLM Response: +----------------------------------------------------------- +Text: +I have rolled a 6 sided die, and the result is 2. +... +``` + +From this output you can verify: + +- Is the system instruction correct? +- Is the conversation history (`user` and `model` turns) accurate? +- Are the correct tools being provided to the model? +- Are the tools correctly called by the model? +- How long it takes for the model to respond?