[googlecloudspanner receiver] Receiver fails with "Invalid parameter name" error from Spanner

[marijakljestan]

Component(s)

receiver/googlecloudspanner

What happened?

Description

The googlecloudspanner receiver in the OpenTelemetry Collector is failing to scrape metrics from Google Cloud Spanner. Spanner returns an error indicating "Invalid parameter name" for parameters like topmetricsquerymaxrows and pulltimestamp. This suggests the receiver is constructing SQL queries with parameter names that Spanner does not accept in that format.

Steps to Reproduce

1. Configure the OpenTelemetry Collector with the googlecloudspanner receiver.
2. Use an OpenTelemetry Collector image: otel/opentelemetry-collector-contrib: 0.127.0
3. Observe the collector logs for errors originating from the googlecloudspanner receiver.

Expected Result

The googlecloudspanner receiver should successfully scrape metrics from the configured Spanner instances and databases without errors.

Actual Result

The collector logs show errors similar to the following:
Error scraping metrics {"otelcol.component.id": "googlecloudspanner", "otelcol.component.kind": "Receiver", "otelcol.signal": "metrics", "error": "cannot read data for data points databaseReader "active queries summary ::<instance_id>::
" because of an error: query "SELECT * FROM SPANNER_SYS.ACTIVE_QUERIES_SUMMARY LIMIT @topMetricsQueryMaxRows" failed with error: spanner: code = "InvalidArgument", desc = "Invalid parameter name: topmetricsquerymaxrows. Expected one of 'p1', 'p2', ..., 'p65535'",

Collector version

0.127.0

Environment information

Environment

Platform: GKE (v1.31.8-gke.1045000)
OpenTelemetry Helm Chart: open-telemetry/opentelemetry-collector, version: 0.127.1
Container Image: otel/opentelemetry-collector-contrib: 0.127.0

OpenTelemetry Collector configuration

image:
  repository: "otel/opentelemetry-collector-contrib"
  tag: "0.127.0"

config:
  receivers:
    googlecloudspanner:
      collection_interval: 60s
      # top_metrics_query_max_rows: 100
      # backfill_enabled: true
      projects:
        - project_id: "<project>"
          instances:
            - instance_id: "<instance-id>
              databases:
                - "<database>"

  processors:
    resource:
    batch:
      send_batch_size: 2048
      timeout: 200ms
      send_batch_max_size: 8192
  exporters:
    prometheusremotewrite:
      endpoint: "<prometheus-endpoint>"

  service:
    pipelines:
      metrics:
        receivers: [googlecloudspanner]
        processors: [batch]
        exporters: [prometheusremotewrite]

Log output

Error scraping metrics	{"otelcol.component.id": "googlecloudspanner", "otelcol.component.kind": "Receiver", "otelcol.signal": "metrics", "error": "cannot read data for data points databaseReader \"active queries summary <project>::<instance_id>::
<db>\" because of an error: query \"SELECT * FROM SPANNER_SYS.ACTIVE_QUERIES_SUMMARY LIMIT @topMetricsQueryMaxRows\" failed with error: spanner: code = \"InvalidArgument\", desc = \"Invalid parameter name: topmetricsquerymaxrows. Expected one of 'p1', 'p2', ..., 'p65535'\", ...
cannot read data for data points databaseReader \"top minute lock stats <project>::<instance>::<db>\" because of an error: query \"SELECT * FROM SPANNER_SYS.LOCK_STATS_TOP_MINUTE WHERE INTERVAL_END = @pullTimestamp ORDER BY INTERVAL_END DESC, LOCK_WAIT_SECONDS DESC LIMIT @topMetricsQueryMaxRows\" failed with error: spanner: code = \"InvalidArgument\", desc = \"Invalid parameter name: pulltimestamp. Expected one of 'p1', 'p2', ..., 'p65535'

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/googlecloudspanner: @dashpole @KiranmayiB @nsj07

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[nsj07]

Looking at this.
@marijakljestan: Is it working perfectly fine earlier? When did you start seeing this error?
Can you try with new version: v0.128.0 or fallback to older version which is already working earlier for you.

[marijakljestan]

Looking at this. @marijakljestan: Is it working perfectly fine earlier? When did you start seeing this error? Can you try with new version: v0.128.0 or fallback to older version which is already working earlier for you.

Thanks!
I set it up for the first time today using the latest version (0.128.0).
I also tested older versions (0.120.0 and 0.127.0, as referenced here), but the same error appeared across all of them.

[nsj07]

@marijakljestan : It could be access issue if you don't have access for the DB mentioned. Can you try via 'service_account_key' as mentioned here.

[marijakljestan]

@nsj07 I don’t think this is a permission issue.
For security reasons, we don’t use service account keys. Instead, the OpenTelemetry Collector has a Kubernetes service account (named otel-collector), which is mapped via GCP Workload Identity to an IAM service account (otel-collector@<project>iam.gserviceaccount.com) that has the roles/spanner.databaseUser role on the folder-level.
Apologies for not including this part of the configuration earlier, here it is:

serviceAccount:
  create: true
  name: otel-collector
  annotations:
    iam.gke.io/gcp-service-account: otel-collector@<project>iam.gserviceaccount.com