Add Azure VM and AKS integration tests for the default OTel config - #730
Merged
Conversation
Adds test/azurevm (TestAzureVM, //go:build integration, TestMain/pre-provisioned pattern) validating the agent on a real Azure VM: Azure IMDS detection, OTLP metrics/logs/traces reaching CloudWatch, and the Azure web-identity credential chain (oidctoken -> AssumeRoleWithWebIdentity -> sigv4auth). Includes the terraform/azurevm harness (references the AWS OIDC provider by ARN; no role creation; uploads the runner-built .deb to the VM over SSH), AZUREVM compute type, and a one-time Azure-side setup.sh.
- Default VM size Standard_D2s_v5 is capacity-restricted in eastus for some subscriptions (SkuNotAvailable 409); switch to Standard_D2s_v7. - Add a network security group with an inbound SSH rule scoped to runner_ip and associate it to the NIC; Azure's implicit default denies all inbound, so the file/remote-exec provisioners could not reach port 22. null_resource now depends on the association so SSH is reachable before provisioning.
Azure VMs have no AWS instance metadata, so the agent (with the new env fallback) resolves region from AWS_REGION at translation time - pass it on the sudo'd fetch-config invocation. Replace the 'tee -a etc/env-config' lines: the agent reads etc/env-config.json (strict JSON) at service startup, so plain KEY=VALUE lines in etc/env-config were never loaded and the systemd-spawned agent would miss CWAGENT_ROLE_ARN needed to expand the role_arn placeholder in the translated YAML. Use the supported '-setenv ... -envconfig' merge instead. USE_DEFAULT_CONFIG is translation-time only and needs no persistence.
Namespaced as test/azure/vm and terraform/azure/vm so future Azure compute targets (e.g. AKS) can live alongside under the same prefix. Assertion design: - Use lowercase "azure" marker (OTel structured logs are lowercase) - No negative log-marker checks in the CredentialChain subtest: AssumeRoleWithWebIdentity is the successful API call name, and transient retries leave permanent markers even on recovery. The Metrics/Logs/Traces subtests already prove delivery end-to-end.
… retry window X-Ray's OTLP endpoint silently drops traces whose first 4 bytes are not a valid Unix epoch timestamp (seconds). Our previous format used nanoseconds which overflowed the 4-byte field. Generate compliant IDs (first 4 bytes = unix seconds, remaining 12 bytes = sequence padding). Also revert the t.Skip() workaround — Traces now hard-fails — and extend the retry window to 5 attempts × 90s to accommodate X-Ray OTLP indexing latency (typically 3-5 minutes).
GetTraceSummaries depends on X-Ray's indexing pipeline which can lag
5-10+ minutes for OTLP-ingested traces, causing persistent test failures.
Switch to BatchGetTraces with the exact trace IDs we generated — this
queries the raw trace store directly and bypasses indexing entirely.
Record generated OTLP trace IDs during the load window, convert to
X-Ray format (1-{8hex}-{24hex}), and fetch them directly.
The X-Ray OTLP endpoint (xray.{region}.amazonaws.com/v1/traces) requires
xray:PutSpans — a separate IAM action from the legacy PutTraceSegments.
Without it, the agent's trace exports are silently rejected with 403,
which is why traces never appeared in X-Ray despite the endpoint
accepting connections.
Also add xray:BatchGetTraces which the test binary needs for the
direct trace-ID-based validation (bypasses GetTraceSummaries indexing).
Dump agent log lines containing error/warn/xray/traces keywords after the load window to diagnose why the otlphttp/traces exporter silently fails to deliver spans to X-Ray. The IAM policy is now correct (PutSpans granted) but traces still don't arrive — need visibility into agent-side export errors.
The X-Ray OTLP endpoint requires Transaction Search (trace segment destination = CloudWatchLogs). Spans ingested this way are stored in the aws/spans log group in full; GetTraceSummaries and BatchGetTraces only see the indexed subset (1% by default), so they return empty for low-volume test traffic. Query aws/spans for the exact generated trace IDs instead and require all of them to be present.
Provisions an AKS cluster with OIDC issuer, creates an AWS IAM role trusting the cluster's service account token, deploys the CWA DaemonSet from the pre-built ECR image with projected STS token auth, runs a hostNetwork load-generator Job pushing OTLP for 3 minutes, then validates all three signals (metrics/logs/traces) reach CloudWatch. Traces validated via aws/spans (Transaction Search) same as the VM test.
The agent entrypoint checks IsRunningInContainer() to decide whether to use the container config path (--input-dir). Without this env var set, translateConfig falls back to host paths and fails inside the container.
…sform
The agent's transform_identity_k8s fills service.namespace from
k8s.namespace.name, so the routed stream is
{k8s.namespace.name}/{service.namespace}/{service.name} — verified live
in CloudWatch for the previous run's cluster.
- Protect generatedTraceIDs/traceSeq with sync.Mutex; snapshot under lock after sendTelemetry stops so the read has a happens-before. - Add AssertLogsNotEmpty to validateLogs (matches the AKS path). - Close resp.Body in post() to avoid leaking connections across the ~54 POST calls per test run.
movence
force-pushed
the
hsookim/azurevm-integ-test
branch
from
July 24, 2026 15:37
ba81e53 to
03d7a2a
Compare
movence
force-pushed
the
hsookim/azurevm-integ-test
branch
from
July 28, 2026 18:08
5ff2881 to
23c3435
Compare
jefchien
reviewed
Jul 28, 2026
Comment on lines
+134
to
+135
| "sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -setenv 'AWS_REGION=${var.region}' -envconfig /opt/aws/amazon-cloudwatch-agent/etc/env-config.json", | ||
| "sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -setenv 'CWAGENT_ROLE_ARN=${aws_iam_role.cwagent.arn}' -envconfig /opt/aws/amazon-cloudwatch-agent/etc/env-config.json", |
Contributor
There was a problem hiding this comment.
nit: Use the ctl script. There's a new set-env action aws/amazon-cloudwatch-agent#2211
| "export PATH=$PATH:/usr/local/go/bin", | ||
| "sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -setenv 'AWS_REGION=${var.region}' -envconfig /opt/aws/amazon-cloudwatch-agent/etc/env-config.json", | ||
| "sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -setenv 'CWAGENT_ROLE_ARN=${aws_iam_role.cwagent.arn}' -envconfig /opt/aws/amazon-cloudwatch-agent/etc/env-config.json", | ||
| "sudo USE_DEFAULT_CONFIG=otel /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m auto -s -c default:otel", |
Contributor
There was a problem hiding this comment.
I don't think USE_DEFAULT_CONFIG=otel should be necessary or does anything in this case.
Comment on lines
+138
to
+139
| "curl -s -H Metadata:true \"http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=${var.azure_token_audience}\" | python3 -c \"import sys,json; print(json.load(sys.stdin)['access_token'])\" > /tmp/azure-identity-token", | ||
| "export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/azure-identity-token AWS_ROLE_ARN=${aws_iam_role.cwagent.arn} AWS_REGION=${var.region}", |
Contributor
There was a problem hiding this comment.
nit: I wonder if the token file could just point to the token file that the agent will manage or will the agent remove it during shutdown.
|
|
||
| variable "region" { | ||
| type = string | ||
| default = "us-west-2" |
Contributor
There was a problem hiding this comment.
Any reason why we default to us-west-2 for azure/vm and us-east-2 for azure/aks? Don't they both need to be a non us-west-2 region so we can avoid the transaction search enablement clashing with our existing tests?
| } | ||
|
|
||
| // filterLogLines returns lines containing any of the given substrings (case-insensitive). | ||
| func filterLogLines(text string, substrs ...string) []string { |
Contributor
There was a problem hiding this comment.
Doesn't look like this is ever used. Do we need it for our validation?
Comment on lines
+48
to
+72
| data "aws_iam_policy_document" "cwagent_permissions" { | ||
| statement { | ||
| effect = "Allow" | ||
| actions = [ | ||
| "cloudwatch:PutMetricData", | ||
| "cloudwatch:ListMetrics", | ||
| "cloudwatch:GetMetricData", | ||
| "logs:PutLogEvents", | ||
| "logs:CreateLogGroup", | ||
| "logs:CreateLogStream", | ||
| "logs:DescribeLogGroups", | ||
| "logs:DescribeLogStreams", | ||
| "logs:GetLogEvents", | ||
| # StartQuery/GetQueryResults let the test binary validate OTLP trace delivery via the aws/spans | ||
| # log group; the X-Ray OTLP endpoint requires account-level Transaction Search (trace segment | ||
| # destination = CloudWatchLogs), which stores 100% of spans there. | ||
| "logs:StartQuery", | ||
| "logs:GetQueryResults", | ||
| "xray:PutSpans", | ||
| "xray:PutTraceSegments", | ||
| "xray:PutTelemetryRecords", | ||
| ] | ||
| resources = ["*"] | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Does this differ from the policy we end up using for AKS? Could we use the agent's managed policy + the permissions we need for validation in a separate inline policy document?
| resourcedetection detector sets `host.id = compute.VMID`, so the test's `-instanceId=<virtual_machine_id>` | ||
| correctly matches the `@resource.host.id` metric label. | ||
|
|
||
| 5. **Logs path depends on agent PR #2197 (OPEN).** Until it merges, `awscloudwatchlogsprovisioner` cannot |
Contributor
There was a problem hiding this comment.
This is outdated. 2197 is merged.
Contributor
There was a problem hiding this comment.
Might want to have another pass at this README if we want to keep it. Some of the information is outdated.
| "logs:DescribeLogStreams", | ||
| "logs:GetLogEvents", | ||
| # StartQuery/GetQueryResults let the test binary validate OTLP trace delivery via the aws/spans | ||
| # log group; the X-Ray OTLP endpoint requires account-level Transaction Search (trace segment |
Contributor
There was a problem hiding this comment.
Is it account-level? I thought it was regional?
Comment on lines
+86
to
+88
| // buildTracesPayload emits an OTLP span with X-Ray-compatible trace IDs. | ||
| // X-Ray requires the first 4 bytes of the 16-byte trace ID to be a Unix epoch timestamp (seconds); | ||
| // IDs that violate this are silently dropped during ingestion. |
Contributor
There was a problem hiding this comment.
Is this still true for transaction search?
jefchien
reviewed
Jul 30, 2026
Comment on lines
+27
to
+35
| # Terraform drives the cluster over the public API server, so restrict it to the runner that created | ||
| # it. The block is omitted entirely when runner_ip is unset rather than emitted with an empty list -- | ||
| # an empty authorized_ip_ranges means "open to all", which would read as restricted while being open. | ||
| dynamic "api_server_access_profile" { | ||
| for_each = var.runner_ip != "" ? [var.runner_ip] : [] | ||
| content { | ||
| authorized_ip_ranges = [api_server_access_profile.value] | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
nit: Should runner_ip just be required then?
Comment on lines
+362
to
+365
| # The payloads carry k8s.cluster.name and k8s.namespace.name resource attributes so the | ||
| # agent's k8s logs-routing template produces a deterministic per-cluster destination | ||
| # (/aws/cwagent/<cluster>/otlp, stream amazon-cloudwatch/<service>); resourcedetection | ||
| # only overrides keys it detects (e.g. host.id), so these pass through intact. |
Contributor
There was a problem hiding this comment.
nit: Consider moving the comment into the script instead of in the module.
Comment on lines
+123
to
+125
| # Agent write omitted from CloudWatchAgentServerPolicy: the X-Ray OTLP endpoint needs PutSpans, | ||
| # which is a different action from PutTraceSegments. | ||
| "xray:PutSpans", |
Contributor
There was a problem hiding this comment.
I don't think this is true. Can you verify this? I've been testing with just the CloudWatchAgentServerPolicy and have been able to send traces via OTLP with the permissions it has. Can see traces from my AKS cluster in aws/spans with resource.attributes.k8s.cluster.name set to my cluster name.
Comment on lines
+86
to
+88
| const maxRetries = 4 | ||
| const retryInterval = 30 * time.Second | ||
| for attempt := 1; attempt <= maxRetries; attempt++ { |
Contributor
There was a problem hiding this comment.
nit: Is there a reason we retry 4 times for the AKS validation and only have one attempt for the VM one? Should the VM one also have retries?
- CloudWatchAgentServerPolicy already covers the agent's OTLP trace writes, so xray:PutSpans is unnecessary. Removing it leaves the AKS role with no inline policy at all; the VM inline policy is now validation reads only. - runner_ip is required in both modules instead of defaulting to "". This removes the path where an unset value left the AKS API server reachable from any IP while reading as restricted, and drops the dynamic block that worked around it. - VM log validation now retries on the same schedule as the AKS path, so a slow first ingestion no longer reports as a delivery failure. - Scope AKS metric validation on a test_id datapoint attribute and assert cloud.platform=azure_aks. Datapoint attributes are the one surface no resource processor rewrites, and cloud.platform is emitted only by the aks detector, so the assertion covers detection rather than echoing back the injected payload. - Match the kubernetes provider constraint already used by eks/daemon/efa. - Move the load-generator payload rationale into the script, and drop the stale USE_DEFAULT_CONFIG references and all-merged prerequisite list from the README.
The README had drifted out of date and described a CI path that does not exist (an azure-integration-test.yml workflow and a generator/ matrix, rather than the jobs in test-artifacts.yml). Its two load-bearing facts are already recorded where they apply: iam.tf explains why the OIDC provider is referenced rather than created, and both iam.tf and variables.tf explain the per-region trace destination. Nothing referenced the file. Leftovers found while auditing for stale references: - The load generator carried the same per-run value under three different attribute names -- test_id on metrics, ClusterName on logs, cluster_name on spans -- and only the metrics one was read. All three now use test_id. - vm/variables.tf pointed at the deleted README for the OIDC provider rationale, so that rationale is now inline, and an image comment said "used below" while referring to a different file. - The role comment in vm/iam.tf described writes only, but the role also carries the reads the on-VM test needs.
jefchien
reviewed
Jul 30, 2026
Comment on lines
+134
to
+150
| streams := awsservice.GetLogStreams(otlpLogGroup) | ||
| if len(streams) == 0 { | ||
| testResult.Reason = fmt.Errorf("attempt %d: no log streams found in %s", attempt, otlpLogGroup) | ||
| } | ||
| for _, stream := range streams { | ||
| log.Printf("[AzureVM_Logs] attempt %d: checking %s/%s", attempt, otlpLogGroup, *stream.LogStreamName) | ||
| err := awsservice.ValidateLogs( | ||
| otlpLogGroup, *stream.LogStreamName, &since, &until, | ||
| awsservice.AssertLogsNotEmpty(), | ||
| awsservice.AssertPerLog(awsservice.AssertLogContainsSubstring(marker)), | ||
| ) | ||
| if err == nil { | ||
| testResult.Status = status.SUCCESSFUL | ||
| return testResult | ||
| } | ||
| testResult.Reason = err | ||
| } |
Contributor
There was a problem hiding this comment.
Why are we checking all log streams? Shouldn't we filter it down by the host.id? The log stream names are deterministic. The number of log streams will continue to grow for the same log group as new VMs are spun up. Locking it down to a fixed log stream also validates that the log routing is working correctly.
…uccess
Log validation scanned every stream in the group and passed if any one of them
matched. That grew with each new VM writing to the shared group, and it could not
tell a routing bug from a delivery failure -- a stream named wrongly was simply
skipped. The agent derives the stream as {host.id}/{service.name}, both of which
the test already knows, so it now asserts that exact name and the check covers
routing as well as delivery.
Both suites now also remove the log destination they created, but only when
validation passed, so a failing run leaves its logs behind as evidence. The VM
deletes just its own stream because /aws/cwagent/otlp is shared across runs; the
AKS group name carries the per-run cluster, so the whole group goes. Neither
touches aws/spans, which is service-managed and shared with other suites.
The VM role gains logs:DeleteLogStream since that test runs on the VM under it.
The AKS test runs on the runner under the runner's own credentials.
The cluster role binding, ECR pull secret, DaemonSet and load-generator Job all named the namespace with the local string, which gives terraform no dependency edge. All four were therefore launched in the same parallel batch as kubernetes_namespace.cwagent instead of after it, and whichever lost the race against the namespace create failed with "namespaces not found". Run 30580467483 lost it on the secret: the namespace took 437ms to create while the secret started 3ms behind it. The preceding run took 150ms for the same call and the secret landed just after the namespace existed, which is why this had passed until now. The service account was already immune because it referenced the namespace resource rather than the string. All four now reference kubernetes_namespace.cwagent.metadata[0].name, matching the service account, so terraform orders them after the namespace.
jefchien
approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the issue
The agent supports running on non-AWS hosts (Azure VM and AKS) with the default OTel config, authenticating to AWS via web-identity federation — but there was no integration test coverage proving the credential chain and end-to-end telemetry delivery (metrics, logs, traces) to CloudWatch from either environment.
Description of changes
Adds two new test suites, each provisioned by terraform and dispatched from the companion workflow PR (aws/amazon-cloudwatch-agent#2216):
Azure VM (
terraform/azure/vm,test/azure/vm).deb, and starts the agent with the default OTel config viaamazon-cloudwatch-agent-ctl -a set-env+-a fetch-config -m auto -s -c default:otel.AssumeRoleWithWebIdentity), with the role's trust policy pinned on both:audand:sub(the VM's system-assigned identity principal) — the audience is a tenant-wide Azure resource, so it is not sufficient alone.aws/spansvia Logs Insights — the OTLP endpoint requiresxray:PutSpansand X-Ray trace IDs with Unix seconds in the first 4 bytes).AKS (
terraform/azure/aks,test/azure/aks)sub/audconditions), and deploys the agent as a DaemonSet from the integration-test ECR image using a projected service-account token (audiencests.amazonaws.com).test_iddatapoint attribute and assertscloud.platform=azure_aks, so it isolates the run and covers resource detection rather than echoing back an injected attribute.AZUREVMandAKScompute types inenvironment/computetype, and adds-aksClusterNameto the environment metadata.License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
Latest green run: 30584238599 — test-repo
38d1c2a5, agentf039ce4c, first attempt. Both Azure jobs passed:This run specifically proves the two changes no earlier run could:
CloudWatchAgentServerPolicyalone.xray:PutSpanswas removed from both roles, and both trace assertions pass with zeroAccessDeniedin either job — so that action is not required for the OTLP trace path. The AKS role now has no inline policy at all.runner_ipis required in both modules and both applies succeeded, confirming no caller depended on the old""default.Resolved providers:
azurerm v4.81.0,kubernetes v2.38.0(under~> 2.0),aws v6.57.1. The run-level conclusion readsfailurefor a pre-existing reason unrelated to these jobs — see the note in aws/amazon-cloudwatch-agent#2216; job-level results are the meaningful signal.The preceding run surfaced a latent ordering bug: four Kubernetes resources named the namespace by string rather than by resource, so
terraform applylaunched them alongside it and whichever lost the race failed withnamespaces not found. All four now referencekubernetes_namespace.cwagent, and this run shows the secret starting 10 ms after the namespace completes rather than 3 ms before.Prior runs: 30459712300, 30113003649, 30048181167, 30029802668, 30018222183.
AssertLogsNotEmptyto prevent vacuous passes.go vet/gofmtclean on new packages;terraform validateandterraform fmt -checkclean on both stacks.Worth noting
us-east-2is load-bearing, not arbitrary.aws/spansis only populated where the X-Ray trace segment destination is CloudWatch Logs, which is a per-region setting. Moving these to us-west-2 would fail the trace assertion and disturb the App Signals suite that relies on theXRaydestination there.CWAGENT_ROLE_ARNso sigv4auth falls through to the default chain; the VM sets it because the translator only wires the web-identity token file whenrole_arnis non-empty. The IAM shape differs for a separate reason: both attachCloudWatchAgentServerPolicy, but the VM adds an inline policy for the validation reads because the test runs on the VM under that role, while the AKS test runs on the runner and needs no inline policy.