Skip to content

appsec: assert app-extended-heartbeat re-emits full telemetry state#3865

Merged
cataphract merged 1 commit into
masterfrom
glopes/fix-extended-hearbeat
May 11, 2026
Merged

appsec: assert app-extended-heartbeat re-emits full telemetry state#3865
cataphract merged 1 commit into
masterfrom
glopes/fix-extended-hearbeat

Conversation

@cataphract
Copy link
Copy Markdown
Contributor

@cataphract cataphract commented May 7, 2026

Updates libdatadog to a version that includes dependencies and integrations in app-extended-heartbeat. The previous behaviour omitted those fields and caused a duplicate app-integrations-change on the next regular flush.

Adds an integration test (TelemetryExtendedHeartbeatTests) that drives phpredis and exec via custom_integrations.php and asserts:

  1. They appear in a regular app-started/app-integrations-change.
  2. The next app-extended-heartbeat re-emits configuration, dependencies, and integrations (with phpredis present).
  3. Loading a fresh integration afterwards produces an app-integrations-change containing only the new integration — phpredis is not re-emitted, confirming the leak is fixed.

The new test runs in its own container so DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL=15 applies before the worker spawns. The existing TelemetryTests no longer sets that env var, so its 24h default no longer interferes with tests that are not about the extended heartbeat (this addresses the intermittent 'telemetry reflects the loading of a new integration' failure caused by the heartbeat re-queueing data into the next regular flush).

Folds two helpers into TelemetryHelpers:

  • filterMessages(...) gains a userAppOnly flag (default true) that skips the sidecar's own datadog-ipc-helper telemetry worker.
  • WithExtendedHeartbeat / waitForExtendedHeartbeat mirror the existing WithIntegrations / waitForIntegrations pattern.

custom_integrations.php gains an "fs" branch (fopen on /dev/null) so tests can trigger the filesystem deferred-loading hook.

Description

Reviewer checklist

  • Test coverage seems ok.
  • Appropriate labels assigned.

@cataphract cataphract requested review from a team as code owners May 7, 2026 16:29
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 987505e92c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-official
Copy link
Copy Markdown

datadog-official Bot commented May 7, 2026

Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 60.67% (-0.05%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ef19035 | Docs | Datadog PR Page | Give us feedback!

gh-worker-dd-mergequeue-cf854d Bot pushed a commit to DataDog/libdatadog that referenced this pull request May 8, 2026
…ed-heartbeat (#1962)

Per the instrumentation-telemetry-api-docs schema, app-extended-heartbeat must carry the full state — configuration, dependencies, and integrations — so the agent can reconstruct application records on data loss. dd-trace-go and dd-trace-dotnet both ship the full triple. The Rust worker, however, defines AppStarted as a configuration-only struct and reuses it for app-extended-heartbeat, so dependencies and integrations are not included in the heartbeat payload.

The ExtendedHeartbeat handler does call unflush_stored() on all three collectors, evidently with the intent of re-emitting the full state. Because the heartbeat payload omits the re-queued dependencies and integrations and app_started_sent_success only pops configurations from unflushed, the re-queued items remain in unflushed and are sent on the next FlushData as a duplicate app-integrations-change / app-dependencies-loaded.

Updating the shared AppStarted struct, build_app_started, and app_started_sent_success addresses both Lifecycle(Start) and Lifecycle(ExtendedHeartbeat) call sites.

# How to test the change?

Tested in DataDog/dd-trace-php#3865 . Fixes an integration test and adds a new one to validate the behavior more thoroughly.


Co-authored-by: gustavo.lopes <gustavo.lopes@datadoghq.com>
Updates libdatadog to a version that includes dependencies and integrations
in app-extended-heartbeat. The previous behaviour omitted those fields and
caused a duplicate app-integrations-change / app-dependencies-loaded on the
next regular flush.

Adds an integration test (TelemetryExtendedHeartbeatTests) that drives
phpredis and exec via custom_integrations.php and asserts:
  1. They appear in a regular app-started/app-integrations-change.
  2. The next app-extended-heartbeat re-emits configuration, dependencies,
     and integrations (with phpredis present).
  3. Loading a fresh integration afterwards produces an
     app-integrations-change containing only the new integration —
     phpredis is not re-emitted, confirming the leak is fixed.

The new test runs in its own container so DD_TELEMETRY_EXTENDED_HEARTBEAT_INTERVAL=15
applies before the worker spawns. The existing TelemetryTests no longer
sets that env var, so its 24h default no longer interferes with tests
that are not about the extended heartbeat (this addresses the intermittent
'telemetry reflects the loading of a new integration' failure caused by
the heartbeat re-queueing data into the next regular flush).

Folds two helpers into TelemetryHelpers:
  - filterMessages(...) gains a userAppOnly flag (default true) that
    skips the sidecar's own datadog-ipc-helper telemetry worker.
  - WithExtendedHeartbeat / waitForExtendedHeartbeat mirror the existing
    WithIntegrations / waitForIntegrations pattern.

custom_integrations.php gains an "fs" branch (fopen on /dev/null) so
tests can trigger the filesystem deferred-loading hook.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@cataphract cataphract force-pushed the glopes/fix-extended-hearbeat branch from 8d1c686 to ef19035 Compare May 11, 2026 09:27
@cataphract cataphract requested review from a team as code owners May 11, 2026 09:27
@cataphract cataphract merged commit 7d17869 into master May 11, 2026
2116 of 2123 checks passed
@cataphract cataphract deleted the glopes/fix-extended-hearbeat branch May 11, 2026 13:34
@github-actions github-actions Bot added this to the 1.20.0 milestone May 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants