Skip to content

Redriven Step Functions Trace Merging#598

Merged
dd-mergequeue[bot] merged 6 commits into
mainfrom
avedmala/sfn-redrive-merging
Dec 17, 2024
Merged

Redriven Step Functions Trace Merging#598
dd-mergequeue[bot] merged 6 commits into
mainfrom
avedmala/sfn-redrive-merging

Conversation

@avedmala
Copy link
Copy Markdown
Contributor

@avedmala avedmala commented Dec 4, 2024 •
edited
Loading

What does this PR do?

Adds support for Step Functions trace merging in Redrive cases

We previously used hash(ExecutionId # StateName # StateEnteredTime) for spanID calculation but these values are identical across redrives for a Lambda task state. The new approach also adds a RedriveCount to the end of the hash but omits this value when it is 0 to have easy backwards compatability.

Motivation

https://github.com/DataDog/logs-backend/pull/84188

Testing Guidelines

Redriven Trace with Lambda and Trace Merging
Screenshot 2024-12-04 at 2 36 08 PM

Types of Changes

  • Bug fix
  • New feature
  • Breaking change
  • Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

  • This PR's description is comprehensive
  • This PR contains breaking changes that are documented in the description
  • This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
  • This PR impacts documentation, and it has been updated (or a ticket has been logged)
  • This PR's changes are covered by the automated tests
  • This PR collects user input/sensitive content into Datadog
  • This PR passes the integration tests (ask a Datadog member to run the tests)

@avedmala avedmala marked this pull request as ready for review December 16, 2024 14:26
@avedmala avedmala requested a review from a team as a code owner December 16, 2024 14:26
@avedmala avedmala changed the title use redrive count to generate step functions parent ID Redriven Step Functions Trace Merging Dec 16, 2024
kimi-p
kimi-p reviewed Dec 16, 2024
View reviewed changes
Comment thread src/trace/context/extractors/step-function.spec.ts Outdated
expect(traceContext).not.toBeNull();

expect(traceContext?.toTraceId()).toBe("1139193989631387307");
expect(traceContext?.toSpanId()).toBe("3615526381891637131");
Copy link
Copy Markdown
Contributor

@kimi-p kimi-p Dec 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you link the PR line in the reducer that matches this test?

Also, is the plan to match traceID only but not spanID?

Copy link
Copy Markdown
Contributor Author

@avedmala avedmala Dec 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the matching line from logs-backend

Yes the lambda span should be in the same trace but we want the spanID to be different from the original when redrive count was 0

Copy link
Copy Markdown
Contributor

@kimi-p kimi-p Dec 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you link the test just to verify that the two matches? Thanks!

kimi-p
kimi-p approved these changes Dec 16, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@kimi-p kimi-p left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@avedmala avedmala mentioned this pull request Dec 17, 2024
Merged
11 tasks
@avedmala
Copy link
Copy Markdown
Contributor Author

avedmala commented Dec 17, 2024

/merge

@dd-devflow
Copy link
Copy Markdown

dd-devflow Bot commented Dec 17, 2024 •
edited
Loading

Devflow running: /merge

View all feedbacks in Devflow UI.

2024-12-17 17:12:15 UTC ℹ️ MergeQueue: pull request added to the queue

The median merge time in main is 0s.

2024-12-17 17:21:39 UTC 🚨 MergeQueue: This merge request is in error

mergequeue build completed successfully, but the github api returned an error while merging the pr.
It's probably because:

  • target branch of PR is restricted to only allow up-to-date branches, but the pr is now outdated
Details

Error: PUT https://api.github.com/repos/DataDog/datadog-lambda-js/pulls/598/merge: 405 15 of 15 required status checks are expected. []

FullStacktrace:
activity error (type: github.GithubService_MergePullRequest, scheduledEventID: 41, startedEventID: 42, identity: 1@github-worker-66d47658fd-jwq4x@): PUT https://api.github.com/repos/DataDog/datadog-lambda-js/pulls/598/merge: 405 15 of 15 required status checks are expected. [] (type: GitFailure, retryable: false): PUT https://api.github.com/repos/DataDog/datadog-lambda-js/pulls/598/merge: 405 15 of 15 required status checks are expected. [] (type: ErrorResponse, retryable: true)

@avedmala
Copy link
Copy Markdown
Contributor Author

avedmala commented Dec 17, 2024

/merge

@dd-devflow
Copy link
Copy Markdown

dd-devflow Bot commented Dec 17, 2024 •
edited
Loading

Devflow running: /merge

View all feedbacks in Devflow UI.

2024-12-17 18:00:19 UTC ℹ️ MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2024-12-17 18:02:25 UTC ℹ️ MergeQueue: merge request added to the queue

The median merge time in main is 0s.

2024-12-17 18:09:19 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 2fae7e7:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

  • Investigate the failures and when ready, re-add your pull request to the queue!
  • Any question, go check the FAQ.
Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

@avedmala
Copy link
Copy Markdown
Contributor Author

avedmala commented Dec 17, 2024

/merge

@dd-devflow
Copy link
Copy Markdown

dd-devflow Bot commented Dec 17, 2024 •
edited
Loading

Devflow running: /merge

View all feedbacks in Devflow UI.

2024-12-17 21:04:12 UTC ℹ️ MergeQueue: pull request added to the queue

The median merge time in main is 0s.

2024-12-17 21:12:59 UTC ℹ️ MergeQueue: This merge request was merged

@dd-mergequeue dd-mergequeue Bot merged commit 1a75734 into main Dec 17, 2024
@dd-mergequeue dd-mergequeue Bot deleted the avedmala/sfn-redrive-merging branch December 17, 2024 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kimi-p kimi-p kimi-p approved these changes

Assignees

No one assigned

Labels

mergequeue-status: done

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@avedmala @kimi-p