feat(data-pipeline): port dd-trace-rs trace buffer implementation#1826
Conversation
📚 Documentation Check Results📦
|
3ede0c0 to
5938b42
Compare
Clippy Allow Annotation ReportComparing clippy allow annotations between branches:
Summary by Rule
Annotation Counts by File
Annotation Stats by Crate
About This ReportThis report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality. |
🔒 Cargo Deny Results📦
|
🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: d366625 | Docs | Datadog PR Page | Give us feedback! |
BenchmarksComparisonBenchmark execution time: 2026-03-31 09:40:19 Comparing candidate commit 5938b42 in PR branch Found 0 performance improvements and 1 performance regressions! Performance is the same for 61 metrics, 0 unstable metrics.
|
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | 145.162µs | 147.194µs ± 1.851µs | 146.987µs ± 0.608µs | 147.504µs | 148.894µs | 154.489µs | 165.669µs | 12.71% | 6.214 | 53.332 | 1.25% | 0.131µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | [146.937µs; 147.450µs] or [-0.174%; +0.174%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| normalization/normalize_trace/test_trace | execution_time | 244.115ns | 254.188ns ± 13.661ns | 247.446ns ± 1.207ns | 256.824ns | 284.718ns | 299.800ns | 301.808ns | 21.97% | 1.954 | 2.976 | 5.36% | 0.966ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| normalization/normalize_trace/test_trace | execution_time | [252.294ns; 256.081ns] or [-0.745%; +0.745%] | None | None | None |
Group 3
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| profile_add_sample2_frames_x1000 | execution_time | 755.490µs | 757.180µs ± 0.605µs | 757.134µs ± 0.376µs | 757.532µs | 758.231µs | 758.721µs | 758.947µs | 0.24% | 0.326 | 0.400 | 0.08% | 0.043µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| profile_add_sample2_frames_x1000 | execution_time | [757.096µs; 757.264µs] or [-0.011%; +0.011%] | None | None | None |
Group 4
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| receiver_entry_point/report/2598 | execution_time | 3.425ms | 3.453ms ± 0.018ms | 3.448ms ± 0.011ms | 3.464ms | 3.487ms | 3.498ms | 3.525ms | 2.24% | 0.984 | 0.832 | 0.51% | 0.001ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| receiver_entry_point/report/2598 | execution_time | [3.450ms; 3.455ms] or [-0.071%; +0.071%] | None | None | None |
Group 5
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| profile_add_sample_frames_x1000 | execution_time | 4.372ms | 4.377ms ± 0.002ms | 4.377ms ± 0.002ms | 4.378ms | 4.381ms | 4.383ms | 4.390ms | 0.30% | 1.067 | 3.179 | 0.06% | 0.000ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| profile_add_sample_frames_x1000 | execution_time | [4.377ms; 4.377ms] or [-0.008%; +0.008%] | None | None | None |
Group 6
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| tags/replace_trace_tags | execution_time | 2.536µs | 2.564µs ± 0.017µs | 2.560µs ± 0.011µs | 2.573µs | 2.596µs | 2.608µs | 2.616µs | 2.18% | 0.735 | -0.111 | 0.65% | 0.001µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| tags/replace_trace_tags | execution_time | [2.561µs; 2.566µs] or [-0.091%; +0.091%] | None | None | None |
Group 7
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| credit_card/is_card_number/ | execution_time | 3.893µs | 3.914µs ± 0.004µs | 3.913µs ± 0.002µs | 3.916µs | 3.919µs | 3.922µs | 3.928µs | 0.38% | -0.124 | 7.013 | 0.09% | 0.000µs | 1 | 200 |
| credit_card/is_card_number/ | throughput | 254600717.643op/s | 255512513.409op/s ± 228583.913op/s | 255579476.992op/s ± 124824.793op/s | 255664568.087op/s | 255737052.045op/s | 255777886.821op/s | 256886255.168op/s | 0.51% | 0.148 | 7.108 | 0.09% | 16163.323op/s | 1 | 200 |
| credit_card/is_card_number/ 3782-8224-6310-005 | execution_time | 79.549µs | 80.012µs ± 0.253µs | 79.965µs ± 0.155µs | 80.140µs | 80.549µs | 80.616µs | 80.707µs | 0.93% | 0.751 | -0.044 | 0.32% | 0.018µs | 1 | 200 |
| credit_card/is_card_number/ 3782-8224-6310-005 | throughput | 12390503.878op/s | 12498179.354op/s ± 39470.673op/s | 12505514.939op/s ± 24184.513op/s | 12527126.558op/s | 12550250.639op/s | 12562153.382op/s | 12570921.367op/s | 0.52% | -0.738 | -0.064 | 0.32% | 2790.998op/s | 1 | 200 |
| credit_card/is_card_number/ 378282246310005 | execution_time | 73.143µs | 73.591µs ± 0.229µs | 73.555µs ± 0.142µs | 73.725µs | 74.000µs | 74.257µs | 74.528µs | 1.32% | 0.743 | 0.958 | 0.31% | 0.016µs | 1 | 200 |
| credit_card/is_card_number/ 378282246310005 | throughput | 13417733.508op/s | 13588742.076op/s ± 42218.429op/s | 13595181.912op/s ± 26183.206op/s | 13618234.480op/s | 13650845.305op/s | 13665220.412op/s | 13671818.060op/s | 0.56% | -0.721 | 0.894 | 0.31% | 2985.294op/s | 1 | 200 |
| credit_card/is_card_number/37828224631 | execution_time | 3.894µs | 3.917µs ± 0.005µs | 3.916µs ± 0.003µs | 3.919µs | 3.925µs | 3.942µs | 3.947µs | 0.79% | 1.898 | 12.307 | 0.13% | 0.000µs | 1 | 200 |
| credit_card/is_card_number/37828224631 | throughput | 253347052.827op/s | 255293143.117op/s ± 331215.054op/s | 255342918.091op/s ± 170479.472op/s | 255491747.091op/s | 255620352.412op/s | 255701130.017op/s | 256832818.930op/s | 0.58% | -1.856 | 12.169 | 0.13% | 23420.441op/s | 1 | 200 |
| credit_card/is_card_number/378282246310005 | execution_time | 69.605µs | 69.979µs ± 0.230µs | 69.940µs ± 0.142µs | 70.124µs | 70.418µs | 70.558µs | 70.840µs | 1.29% | 0.778 | 0.470 | 0.33% | 0.016µs | 1 | 200 |
| credit_card/is_card_number/378282246310005 | throughput | 14116301.742op/s | 14290119.381op/s ± 46778.607op/s | 14297951.818op/s ± 29155.792op/s | 14323720.442op/s | 14354096.083op/s | 14365366.281op/s | 14366723.373op/s | 0.48% | -0.759 | 0.423 | 0.33% | 3307.747op/s | 1 | 200 |
| credit_card/is_card_number/37828224631000521389798 | execution_time | 51.561µs | 52.017µs ± 0.192µs | 52.039µs ± 0.140µs | 52.163µs | 52.296µs | 52.358µs | 52.389µs | 0.67% | -0.308 | -0.741 | 0.37% | 0.014µs | 1 | 200 |
| credit_card/is_card_number/37828224631000521389798 | throughput | 19087893.420op/s | 19224691.039op/s ± 71022.559op/s | 19216312.118op/s ± 51643.365op/s | 19277026.130op/s | 19345199.103op/s | 19377081.744op/s | 19394638.076op/s | 0.93% | 0.321 | -0.731 | 0.37% | 5022.053op/s | 1 | 200 |
| credit_card/is_card_number/x371413321323331 | execution_time | 5.746µs | 5.884µs ± 0.054µs | 5.882µs ± 0.035µs | 5.915µs | 5.977µs | 6.011µs | 6.015µs | 2.27% | 0.035 | -0.230 | 0.91% | 0.004µs | 1 | 200 |
| credit_card/is_card_number/x371413321323331 | throughput | 166238848.022op/s | 169964129.026op/s ± 1548728.618op/s | 170004704.233op/s ± 1015979.168op/s | 171110156.247op/s | 172437366.587op/s | 173319885.316op/s | 174046044.711op/s | 2.38% | 0.013 | -0.229 | 0.91% | 109511.651op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/ | execution_time | 3.894µs | 3.918µs ± 0.004µs | 3.917µs ± 0.003µs | 3.920µs | 3.925µs | 3.927µs | 3.931µs | 0.35% | -0.469 | 4.445 | 0.11% | 0.000µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/ | throughput | 254401296.829op/s | 255250206.334op/s ± 269425.943op/s | 255286703.763op/s ± 171973.284op/s | 255432114.207op/s | 255586331.382op/s | 255684132.656op/s | 256796304.225op/s | 0.59% | 0.489 | 4.547 | 0.11% | 19051.291op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 | execution_time | 64.193µs | 64.393µs ± 0.117µs | 64.363µs ± 0.057µs | 64.435µs | 64.631µs | 64.801µs | 64.873µs | 0.79% | 1.493 | 2.753 | 0.18% | 0.008µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 | throughput | 15414672.476op/s | 15529724.925op/s ± 28134.668op/s | 15536833.450op/s ± 13794.405op/s | 15547075.037op/s | 15562010.099op/s | 15568728.678op/s | 15578011.637op/s | 0.27% | -1.479 | 2.700 | 0.18% | 1989.421op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/ 378282246310005 | execution_time | 57.487µs | 57.615µs ± 0.092µs | 57.599µs ± 0.057µs | 57.659µs | 57.786µs | 57.851µs | 57.995µs | 0.69% | 1.149 | 1.872 | 0.16% | 0.006µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/ 378282246310005 | throughput | 17242920.443op/s | 17356621.217op/s ± 27547.458op/s | 17361332.840op/s ± 17136.325op/s | 17376897.436op/s | 17391244.900op/s | 17393536.631op/s | 17395174.579op/s | 0.19% | -1.136 | 1.824 | 0.16% | 1947.899op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/37828224631 | execution_time | 3.901µs | 3.917µs ± 0.004µs | 3.916µs ± 0.003µs | 3.919µs | 3.924µs | 3.928µs | 3.931µs | 0.39% | 0.536 | 0.951 | 0.11% | 0.000µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/37828224631 | throughput | 254381684.678op/s | 255329191.025op/s ± 274053.523op/s | 255373173.071op/s ± 174269.807op/s | 255522235.569op/s | 255677287.135op/s | 255792287.145op/s | 256329373.409op/s | 0.37% | -0.528 | 0.950 | 0.11% | 19378.510op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/378282246310005 | execution_time | 54.606µs | 54.791µs ± 0.135µs | 54.762µs ± 0.061µs | 54.827µs | 55.113µs | 55.255µs | 55.452µs | 1.26% | 2.060 | 5.210 | 0.25% | 0.010µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/378282246310005 | throughput | 18033460.050op/s | 18251216.346op/s ± 44717.898op/s | 18260881.540op/s ± 20178.563op/s | 18279949.638op/s | 18298739.401op/s | 18309197.391op/s | 18313075.536op/s | 0.29% | -2.038 | 5.102 | 0.24% | 3162.033op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/37828224631000521389798 | execution_time | 51.404µs | 52.145µs ± 0.239µs | 52.158µs ± 0.186µs | 52.359µs | 52.443µs | 52.482µs | 52.532µs | 0.72% | -0.713 | 0.140 | 0.46% | 0.017µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/37828224631000521389798 | throughput | 19035976.714op/s | 19177654.413op/s ± 88263.922op/s | 19172533.331op/s ± 68386.954op/s | 19231939.616op/s | 19346709.831op/s | 19437837.774op/s | 19453846.857op/s | 1.47% | 0.736 | 0.199 | 0.46% | 6241.202op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/x371413321323331 | execution_time | 5.706µs | 5.887µs ± 0.055µs | 5.898µs ± 0.035µs | 5.922µs | 5.969µs | 6.001µs | 6.050µs | 2.57% | -0.367 | 0.332 | 0.94% | 0.004µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/x371413321323331 | throughput | 165296818.766op/s | 169884924.669op/s ± 1606249.539op/s | 169552986.733op/s ± 996677.214op/s | 170742116.737op/s | 172520793.526op/s | 174112874.907op/s | 175244568.347op/s | 3.36% | 0.429 | 0.392 | 0.94% | 113578.994op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| credit_card/is_card_number/ | execution_time | [3.913µs; 3.914µs] or [-0.012%; +0.012%] | None | None | None |
| credit_card/is_card_number/ | throughput | [255480833.877op/s; 255544192.941op/s] or [-0.012%; +0.012%] | None | None | None |
| credit_card/is_card_number/ 3782-8224-6310-005 | execution_time | [79.977µs; 80.048µs] or [-0.044%; +0.044%] | None | None | None |
| credit_card/is_card_number/ 3782-8224-6310-005 | throughput | [12492709.099op/s; 12503649.610op/s] or [-0.044%; +0.044%] | None | None | None |
| credit_card/is_card_number/ 378282246310005 | execution_time | [73.559µs; 73.623µs] or [-0.043%; +0.043%] | None | None | None |
| credit_card/is_card_number/ 378282246310005 | throughput | [13582891.008op/s; 13594593.144op/s] or [-0.043%; +0.043%] | None | None | None |
| credit_card/is_card_number/37828224631 | execution_time | [3.916µs; 3.918µs] or [-0.018%; +0.018%] | None | None | None |
| credit_card/is_card_number/37828224631 | throughput | [255247239.896op/s; 255339046.338op/s] or [-0.018%; +0.018%] | None | None | None |
| credit_card/is_card_number/378282246310005 | execution_time | [69.947µs; 70.011µs] or [-0.045%; +0.045%] | None | None | None |
| credit_card/is_card_number/378282246310005 | throughput | [14283636.316op/s; 14296602.446op/s] or [-0.045%; +0.045%] | None | None | None |
| credit_card/is_card_number/37828224631000521389798 | execution_time | [51.991µs; 52.044µs] or [-0.051%; +0.051%] | None | None | None |
| credit_card/is_card_number/37828224631000521389798 | throughput | [19214847.995op/s; 19234534.082op/s] or [-0.051%; +0.051%] | None | None | None |
| credit_card/is_card_number/x371413321323331 | execution_time | [5.877µs; 5.892µs] or [-0.126%; +0.126%] | None | None | None |
| credit_card/is_card_number/x371413321323331 | throughput | [169749490.134op/s; 170178767.917op/s] or [-0.126%; +0.126%] | None | None | None |
| credit_card/is_card_number_no_luhn/ | execution_time | [3.917µs; 3.918µs] or [-0.015%; +0.015%] | None | None | None |
| credit_card/is_card_number_no_luhn/ | throughput | [255212866.489op/s; 255287546.178op/s] or [-0.015%; +0.015%] | None | None | None |
| credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 | execution_time | [64.377µs; 64.409µs] or [-0.025%; +0.025%] | None | None | None |
| credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 | throughput | [15525825.730op/s; 15533624.119op/s] or [-0.025%; +0.025%] | None | None | None |
| credit_card/is_card_number_no_luhn/ 378282246310005 | execution_time | [57.602µs; 57.628µs] or [-0.022%; +0.022%] | None | None | None |
| credit_card/is_card_number_no_luhn/ 378282246310005 | throughput | [17352803.404op/s; 17360439.030op/s] or [-0.022%; +0.022%] | None | None | None |
| credit_card/is_card_number_no_luhn/37828224631 | execution_time | [3.916µs; 3.917µs] or [-0.015%; +0.015%] | None | None | None |
| credit_card/is_card_number_no_luhn/37828224631 | throughput | [255291209.843op/s; 255367172.208op/s] or [-0.015%; +0.015%] | None | None | None |
| credit_card/is_card_number_no_luhn/378282246310005 | execution_time | [54.772µs; 54.810µs] or [-0.034%; +0.034%] | None | None | None |
| credit_card/is_card_number_no_luhn/378282246310005 | throughput | [18245018.876op/s; 18257413.817op/s] or [-0.034%; +0.034%] | None | None | None |
| credit_card/is_card_number_no_luhn/37828224631000521389798 | execution_time | [52.112µs; 52.178µs] or [-0.064%; +0.064%] | None | None | None |
| credit_card/is_card_number_no_luhn/37828224631000521389798 | throughput | [19165421.883op/s; 19189886.944op/s] or [-0.064%; +0.064%] | None | None | None |
| credit_card/is_card_number_no_luhn/x371413321323331 | execution_time | [5.879µs; 5.895µs] or [-0.131%; +0.131%] | None | None | None |
| credit_card/is_card_number_no_luhn/x371413321323331 | throughput | [169662313.931op/s; 170107535.407op/s] or [-0.131%; +0.131%] | None | None | None |
Group 8
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... | execution_time | 534.633µs | 535.331µs ± 0.508µs | 535.168µs ± 0.278µs | 535.624µs | 536.259µs | 536.822µs | 537.367µs | 0.41% | 1.130 | 1.071 | 0.09% | 0.036µs | 1 | 200 |
| normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... | throughput | 1860925.180op/s | 1868003.766op/s ± 1770.247op/s | 1868573.345op/s ± 970.945op/s | 1869242.578op/s | 1870152.146op/s | 1870372.134op/s | 1870441.555op/s | 0.10% | -1.125 | 1.052 | 0.09% | 125.175op/s | 1 | 200 |
| normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて | execution_time | 381.913µs | 382.847µs ± 0.462µs | 382.768µs ± 0.306µs | 383.111µs | 383.632µs | 384.109µs | 384.633µs | 0.49% | 0.821 | 1.012 | 0.12% | 0.033µs | 1 | 200 |
| normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて | throughput | 2599877.693op/s | 2612015.951op/s ± 3149.675op/s | 2612545.626op/s ± 2091.495op/s | 2614298.777op/s | 2615896.217op/s | 2617295.626op/s | 2618394.105op/s | 0.22% | -0.812 | 0.987 | 0.12% | 222.716op/s | 1 | 200 |
| normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters | execution_time | 194.544µs | 195.514µs ± 0.494µs | 195.557µs ± 0.254µs | 195.778µs | 196.104µs | 196.446µs | 199.155µs | 1.84% | 1.821 | 13.424 | 0.25% | 0.035µs | 1 | 200 |
| normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters | throughput | 5021211.978op/s | 5114754.570op/s ± 12878.646op/s | 5113610.637op/s ± 6643.531op/s | 5121482.960op/s | 5135117.765op/s | 5138075.153op/s | 5140234.387op/s | 0.52% | -1.731 | 12.663 | 0.25% | 910.658op/s | 1 | 200 |
| normalization/normalize_service/normalize_service/[empty string] | execution_time | 37.963µs | 38.315µs ± 0.130µs | 38.306µs ± 0.082µs | 38.406µs | 38.534µs | 38.604µs | 38.736µs | 1.12% | 0.080 | 0.277 | 0.34% | 0.009µs | 1 | 200 |
| normalization/normalize_service/normalize_service/[empty string] | throughput | 25815922.272op/s | 26099776.312op/s ± 88862.624op/s | 26105454.639op/s ± 56048.329op/s | 26153468.819op/s | 26245179.056op/s | 26322381.104op/s | 26341483.420op/s | 0.90% | -0.057 | 0.269 | 0.34% | 6283.536op/s | 1 | 200 |
| normalization/normalize_service/normalize_service/test_ASCII | execution_time | 45.801µs | 45.943µs ± 0.080µs | 45.923µs ± 0.039µs | 45.982µs | 46.106µs | 46.191µs | 46.281µs | 0.78% | 1.323 | 2.228 | 0.17% | 0.006µs | 1 | 200 |
| normalization/normalize_service/normalize_service/test_ASCII | throughput | 21607035.251op/s | 21766200.000op/s ± 37815.879op/s | 21775786.351op/s ± 18332.460op/s | 21790339.048op/s | 21810108.906op/s | 21827075.042op/s | 21833399.036op/s | 0.26% | -1.310 | 2.181 | 0.17% | 2673.986op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... | execution_time | [535.261µs; 535.402µs] or [-0.013%; +0.013%] | None | None | None |
| normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... | throughput | [1867758.427op/s; 1868249.105op/s] or [-0.013%; +0.013%] | None | None | None |
| normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて | execution_time | [382.783µs; 382.911µs] or [-0.017%; +0.017%] | None | None | None |
| normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて | throughput | [2611579.437op/s; 2612452.466op/s] or [-0.017%; +0.017%] | None | None | None |
| normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters | execution_time | [195.446µs; 195.583µs] or [-0.035%; +0.035%] | None | None | None |
| normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters | throughput | [5112969.714op/s; 5116539.427op/s] or [-0.035%; +0.035%] | None | None | None |
| normalization/normalize_service/normalize_service/[empty string] | execution_time | [38.297µs; 38.333µs] or [-0.047%; +0.047%] | None | None | None |
| normalization/normalize_service/normalize_service/[empty string] | throughput | [26087460.807op/s; 26112091.817op/s] or [-0.047%; +0.047%] | None | None | None |
| normalization/normalize_service/normalize_service/test_ASCII | execution_time | [45.932µs; 45.954µs] or [-0.024%; +0.024%] | None | None | None |
| normalization/normalize_service/normalize_service/test_ASCII | throughput | [21760959.083op/s; 21771440.917op/s] or [-0.024%; +0.024%] | None | None | None |
Group 9
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| two way interface | execution_time | 14.987µs | 15.554µs ± 0.201µs | 15.537µs ± 0.120µs | 15.665µs | 15.953µs | 16.031µs | 16.179µs | 4.13% | 0.347 | 0.427 | 1.29% | 0.014µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| two way interface | execution_time | [15.526µs; 15.581µs] or [-0.179%; +0.179%] | None | None | None |
Group 10
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| benching serializing traces from their internal representation to msgpack | execution_time | 14.724ms | 14.797ms ± 0.030ms | 14.793ms ± 0.017ms | 14.813ms | 14.846ms | 14.863ms | 14.944ms | 1.01% | 1.270 | 4.631 | 0.20% | 0.002ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| benching serializing traces from their internal representation to msgpack | execution_time | [14.793ms; 14.801ms] or [-0.028%; +0.028%] | None | None | None |
Group 11
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | 191.119ns | 193.821ns ± 2.150ns | 193.349ns ± 1.179ns | 194.677ns | 197.961ns | 201.344ns | 203.578ns | 5.29% | 1.519 | 3.216 | 1.11% | 0.152ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | [193.523ns; 194.119ns] or [-0.154%; +0.154%] | None | None | None |
Group 12
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| write only interface | execution_time | 5.222µs | 5.418µs ± 0.037µs | 5.416µs ± 0.026µs | 5.445µs | 5.477µs | 5.492µs | 5.500µs | 1.54% | -0.476 | 2.764 | 0.68% | 0.003µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| write only interface | execution_time | [5.413µs; 5.424µs] or [-0.095%; +0.095%] | None | None | None |
Group 13
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| concentrator/add_spans_to_concentrator | execution_time | 14.862ms | 14.910ms ± 0.017ms | 14.909ms ± 0.010ms | 14.919ms | 14.936ms | 14.960ms | 14.985ms | 0.51% | 0.492 | 2.000 | 0.12% | 0.001ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| concentrator/add_spans_to_concentrator | execution_time | [14.907ms; 14.912ms] or [-0.016%; +0.016%] | None | None | None |
Group 14
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sql/obfuscate_sql_string | execution_time | 288.634µs | 289.526µs ± 0.576µs | 289.438µs ± 0.190µs | 289.612µs | 290.412µs | 291.836µs | 293.627µs | 1.45% | 3.600 | 20.030 | 0.20% | 0.041µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| sql/obfuscate_sql_string | execution_time | [289.446µs; 289.605µs] or [-0.028%; +0.028%] | None | None | None |
Group 15
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... | execution_time | 185.650µs | 186.487µs ± 0.660µs | 186.242µs ± 0.295µs | 186.951µs | 187.750µs | 188.363µs | 188.613µs | 1.27% | 1.102 | 0.272 | 0.35% | 0.047µs | 1 | 200 |
| normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... | throughput | 5301872.025op/s | 5362364.084op/s ± 18900.401op/s | 5369366.076op/s ± 8519.453op/s | 5376292.601op/s | 5381953.072op/s | 5385146.412op/s | 5386485.013op/s | 0.32% | -1.091 | 0.237 | 0.35% | 1336.460op/s | 1 | 200 |
| normalization/normalize_name/normalize_name/bad-name | execution_time | 17.850µs | 17.979µs ± 0.065µs | 17.977µs ± 0.035µs | 18.007µs | 18.102µs | 18.168µs | 18.229µs | 1.40% | 0.898 | 1.787 | 0.36% | 0.005µs | 1 | 200 |
| normalization/normalize_name/normalize_name/bad-name | throughput | 54857142.857op/s | 55620139.313op/s ± 200755.788op/s | 55626646.928op/s ± 107684.306op/s | 55755093.011op/s | 55924799.103op/s | 55994131.933op/s | 56020865.905op/s | 0.71% | -0.866 | 1.698 | 0.36% | 14195.578op/s | 1 | 200 |
| normalization/normalize_name/normalize_name/good | execution_time | 9.864µs | 9.929µs ± 0.072µs | 9.885µs ± 0.015µs | 9.978µs | 10.073µs | 10.122µs | 10.129µs | 2.47% | 1.090 | -0.146 | 0.72% | 0.005µs | 1 | 200 |
| normalization/normalize_name/normalize_name/good | throughput | 98726648.921op/s | 100717184.066op/s ± 725984.940op/s | 101166649.247op/s ± 148775.520op/s | 101249768.524op/s | 101322506.300op/s | 101361822.768op/s | 101379910.161op/s | 0.21% | -1.075 | -0.189 | 0.72% | 51334.887op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... | execution_time | [186.396µs; 186.579µs] or [-0.049%; +0.049%] | None | None | None |
| normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... | throughput | [5359744.670op/s; 5364983.498op/s] or [-0.049%; +0.049%] | None | None | None |
| normalization/normalize_name/normalize_name/bad-name | execution_time | [17.970µs; 17.988µs] or [-0.050%; +0.050%] | None | None | None |
| normalization/normalize_name/normalize_name/bad-name | throughput | [55592316.492op/s; 55647962.135op/s] or [-0.050%; +0.050%] | None | None | None |
| normalization/normalize_name/normalize_name/good | execution_time | [9.919µs; 9.939µs] or [-0.101%; +0.101%] | None | None | None |
| normalization/normalize_name/normalize_name/good | throughput | [100616569.535op/s; 100817798.596op/s] or [-0.100%; +0.100%] | None | None | None |
Group 16
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| profile_add_sample_timestamped_x1000 | execution_time | 4.374ms | 4.381ms ± 0.009ms | 4.379ms ± 0.002ms | 4.382ms | 4.385ms | 4.393ms | 4.501ms | 2.78% | 11.444 | 145.811 | 0.21% | 0.001ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| profile_add_sample_timestamped_x1000 | execution_time | [4.379ms; 4.382ms] or [-0.029%; +0.029%] | None | None | None |
Group 17
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| redis/obfuscate_redis_string | execution_time | 33.647µs | 34.231µs ± 0.984µs | 33.772µs ± 0.056µs | 33.889µs | 36.332µs | 36.419µs | 37.277µs | 10.38% | 1.703 | 1.004 | 2.87% | 0.070µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| redis/obfuscate_redis_string | execution_time | [34.094µs; 34.367µs] or [-0.399%; +0.399%] | None | None | None |
Group 18
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| profile_serialize_compressed_pprof_timestamped_x1000 | execution_time | 934.320µs | 937.751µs ± 2.304µs | 937.143µs ± 1.085µs | 938.459µs | 941.935µs | 945.090µs | 951.054µs | 1.48% | 1.823 | 5.759 | 0.25% | 0.163µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| profile_serialize_compressed_pprof_timestamped_x1000 | execution_time | [937.431µs; 938.070µs] or [-0.034%; +0.034%] | None | None | None |
Group 19
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ip_address/quantize_peer_ip_address_benchmark | execution_time | 4.929µs | 5.007µs ± 0.052µs | 5.013µs ± 0.055µs | 5.046µs | 5.091µs | 5.093µs | 5.097µs | 1.68% | 0.231 | -1.377 | 1.03% | 0.004µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| ip_address/quantize_peer_ip_address_benchmark | execution_time | [5.000µs; 5.014µs] or [-0.143%; +0.143%] | None | None | None |
Group 20
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| benching string interning on wordpress profile | execution_time | 160.220µs | 161.100µs ± 0.614µs | 161.078µs ± 0.227µs | 161.283µs | 161.677µs | 162.973µs | 165.895µs | 2.99% | 4.142 | 27.671 | 0.38% | 0.043µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| benching string interning on wordpress profile | execution_time | [161.015µs; 161.185µs] or [-0.053%; +0.053%] | None | None | None |
Group 21
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| benching deserializing traces from msgpack to their internal representation | execution_time | 47.873ms | 48.280ms ± 0.975ms | 48.127ms ± 0.095ms | 48.260ms | 48.438ms | 53.542ms | 58.876ms | 22.33% | 8.559 | 79.737 | 2.01% | 0.069ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| benching deserializing traces from msgpack to their internal representation | execution_time | [48.145ms; 48.416ms] or [-0.280%; +0.280%] | None | None | None |
Baseline
Omitted due to size.
| if res.timed_out() { | ||
| return Err(TraceBufferError::TimedOut(timeout)); | ||
| } | ||
| Ok(()) |
There was a problem hiding this comment.
Or am I missing something?
Doesn't this always return Ok, even if the last flush didn't happen because of shut down?
I think we need to look at the state.last_flush_generation
There was a problem hiding this comment.
res is a WaitTimeoutResult so I cannot just return it
There was a problem hiding this comment.
Sorry for the confusion. I was editing my question while you answered.
Doesn't this always return Ok, even if the last flush didn't happen because of shut down?
I think we need to look at the
state.last_flush_generation
| } | ||
|
|
||
| fn reset(&mut self) { | ||
| *self = Self::new(self.max_buffered_spans); |
There was a problem hiding this comment.
Not sure if it is worth it (is this remotely perf-sensitive?), but clear()ing chunks instead of rea might avoid dropping the old vector to just re-allocate a new one.
There was a problem hiding this comment.
I changed to reset each field, and clear the Vec.
Right now this is not perf sensitive as this method is only used after a process forks but better make the code a bit nicer just in case
| let gen = state.batch.batch_gen; | ||
| if state.batch.span_count() > self.flush_trigger_number_of_spans || self.synchronous_write { | ||
| state.flush_needed = true; | ||
| self.waiter.notify_receiver(state); |
There was a problem hiding this comment.
I wonder if we can avoid notifying if state.flush_needed was already set to true here. I think, but it must be double checked that this won't let a waiter be stuck forever. I'm saying that because notifying is basically a futex syscall, and a syscall is costly, so usually we try to avoid unnecessary wait or notify. That being said, it's a bit of a detail and it might be for something to think about later, or be ignored entirely if this part is not perf sensitive.
There was a problem hiding this comment.
after thinking a bit, I agree that we can spare the notification if flush_needed is already true
There was a problem hiding this comment.
Do you want to leave a comment here for a future TODO, so that it's not lost in limbo? (There's also a similar situation in trigger_flush below)
| /// This allows mapping from the buffered spans to another type, and | ||
| /// calling any method on the trace exporter to send traces | ||
| pub trait Export<T>: Send + Debug { | ||
| fn export_trace_chunks<'a: 'c, 'b: 'c, 'c>( |
There was a problem hiding this comment.
Do 'a and 'b need to be different here?
There was a problem hiding this comment.
No, refactored to have only one lifetime
# Motivation As we implement native spans in ruby in python, we have opportuinities to replace more components of the span pipeline with native code. One of them i the span buffer which we is before the trace exporter. # Changes Add a TraceBuffer implementation. The TraceBuffer is split in two part, a Sender which allows putting trace chunks in the queue and a background task, spawned on the shared runtime which periodically pulls data from the chunk and submit them using the TraceExporter.
b08bedf to
7252dcb
Compare
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1826 +/- ##
==========================================
+ Coverage 71.62% 71.76% +0.13%
==========================================
Files 432 433 +1
Lines 69183 69713 +530
==========================================
+ Hits 49554 50027 +473
- Misses 19629 19686 +57
🚀 New features to boost your workflow:
|
a7b5c98 to
04c44e7
Compare
|
The sidecar has a TraceFlusher - this seems to duplicate this functionality, instead of moving the battle-tested implememtation to some common place and adapting it in a way that it fits both use cases perfectly. The approach used seems also quite a bit more (unnecessarily?) complicated over simply pushing to a Mutex (what the TraceFlusher does) ... and what you seem to be doing as well, a few layers down in Batch. Not sure what the Waiter is about - it feels like what the TraceFlusher impl ultimately solves with a simple ManualFuture. |
| }; | ||
| } | ||
|
|
||
| fn span_count(&self) -> usize { |
There was a problem hiding this comment.
Not very important, but any caller that sees span_count is also able to see the field directly, so could access the field instead of going through a getter. I would just access the field directly, FWIW.
|
|
||
| let chunk_len = chunk.len(); | ||
| self.chunks.push(chunk); | ||
| self.span_count += chunk_len; |
There was a problem hiding this comment.
Nit: you can move this increment up and avoid storing the length in a variable (well, the only difference is if push panics, not sure if it's relevant here).
| /// # Synchronous mode | ||
| /// | ||
| /// If [`TraceBufferConfig::synchronous_writes`] is true and | ||
| /// * Either until the chunks have been flushed the agent |
There was a problem hiding this comment.
| /// * Either until the chunks have been flushed the agent | |
| /// * Either until the chunks have been flushed to the agent |
| /// If [`TraceBufferConfig::synchronous_writes`] is true and | ||
| /// * Either until the chunks have been flushed the agent | ||
| /// * Or if `synchronous_writes_timeout` is Some, until the timeout is reached. At which point the | ||
| /// flush might continue in the background |
There was a problem hiding this comment.
I don't understand this paragraph. It starts with If but it's not clear which part is the then part.
| /// flush might continue in the background | ||
| pub struct TraceBuffer<T> { | ||
| tx: Sender<T>, | ||
| /// Enables synchronous exports if Some |
There was a problem hiding this comment.
?
| /// Enables synchronous exports if Some | |
| /// Enables synchronous exports if true |
| let gen = state.batch.batch_gen; | ||
| if state.batch.span_count() > self.flush_trigger_number_of_spans || self.synchronous_write { | ||
| state.flush_needed = true; | ||
| self.waiter.notify_receiver(state); |
There was a problem hiding this comment.
Do you want to leave a comment here for a future TODO, so that it's not lost in limbo? (There's also a similar situation in trigger_flush below)
| export_operation: Box<dyn Export<T> + Send + Sync>, | ||
| agent_response_handler: ResponseHandler, | ||
| config: TraceBufferConfig, | ||
|
|
| // The MutexGuard must not be held across .await points | ||
| let leftover; | ||
| { | ||
| let mut state = self.waiter.state.lock().map_err(|_| MutexPoisonedError)?; |
There was a problem hiding this comment.
This line comes up a lot. Should we have a smaller helper similar to get_state for this?
| // Enable the notify future BEFORE acquiring the lock to avoid lost wakeups: | ||
| // any notify_waiters() call that fires between enable() and .await is captured. | ||
| let notified = self.waiter.receiver_notifier.notified(); | ||
| let mut notified = std::pin::pin!(notified); |
There was a problem hiding this comment.
Out of curiosity, is there any difference between tokio::pin!(notified) and let mut notifier = std::pin::pin!(notified)?
There was a problem hiding this comment.
The std::pin::pin! macro uses super_let, an unstable feature which allows defining variables with the lifetime of the upper block https://rust-lang.github.io/rust-project-goals/2026/redesigning-super-let.html
Which is why it allows this syntax when tokio::pin! doesn't.
Otherwise it's the same, create a variable to have a place for the pinned value, then shadow the binding with a mutable borrow and call Pin::new_unchecked
| // The MutexGuard must not be held across .await points | ||
| let leftover; | ||
| { | ||
| let mut state = self.waiter.state.lock().map_err(|_| MutexPoisonedError)?; |
There was a problem hiding this comment.
Note: while we do not hold a mutex guard across .await points, this will still block the executor waiting for the mutex, right? Maybe it's ok in this case, but just to make sure this is a conscious decision.
There was a problem hiding this comment.
The comment is not clear but the idea is that all of the critical section protected by this mutex are fairly quick, do not call to blocking code, or get suspended so even if it blocks it should be for a short time. (well under high contention it might take a bit of time)
Maybe it'd be better to use parking_lot, which Mutex guarantees fairness 🤔
| } | ||
| } | ||
|
|
||
| /// The number of spans that will be buffered before we decide to flush |
There was a problem hiding this comment.
should this be based on size of payload and not number of spans? since we have payload size limits to the agent, not number of span limits. not all spans are the same size
# Release proposal for libdd-data-pipeline and its dependencies This PR contains version bumps based on public API changes and commits since last release. ## libdd-dogstatsd-client **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-dogstatsd-client-v3.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^3.0.2 → ^4.1.0 ### Commits - fix(crypto): gate libdd-common TLS features in remaining internal crates + add CI guard (#1943) ## libdd-trace-obfuscation **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-obfuscation-v3.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^3.0.2 → ^4.1.0 - `libdd-trace-utils`: ^3.0.1 → ^4.0.0 ### Commits - feat!: integrate obfuscation to the stats exporter [APMSP-2764] (#1819) - feat!: added regex-lite feature (#1939) - chore: clippy (#1889) - fix(crypto): gate libdd-common TLS features in obfuscation and capabilities-impl (#1872) - feat(obfuscation)!: feature parity on span obfuscation [APMSP-2671] (#1788) - feat(obfuscation/sql): feature parity on sql obfuscation [APMSP-2667] (#1708) ## libdd-trace-stats **Next version:** `3.0.0` **Semver bump:** `major` **Tag:** `libdd-trace-stats-v3.0.0` ###⚠️ major bump forced due to: - `libdd-trace-utils`: ^3.0.1 → ^4.0.0 ### Commits - perf: pre-compute string messagepack encoding (#1948) - feat!: integrate obfuscation to the stats exporter [APMSP-2764] (#1819) - feat(capablities)!: sleep & spawn capabilities (#1873) - feat: use ip quantization when aggregating peer tags for trace stats (#1944) - fix(crypto): gate libdd-common TLS features in remaining internal crates + add CI guard (#1943) - feat(shared_runtime)!: allow worker to be stopped after fork (#1893) - feat(sidecar)!: Add stats computation via SHM (#1821) - feat(stats): propagate service source from span meta to client stats payload (#1803) - fix(stats): align with css spec (#1790) ## libdd-data-pipeline **Next version:** `4.0.0` **Semver bump:** `major` **Tag:** `libdd-data-pipeline-v4.0.0` ###⚠️ major bump forced due to: - `libdd-common`: ^3.0.2 → ^4.1.0 - `libdd-telemetry`: ^4.0.0 → ^5.0.0 - `libdd-trace-utils`: ^3.0.1 → ^4.0.0 ### Commits - fix(libdd-telemetry): restore previous Cargo.toml version (#1993) - fix(data-pipeline): remove default-features from of trace-obfuscation (#1981) - fix(trace_exporter: shared_runtime): unwrap_or being eager is not good (#1983) - perf: pre-compute string messagepack encoding (#1948) - feat!: integrate obfuscation to the stats exporter [APMSP-2764] (#1819) - feat(capablities)!: sleep & spawn capabilities (#1873) - fix(telemetry): avoid trigger loop in telemetry worker (#1950) - feat(telemetry)!: include dependencies and integrations in app-extended-heartbeat (#1962) - perf(trace-serializer): pre-allocate serialization buffer (#1949) - feat!: added regex-lite feature (#1939) - fix(crypto): gate libdd-common TLS features in remaining internal crates + add CI guard (#1943) - feat(telemetry): add session id support to trace export (#1822) - fix(path): missing bench path in data-pipeline (#1907) - feat(data-pipeline): port dd-trace-rs trace buffer implementation (#1826) - feat(info_fetcher): add timeout to info fetcher (#1890) - feat(shared_runtime)!: allow worker to be stopped after fork (#1893) - feat(sidecar)!: Add stats computation via SHM (#1821) - ci(libdd-shared-runtime): downgrade version so publish workflow succeeds (#1870) - feat(runtime)!: add shared runtime (#1602) - ci: compilation of libdd-data-pipeline to wasm32 (#1830) - feat(capabilities)!: trait architecture http (#1555) - feat(otel): add support for OTLP trace export (#1641) - fix(stats): align with css spec (#1790) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: iunanua <18325288+iunanua@users.noreply.github.com>
Motivation
As we implement native spans in ruby in python, we have opportuinities to replace more components of the span pipeline with native code.
One of them i the span buffer which we is before the trace exporter.
Changes
Add a TraceBuffer implementation.
The TraceBuffer is split in two part, a Sender which allows putting trace chunks in the queue and a background task, spawned on the shared runtime which periodically pulls data from the chunk and submit them using the TraceExporter.