TraceContinuationStrategy=restart_external without TraceParent header causes NullReferenceException

[Vaahik]

APM Agent version

elastic_apm_profiler_1.17.0-beta.1.zip

Environment

Operating system and version:
CentOS Stream 8

.NET Framework/Core name and version
Host (useful for support):
Version: 5.0.17
Commit: 6a98414363

.NET runtimes installed:
Microsoft.AspNetCore.App 5.0.17 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 5.0.17 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Application Target Framework(s) (netcoreapp5.0):

Describe the bug

+-> Exception: System.NullReferenceException: Object reference not set to an instance of an object.
at Elastic.Apm.Model.Transaction..ctor(IApmLogger logger, String name, String type, Sampler sampler, DistributedTracingData distributedTracingData, IPayloadSender sender, IConfiguration configuration, ICurrentExecutionSegmentsContainer currentExecutionSegmentsContainer, IApmServerInfo apmServerInfo, BreakdownMetricsProvider breakdownMetricsProvider, Boolean ignoreActivity, Nullable1 timestamp, String id, String traceId, IEnumerable1 links) in C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1797\apm-agent-dotnet\src\Elastic.Apm\Model\Transaction.cs:line 319
at Elastic.Apm.Api.Tracer.StartTransactionInternal(String name, String type, DistributedTracingData distributedTracingData, Boolean ignoreActivity, Nullable1 timestamp, String id, String traceId, IEnumerable1 links) in C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1797\apm-agent-dotnet\src\Elastic.Apm\Api\Tracer.cs:line 87
at Elastic.Apm.Api.Tracer.StartTransaction(String name, String type, DistributedTracingData distributedTracingData, Boolean ignoreActivity, IEnumerable`1 links) in C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1797\apm-agent-dotnet\src\Elastic.Apm\Api\Tracer.cs:line 66
at Elastic.Apm.AspNetCore.WebRequestTransactionCreator.StartTransactionAsync(HttpContext context, IApmLogger logger, ITracer tracer, IConfiguration configuration) in C:\Users\jenkins\workspace\net_apm-agent-dotnet-mbp_PR-1797\apm-agent-dotnet\src\Elastic.Apm.AspNetCore\WebRequestTransactionCreator.cs:line 86

elastic_apm_profiler_1.16.1-beta.1.zip - working done

[gregkalapos]

Thanks for reporting @Vaahik!

Haven't tested on Linux yet, but a default net5.0 MVC app works without any issue on Windows for me. We also have multiple tests in CI running both on Linux and Windows - so far all green.

It seems Elastic.Apm.Model.Transaction..ctor throws the exception - so looks like an issue in the managed code (and not directly in the profiler). The line numbers don't seem to be correct, but I guess you have release build anyway...

Could you please help us by answering a few questions:

• Do you reference any Elastic APM packages via NuGet? (The thinking here is that maybe somehow an older version is loaded) Or you add the profiler via Environment variables and don't have any Elastic APM package directly in the app?
• Did you change any config? If so, could you please list them? (Ignore the ServerUrl and security related configs...)
• Do you have only a single app where it fails or do you have multiple ones where all apps fail? Have you seen it working with another app?
• I guess this always happens once an HTTP request hits your application - can you confirm this?
• Do you have a crash dump which you could upload?
• You updated the whole profiler, right? So downloaded the zip file, removed the old folder from the old zip file and unzipped the whole thing as is and started using it, right?

Thanks!

[gregkalapos]

Hmm - by looking at it a bit more: is it possible that you set TraceContinuationStrategy to restart_external? This is a new setting.

[Vaahik]

Hmm - by looking at it a bit more: is it possible that you set TraceContinuationStrategy to restart_external? This is a new setting.

Yes, i use ELASTIC_APM_TRACE_CONTINUATION_STRATEGY=restart_external
In old version (elastic_apm_profiler_1.16.1-beta) i use ELASTIC_APM_TRACE_CONTEXT_IGNORE_SAMPLED_FALSE=true

Upd:
I change ELASTIC_APM_TRACE_CONTINUATION_STRATEGY=restart_external to ELASTIC_APM_TRACE_CONTEXT_IGNORE_SAMPLED_FALSE=true in new version elastic_apm_profiler_1.17.0, working normaly.

[gregkalapos]

Thank you for the response @Vaahik, super helpful!

Ok, so this verified my theory and #1803 (already merged) fixes it.

Summary:

• NullreferenceException happens in combination with ELASTIC_APM_TRACE_CONTINUATION_STRATEGY=restart_external and incoming HTTP requests without traceparent and/or tracestate header.
• Other values in the ELASTIC_APM_TRACE_CONTINUATION_STRATEG aren't effected.
• This happens in all setups and it isn't directly related to the profiler (I rename the issue accordingly)

Workaround which everyone can use in 1.17.0:

• Leave ELASTIC_APM_TRACE_CONTINUATION_STRATEGY on default and still use the old ELASTIC_APM_TRACE_CONTEXT_IGNORE_SAMPLED_FALSE config - it'll will still work - that code is still active and we have tests covering that.

cc @z1c0

[Vaahik]

Thank you for the response @Vaahik, super helpful!

Ok, so this verified my theory and #1803 (already merged) fixes it.

Summary:

• NullreferenceException happens in combination with ELASTIC_APM_TRACE_CONTINUATION_STRATEGY=restart_external and incoming HTTP requests without traceparent header
• Other values in the ELASTIC_APM_TRACE_CONTINUATION_STRATEG aren't effected.
• This happens in all setups and it isn't directly related to the profiler (I rename the issue accordingly)

Workaround which everyone can use in 1.17.0:

• Leave ELASTIC_APM_TRACE_CONTINUATION_STRATEGY on default and still use the old ELASTIC_APM_TRACE_CONTEXT_IGNORE_SAMPLED_FALSE config - it'll will still work - that code is still active and we have tests covering that.

cc @z1c0

"traceparent" header is present , caller service (net core 5) add it from default configuration to outgoing http request.
"tracestate" is not present from header.
I think both situations need to be handled.
incoming HTTP requests without traceparent and tracestate header.

[gregkalapos]

"traceparent" header is present , caller service (net core 5) add it from default configuration to outgoing http request.
"tracestate" is not present from header.
I think both situations need to be handled.
incoming HTTP requests without traceparent and tracestate header.

Thanks! The original PR fixed the nullref for both cases, but after looking at it again, I realize the behaviour still wasn't correct when tracestate was not present (the trace wasn't restarted). I opened #1804 - that handles that case as well.

Thanks again for the input @Vaahik .

[gregkalapos]

This is fixed in #1803 and in #1804. The fix will be included in the next release.