[BUG] PayloadSenderV2 throws Exception during execution of the filter on transaction

[kbalys]

APM Agent version

1.11.1

Environment

APM server 7.14

.NET Framework/Core name and version
NET Core 3.1.413

Application Target Framework(s)
netcoreapp3.1

Describe the bug

PayloadSenderV2 throws exception when Asp.net core HealthChecks is checking a dependency which fails.

To Reproduce

I noticed a Warning in log when Health Check in my application is testing a failed dependency which is down and health check is reporting an error. I've tried to reproduce this issue in smaller sample but so far with no luck.

Expected behavior

No Exception

Actual behavior

	[11:41:45 WRN] {PayloadSenderV2} Exception during execution of the filter on transaction
System.NullReferenceException: Object reference not set to an instance of an object.
   at Elastic.Apm.Filters.ErrorContextSanitizerFilter.Filter(IError error)
   at Elastic.Apm.Report.PayloadSenderV2.TryExecuteFilter[T](List`1 filters, T item)

Error value which is received doesn't have Context filled. I think this may be the case for health check requests that they are not normal Http Request but they are executed from healthcheck-ui automatically every x seconds, but this is only my guess.

{Error{Id: 71062bdb2bbf2fcf2f5ce5307c818e7c, TraceId: fce5e93fe1dcc643b1b69e1172abd2ae, ParentId: 9a8d3663dd710241, TransactionId: 9a8d3663dd710241}}
    ConfigSnapshot: {Elastic.Apm.Config.ConfigSnapshotFromReader}
    Context: null
    Culprit: "Failed outgoing HTTP request"
    Exception: {CapturedException{Type: System.Net.Http.HttpRequestException, Message: No connection could be made because the target machine actively refused it., Handled: False, Code: null}}
    Id: "71062bdb2bbf2fcf2f5ce5307c818e7c"
    Log: null
    ParentId: "9a8d3663dd710241"
    Timestamp: 1633085851482142
    TraceId: "fce5e93fe1dcc643b1b69e1172abd2ae"
    Transaction: {TransactionData{IsSampled: False, Type: request}}
    TransactionId: "9a8d3663dd710241"

https://github.com/elastic/apm-agent-dotnet/blob/1.11/src/Elastic.Apm/Filters/ErrorContextSanitizerFilter.cs
internal class ErrorContextSanitizerFilter
	{
		public IError Filter(IError error)
		{
			if (error is Error realError)
			{
                                // I suspect this realError.Context throws exception as it was null
				if (realError.Context.Request?.Headers != null && realError.ConfigSnapshot != null)
				{
					foreach (var key in realError.Context?.Request?.Headers?.Keys.ToList())
					{
						if (WildcardMatcher.IsAnyMatch(realError.ConfigSnapshot.SanitizeFieldNames, key))
							realError.Context.Request.Headers[key] = Consts.Redacted;
					}
				}
			}

			return error;
		}
	}

Could you give some guidance how I could track this error?

[russcam]

Thanks for reporting @kbalys. It looks like the line

if (realError.Context.Request?.Headers != null && realError.ConfigSnapshot != null)

might be the culprit, where realError.Context is null, which can be the case when there's no transaction or the transaction is not sampled, the latter of which looks to be the case based on the error output. We'll look to fix this.