[BUG] Possible memory leak on AspNetCoreDiagnosticListener

[nugroho-exp]

APM Agent version

1.14.1

Environment

Operating system and version:
Linux (dockerized)

.NET Framework/Core name and version (e.g. .NET 4.6.1, NET Core 3.1.100) :
.NET 6

Application Target Framework(s) (e.g. net461, netcoreapp3.1):
net6.0

SpanCompression is enabled.

Describe the bug

Our application memory keeps growing and ended up with OutOfMemoryException error.

Please see the attached dotMemory screenshot. It looks like AspNetCoreDiagnosticListener consume 695MB in _processingRequests and does not release it.

Created dashboard using TSVB from APM metrics

Could you please check.

Thank you.
Nugroho

[gregkalapos]

Hi @nugroho-exp

thanks for reporting.

Do you have agent logs for this? Based on the memdump it looks like transactions never end - there is a theoretical possibility for this in the code, but so far I did not thought this could happen. If you could attach agent logs on trace level, we'd know more about these transactions.

[gregkalapos]

Thanks for the logs @nugroho-exp

Unfortunately I wasn't really able to read out much of it. There were useful things in it, but it does not 100% verify my theory.

Nevertheless I opened #1685 which I believe will fix this. That PR is merged and it'll be included in the next release. I'll keep this issue open; if possible could you please verify that the memory issue goes away, once the next release is out?

[nugroho-exp]

Thanks @gregkalapos, will try it once it is released.

[nugroho-exp]

Hi @gregkalapos, When will the fix be released? Thank you.

[gregkalapos]

Hi @nugroho-exp, a release with the fix is out: https://github.com/elastic/apm-agent-dotnet/releases/tag/v1.15.0

Could you maybe give it a try and let me know if the issue is really fixed? Thanks.

[gregkalapos]

Closing now, feel free to report here if something is not ok. The issue should be fixed.