Add configuration setting to control whether a stack trace should be captured for each span

[SergeyKleyman]

We need to decide

• Would it make sense to add an additional parameter to StartSpan/CaptureSpan APIs to allow user to control it on more fine grained level?
• Should it be a simple boolean configuration setting resulting in the same behavior for all spans (unless overridden by a new parameter to public API) or a more sophisticated setting - for example list of span types for which stack trace should be captured? Maybe some aspect of a span other than its type to decide whether to capture stack trace - for example Java agent uses span duration. In case of the duration I assume the stack trace is for the end of the span not the start.
• What should be the default value?

[SergeyKleyman]

We can have a configuration setting (let's say something like span_stack_trace_capture) with the following values:

• disabled (agent should not automatically capture stack trace for any span). Stack trace can still be captured via public API.
• all_spans_on_start (agent should capture stack trace for each span's start)
• long_spans_on_end (agent should capture stack trace for long spans' end where long can be controlled by another setting - span_stack_trace_min_duration)

[SergeyKleyman]

Indirectly related to #225

[gregkalapos]

Ran a perf. test to measure the perf. impact of capturing stacktraces.

Test case: Creating 100 transactions, each with 10 spans. - Transactions and spans don't do anything, work is done only by the agent - so only agent work is measured here.

With capturing stacktrace:

|                      Method |     Mean |    Error |   StdDev | Gen 0/1k Op | Gen 1/1k Op | Gen 2/1k Op | Allocated Memory/Op |
|---------------------------- |---------:|---------:|---------:|------------:|------------:|------------:|--------------------:|
| Simple100Transaction10Spans | 84.18 ms | 1.665 ms | 3.402 ms |   5666.6667 |   2166.6667 |           - |            34.49 MB |

// * Legends *
  Mean                : Arithmetic mean of all measurements
  Error               : Half of 99.9% confidence interval
  StdDev              : Standard deviation of all measurements
  Gen 0/1k Op         : GC Generation 0 collects per 1k Operations
  Gen 1/1k Op         : GC Generation 1 collects per 1k Operations
  Gen 2/1k Op         : GC Generation 2 collects per 1k Operations
  Allocated Memory/Op : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
  1 ms                : 1 Millisecond (0.001 sec)

With not capturing stacktraces:

|                      Method |     Mean |    Error |   StdDev | Gen 0/1k Op | Gen 1/1k Op | Gen 2/1k Op | Allocated Memory/Op |
|---------------------------- |---------:|---------:|---------:|------------:|------------:|------------:|--------------------:|
| Simple100Transaction10Spans | 24.24 ms | 1.117 ms | 2.864 ms |   1937.5000 |    562.5000 |           - |            11.63 MB |

[SergeyKleyman]

@felixbarny mentioned that Java agent resolves stack traces in deferred fashion (not on app thread thus avoid blocking the app) - maybe it is worth measuring how much it will help if we resolve stack traces on payload thread instead.

[gregkalapos]

@felixbarny mentioned that Java agent resolves stack traces in deferred fashion (not on app thread thus avoid blocking the app) - maybe it is worth measuring how much it will help if we resolve stack traces on payload thread instead.

Definitely something we can think about, but I think it won't be trivial.

Here is how we do it now:

StackTrace = StacktraceHelper.GenerateApmStackTrace(new StackTrace(true).GetFrames(), _logger, $"Span `{Name}'");

In GenerateApmStackTrace (which is our method) we basically just create our own data structure.

The main part of the work is done by new StackTrace(true).GetFrames() and I suspect that this API already resolves the stack frames.

I did 1 test run with only executing new StackTrace(true) - by only doing this we'd have the option to defer the rest of the work, so this would be the prerequisite to implement the suggestion and we'd do the rest of the work async later.

So I changed

StackTrace = StacktraceHelper.GenerateApmStackTrace(new StackTrace(true).GetFrames(), _logger, $"Span `{Name}'");

To

new StackTrace(true);

And here is the result:

|                      Method |     Mean |     Error |    StdDev | Gen 0/1k Op | Gen 1/1k Op | Gen 2/1k Op | Allocated Memory/Op |
|---------------------------- |---------:|----------:|----------:|------------:|------------:|------------:|--------------------:|
| Simple100Transaction10Spans | 50.93 ms | 0.9664 ms | 0.8567 ms |   5300.0000 |   1000.0000 |           - |            31.82 MB |

As you can see, this already caused most of the allocation, and the runtime overhead also increased drastically. So I think if we want to implement the suggestion we need to look for another API, because the one we use (System.Diagnostics.StackTrace.StackTrace) seems to resolve everything immediately.

[gregkalapos]

Yeah, the API we use resolves everything synchronously

https://github.com/dotnet/coreclr/blob/9773db1e7b1acb3ec75c9cc0e36bd62dcbacd6d5/src/System.Private.CoreLib/src/System/Diagnostics/StackTrace.CoreCLR.cs#L58

https://github.com/dotnet/coreclr/blob/9773db1e7b1acb3ec75c9cc0e36bd62dcbacd6d5/src/System.Private.CoreLib/src/System/Diagnostics/StackFrame.CoreCLR.cs#L29

[SergeyKleyman]

What about new StackTrace(false) instead of new StackTrace(true) ?
According to https://ayende.com/blog/3879/reducing-the-cost-of-getting-a-stack-trace new StackTrace(false) might be quite faster

[SergeyKleyman]

As you can see, this already caused most of the allocation

Of course, allocations number will not be reduced by deferring stack trace resolution (even if new StackTrace(true) didn't make most of allocations anyway). The goal of deferring is to minimize immediate latency incurred by application.