Migrating to the OpenChoreo Observer backend for fetching traces.

[AnoshanJ]

The agent management platforms needs to migrate to the OpenChoreo observer for fetching traces, and OpenChoreo Observer needs support its tracing requirements in a generic way. Currently we are using and maintaining a separate service called traces-observer-service which queries opensearch. This carries a maintantence overhead and causes a redundancy with the OpenChoreo observer as it supports tracing too. To improve our design we can fully migrate to the OpenChoreo observer, once it supports our requirements.

[AnoshanJ]

We propose a change to the observer from the current single api design that fetches traces to three apis, to address different requirements of the agent management platform:

1. Api for fetching trace list :

This should contain:

• Root span name
• Start time
• Duration
• No. of Spans - This value will be accurately taken by running a nested opensearch query that queries the count of documents with the traceid
• Error or Not (If feasible)

2. Api for fetching spans of a specific trace

This should contain:

• Span attributes

This cannot be paginated (Since all spans are required to build the trace tree)

When we need to query opensearch we need to specify the number of documents(span) to retrieve, but we don’t know how many spans are in a trace. So we query 100 documents by default and programmatically group. This can cause issues if there are more spans in a trace than that ( UI cannot render the full trace tree). We can adjust this value as a configuration since opensearch supports a maximum of 10000. This can be provided as a helm configuration.

3. Another API to fetch the attributes of a specific span

By specifying the spanid we can get the attributes of that span and this will allow usecases for filtering and displaying customized views.

[a5anka]

@AnoshanJ Let's try to keep the top document updated for ideas as well. Are there any related GitHub issue tracking this task in OpenChoreo side?

[AnoshanJ]

openchoreo/openchoreo#2016