[HCP Telemetry] Periodic Refresh for Dynamic Telemetry Configuration

[Achooo]

Description

New feature for HCP Telemetry that allows us to dynamically re-configure metrics export to HCP.

Background
Currently, we need to restart the Consul server to bring in any updates to filters, a new endpoint or new labels from the Consul Telemetry Gateway in HCP.

Ideally, we are able to modify these dynamically, which requires logic to continuously fetch new configuration and update it as necessary.

Implementation
This PR enables this functionality, with changes to :

1. New object: TelemetryConfigProvider:

• Periodically fetches telemetry configuration from HCP, using a time.Ticker.
• Holds configuration and exposes it via methods : GetEndpoint(), GetFilters() and GetLabels()
• Updates configuration, ONLY if it has changed. Comparisons are done using a hash function on the structs holding the config.
• Handles concurrency access to read the config and modify the config.

2. OTELSink: uses a new ConfigProvider interface to access filters and labels.

• Since labels are now dynamic, we can't add them to the OTEL resource (immutable) so a new function is added to compute OTEL labels dynamically.
• Handle dynamic filters

3. OTELExporter: uses a new EndpointProvider interface to access the endpoint.

** Cleanup / Refactor ***

1. The client package was cleaned to separate out the TelemetryConfig object to parse and convert objects.
2. A logger was added to the client to log bad filter information

This refactor was necessary since now the TelemetryConfigProvider also fetches telemetry config. This avoid duplication of logic for the parsing of the endpoint, labels ,etc.

Testing & Reproduction steps

• End to end tests:
• Unit tests:

Links

PR Checklist

• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern