[receiver/kafka_metrics_receiver] Provide a way to enable cluster metadata refresh on custom intervals #37896
Description
Component(s)
No response
Is your feature request related to a problem? Please describe.
Say there are multiple kafka brokers running, if a broker goes down, instead of refreshing the metadata immediately the sarama library uses the cached cluster metadata for it.
2025-02-13T10:05:41.990Z error scraperhelper@v0.119.1-0.20250210123122-44b3eeda354c/obs_metrics.go:61 Error scraping metrics {"otelcol.component.id": "kafkametrics", "otelcol.component.kind": "Receiver", "otelcol.signal": "metrics", "error": "write tcp 10.244.0.115:47710->10.244.0.120:9092: write: broken pipe"}
go.opentelemetry.io/collector/scraper/scraperhelper.wrapObsMetrics.func1
/go/pkg/mod/go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.1-0.20250210123122-44b3eeda354c/obs_metrics.go:61
go.opentelemetry.io/collector/scraper.ScrapeMetricsFunc.ScrapeMetrics
/go/pkg/mod/go.opentelemetry.io/collector/scraper@v0.119.1-0.20250210123122-44b3eeda354c/metrics.go:24
go.opentelemetry.io/collector/scraper/scraperhelper.scrapeMetrics
/go/pkg/mod/go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.1-0.20250210123122-44b3eeda354c/controller.go:256
go.opentelemetry.io/collector/scraper/scraperhelper.NewMetricsController.func1
/go/pkg/mod/go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.1-0.20250210123122-44b3eeda354c/controller.go:228
go.opentelemetry.io/collector/scraper/scraperhelper.(*controller[...]).startScraping.func1
/go/pkg/mod/go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.1-0.20250210123122-44b3eeda354c/controller.go:175
And also during leader election, there's similar problems
2025-02-13T10:05:43.990Z error scraperhelper@v0.119.1-0.20250210123122-44b3eeda354c/obs_metrics.go:61 Error scraping metrics {"kind": "receiver", "name": "kafkametrics", "data_type": "metrics", "error": "kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes", "scraper": "topics"}
Describe the solution you'd like
Similar problems are seen in issues like:
IBM/sarama#2173
IBM/sarama#2237
IBM/sarama#272
There could be other ways to solve this problem like, but I propose exposing sarama config to let the user configure the RefreshFrequency https://github.com/IBM/sarama/blob/main/config.go#L151
Describe alternatives you've considered
No response
Additional context
No response