[receiver/vcenter]: Optimize vCenter Receiver to use concurrency

[schmikei]

Component(s)

receiver/vcenter

Is your feature request related to a problem? Please describe.

I'm lead to believe that larger environments have a harder time using the receiver due to the large number of requests that need to be made.

For larger environments the amount of requests can show how the current synchronous code has some room for improvement. I think once we scaled up it was pretty evident that we were bottlenecking on requests, particularly for VMs.

Describe the solution you'd like

From my recollection I don't think that the MetricsBuilder maintains some state that prevented easy parallelization, however we could parallelize/batch the requests to get better performance.

The tooling mitmproxy really helped us identify the issue as the massive number of requests made individually were stacking up to be quite a time sink.

Describe alternatives you've considered

No response

Additional context

For an environment of 2000 VMs it was taking sometimes minutes to complete a collection, based off of early experiments, hoping to cut it down to a collection interval of seconds.

[github-actions]

Pinging code owners for receiver/vcenter: @djaglowski @schmikei. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[schmikei]

Me and @StefanKurek plan to be taking this effort on, feel free to assign us to it

[atoulme]

maybe this spike can help: #30624

[StefanKurek]

@schmikei I think this can be closed now with the recent performance enhancements. I'll let you decide.