[prometheus] instrument metrics with createdtimestamps #3846
Description
container_ and machine_ counters can exhibit the uninitialized counter problem whereby the rate/increase contribution of the first scraped sample is dropped. CreatedTimestamp is an additional field supported by Counters/Summaries/Histograms to propagate additional information to infer a zero datapoint for a newly initialized counter.
There's active development in various prometheus-conformant tsdb projects (i.e prometheus/proposals#60) to support the ingest of CT values.
The change to support CT for cadvisor via prom-proto should be trivial to implement.
Generally speaking CT value should be set as close to source as possible (i.e the container). cadvisor itself can be seen as a collector rather than source of metrics. Thus the natural source of CT values for container_* counters would be the creation_time field in the container spec.