Skip to content

Container metadata labels lost when Docker version contains multiple suffixes #3830

@mihua336

Description

@mihua336

What happened:
When using Docker versions with complex suffixes like v20.10.12-v1.0.2, cAdvisor fails to parse container metadata labels (e.g., container, pod, namespace, image), resulting in empty label values in metrics.

Example metric:

container_cpu_usage_seconds_total{
  container="", 
  pod="", 
  namespace="", 
  image=""
} 0.184540494

What expected to happen :
Container labels should be correctly populated from Docker metadata, allowing proper association of metrics with Kubernetes pods and containers.

How to reproduce it :

  1. Install Docker with version containing multiple suffixes (e.g., v 20.10.12-v 1.0.2)
  2. Start kubelet and scrape /metrics/cadvisor
  3. Observe that container-level metrics have empty values for Kubernetes-related labels

Root cause:
cAdvisor uses a strict regex (\d+)\.(\d+)\.(\d+) to parse Docker version strings. This regex fails to match versions that contain multiple hyphens (e.g., v 20.10.12-v 1.0.2), causing the Docker-specific handler to fail registration and fall back to the raw handler, which does not enrich metrics with Kubernetes metadata.

cadvisor/container/docker/docker.go at master · google/cadvisor

func ParseVersion(versionString string, regex *regexp.Regexp, length int) ([]int, error) {
	matches := regex.FindAllStringSubmatch(versionString, -1)
	if len(matches) != 1 {
		return nil, fmt.Errorf("version string \"%v\" doesn't match expected regular expression: \"%v\"", versionString, regex.String())
	}
	versionStringArray := matches[0][1:]
	versionArray := make([]int, length)
	for index, versionStr := range versionStringArray {
		version, err := strconv.Atoi(versionStr)
		if err != nil {
			return nil, fmt.Errorf("error while parsing \"%v\" in \"%v\"", versionStr, versionString)
		}
		versionArray[index] = version
	}
	return versionArray, nil
}

Proposed fix :
To improve compatibility with custom or extended Docker version strings, I propose the following fix:
Even if the Docker version string contains non-standard suffixes, cAdvisor should still attempt to extract and use the first valid semantic version part (e.g., X.Y.Z) to register the Docker handler. This ensures that container metadata enrichment (e.g., pod, namespace, container, image) continues to function normally, without falling back to the raw handler.

  1. Improve version parsing logic
  • Use the regular expression (\d+\.\d+\.\d+) to extract the first matching semantic version part (e.g., 20.10.12).
  • If no valid version is found, return an error.
  • If a match is found, continue to register and use the Docker handler normally.
  1. Add warning logging
  • When a version string contains non-standard suffixes is detected, log a warning message in English for better compatibility with internationalized logging systems.
  • Example log:
[WARNING] Detected non-standard Docker version "v20.10.12-v1.0.2", using base version "20.10.12" for handler registration

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions