CI builds are breaking after we stopped immediately building rc.0

[puerco]

Immediately after cutting the March patch releases, the ci builds started failing:

The error as shown in the build log shows its related to the version string:

level=info msg="Using build version: v1.25.8-1+3a14fe1af239a0"
level=fatal msg="failed to run: find latest version: build version v1.25.8-1+3a14fe1af239a0 is not valid for release"

Upon further investigation, the builds are failing because the regular expressions that check the version string don't catch the gitCommt string without the rc.0:

👉 This is matched: v1.25.8-rc.0.31+36b707f892c523
👉 This is not matched: v1.25.8-1+3a14fe1af239a0

We only started seeing this as these are the first releases we cut after we stopped building the rc.0 immediately after tagging the repo.

More details on this slack thread in #release-ci-signal.

Thanks to @BenTheElder for reporting.

/cc @saschagrunert @kubernetes/release-engineering

[BenTheElder]

We could change the regex, but I think it's not expected that version string format would change.
We have other code that parses and reacts to version strings, so for now without digging too deep into that I recommend that we add an rc.0 tag (but just the tag, no release).

[BenTheElder]

/kind bug
/sig release

[saschagrunert]

We could change the regex, but I think it's not expected that version string format would change.

It's not the format which changes, it's just a case which we have not considered yet. I think we should fix the regex, because it looks like that it matches now incorrectly group 4 of the versionReleaseRE:

release/pkg/release/release.go

Line 61 in cbd865f

versionReleaseRE = `v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-[a-zA-Z0-9]+)*\.*(0|[1-9][0-9]*)?`

The last matching group (group 4) is wrongly matched in the regex, because it should be part of the versionBuildRE:

release/pkg/release/release.go

Line 62 in cbd865f

versionBuildRE = `([0-9]{1,})\+([0-9a-f]{5,40})`

The assembling of the regular expressions make them fail, because in case of v1.25.8-1+3a14fe1af239a0 it's now invalid:

release/pkg/release/release.go

Line 244 in cbd865f

return regexp.MatchString("("+versionReleaseRE+`(\.`+versionBuildRE+")"+versionDirtyRE+"?)", build)

[xmudrii]

I'm +1 for changing the regex, it seems wrong.

I see two options here:

• We reintroduce the rc.0 tag, but without building artifacts
  • This is IMO not ideal because we almost always have artifacts for pushed tags. Not having artifacts might break someone or some use cases and then we have an issue again
  • I expect that in the future we'll have questions why we do it this way instead of completely dropping the tag and fixing the issue
• We fix the regex to match the new case
  • This feels like a proper and more sustainable fix to me, but I'm not sure if this is going to break some consumers/users. In general, I think it should be safe, but we can never know for sure

[saschagrunert]

I agree having a tag without artifacts seems not correct.

[liggitt]

is this also blocking presubmit e2e runs on release branches? seeing

W0320 12:43:01.804] 2023/03/20 12:43:01 main.go:328: Something went wrong: failed to acquire k8s binaries: U=file:///go/src/k8s.io/kubernetes/_output/gcs-stage R=v1.26.3-3+df96302de9bd79 get-kube.sh failed: error during /workspace/get-kube.sh: exit status 1

in presubmits as well

[BenTheElder]

I'm not awake enough to vet version formats but please be careful with that, we have other packages (including one exported for external use) that know how to parse and compare / order Kubernetes versions and some tools take action based on this

[saschagrunert]

@xmudrii are you planning to work on this or should I take a stab on the regex?