inclusion proof failure: wrong proof size 11, want 12

[laurentsimon]

As part of our SLSA native GitHub verifier https://github.com/slsa-framework/slsa-verifier, we intermittently see the following error in our e2e tests:

https://github.com/slsa-framework/slsa-verifier/blob/8c4373c533dfa2d9ec74ec042a12811a75c205de/pkg/rekor.go#L176-L179

validating log entry: verifying inclusion proof: wrong proof size 11, want 12

We have seen this error happen every once in a while. Are we doing something wrong in the code? How can we troubleshoot the problem?

If nothing is wrong in our code, this is something that will need to be fixed before GA... but it's hard to reproduce.

Any thoughts?

/cc @asraa @ianlewis

[asraa]

@bobcallaway @priyawadhwa
anything for some more Rekor help. It's very difficult to reproduce it since the inclusion proof changes over time.

[bobcallaway]

i'm suspect of the tree size calculation due to sharding. if this is being done against the staging instance (where we have sharded), it's possible we're returning an incorrect treesize which causes the merkle inclusion proof to think there should be a different number of hashes than are actually there returned in the response.

there's also everything we would need to verifying the inclusion proof in logEntry returned in https://github.com/slsa-framework/slsa-github-generator/blob/bd5d98277f41bbe5ef9f7a4fd7e0031c9b8aef68/signing/sigstore/rekor.go#L68, so there's no need to re-query it from the log in lines 73-78.

[priyawadhwa]

seems like it's being run against prod:

	defaultRekorAddr = "https://rekor.sigstore.dev"

[laurentsimon]

Good point. I thought we'd updated to use staging. My bad. Good catch!

[asraa]

Investigating this now. Actually, the workflow run it failed in is coming from https://github.com/slsa-framework/slsa-github-generator/blob/4a8e7ad5fd41988c38beb9cf3ff1d1dd04ae4ce8/signing/sigstore/rekor.go#L81
(it is happening during provenance generation and signing)

the cosign.VerifyInclusionProof, not from code in the slsa-verifier.

[asraa]

I've been spamming the log (no one says not to? :P)

and i got one reproducer. I should have logged the root size etc, but this is what I got for the inclusion proof for a certain entry.

ab6673ff0fb2591e2a645f1de6fe4bac5aecc3e301bebd0c72413148a685c175, d25ef569c5311766fe83011aa7d3619387af6701c81501ffbb91be4678b65406, 3afedbc670b585f544fbd07c344381ffe9ade711b10d1394628e48464a5ddbf4, a80ecef81f1a4592acc765cbcbaf094d5077137c4a186271b0d6fa234ed463a2, b6c0647b284ecd6e287029f5216322cf5f775800bd304ce1778baac88f1a0794, 3b7139c62c7ef1b5f1f0df7504a55f724212f336bfd7a5ca2857c22ad44f7658, e13754ea0196020d7246ec2508becc0aff62658bb278f9f67ebe5980510b7233, 553c1d557a02e7f3e7e78ca3bd299c3c2e3e1aa7bfd3482b10f6fb30c841aa77, f06c801b1fe5cbe5c83f295e9602e171a4f9456d9fa389a0922c68549ea78d22, be550466660c3004f7fa1e06d8195a4d5ffdd4a9bd01167d562528db3cc04422, 74f9a2f24c76d1ced3a175cdf0b9a79b1d3ba855f085d02c5e60a05cc0c8b970, 860efc785c66773fe35db221860cfb482dfe0a6bfe602aaade78149084c9ab86, 39da4af7f0fbe095e6bc78c146ad59c70881d3fcb22e90c7675dee5e6736bc76, 806e7153f439cdb4558d7ac89defb0e17aa9742e888cb660774b94b5399c3322, efb36cfc54705d8cd921a621a9389ffa03956b15d68bfabadac2b4853852079b2022/08/03 16:50:18 
validating log entry 3103867: verifying inclusion proof: wrong proof size 15, want 16

[laurentsimon]

Here's another from 5mn ago (pre-submits):

https://github.com/slsa-framework/slsa-verifier/runs/7662172712?check_suite_focus=true

"could not find a matching valid signature entry: got unexpected errors calculated root:\n[189 142 83 90 221 226 152 150 71 82 58 68 231 213 81 93 123 224 236 130 197 205 216 39 134 143 157 180 7 254 67 184]\n does not match expected root:\n[220 125 230 194 30"...,

[asraa]

got one more reproducer. here's more info.

log index: 3111199
root hash: 9fefcec8d664a5a4e20131ef23cfd0e48006e1ea4266c607c127967cae4eab42
leaf hash/uuid: 8238c7f97d0e9ade9499f91f80e11c96d188a73a19fb6bb26b4953d5572fcfac

inclusion proof

 ae7356dd7ac80f165f316606b23984a54200cc319d4d8eec639317bf0afeb66e
 ab889617c36e78653debb0fabc4111bd66bd392420b63f1da286e2799fcc0cee
 9501e59210f8b611c5351fbe9c2e7de2b5597248be4f4d753c005aa077620957
 0a8959fcdf152a22c2ee151d1fdde428a704ede149b8008eb5978b1c2458eb5d
 5463d1fcee4690a9711d80c5611092c23588d2cc1435e95a0dff3a13cda6853d
 e06474518cecdb71ee9c619bf1f2546d7f5ff0c866ed4ddafb415389b98d6cda
 6def364a0cce5f8a3db4f212fc197da4b859a535acdba9edeb43db608f703811
 d00111c8beb3afa536bc9c51760325c5c78c2a3efa1a7e13e2e3c8374f2d1818
 c0be2bbfdeffa203cd54b7d18bbaab3dc2dcdc97d3133febc410b4c3c5503e7c
 730bcf75857d37531e56edcdde3741ef9c814dfbe53528ab5180fa0860819148
 be550466660c3004f7fa1e06d8195a4d5ffdd4a9bd01167d562528db3cc04422
 74f9a2f24c76d1ced3a175cdf0b9a79b1d3ba855f085d02c5e60a05cc0c8b970
 860efc785c66773fe35db221860cfb482dfe0a6bfe602aaade78149084c9ab86
 39da4af7f0fbe095e6bc78c146ad59c70881d3fcb22e90c7675dee5e6736bc76
 806e7153f439cdb4558d7ac89defb0e17aa9742e888cb660774b94b5399c3322
 efb36cfc54705d8cd921a621a9389ffa03956b15d68bfabadac2b4853852079b

error:

verifying inclusion proof: calculated root:
[187 6 124 179 86 16 100 140 94 164 184 152 53 20 133 70 226 221 202 66 48 90 164 164 227 216 148 70 183 55 95 102]
 does not match expected root:
[159 239 206 200 214 100 165 164 226 1 49 239 35 207 208 228 128 6 225 234 66 102 198 7 193 39 150 124 174 78 171 66]

have not been able to get a reproducer on a local rekor

[asraa]

another one, with the error on proof size

log index 3112036
root hash 69c1cb658a4a7f01e2a3677fde09067a26a537e81af539070a2fee96f131e051
leaf hash c720f72a3086298291bc00f60b94098f0aa4c97f4ca205e7c3a8033c0f7da5a1
inclusion proof

 9770922f93e570b842eeb7f62943d0265c07f17621ec2aba85511b28b41dae42
 ff4bddbbf71c4cc5f88671762280fa971c0322637abb8b561c66b08b2442db38
 c8798fc04f8f3c0ad7bad40da786a588e80320804dd7175ac5ad9d8cfc3ad535
 2ab3fa24c94130948d918a9c380db9d406163feb298ac458a49b4a53047c495c
 d00111c8beb3afa536bc9c51760325c5c78c2a3efa1a7e13e2e3c8374f2d1818
 c0be2bbfdeffa203cd54b7d18bbaab3dc2dcdc97d3133febc410b4c3c5503e7c
 730bcf75857d37531e56edcdde3741ef9c814dfbe53528ab5180fa0860819148
 be550466660c3004f7fa1e06d8195a4d5ffdd4a9bd01167d562528db3cc04422
 74f9a2f24c76d1ced3a175cdf0b9a79b1d3ba855f085d02c5e60a05cc0c8b970
 860efc785c66773fe35db221860cfb482dfe0a6bfe602aaade78149084c9ab86
 39da4af7f0fbe095e6bc78c146ad59c70881d3fcb22e90c7675dee5e6736bc76
 806e7153f439cdb4558d7ac89defb0e17aa9742e888cb660774b94b5399c3322
 efb36cfc54705d8cd921a621a9389ffa03956b15d68bfabadac2b4853852079b

error

validating log entry 3112036: verifying inclusion proof: wrong proof size 13, want 14

[asraa]

Fix is merged, this will be resolved with the next deployment!