Parallel signing

[jku]

This uses threads to sign multiple artifacts in parallel. The issue that is being solved here is that rekor v2 only responds to the entry request after the entry has been included in the log and the inclusion proof is available. When entry requests are done in parallel, they all get responses in the next "inclusion batch" (instead of one response per batch), speeding up multi-artifact signing by a lot. A practical example of the use case could be cpython releases where they sign ~10 artifacts.

There are multiple changes:

• Refactor signing so the "threaded method" is as small as possible -- the threadpool code is from ramon
• Avoid non-thread-safe print() in threaded method
• Avoid requests Session use from multiple threads. It's not 100% certain this is required as the thread safety of session is unclear -- it used to be unsafe but the situation has improved -- so I tried to be conservative and use single-use Sessions since doing so does not seem like a major issue for the sigstore client. With Refactor: replace requests with urllib3 #1040 we can likely get rid of this.
• refactor CLI output so that it remains in logical order even with threads
• Add test that signs multiple artifacts

Signer.sign_artifact() and Signer.sign_dsse() are now expected to be thread safe.

[jku]

The included test only signs a few artifacts at a time but I have run this on more pathological cases:

touch a b c d e f g h i j k l m n o p q r s t u v x y z
time sigstore sign --overwrite a b c d e f g h i j k l m n o p q r s t u v x y z

Runtime on current main branch

• rekor1: 20 seconds
• rekor2: 63 seconds

Runtime on parallel-sign branch

• rekor1: 7 seconds (reduction of 65%)
• rekor2: 12 seconds (reduction of 80%)

The time differences here are not that big but I think that's mostly because staging rekor 2 is currently configured with small and quick batches: With longer batching times the rekor2 difference will only grow

[jku]

The time differences here are not that big but I think that's mostly because staging rekor 2 is currently configured with small and quick batches: With longer batching times the rekor2 difference will only grow

Colleen confirmed this for me: tessera checkpoint interval is currently configured at 2 seconds in staging, this might get bumped up to 10 seconds

The effect of this with the 25 artifact rekor2 test (with the current 10 thread limit) is :

• with current staging tessera configuration (2 second checkpoint interval)
  • sigstore-python main waits for 25 checkpoints: ~49 secs
  • parallel-signing branch waits for 3 checkpoints: ~5 secs
• with possible staging tessera configuration (10 second checkpoint interval)
  • sigstore-python main waits for 25 checkpoints: ~ 240 secs
  • sigstore-python main waits for 3 checkpoints: ~ 25 secs

The timings for this branch depend on the max number of threads: current 10 threads means 25 artifacts splits to 3 checkpoints.

The takeaways to me are:

• we can/should increase the max number of threads as far as is reasonable
• if there are more than max_threads artifacts to sign, we could warn the user that the signing might take a while -- basically signing max_threads artifacts takes ~0.5 checkpoint intervals on average but even one more artifact takes ~1.5 checkpoint intervals on average

[woodruffw]

Thanks @jku! I left two small comments/nitpicks 🙂

[woodruffw]

LGTM, great work @jku!

One thing I was thinking about it whether the increased latency with Rekor v2 means that it might make sense to show a progress bar while signing (i.e. when we can detect an interactive user terminal). Do you have thoughts on that? Not a blocker here either way.

[jku]

One thing I was thinking about it whether the increased latency with Rekor v2 means that it might make sense to show a progress bar while signing (i.e. when we can detect an interactive user terminal). Do you have thoughts on that? Not a blocker here either way.

I'd like it... but I don't think it's possible to implement one that is useful: for the typical case (signing less than max_workers artifacts) the runtime can be anything from X seconds to X+Y seconds where X is the "current baseline" and Y is the checkpoint interval that rekor-tiles uses... So there's problems:

• currently X and Y are similar (a couple of seconds) but Y may be increased to up to 10 secs so it could dominate the progress bar
• we don't know for sure what Y is (since it's a knob that rekor-tiles admins can tweak)
• even if we did, our runtime depends on when during the interval we happen to make our request -- if we are lucky we get a response immediately