Skip to content

optimize memory for DSSE v0.0.1 processing#2766

Merged
Hayden-IO merged 1 commit intosigstore:mainfrom
Hayden-IO:memory
Mar 4, 2026
Merged

optimize memory for DSSE v0.0.1 processing#2766
Hayden-IO merged 1 commit intosigstore:mainfrom
Hayden-IO:memory

Conversation

@Hayden-IO
Copy link
Contributor

@Hayden-IO Hayden-IO commented Mar 4, 2026

Summary

This change reduces the memory footprint of DSSE v0.0.1 entries by:

  • Extracting index keys (subjects and SLSA materials) during initial unmarshaling
  • Clearing the large base64-encoded envelope and decoded payload early
  • Avoiding redundant payload decoding and JSON unmarshaling in IndexKeys

These optimizations prevent out-of-memory errors when processing large attestations.

Release Note

Documentation

This change reduces the memory footprint of DSSE v0.0.1 entries by:
- Extracting index keys (subjects and SLSA materials) during initial unmarshaling
- Clearing the large base64-encoded envelope and decoded payload early
- Avoiding redundant payload decoding and JSON unmarshaling in IndexKeys

These optimizations prevent out-of-memory errors when processing large
attestations.

Signed-off-by: Hayden <8418760+Hayden-IO@users.noreply.github.com>
@Hayden-IO Hayden-IO requested a review from a team as a code owner March 4, 2026 00:39
@codecov
Copy link

codecov bot commented Mar 4, 2026

Codecov Report

❌ Patch coverage is 95.23810% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 26.17%. Comparing base (488eb97) to head (42984af).
⚠️ Report is 644 commits behind head on main.

Files with missing lines Patch % Lines
pkg/types/dsse/v0.0.1/entry.go 95.23% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #2766       +/-   ##
===========================================
- Coverage   66.46%   26.17%   -40.29%     
===========================================
  Files          92      191       +99     
  Lines        9258    20122    +10864     
===========================================
- Hits         6153     5267      -886     
- Misses       2359    14027    +11668     
- Partials      746      828       +82     
Flag Coverage Δ
e2etests 49.58% <80.95%> (+2.02%) ⬆️
unittests 16.71% <95.23%> (-30.97%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Member

@bobcallaway bobcallaway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i attempted to do this via streaming, io.MultiReader, etc - and this kept coming back as the optimal approach under benchmarking

@Hayden-IO Hayden-IO merged commit 2d46808 into sigstore:main Mar 4, 2026
16 checks passed
@Hayden-IO Hayden-IO deleted the memory branch March 4, 2026 23:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants