Skip to content

Add methods for bloom filter compression#458

Merged
quantumagi merged 1 commit intostratisproject:masterfrom
quantumagi:bloomcompression
Mar 15, 2021
Merged

Add methods for bloom filter compression#458
quantumagi merged 1 commit intostratisproject:masterfrom
quantumagi:bloomcompression

Conversation

@quantumagi
Copy link
Contributor

@quantumagi quantumagi commented Mar 12, 2021

The purpose of this PR is to reduce the serialized size of the bloom filter used by the PosBlockHeader. Since the bloom filter is basically a bit map of 2048 bits contained in 256 bytes it is quite possible that many of those bytes may be 0. This compression scheme reduces the number of 0 bytes by storing the number of 0 bytes instead of the bytes themselves. As an example 256 0 bytes would be encoded as 1 255 0 - where 1 is the total compressed length - 1, followed by 255 zeros and finally one more zero. The general pattern is <compressed length - 1> [<number of zeros> <explicit byte>] x n where n > 0. The last explicit byte is added only if required. If the compressed length exceeds 255 then the bloom filter is not compressed when serialized, which is flagged by 255 in the first byte. A 0 in the first byte is not possible and is reserved for future use.

This is a backwards compatible change required by system contract and does not affect legacy bloom filters.

This PR is being split off from PR #450 to reduce the size of that PR.

@fassadlr
Copy link
Contributor

@quantumagi can you detail what the requirement here is for the compression?

@quantumagi
Copy link
Contributor Author

@fassadlr , Done.

@fassadlr
Copy link
Contributor

@quantumagi merge this to #450 👍

@quantumagi quantumagi merged commit f69bb99 into stratisproject:master Mar 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants