Skip to content

Add binary operators for the BitStream class #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
WibblyGhost merged 4 commits into from
Aug 3, 2026
Merged

Add binary operators for the BitStream class #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b9eb28a
Add binary operators for the BitStream class
WibblyGhost Jul 12, 2026
dbbef96
Use collections Mapping for typehinting
WibblyGhost Aug 3, 2026
10009f8
Update README.md
WibblyGhost Aug 3, 2026
2da167c
Update .pre-commit-config.yaml
WibblyGhost Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 20 additions & 5 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,14 @@ repos:
- id: check-docstring-first

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.12
rev: v0.16.1
hooks:
# Run the linter.
- id: ruff-check
name: ruff-check
args: [--fix]

# Run the formatter.
- id: ruff-format
name: ruff-format

- repo: local
hooks:
Expand All @@ -56,12 +54,29 @@ repos:
language: system
entry: uv run mypy
files: .*(\.py|\.pyi)$
args: [features, ci, scripts]
args: [src]
pass_filenames: false

# - repo: local
# hooks:
# - id: pytest
# name: pytest
# stages:
# - pre-commit
# - pre-push
# language: system
# entry: uv run pytest
# files: (src|tests)/.*(\.py|\.pyi)$
# args:
# - tests
# - -q
# - --no-summary
# - --no-header
# pass_filenames: false

- repo: https://github.com/astral-sh/uv-pre-commit
# uv version.
rev: 0.11.8
rev: 0.12.1
hooks:
# Ensure the lockfile is up-to-date
- id: uv-lock
Expand Down
58 changes: 30 additions & 28 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Writing and reading to/from the stream modifies the underlying buffer.
```python
class BitStream:
"""Custom IO class which converts a bytestream into a bitstream with read and write methods."""
stream: StringIO
stream: bitarray
@property
def bin(self) -> str: ...
def __init__(self, buffer: bytes | str = b"", /) -> None:
Expand All @@ -67,7 +67,6 @@ The encoders/decoders handle `Codec`'s by pushing them ontop of a stack, which h
`Containers` are also used as a special dictionary holding *key, value*, they pass on their members to the encoders/decoders. They also contain a *parent* property under the attribute `Container._` which helps with indexing the parent container within the encoding/decoding process. `Container`'s also include special attribute access functionality, meaning that you can access the items of the Container with direct `container.item` analogy, upon failure to find an attribute it will search the underlying data store for that attribute too. The Container has a nice **pretty print** function built in.

```python

class Container[VT: Any = Any]:
"""
Wrapper for a dictionary-like object, we use this to add extra functionality to
Expand All @@ -89,6 +88,7 @@ All stacks have definitions for pushing, setting, clearing, popping and freezing
```python
class Stack[T: SupportsName](FrozenSlots):
items: list[T]

def pop(self, index: SupportsIndex = -1) -> T: ...
def set(self, index: SupportsIndex, value: T) -> None: ...
def empty(self) -> bool: ...
Expand All @@ -103,7 +103,9 @@ There's also a **pretty print** function built into this Stack to help print out
```python
class StackC(Stack["Codec | StackC"]):
"""Version of the stack which contains methods for holding Codec's."""

name: str

def pprint(self, *, depth: int = 1) -> str: ...
```

Expand Down Expand Up @@ -156,6 +158,7 @@ class Codec(CodecProtocol):
self.name: Name of this codec, can use "name" / Codec to name this codec
self.size: Size in bits of this codec.
"""

@property
def name(self) -> str: ...
@property
Expand All @@ -165,6 +168,7 @@ class Codec(CodecProtocol):
NOTE: This will not always work, or be accurate, please double check the
output size is what you expect.
"""

def __rtruediv__(self, other: Any) -> Self:
"""
Method which defines the behaviour of right side '/' operator.
Expand All @@ -176,6 +180,7 @@ class Codec(CodecProtocol):
>>> print(codec.name)
>>>> "name"
"""

def rename(self, name: str) -> None: ...
def sizeof(self, io: BitStream, context: Container, codecs: StackC) -> int: ...
```
Expand All @@ -185,8 +190,8 @@ class Codec(CodecProtocol):
> [!NOTE]
> This section is different to the Default Codec type defined later down.

Some Codec's can take a *default* argument which will take a Singleton object of `Pass` or `Error`, if this default is triggered then it will either ignore the failed conditional or error out of building/parsing.
These Codec's don't need to be initialised via `Pass()` or `Error()`, since they are **singletons** assigned to a variable.
Some `Codec`'s can take a *default* argument which will take a Singleton object of `Pass` or `Error`, if this default is triggered then it will either ignore the failed conditional or error out of building/parsing.
These Codec's don't need to be initialised via `Pass()` or `Error()`, since they are Singletons assigned to a variable.
Please see the *Pass* and *Error* Codec headers for more information around these Codecs

#### Pass
Expand All @@ -203,7 +208,7 @@ class Pass(Codec):

#### Error

Much like the `Pass` codec but instead when this Codec is parsed or built, will raise an exception.
Much like the Pass codec but instead when this Codec is parsed or built, will raise an exception.

```python
class Error(Codec):
Expand All @@ -213,12 +218,14 @@ class Error(Codec):
"""

@classmethod
def raise_error(cls, io: BitStream, context: Container, codecs: StackC, **kwargs: Any) -> NoReturn: ...
def raise_error(
cls, io: BitStream, context: Container, codecs: StackC, **kwargs: Any
) -> NoReturn: ...
```

#### NotImplementedCodec

Acts exactly the same as the `Error` Codec, but is useful for flagging future development for this Codec.
Acts exactly the same as the Error Codec, but is useful for flagging future development for this Codec.

```python
NotImplementedCodec = Error
Expand All @@ -228,11 +235,11 @@ NotImplementedCodec = Error

#### Struct

`Struct`'s are the main building block and wrapper of codecs, these are what we use to call the `parse()` and `build()` methods and contain an array of Codec's.
`Struct`'s are the main building block and wrapper of codecs, these are what we use to call the `parse()` and `build()` methods and contain an array of `Codec`'s.
These can be nested inside each other, and may be either embedded into the current structure or wrapped into a seperate container upon parsing and building.


Most `Struct`'s contain an `embedded` attribute which is enabled by default when it has no name assigned, but can also be manually assigned. This make all the defined Codec's inside the `Struct` embed to the parent `Codec`.
Most `Struct`'s contain an `embedded` attribute which is enabled by default when it has no name assigned, but can also be manually assigned. This make all the defined `Codec`'s inside the `Struct` embed to the parent `Codec`.

> [!NOTE]
> `embedded` toggling is still an experimental feature when toggled manually, and can cause unpredictable indexing of other `Codec`'s particularly the `Pointer` for example.
Expand All @@ -254,7 +261,7 @@ class Struct(Codec, StructProtocol):

#### Pointers

Sometimes you need advanced IO handling so that you can read IO out of sequence. The `Pointer` class helps forward the
Sometimes you need advanced IO handling so that you can read IO out of sequence. The Pointer class helps forward the
IO stream, parse the later stream, then reverse the stream and parse the original segment.

```python
Expand Down Expand Up @@ -342,7 +349,7 @@ class Switch[MKey: Any, MValue: Codec | Struct = Codec](Codec):
def __init__(
self,
funct: FunctType[MKey],
mapping: dict[MKey, MValue],
mapping: Mapping[MKey, MValue],
*,
default: DefaultType | MValue = ...,
embedded: bool = False,
Expand Down Expand Up @@ -400,7 +407,7 @@ class Bits(Codec):

#### Enum

The `Enum` Codec is a Bits type which contains a string to integer mapping for its parsed/built values.
The Enum Codec is a Bits type which contains a string to integer mapping for its parsed/built values.
By default, if a value cannot be mapped, it will raise an exception, but this can be changed to ignore missing
mappings via the `default=Pass` keyword argument.

Expand Down Expand Up @@ -429,7 +436,7 @@ class Enum(Bits):

#### Mapping

Works exactly the same as the `Enum` Codec except that it takes a dictionary as its initialization.
Works exactly the same as the Enum Codec except that it takes a dictionary as its initialization.

```python
class Mapping(Enum):
Expand Down Expand Up @@ -526,7 +533,7 @@ class Array(Codec):

#### Raw Bits

This codec just copies over the bitstream into the `Container` and vice versa.
This codec just copies over the bitstream into the Container and vice versa.
Useful if you want to include a payload but don't want to perform any calculations on the output values.

```python
Expand All @@ -545,10 +552,7 @@ class RawBits(Codec):

#### Computed

Special Codec type which can perform calculations and representations of values **without** modifying the IO stream upon parsing.

> [!note]
> This doesn't contain any `io_build` functionality and therefore cannot be built into a bitstream. Acts identically to the `Pass` Codec in that mannor.
Special Codec type which can perform calculations and representations of values **without** modifying the IO stream.

```python
class Computed[T: ValueType](Codec):
Expand Down Expand Up @@ -590,7 +594,7 @@ class Bitshift[T: Any = int](Codec):

#### Checksum

`Checksum` is a very special Codec type which will perform checksum calculations on your packet after building.
Checksum is a very special Codec type which will perform checksum calculations on your packet after building.
The fields to perform the checksum calculations must be listed and present before the checksum Codec.

```python
Expand Down Expand Up @@ -633,7 +637,7 @@ class Checksum(Bits):

#### Greedy Array

Subset of the `Array` Codec but instead of giving the Codec a hardcoded container count, this Codec will keep parsing until an end of stream (EOS).
Subset of the Array Codec but instead of giving the Codec a hardcoded container count, this Codec will keep parsing until an end of stream (EOS).
It can also take a *max_count* argument which will only parse the stream up to the specified count and no further.
Or the *max_count* argument can take a *lambda* expression.

Expand Down Expand Up @@ -663,7 +667,7 @@ class GreedyArray(Array):

#### Greedy Bits

Defines a greedy bits consumer which will keep consuming the IO stream until an end of stream (EOS).
Defines a Greedy bits consumer which will keep consuming the IO stream until an end of stream (EOS).
The output value will be a `BitStream` type. It works the same way as the `RawBits` Codec.

It can also take a *max_size* argument which will only parse the stream up to the specified length and no further.
Expand Down Expand Up @@ -703,7 +707,7 @@ class GreedyBits(Codec):

#### Blacklisted

Simple wrapper for the `Bits` Codec which will raise an exception if the parsed value is in the blacklisted range.
Simple wrapper for the Bits Codec which will raise an exception if the parsed value is in the blacklisted range.

```python
class Blacklisted(Bits):
Expand All @@ -719,7 +723,7 @@ class Blacklisted(Bits):

#### Whitelisted

Simple wrapper for the `Bits` Codec which will raise an exception if the parsed value is not in the whitelisted range.
Simple wrapper for the Bits Codec which will raise an exception if the parsed value is not in the whitelisted range.

```python
class Whitelisted(Bits):
Expand Down Expand Up @@ -760,7 +764,7 @@ class Adapter(Codec, AdapterProtocol):

#### IpAddress

This adapter converts the defined Codec into an `IpAddress` string and back into an integer field.
This adapter converts the defined Codec into an IpAddress string and back into an integer field.

```python
class IpAddress(Adapter):
Expand Down Expand Up @@ -809,7 +813,6 @@ class ExprAdapter(Adapter):

[Simple IPv4 Codecs](src/examples/ip_codec.py)
```python

IPV4_HEADER = Struct(
"version" / Const(Bits(4), const=4),
"header_length"
Expand Down Expand Up @@ -901,6 +904,5 @@ IP_PACKET = Struct(

Whilst building this package, here's a few decisions and thoughts I had:

- `StringIO`/`BytesIO` was my original plan to use but it didn't handle seeking IO very well, and copying one `IO` object into a new one everytime I needed to modify the stream was inefficient.
- Then I checked out [`bitstring`](https://pypi.org/project/bitstring/) which looked reasonable but I found the parse/build speed slow. It uses the `bitarray` library underneath, implementing wrapper functions.
- Then I checked out [`bitarray`](https://pypi.org/project/bitarray/) which used C DLL's to efficiently work on boolean arrays, this was quite effective and provided the low-level interfaces I required.
Originally I was going to use [bitstring](https://pypi.org/project/bitstring/) which looked reasonable but I found the parse/build speed slow.
Then I checked out [bitarray](https://pypi.org/project/bitarray/) which used C DLL's to efficiently work on boolean arrays, this was quite effective and I was using this package as our base for a while. However after a variety of testing their speeds and footprints, I actually discovered that storing the bitstream as a string was actually the fastest way of handling our streams. I even considered using the bitarray and BytesIO builtin packages.
8 changes: 4 additions & 4 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,10 +120,8 @@ select = [
]
fixable = ["ALL"]
ignore = [
"D203", # Incorrect black line before class
"D100", # Docstring in public module
"D102", # Docstring in public method
"D203", # Blank line before class
"D104", # Docstring in public package
"D105", # Docstring in magic method
"D212", # Multiline Docstring Start
"D205", # Line between summary and description
Expand All @@ -133,9 +131,11 @@ ignore = [
isort.split-on-trailing-comma = false

[tool.ruff.lint.per-file-ignores]
"**/__init__.py" = ["D104"]
"tests/*.py" = ["SLF001", "PT013"]
"typing.py" = ["D101", "D107"]
"typing.py" = ["D101", "D107", "D102", "D100"]
"exceptions.py" = ["D101", "D107"]
"test_*.py" = ["D100", "D101", "D102"]

[tool.ruff.format]
# skip-magic-trailing-comma = true
Expand Down
Loading
Loading