Skip to content

feat: Zero-Memory Streaming Architecture for High-Performance Generation#14

Merged
Sauvikn98 merged 2 commits into
devfrom
feature/streaming-architecture
Apr 16, 2026
Merged

feat: Zero-Memory Streaming Architecture for High-Performance Generation#14
Sauvikn98 merged 2 commits into
devfrom
feature/streaming-architecture

Conversation

@Sauvikn98
Copy link
Copy Markdown
Contributor

@Sauvikn98 Sauvikn98 commented Apr 16, 2026

Description: This PR implements transforms the drawline-core generation engine from a memory-bound sequential process into a high-performance streaming pipeline.

Key Changes:

  • Zero-Memory Pipeline: Refactored BaseAdapter and TestDataGeneratorService to use AsyncGenerators. Data is now generated and piped to the database in batches, maintaining a constant memory footprint (< 100MB) even for millions of records.
  • Deterministic Sharding Foundation: Updated the generation loop to support rangeStart, enabling perfect reproducibility across future multi-core worker shards.
  • Batch Optimization: Introduced writeBatchStream to allow adapters to handle massive data flushing efficiently.
  • NullAdapter & Benchmarking: Added a NullAdapter for stress-testing the generation logic without storage overhead. Verified 4,300+ RPS on a single thread.
  • Stability Fixes: Resolved persistent Faker localization errors by optimizing the FieldInferenceEngine initialization.

Verification Results:

100k records generated in 23s.
Total memory increase: ~47MB.
All existing relationship resolution logic remains functional within the stream.

@Sauvikn98 Sauvikn98 merged commit 1fc2708 into dev Apr 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant