perf(kaos): stream readLines to avoid loading whole file into memory - #962
Closed
7Sageer wants to merge 1 commit into
Closed
perf(kaos): stream readLines to avoid loading whole file into memory#9627Sageer wants to merge 1 commit into
7Sageer wants to merge 1 commit into
Conversation
Read files in 64KiB chunks and split on the LF byte so peak memory stays bounded by the longest line instead of the whole file. readTail/readForward keep the same behavior and Total lines reporting.
🦋 Changeset detectedLatest commit: f6d584e The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
itxaiohanglover
approved these changes
Jun 22, 2026
itxaiohanglover
left a comment
There was a problem hiding this comment.
Good performance fix — streaming file reads in 64KB chunks instead of loading the entire file into memory is the right approach for large files. The pending buffer handling for partial lines is correct. Changeset included.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue
No related issue. The problem is explained below.
Problem
Reading a large text file with the
Readtool (including tail mode via a negativeline_offset) loads the entire file into memory.LocalKaos.readLinesusedreadFile+split('\n'), which holds the whole file as aBuffer, a decoded string, and a split array at the same time. On a 100 MB file this peaks at ~330 MB of resident memory even when only the last 100 lines are needed, and grows linearly with file size.What changed
LocalKaos.readLinesnow reads the file in 64 KiB chunks and splits on the LF byte, decoding one line at a time. Peak memory is bounded by the chunk size plus the longest line rather than the whole file (~57 MB for the same 100 MB file, most of which is the Node.js baseline).Behavior is unchanged:
readTail/readForwardkeep their ring buffer, theTotal lines in filereport, and theMAX_LINES/MAX_BYTEScaps. CRLF, lone CR, empty files, and thestrict/replace/ignoredecode modes are preserved. Added tests for line-ending fidelity, multi-chunk files, multibyte characters across the chunk boundary, and the three decode modes.Checklist
gen-changesetsskill, or this PR needs no changeset.gen-docsskill, or this PR needs no doc update.