Do not allocate when flushing#279
Merged
craigfe merged 2 commits intomirage:masterfrom Jun 5, 2021
pascutto:io-allocations
Merged
Do not allocate when flushing#279craigfe merged 2 commits intomirage:masterfrom pascutto:io-allocations
craigfe merged 2 commits intomirage:masterfrom
pascutto:io-allocations
Conversation
Member
|
This needs to cleaning up - @pascutto will do it one day :-) |
Member
|
Rebased. |
samoht
reviewed
May 19, 2021
icristescu
approved these changes
May 20, 2021
Contributor
icristescu
left a comment
icristescu
left a comment
There was a problem hiding this comment.
Looks good, thanks! can you add some memtrace graphs to track the effect of this PR?
This value wasn't provided in the standard library until 4.09.
Member
|
Thanks! Merging. This is a very significant performance improvement for write-intensive loads. Running benchmarks with: ; dune exec bench/bench.exe -- --nb-entries 10_000_000 --filter '^replace_random$' --log-size 200_000For this benchmark, this PR reduces total allocations from 35G to 24G, and makes memory usage more stable. Before:
After:
|
craigfe
added a commit
to craigfe/opam-repository
that referenced
this pull request
Jun 16, 2021
CHANGES: ## Fixed - Fixed a crash-consistency bug due to a potential flush of an incomplete entry to disk. Entries are now flushed as complete strings. (mirage/index#301) - Fixed a performance issue for `Index.sync` when there is a blocking merge in progress: the `log_async` file was not cached properly and fully reloaded from disk every time. (mirage/index#310) - Added fsync after `Index.clear` to signal more quickly to read-only instances than something has changed in the file (mirage/index#308) - Attempt to recover from `log_async` invariant violations during an explicit sync operation, rather than failing immediately. (mirage/index#329) ## Changed - Release overly defensive warnings occuring when pre-fetching the disk. (mirage/index#322) - Specialise `IO.v` to create read-only or read-write instances. (mirage/index#291) - Optimised the in-memory representation of index handles and intermediate buffers, resulting in a significant reduction in memory use. (mirage/index#273, mirage/index#279) - Benches are now executed 3 times and a new option `nb-exec` has been added (mirage/index#292) - `clear` removes the files on disks and opens new ones containing only the header. (mirage/index#288) - `Index.Make` now requires an implementation of a monotonic time source. (mirage/index#321) - The `Index.Make` functor now takes a single `Platform` argument containing all system dependencies (i.e. `IO`, `Clock`, `Semaphore` and `Thread`). The `Platform` module holds the necessary types for these modules. (mirage/index#321, mirage/index#330) ## Added - Added benchmarks that replay a trace of index operations. (mirage/index#300) - Log reporter for the benches
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.