Pipeline execute_many and wrap in a transaction#171
Merged
Conversation
Replaces the per-row sequential await loop in execute_many with concurrent futures driven via FuturesOrdered, brackets the batch in BEGIN/COMMIT when not already in a transaction, and uses a SAVEPOINT when invoked from Transaction.execute_many so a failed batch can never poison the caller's surrounding transaction. The order-of-magnitude speedup comes from collapsing N implicit auto-commits into one WAL fsync; pipelining alone is insufficient. Locally measured against the forked tokio-postgres: 1000-row INSERT batch ~1326 ms sequential -> ~32 ms pipelined-in-transaction. End-to-end through pyo3: ~128 ms for 1000 rows (~7,800 rows/s), versus the ~3 batches/sec reported in psqlpy-python#167. Fixes psqlpy-python#167 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Connection.execute_many/Transaction.execute_manyno longer issue one round-trip per row. The implementation insrc/connection/impls.rsnow:Bind/Executemessages on the same connection viaFuturesOrdered(tokio-postgres dispatches them back-to-back instead of stalling on each reply).When invoked from
Connection.execute_many, the wrap isBEGIN/COMMIT(withROLLBACKon failure). When invoked fromTransaction.execute_many, the wrap is aSAVEPOINT psqlpy_execute_many(RELEASEon success;ROLLBACK TO+RELEASEon failure) so a failed batch can never poison the caller's surrounding transaction. Internal docs on the method body explain the rationale, the asyncpg comparison, and the deliberate divergence on savepoint behaviour.Motivation and Context
Fixes #167. Reported behaviour:
execute_manywas ~93× slower thanasyncpg.executemanyfor the same workload because it issued one full round-trip per row and never amortized fsync cost. The bottleneck was visible insrc/connection/impls.rsas a sequentialfor ... awaitoverself.query(&stmt, ¶ms).The change also introduces a behavioural shift worth flagging in release notes: a mid-batch failure now rolls back earlier rows in the batch (previously each row auto-committed independently). This matches
asyncpg/psycopgexecutemanysemantics and the way bulk APIs are generally expected to behave.How has this been tested?
Environment: PostgreSQL 14 in Docker on localhost (sub-millisecond RTT), CPython 3.13, Linux.
tokio-postgres(1000-row INSERT batch): ~1326 ms sequential → ~32 ms pipelined-in-transaction (41× speedup, ~31 k rows/s). Pipelining without the transaction wrap only got ~1024 ms — confirms the fsync floor is the real bottleneck.execute_many#167: ~128 ms / ~7,800 rows/s, versus the ~3 batches/sec the issue reports.tx.execute_manythat fails on a PK violation no longer aborts the surrounding transaction; subsequent statements in the sametxcontinue to succeed.python/tests/test_connection.py,python/tests/test_transaction.py): 37 passed, no failures attributable to this change.cargo build --release,cargo clippy --release, and the project's pre-commit chain (rustfmt, clippy, cargo check, ruff, mypy) all clean.Screenshots (if appropriate):
N/A.
Types of changes
Checklist: