Add copy_records_to_table for COPY FROM STDIN bulk-load#169
Open
Dev-iL wants to merge 1 commit into
Open
Conversation
Closes psqlpy-python#166. The existing binary_copy_to_table required callers to pre-encode PostgreSQL's binary COPY wire format, leaving no ergonomic bulk-load path comparable to asyncpg's copy_records_to_table or psycopg3's cursor.copy(...). The new method on Connection and Transaction accepts an iterable of records, introspects column types from the target table, and streams rows via tokio-postgres' BinaryCopyInWriter using the same PythonDTO conversions used by execute().
7794063 to
65b9d12
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
copy_records_to_tableonConnectionandTransaction, mirroringasyncpg.Connection.copy_records_to_table. The new method accepts an iterable of records, introspects column types from the target table, and streams rows overCOPY FROM STDIN (FORMAT binary)viatokio-postgres'BinaryCopyInWriter.Signature:
Internals:
SELECT <cols|*> FROM <qualified_table> WHERE falseis prepared (non-cached) to fetch the columntokio_postgres::types::Typelist — nopg_catalogquery needed.PythonDTOwith the existingfrom_python_typed, so the type coverage matchesexecute()(including JSON/JSONB, arrays, geometry, decimals, etc.).table_name,schema_name,columns) are routed through the existingquote_identhelper.Motivation and Context
Closes #166.
Prior to this change, the only COPY API was
binary_copy_to_table, which required callers to pre-encode PostgreSQL's binary COPY wire format (e.g. viapgpq/Arrow). There was no record-list API and no streaming-writer API — so no ergonomic bulk-load path comparable toasyncpg.copy_records_to_tableorpsycopg3'scursor.copy(...). Applications falling back toexecute_manypaid a large throughput penalty.This PR adds the asyncpg-style record-list variant (Option A from the issue). The psycopg3-style
async with conn.copy(...) as writerstreaming variant (Option B) is intentionally left as a follow-up — it's a separate, larger API surface (async context manager, partial-write semantics) and worth its own issue.How has this been tested?
cargo build— passes, no warnings.cargo clippy— passes, no warnings.python/tests/test_copy_records.py, run against PostgreSQL 14:test_copy_records_to_table_on_connection— round-trips mixed INT/TEXT/FLOAT8/TIMESTAMPTZ, including aNULL.test_copy_records_to_table_with_columns_subset— explicitcolumns=leaves untouched columns NULL.test_copy_records_to_table_in_transaction— same API onTransaction.test_copy_records_to_table_rejects_record_arity_mismatch— clean error when a record's field count doesn't match the resolved columns.test_copy_records_to_table_uses_schema_qualifier—schema_name=qualifies both introspection and the COPY statement.Test run:
Existing
test_binary_copy.pyis unchanged and thebinary_copy_to_tablecode path is untouched.Screenshots (if appropriate):
N/A — backend driver change, no UI.
Types of changes
Checklist:
Notes on the checklist:
python/psqlpy/_internal/__init__.pyistubs have been added on bothTransactionandConnectionso users get full type/IDE support for the new method.docs/components/connection.mdanddocs/components/transaction.mddo not currently documentbinary_copy_to_tableeither, so no existing section needed updating. Happy to add aCOPY FROM STDINdoc page in this PR or a follow-up if maintainers prefer.