Updating the hash for a transaction is very slow for certain statements with larger Ion inputs when using QLDB python driver. In some cases over 60% of client-side time on a transaction is spent in this method (_update_hash) for a request that has a total client-side latency > 500 ms. In comparison, hashing the same data in the standard Python hashlib takes <1 millisecond.
https://github.com/awslabs/amazon-qldb-driver-python/blob/v3.2.0/pyqldb/transaction/transaction.py#L116
An example:
In my environment, computing the Ion hash of the following Ion document takes ~2.5 ms using the Python Ion hash library (which _update_hash uses) while computing the native hash is < 0.01 ms. Computing the hash of a list of 100 of these entries in a single Ion object scales linearly and takes ~200ms using Python Ion hash and <1 ms with the native library. Hence if a document containing a list of 100 entries similar to below is inserted into QLDB, the driver will spend 200ms hashing the data.
$ion_1_0 { id:"0001", type:"donut", name:"Cake", ppu:0.55, batters:{ batter:[ { id:"1001", type:"Regular" }, { id:"1002", type:"Chocolate" }, { id:"1003", type:"Blueberry" }, { id:"1004", type:"Devil's Food" } ] }, topping:[ { id:"5001", type:"None" }, { id:"5002", type:"Glazed" }, { id:"5005", type:"Sugar" }, { id:"5007", type:"Powdered Sugar" }, { id:"5006", type:"Chocolate with Sprinkles" }, { id:"5003", type:"Chocolate" }, { id:"5004", type:"Maple" } ] }
Updating the hash for a transaction is very slow for certain statements with larger Ion inputs when using QLDB python driver. In some cases over 60% of client-side time on a transaction is spent in this method (
_update_hash) for a request that has a total client-side latency > 500 ms. In comparison, hashing the same data in the standard Python hashlib takes <1 millisecond.https://github.com/awslabs/amazon-qldb-driver-python/blob/v3.2.0/pyqldb/transaction/transaction.py#L116
An example:
In my environment, computing the Ion hash of the following Ion document takes ~2.5 ms using the Python Ion hash library (which
_update_hashuses) while computing the native hash is < 0.01 ms. Computing the hash of a list of 100 of these entries in a single Ion object scales linearly and takes ~200ms using Python Ion hash and <1 ms with the native library. Hence if a document containing a list of 100 entries similar to below is inserted into QLDB, the driver will spend 200ms hashing the data.$ion_1_0 { id:"0001", type:"donut", name:"Cake", ppu:0.55, batters:{ batter:[ { id:"1001", type:"Regular" }, { id:"1002", type:"Chocolate" }, { id:"1003", type:"Blueberry" }, { id:"1004", type:"Devil's Food" } ] }, topping:[ { id:"5001", type:"None" }, { id:"5002", type:"Glazed" }, { id:"5005", type:"Sugar" }, { id:"5007", type:"Powdered Sugar" }, { id:"5006", type:"Chocolate with Sprinkles" }, { id:"5003", type:"Chocolate" }, { id:"5004", type:"Maple" } ] }