bug: concurrent commit and RPC Simulate call causes a data race in IAVL

[p0mvn]

Background

There was a fatal error observed on v7.1.0. On initial investigation, it is being caused by concurrently committing and calling the following RPC.

This happened at least 3 times over the past 3 days

Logs: concurrent_rw_panic.log

Acceptance Criteria

• investigate and synchronize and fix the problem
• create an e2e test to test the fix by running a chain and calling the Simulate RPC several times

[p0mvn]

SDK tag used on v7.1.0: v0.45.1-0.20220327212402-9f71a993afdc
IAVL tag: v0.17.3-osmo-v7

[p0mvn]

There was #137 that might be related. IAVL must be concurrent with this change present

[p0mvn]

More logs from different occurrence that are stemming from the same source:

• concurrent_rw_21_apr.log
• concurrent_rw_23_apr.log

The original report is from April 26

[p0mvn]

This node is used for wallets

[vorotech]

hi @p0mvn
Do you think the following is related?

If I'm "lucky" enough and hit the endpoint https://lcd-osmosis.blockapsis.com/cosmos/tx/v1beta1/txs?events=tx.height%3D3815464&pagination.offset=0 several times, I'm getting:

{
  "code": 8,
  "message": "grpc: received message larger than max (4589341 vs. 4194304)",
  "details": [
  ]
}

[vorotech]

Tbh, I think it might be unrelated issue, the gRPC client should have the grpcClient.MaxCallSendMsgSize() method called here (also missed in the original implementation)

[p0mvn]

Tbh, I think it might be unrelated issue, the gRPC client should have the grpcClient.MaxCallSendMsgSize() method called here (also missed in the original implementation)

@vorotech You're right. We have already updated this on the latest branch of the SDK fork. However, this has not been backported to v7.x branch yet.

We are planning a minor (v7.2.1) release soon and should backport osmosis-labs/cosmos-sdk#170 to it

CC: @czarcas7ic

[catShaark]

So I looked at the log, this is where the error showed up right ? Apr 26 18:43:17 ip-172-31-49-142 osmosisd_osmosis-1[333746]: fatal error: concurrent map iteration and map write

[p0mvn]

So I looked at the log, this is where the error showed up right ? Apr 26 18:43:17 ip-172-31-49-142 osmosisd_osmosis-1[333746]: fatal error: concurrent map iteration and map write

That's correct. It's happening because 2 goroutines are trying to iterate over this map and write to it concurrently.

One goroutine is accessing that IAVL map through Commit() in baseapp and another by calling this RPC

[alexanderbez]

Doesn't Tendermint place a lock on that ABCI flow though? Or is this lower level @p0mvn? What can be do in the SDK to synchronize these calls?

[p0mvn]

@alexanderbez essentially, this PR osmosis-labs/cosmos-sdk#137 removed the synchrony with the ABCI flow. The asynchrony is needed by integrators for handling the load. Otherwise, their queries are blocked at epoch / long commits

I know exactly where the problem is (IAVL - mutable_tree.go) and working on a fix as well as an e2e test to expose this in our suite. Should have an update in the next few days