adr-038 state listener could observe uncommitted writes

[yihuang]

Summary of Bug

adr-038 state listener get feed with uncommitted writes.

adr-038 state listener is expect to only listens to the writes that get committed into chain state at the end of block, but it currently listens to kv store interface, which could includes writes that are not committed eventually, for example write calls happens in a grpc query.

We should only listen to the state writes on the consensus state machine.

Version

all the versions that support adr-038 state listening

Steps to Reproduce

• Setup state listener
• Simulate a cache write in grpc query like this:

      cacheCtx, write := ctx.CacheContext()
      store := cacheCtx.KVStore(k.storeKey)
      store.Set([]byte("debug set"), []byte("value"))
      write()

• Observe that the writes happen in grpc query are observed by state listener, although the writes will be discarded eventually.

[yihuang]

Even in tx delivery, if we write to a nested cache context, commit the underlying cache context, then discard that cache, the write is also propagated to the listener I think.
And if nested cache context all committed, the writes will be observed multiple times.

[yihuang]

This is more tricky than I thought, the most inner cache only commits in abci commit event, so we actually need to listen to the second layer of the cache, and this layer only, which is assuming all the writes happens on this layer, we need to structure the code like this:

BeginBlocker:
  deliverState.ms = app.cms.CacheMultiStore()

  cacheMS = deliverState.ms.CacheMultiStore()
  ... execute begin blocker ...
  cacheMS.Write()
DeliverTx:
  cacheMS = deliverState.ms.CacheMultiStore()
  ... execute ante handler ...
  cacheMS.Write()

  cacheMS = deliverState.ms.CacheMultiStore()
  ... execute msgs ...
  cacheMS.Write()
EndBlocker:
  cacheMS = deliverState.ms.CacheMultiStore()
  ... execute end blocker ...
  cacheMS.Write()
Commit:
  deliverState.ms.Write()
  app.cms.Commit()

Alternatively, we only listen to the uncached rootmulti, in that case, we only observe the writes at commit event, and can't associate the writes with tx, which is ok for our use case, but different from the design purpose of adr-038.

[yihuang]

I find the current api is very tricky to use, for example, set a listener on rootmulti.Store means you get notification when user do store.GetKVStore().Set(...), but set a listener on cachemulti.Store, you don't get notification when store.GetKVStore().Set(...), but get notification when store.Write().
And when you call AddListeners on cachemulti.Store, it don't have effect because the listenkv.Store wiring is done at construction only.

[alexanderbez]

I honestly don't have much context on the ADR 038 API -- but my intuition tells me there should be and only one place where we set listeners, and that should probably be on the root MS only. Even then, only emit writes on commit/write. WDYT?

[yihuang]

I honestly don't have much context on the ADR 038 API -- but my intuition tells me there should be and only one place where we set listeners, and that should probably be on the root MS only. Even then, only emit writes on commit/write. WDYT?

that works for my use case perfectly, I only want states versioned by block numbers.
But the original design seems to be able to associate the writes to abci events, you can tell a certain change set is emitted by begin blocker or delivery of the first tx, that's not possible if we observe the root ms which is only get written to at commit event. That may be useful for some use case.
For example, tx0 set a -> 1, tx1 set a -> 2, observe most inner layer only sees a -> 2, while observe higher level can see both.