Avoid sequence mismatch when using relayer wallet to sign ibc tx msgs

[colin-axner]

Summary

An indeterminstic failure can be seen here

  testsuite.go:431: 
        	Error Trace:	/home/runner/work/ibc-go/ibc-go/e2e/testsuite/testsuite.go:431
        	            				/home/runner/work/ibc-go/ibc-go/e2e/tests/interchain_accounts/incentivized_test.go:292
        	Error:      	Should not be: 0
        	Test:       	TestIncentivizedInterchainAccountsTestSuite/TestMsgSendTx_FailedBankSend_Incentivized
        	Messages:   	code: 32
        	            	codespace: sdk
        	            	data: ""
        	            	events: []
        	            	gas_used: "0"
        	            	gas_wanted: "0"
        	            	height: "0"
        	            	info: ""
        	            	logs: []
        	            	raw_log: 'account sequence mismatch, expected 9, got 8: incorrect account sequence'
        	            	timestamp: ""
        	            	tx: null
        	            	txhash: B915E76EC510455F9A27ECA78521CFC0E97F6FE3059BEC1CF4A2DC5D96C0F528

Problem Definition

We use the relayer wallet to sign the MsgRegisterCounterpartyPayee which makes sense, but this causes a sequence mismatch since the relayer wallet may be submitting a tx at the same time.

Proposal

Not sure since the MsgRegisterCounterpartyPayee needs to be signed by the relayer wallet. Maybe we can stop the relayer/restart while sending this tx?

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged/assigned

[colin-axner]

I think we can actually look for a substring match of account sequence mismatch and then retry the tx for MsgRegisterCounterpartyPayee specifically?

[chatton]

I think for now we can specifically retry on the mismatch error, but it might be useful to have a mechanism to retry on one of a list of errors if more pop up.