chain reaches consensus failure on binary rebuild and restart

[adamewozniak]

Summary of Bug

When running the beta version of the chain locally, rebuilding the binary in beta mode, stopping the chain, and then restarting, it reaches consensus failure via an error from the x/leverage module :

ERR CONSENSUS FAILURE!!! err="-0.000002092846270928 years: negative time elapsed since last interest time" module=consensus ...

Not sure if this is a major issue, but still worth jotting down

Version

Please provide the output of the following commands:

• $ umeed version
  price-feeder/v0.1.0-c66f8961
• $ go version
  go version go1.17.3 darwin/arm64
• $ uname -a
  Darwin Adams-MacBook-Pro-2.local 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T6000 arm64

Steps to Reproduce

Steps to reproduce the behavior:

1. Launch beta umee network

UMEE_ENABLE_BETA=true starport chain serve -c ./starport.ci.beta.yml -v --reset-once

2. Rebuild binary with beta support

UMEE_ENABLE_BETA=true make install

3. Stop & Restart umee network

UMEE_ENABLE_BETA=true starport chain serve -c ./starport.ci.beta.yml -v

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[alexanderbez]

Interesting. @toteki does anything stick out here?

[alexanderbez]

@adamewozniak when you rebuilt, did you change any code?

[toteki]

We were talking about it. and the only way for this to happen is for the LastInterestTime in state to be greater than ctx.BlockTime().Unix() - it comes down to the behavior of BlockTime and EndBlocker (during which AccrueAllInterest sets LastInterestTime to ctx.BlockTime().Unix() after accruing interest using the difference of the two.)

The error message posted, by the way, comes out to be a 3 minute time reversal, longer than a block time, and might be consistent with the time the chain was stopped before relaunch. Still can't figure out why BlockTime would reverse though.

[adamewozniak]

@adamewozniak when you rebuilt, did you change any code?

Nope, you should be able to repro this just using those console commands & no change

[RafilxTenfen]

@adamewozniak This still happen? I couldn't reproduce

[adamewozniak]

@RafilxTenfen I'm now able to repo it by running starport, making a change to force starport to rebuild, and then restarting starport. Let me know if you can repro that way, same issue :

CONSENSUS FAILURE!!! err="-0.000000697615423642 years: negative time elapsed since last interest time" 

cc : @toteki

[toteki]

I suspect this is just blockTime behaving weirdly when starport restarts, but we should see if we run into it during our chain upgrade tests. If that happens, it might be worth treating unexpected elapsed times differently.

[adamewozniak]

@toteki I think we should be worried about upgrading v2 -> v3, since the chain would be required to stop and restart similarly to this. Our upgrade tests should include v2.0.0 -> v2.0.1 (when we cut it), I think that would replicate this better - but agreed, it's probably just blocktime behaving weirdly.

Edit: Looks like it also only happens once, e.g., if you change the codeblock out to this :

	// calculate time elapsed since last interest accrual (measured in years for APR math)
	yearsElapsed := sdk.NewDec(currentTime - prevInterestTime).QuoInt64(types.SecondsPerYear)
	if yearsElapsed.IsNegative() {
		// Because this action is not caused by a message, logging and
		// events are here instead of msg_server.go
		k.Logger(ctx).Debug(
			"negative time elapsed",
			"block_height", fmt.Sprintf("%d", ctx.BlockHeight()),
			"unix_time", fmt.Sprintf("%d", currentTime),
			"yearsElapsed", yearsElapsed.String(),
		)
		return nil
	}

You get the "negative time elapsed" message, and then it continues as normal

[toteki]

We could do that - but negative time elapsed is a panic because if it happens for any amount of time (worst case: if block time were ever zero for some reason) then unjustified interest (worst case: 50 years worth) will accrue instantly on the next block with no error message.

I want to be sure that it's not a starport-only problem before considering removing the current safety. Has anyone run into the bug without starport during this week's gov upgrade testing?

[adamewozniak]

I don't believe this has been seen in a while. Please confirm / close, either @dreamcodez, @RafilxTenfen, or @toteki

[toteki]

Been a while - I suspect it's because our local testing has evolved beyond relaunching with starport (e.g. noob scripts and multinode now).

That would mean that blocktime moving backwards was just a starport edge case we don't have a problem.