refactor: Update median tracking for historacle pricing by rbajollari · Pull Request #1632 · umee-network/umee

rbajollari · 2022-12-01T15:40:48Z

Description

closes: #1621

Things that still need to be done after this PR:

Add new median stat keeper methods as grpc queries
Build out oracle endblocker integration test to test historacle functionality

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed all author checklist items have been addressed
reviewed state machine logic
reviewed API design and naming
reviewed documentation is accurate
reviewed tests and test coverage
manually tested (if applicable)

codecov-commenter · 2022-12-01T15:44:10Z

Codecov Report

Merging #1632 (8e41f41) into main (514380a) will decrease coverage by 0.04%.
The diff coverage is 77.32%.

@@            Coverage Diff             @@
##             main    #1632      +/-   ##
==========================================
- Coverage   54.22%   54.18%   -0.05%     
==========================================
  Files          72       73       +1     
  Lines        7454     7547      +93     
==========================================
+ Hits         4042     4089      +47     
- Misses       3100     3143      +43     
- Partials      312      315       +3

Impacted Files	Coverage Δ
x/oracle/abci.go	`6.79% <0.00%> (-40.78%)`	⬇️
x/oracle/genesis.go	`4.06% <0.00%> (-0.18%)`	⬇️
x/oracle/keeper/migrations.go	`25.00% <0.00%> (ø)`
x/oracle/keeper/params.go	`80.39% <64.70%> (+27.56%)`	⬆️
x/oracle/keeper/historic_price.go	`70.23% <74.74%> (-11.59%)`	⬇️
util/decmath/decmath.go	`98.11% <98.11%> (ø)`
util/bytes.go	`100.00% <100.00%> (ø)`
x/oracle/keeper/genesis.go	`87.23% <100.00%> (+1.86%)`	⬆️
x/oracle/keeper/grpc_query.go	`93.65% <100.00%> (+1.80%)`	⬆️
x/oracle/types/genesis.go	`63.63% <100.00%> (ø)`
... and 7 more

github-advanced-security

CodeQL found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

x/oracle/keeper/historic_price.go

x/oracle/keeper/migrations.go

robert-zaremba

Need to fix IterateMedians

x/oracle/keeper/historic_price.go

zarazan

Great job getting this together! It needed a ton of changes since last time. I think we're almost there, a couple tiny comments from me and some consensus on the iteration method and I think its good.

util/pricestats.go

x/oracle/types/keys.go

x/oracle/keeper/historic_price.go

Co-authored-by: Kyle <zarazan@users.noreply.github.com>

zarazan

I think @robert-zaremba's concerns are resolved but idk if he wants to take a look before merging.

robert-zaremba

thanks for fixing the denom prefix for iterations.

Please verify if these change are in line with the updated: 3909192

x/oracle/keeper/historic_price.go

util/pricestats.go

x/oracle/abci.go

util/pricestats_test.go

x/oracle/keeper/historic_price.go

x/oracle/keeper/migrations.go

rbajollari · 2022-12-06T00:52:50Z

@robert-zaremba @zarazan @adamewozniak Updated the keeper methods in the historacle design doc to match the actual implementation in this PR (some of the stuff agreed upon in #1631 wasn't put into the doc). Lmk if anything looks wrong in there.

docs/design_docs/011-historacle-pricing.md

this is due to e2e tests failing with a change in the oracle parameters query

adamewozniak · 2022-12-06T18:45:22Z

note that this PR makes the most updated version of the price feeder incompatible & we'll need to test PF against the release that includes this PR
cc @rbajollari / @zarazan

docs/design_docs/011-historacle-pricing.md

x/oracle/abci.go

proto/umee/oracle/v1/genesis.proto

util/decmath/decmath.go

x/oracle/keeper/genesis.go

x/oracle/keeper/grpc_query_test.go

x/oracle/keeper/historic_price.go

Co-authored-by: Robert Zaremba <robert@zaremba.ch>

x/oracle/abci.go

-				if isPeriodLastBlock(ctx, params.StampPeriod) && params.HistoricAcceptList.Contains(ballotDenom.Denom) {
+				if isPeriodLastBlock(ctx, params.HistoricStampPeriod) {
 					k.AddHistoricPrice(ctx, ballotDenom.Denom, exchangeRate)
 				}


x/oracle/abci.go

+				// Calculate and stamp median/median deviation if median stamp period has passed
+				if isPeriodLastBlock(ctx, params.MedianStampPeriod) {
+					if err = k.CalcAndSetHistoricMedian(ctx, ballotDenom.Denom); err != nil {
+						return err


x/oracle/abci.go

+				if isPeriodLastBlock(ctx, params.MedianStampPeriod) {
+					if err = k.CalcAndSetHistoricMedian(ctx, ballotDenom.Denom); err != nil {
+						return err
+					}


x/oracle/abci.go

+		pruneMedianPeriod := params.MedianStampPeriod*(params.MaximumMedianStamps) - params.VotePeriod
+		for _, v := range params.AcceptList {
+			k.DeleteHistoricPrice(ctx, v.SymbolDenom, uint64(ctx.BlockHeight())-pruneHistoricPeriod)
+			k.DeleteHistoricMedian(ctx, v.SymbolDenom, uint64(ctx.BlockHeight())-pruneMedianPeriod)


x/oracle/abci.go

+			k.DeleteHistoricMedianDeviation(ctx, v.SymbolDenom, uint64(ctx.BlockHeight())-pruneMedianPeriod)
 		}
 	}



x/oracle/abci.go

robert-zaremba

utACK. Thanks for keeping improving the PR.

x/oracle/keeper/historic_price.go

toteki · 2022-12-09T15:44:57Z

Note: be sure to run make proto-all in these PRs.

In this case, there's a little whitespace that gets changed by make proto-format and some changes not reflected in swagger/ folder.

rbajollari added 2 commits December 1, 2022 10:36

Refactor historacle median tracking

a48bb43

Merge branch 'main' into ryan/historacle-median-tracking

1501fb1

github-advanced-security bot found potential problems Dec 1, 2022

View reviewed changes

lint

a90346e

rbajollari marked this pull request as ready for review December 1, 2022 16:00

rbajollari requested review from a team as code owners December 1, 2022 16:00

Merge branch 'main' into ryan/historacle-median-tracking

7fc2c74

zarazan reviewed Dec 1, 2022

View reviewed changes

x/oracle/keeper/historic_price.go Outdated Show resolved Hide resolved

zarazan reviewed Dec 1, 2022

View reviewed changes

x/oracle/keeper/migrations.go Show resolved Hide resolved

robert-zaremba suggested changes Dec 2, 2022

View reviewed changes

x/oracle/keeper/historic_price.go Outdated Show resolved Hide resolved

x/oracle/keeper/historic_price.go Show resolved Hide resolved

rbajollari and others added 3 commits December 2, 2022 15:37

Merge branch 'main' into ryan/historacle-median-tracking

a545a28

Update historacle keeper methods

c9356a3

Add pricestats unit test

803e219

rbajollari requested review from adamewozniak, robert-zaremba and zarazan December 2, 2022 21:06

rbajollari and others added 2 commits December 2, 2022 16:10

Merge branch 'main' into ryan/historacle-median-tracking

e91f75b

Add Get to getter keeper methods

97627a0

zarazan reviewed Dec 5, 2022

View reviewed changes

util/pricestats.go Outdated Show resolved Hide resolved

util/pricestats.go Outdated Show resolved Hide resolved

x/oracle/types/keys.go Show resolved Hide resolved

x/oracle/keeper/historic_price.go Outdated Show resolved Hide resolved

rbajollari and others added 4 commits December 5, 2022 11:13

PR comments

c6a5f60

Update x/oracle/types/keys.go

dd9f000

Co-authored-by: Kyle <zarazan@users.noreply.github.com>

Add ParseBlockFromKey comment

eb0f1b0

lint

a0630a0

zarazan approved these changes Dec 5, 2022

View reviewed changes

Merge branch 'main' into ryan/historacle-median-tracking

85c2ffc

adamewozniak requested review from robert-zaremba and removed request for robert-zaremba December 5, 2022 23:26

robert-zaremba reviewed Dec 5, 2022

View reviewed changes

Update keeper methods in historacle design doc

2fff271

rbajollari requested a review from a team as a code owner December 6, 2022 00:44

zarazan reviewed Dec 6, 2022

View reviewed changes

docs/design_docs/011-historacle-pricing.md Outdated Show resolved Hide resolved

rbajollari added 2 commits December 6, 2022 10:45

Update WithinMedianDeviation

d8fd83f

revert file picked up by gofmt

fcfcf60

adamewozniak requested a review from robert-zaremba December 6, 2022 17:58

switch umee version from v3.2.0 -> ../ (again)

ff0ed4c

this is due to e2e tests failing with a change in the oracle parameters query

robert-zaremba reviewed Dec 7, 2022

View reviewed changes

docs/design_docs/011-historacle-pricing.md Outdated Show resolved Hide resolved

rbajollari and others added 2 commits December 7, 2022 19:04

Merge branch 'main' into ryan/historacle-median-tracking

ffeda21

Fix keeper method names

8d6ee4f

github-advanced-security bot found potential problems Dec 8, 2022

View reviewed changes

robert-zaremba reviewed Dec 8, 2022

View reviewed changes

rbajollari and others added 8 commits December 8, 2022 10:07

Update util/decmath/decmath.go

e3d3154

Co-authored-by: Robert Zaremba <robert@zaremba.ch>

Update util/decmath/decmath.go

1813826

Co-authored-by: Robert Zaremba <robert@zaremba.ch>

Update util/decmath/decmath.go

78fca1b

Co-authored-by: Robert Zaremba <robert@zaremba.ch>

Update x/oracle/abci.go

4d1160f

Co-authored-by: Robert Zaremba <robert@zaremba.ch>

Merge branch 'main' into ryan/historacle-median-tracking

2b3896a

PR comments

43db6cb

lint

0c0e5eb

update method names in design doc

8e41f41

github-advanced-security bot found potential problems Dec 8, 2022

View reviewed changes

rbajollari added 2 commits December 8, 2022 18:07

Use KVStoreReversePrefixIteratorPaginated

aad1422

Update iterator method names and comments

694d5d3

robert-zaremba approved these changes Dec 8, 2022

View reviewed changes

x/oracle/keeper/historic_price.go Show resolved Hide resolved

rbajollari merged commit 0903f03 into main Dec 9, 2022

rbajollari deleted the ryan/historacle-median-tracking branch December 9, 2022 00:05

robert-zaremba mentioned this pull request Dec 9, 2022

refactor: Update historic medians query #1651

Merged

17 tasks

Conversation

rbajollari commented Dec 1, 2022

Description

Author Checklist

Reviewers Checklist

Uh oh!

codecov-commenter commented Dec 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

robert-zaremba left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zarazan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zarazan left a comment

Choose a reason for hiding this comment

Uh oh!

robert-zaremba left a comment • edited by rbajollari Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rbajollari commented Dec 6, 2022

Uh oh!

Uh oh!

adamewozniak commented Dec 6, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

robert-zaremba left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

toteki commented Dec 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

codecov-commenter commented Dec 1, 2022 •

edited

Loading

robert-zaremba left a comment •

edited by rbajollari

Loading

toteki commented Dec 9, 2022 •

edited

Loading