fix!: allow safe leverage operations during partial oracle outages (#1821)

## Description

This one does a lot of things:
- Allows `account_summary` to work during price outages, treating all unknown prices as zero.
  - In such cases, supplied value and other fields will appear lower than it really is, since some assets were skipped
  - Liquidation threshold will be null when it can't be computed, since there's no safe way to do that with missing prices
- `borrow_limit` in queries as well as messages is computed using only collateral with known prices
  - If the portion of your collateral with known prices is enough to cover a borrow, then it still works.
  - Same for withdraw and decollateralize
- `MaxWithdraw` and `MaxBorrow` (both queries and messages) now function with missing collateral prices, respecting the borrow limit policy above. The queries return zero on missing borrow prices, or a missing price for the specific token being asked for.
- `liquidation_targets` query skips addresses where liquidation threshold cannot be computed, instead of returning an error for the whole query
- `MsgLiquidate` will be able to function with some missing prices on the target's borrowed assets, if their other borrows are  high enough to still put them above their liquidation threshold

API Breaking:
- `liquidation_threshold` field in `account_summary` field can now be null

---

### Author Checklist

_All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues._

I have...

- [x] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [x] added `!` to the type prefix if API or client breaking change
- [x] added appropriate labels to the PR
- [x] targeted the correct branch (see [PR Targeting](https://github.com/umee-network/umee/blob/main/CONTRIBUTING.md#pr-targeting))
- [ ] provided a link to the relevant issue or specification
- [x] added a changelog entry to `CHANGELOG.md`
- [x] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [x] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed

### Reviewers Checklist

_All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items._

I have...

- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed all author checklist items have been addressed
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)

Author: toteki

CHANGELOG.md

@@ -55,6 +55,7 @@ Ref: https://keepachangelog.com/en/1.0.0/
 - [1812](https://github.com/umee-network/umee/pull/1812) MaxCollateralShare now works during partial oracle outages when certain conditions are safe.
 - [1736](https://github.com/umee-network/umee/pull/1736) Blacklisted tokens no longer add themselves back to the oracle accept list.
 - [1807](https://github.com/umee-network/umee/pull/1807) Fixes BNB ibc denom in 4.1 migration
+- [1821](https://github.com/umee-network/umee/pull/1821) Allow safe leverage operations during partial oracle outages.
 
 ## [v4.0.1](https://github.com/umee-network/umee/releases/tag/v4.0.1) - 2023-02-10
 

proto/umee/leverage/v1/query.proto

@@ -222,31 +222,40 @@ message QueryAccountSummary {
 // QueryAccountSummaryResponse defines the response structure for the AccountSummary gRPC service handler.
 message QueryAccountSummaryResponse {
   // Supplied Value is the sum of the USD value of all tokens the account has supplied, including interest earned.
+  // Computation skips assets which are missing oracle prices, potentially resulting in a lower supplied
+  // value than if prices were all available.
   string supplied_value = 1 [
     (gogoproto.customtype) = "github.com/cosmos/cosmos-sdk/types.Dec",
     (gogoproto.nullable)   = false
   ];
   // Collateral Value is the sum of the USD value of all uTokens the account has collateralized.
+  // Computation skips collateral which is missing an oracle price, potentially resulting in a lower collateral
+  // value than if prices were all available.
   string collateral_value = 2 [
     (gogoproto.customtype) = "github.com/cosmos/cosmos-sdk/types.Dec",
     (gogoproto.nullable)   = false
   ];
   // Borrowed Value is the sum of the USD value of all tokens the account has borrowed, including interest owed.
   // It always uses spot prices.
+  // Computation skips borrows which are missing oracle prices, potentially resulting in a lower borrowed
+  // value than if prices were all available.
   string borrowed_value = 3 [
     (gogoproto.customtype) = "github.com/cosmos/cosmos-sdk/types.Dec",
     (gogoproto.nullable)   = false
   ];
   // Borrow Limit is the maximum Borrowed Value the account is allowed to reach through direct borrowing.
   // The lower of spot or historic price for each collateral token is used when calculating borrow limits.
+  // Computation skips collateral which is missing an oracle price, potentially resulting in a lower borrow
+  // limit than if prices were all available.
   string borrow_limit = 4 [
     (gogoproto.customtype) = "github.com/cosmos/cosmos-sdk/types.Dec",
     (gogoproto.nullable)   = false
   ];
   // Liquidation Threshold is the Borrowed Value at which the account becomes eligible for liquidation.
+  // Will be null if an oracle price required for computation is missing.
   string liquidation_threshold = 5 [
     (gogoproto.customtype) = "github.com/cosmos/cosmos-sdk/types.Dec",
-    (gogoproto.nullable)   = false
+    (gogoproto.nullable)   = true
   ];
 }
 

swagger/swagger.yaml

@@ -132,18 +132,33 @@ paths:
                 description: >-
                   Supplied Value is the sum of the USD value of all tokens the
                   account has supplied, including interest earned.
+
+                  Computation skips assets which are missing oracle prices,
+                  potentially resulting in a lower supplied
+
+                  value than if prices were all available.
               collateral_value:
                 type: string
                 description: >-
                   Collateral Value is the sum of the USD value of all uTokens
                   the account has collateralized.
+
+                  Computation skips collateral which is missing an oracle price,
+                  potentially resulting in a lower collateral
+
+                  value than if prices were all available.
               borrowed_value:
                 type: string
                 description: >-
                   Borrowed Value is the sum of the USD value of all tokens the
                   account has borrowed, including interest owed.
 
                   It always uses spot prices.
+
+                  Computation skips borrows which are missing oracle prices,
+                  potentially resulting in a lower borrowed
+
+                  value than if prices were all available.
               borrow_limit:
                 type: string
                 description: >-
@@ -152,11 +167,19 @@ paths:
 
                   The lower of spot or historic price for each collateral token
                   is used when calculating borrow limits.
+
+                  Computation skips collateral which is missing an oracle price,
+                  potentially resulting in a lower borrow
+
+                  limit than if prices were all available.
               liquidation_threshold:
                 type: string
                 description: >-
                   Liquidation Threshold is the Borrowed Value at which the
                   account becomes eligible for liquidation.
+
+                  Will be null if an oracle price required for computation is
+                  missing.
             description: >-
               QueryAccountSummaryResponse defines the response structure for the
               AccountSummary gRPC service handler.
@@ -2023,18 +2046,33 @@ definitions:
         description: >-
           Supplied Value is the sum of the USD value of all tokens the account
           has supplied, including interest earned.
+
+          Computation skips assets which are missing oracle prices, potentially
+          resulting in a lower supplied
+
+          value than if prices were all available.
       collateral_value:
         type: string
         description: >-
           Collateral Value is the sum of the USD value of all uTokens the
           account has collateralized.
+
+          Computation skips collateral which is missing an oracle price,
+          potentially resulting in a lower collateral
+
+          value than if prices were all available.
       borrowed_value:
         type: string
         description: >-
           Borrowed Value is the sum of the USD value of all tokens the account
           has borrowed, including interest owed.
 
           It always uses spot prices.
+
+          Computation skips borrows which are missing oracle prices, potentially
+          resulting in a lower borrowed
+
+          value than if prices were all available.
       borrow_limit:
         type: string
         description: >-
@@ -2043,11 +2081,18 @@ definitions:
 
           The lower of spot or historic price for each collateral token is used
           when calculating borrow limits.
+
+          Computation skips collateral which is missing an oracle price,
+          potentially resulting in a lower borrow
+
+          limit than if prices were all available.
       liquidation_threshold:
         type: string
         description: >-
           Liquidation Threshold is the Borrowed Value at which the account
           becomes eligible for liquidation.
+
+          Will be null if an oracle price required for computation is missing.
     description: >-
       QueryAccountSummaryResponse defines the response structure for the
       AccountSummary gRPC service handler.

x/leverage/client/tests/tests.go

@@ -252,6 +252,8 @@ func (s *IntegrationTestSuite) TestLeverageScenario() {
 		nil,
 	}
 
+	lt1 := sdk.MustNewDecFromStr("0.0085610525")
+
 	nonzeroQueries := []testQuery{
 		{
 			"query account balances",
@@ -294,7 +296,7 @@ func (s *IntegrationTestSuite) TestLeverageScenario() {
 				// (1001 / 1000000) * 34.21 * 0.25 = 0.0085610525
 				BorrowLimit: sdk.MustNewDecFromStr("0.0085610525"),
 				// (1001 / 1000000) * 0.25 * 34.21 = 0.0085610525
-				LiquidationThreshold: sdk.MustNewDecFromStr("0.0085610525"),
+				LiquidationThreshold: &lt1,
 			},
 		},
 		{

x/leverage/keeper/borrows.go

@@ -9,7 +9,9 @@ import (
 
 // assertBorrowerHealth returns an error if a borrower is currently above their borrow limit,
 // under either recent (historic median) or current prices. It returns an error if
-// prices cannot be calculated.
+// borrowed asset prices cannot be calculated, but will try to treat collateral whose prices are
+// unavailable as having zero value. This can still result in a borrow limit being too low,
+// unless the remaining collateral is enough to cover all borrows.
 // This should be checked in msg_server.go at the end of any transaction which is restricted
 // by borrow limits, i.e. Borrow, Decollateralize, Withdraw, MaxWithdraw.
 func (k Keeper) assertBorrowerHealth(ctx sdk.Context, borrowerAddr sdk.AccAddress) error {
@@ -20,7 +22,7 @@ func (k Keeper) assertBorrowerHealth(ctx sdk.Context, borrowerAddr sdk.AccAddres
 	if err != nil {
 		return err
 	}
-	limit, err := k.CalculateBorrowLimit(ctx, collateral)
+	limit, err := k.VisibleBorrowLimit(ctx, collateral)
 	if err != nil {
 		return err
 	}
@@ -127,6 +129,45 @@ func (k Keeper) CalculateBorrowLimit(ctx sdk.Context, collateral sdk.Coins) (sdk
 	return limit, nil
 }
 
+// VisibleBorrowLimit uses the price oracle to determine the borrow limit (in USD) provided by
+// collateral sdk.Coins, using each token's uToken exchange rate and collateral weight.
+// The lower of spot price or historic price is used for each collateral token.
+// An error is returned if any input coins are not uTokens.
+// This function skips assets that are missing prices, which will lead to a lower borrow
+// limit when prices are down instead of a complete loss of borrowing ability.
+func (k Keeper) VisibleBorrowLimit(ctx sdk.Context, collateral sdk.Coins) (sdk.Dec, error) {
+	limit := sdk.ZeroDec()
+
+	for _, coin := range collateral {
+		// convert uToken collateral to base assets
+		baseAsset, err := k.ExchangeUToken(ctx, coin)
+		if err != nil {
+			return sdk.ZeroDec(), err
+		}
+
+		ts, err := k.GetTokenSettings(ctx, baseAsset.Denom)
+		if err != nil {
+			return sdk.ZeroDec(), err
+		}
+
+		// ignore blacklisted tokens
+		if !ts.Blacklist {
+			// get USD value of base assets using the chosen price mode
+			v, err := k.TokenValue(ctx, baseAsset, types.PriceModeLow)
+			if err == nil {
+				// if both spot and historic (if required) prices exist,
+				// add collateral coin's weighted value to borrow limit
+				limit = limit.Add(v.Mul(ts.CollateralWeight))
+			}
+			if nonOracleError(err) {
+				return sdk.ZeroDec(), err
+			}
+		}
+	}
+
+	return limit, nil
+}
+
 // CalculateLiquidationThreshold determines the maximum borrowed value (in USD) that a
 // borrower with given collateral could reach before being eligible for liquidation, using
 // each token's oracle price, uToken exchange rate, and liquidation threshold.

x/leverage/keeper/collateral.go

@@ -94,17 +94,13 @@ func (k Keeper) VisibleCollateralValue(ctx sdk.Context, collateral sdk.Coins) (s
 
 		// get USD value of base assets
 		v, err := k.TokenValue(ctx, baseAsset, types.PriceModeSpot)
-		if err != nil {
-			k.Logger(ctx).Info(
-				"collateral value skipped",
-				"uToken", coin.String(),
-				"error", err.Error(),
-			)
-			continue
+		if err == nil {
+			// for coins that did not error, add their value to the total
+			total = total.Add(v)
+		}
+		if nonOracleError(err) {
+			return sdk.ZeroDec(), err
 		}
-
-		// for coins that did not error, add their value to the total
-		total = total.Add(v)
 	}
 
 	return total, nil

x/leverage/keeper/errors.go

@@ -0,0 +1,33 @@
+package keeper
+
+import (
+	"strings"
+
+	"cosmossdk.io/errors"
+
+	"github.com/umee-network/umee/v4/util/decmath"
+	leveragetypes "github.com/umee-network/umee/v4/x/leverage/types"
+	oracletypes "github.com/umee-network/umee/v4/x/oracle/types"
+)
+
+// nonOracleError returns true if an error is non-nil
+// and also not one of ErrEmptyList, ErrUnknownDenom, or ErrNoHistoricMedians
+// which are errors which can result from missing prices
+func nonOracleError(err error) bool {
+	if err == nil {
+		return false
+	}
+	// check typed errors
+	if errors.IsOf(err,
+		leveragetypes.ErrInvalidOraclePrice,
+		leveragetypes.ErrNoHistoricMedians,
+		oracletypes.ErrUnknownDenom,
+	) {
+		return false
+	}
+	// this error needs to be checked by string comparison
+	if strings.Contains(err.Error(), decmath.ErrEmptyList.Error()) {
+		return false
+	}
+	return true
+}

x/leverage/keeper/errors_test.go

@@ -0,0 +1,38 @@
+package keeper
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+
+	"cosmossdk.io/errors"
+
+	"github.com/umee-network/umee/v4/util/decmath"
+	leveragetypes "github.com/umee-network/umee/v4/x/leverage/types"
+	oracletypes "github.com/umee-network/umee/v4/x/oracle/types"
+)
+
+func TestErrorMatching(t *testing.T) {
+	// oracle errors
+	err1 := errors.Wrap(decmath.ErrEmptyList, "denom: UMEE")
+	err2 := oracletypes.ErrUnknownDenom.Wrap("UMEE")
+	err3 := leveragetypes.ErrNoHistoricMedians.Wrapf(
+		"requested %d, got %d",
+		16,
+		12,
+	)
+	// not oracle errors
+	err4 := leveragetypes.ErrBlacklisted
+	err5 := leveragetypes.ErrUToken
+	err6 := leveragetypes.ErrNotRegisteredToken
+	err7 := errors.New("foo", 1, "bar")
+
+	require.Equal(t, false, nonOracleError(nil))
+	require.Equal(t, false, nonOracleError(err1))
+	require.Equal(t, false, nonOracleError(err2))
+	require.Equal(t, false, nonOracleError(err3))
+	require.Equal(t, true, nonOracleError(err4))
+	require.Equal(t, true, nonOracleError(err5))
+	require.Equal(t, true, nonOracleError(err6))
+	require.Equal(t, true, nonOracleError(err7))
+}