Improve the WAGED rebalancer soft constraint design

[jiajunwang]

Our current implementation is based on POC work, which returns the evaluated score based on the ratio between current usage versus the estimated usage.
However, the estimated usage could be difficult to derive. For example, tags could impact expected usage.
In order to address these corner cases, we need to revisit all the soft constraint definition.
The requirement is basically:

1. Continous distribution.
2. Normalized to a [0.0 ~1.0] number.
3. Any two estimated scores should be comparable and different if the two potential assignment will cause different usages.

[i3wangyi]

The simple inverse function has an issue where max value is not constrained.
I found the sigmoid function (widely used in logistic regression) more aligned with our requirement:
https://en.wikipedia.org/wiki/Logistic_function

[i3wangyi]

A better alternative function is $$f(x) = tanh(1/x)$$

The function plot looks like

It satisfies

• continues distribution between (0, 1]
• The value decreases when x >= 0
• When x == 0, the value is infinitely close to 1

The X in soft constraint mostly defines the usage of some aspect (X >= 0). The lower usage is generally encouraged and higher usage is discouraged.

[jiajunwang]

Which calculation is faster? And I would prefer sigmoid. I think it differentiates better when assigned partition count is a small number. And that is our major concerns.
We won't handle very full clusters usually.

[i3wangyi]

$$tanh(x) = \frac{e^{2x} - 1}{e^{2x} + 1}$$
$$sigmoid(x) = \frac{1}{1 + e ^ {x}}$$
The calculation speed is same. I would prefer the tanh because I can show you to make sigmoid value = 0 when x == 0 is much harder, whereas the tanh by nature closes to 1 when x nears 0.

The second part is we can tune the paramter inside, e.g,
$$f(x) = tanh( 1 / \alpha x)$$
Different alpha parameter would lead to differen kind of shapes

[i3wangyi]

$$f(x) = tanh( 1 / 10x)$$ $$f(x) = tanh( 1 / 0.1x)$$

[jiajunwang]

We have changed to use the sigmoid function to calculate.