Description
As discussed on Discourse, expect_near_rel behaves weirdly for values around zero. Most importantly, the relative tolerance jumps up at the tolerance value - here is the relative tolerance vs. average of the tested values in the current implementation, assuming tol = 1e-8:
This makes it hard to write tests that are strict for values near zero while not being too strict for large values.
Proposed solution
I propose that expect_near_rel has two tolerances relative (tol_rel) and minimal (tol_min). The final tolerance is then computed as
auto avg = 0.5 * (fabs(x1) + fabs(x2));
tol_final = max(tol_min, tol_rel * avg);
EXPECT_NEAR(x1, x2, tol_final);
where x1 and x2 are the values we are comparing.
The relative tolerance now increases smoothly when approaching zero as seen in this plot made with tol_rel = 1e-8, tol_min = 1e-16.
It is slightly unclear what should be the default value for tol_min. Using tol_min = tol_rel means the tolerance stays the same or is increased (relative to the current implementation). Using tol_min = tol_rel ^ 2 means the tolerance stays the same or decreases. Values in between would make the tolerance sometimes increase sometimes decrease.
Currently the default tolerance is 1e-8. Having tol_min = 1e-8 is quite lax, while having tol_min = 1e-16 is very strict (I can't make almost any slightly strong test pass with tol_min = 1e-16).
I would currently prefer somehing in between, possibly tol_min = max(tol_rel ^ 2,1e-14) (as 1e-14 is IMHO achievable for many functions).
Current Version:
v3.1.0
Description
As discussed on Discourse,
expect_near_relbehaves weirdly for values around zero. Most importantly, the relative tolerance jumps up at the tolerance value - here is the relative tolerance vs. average of the tested values in the current implementation, assumingtol = 1e-8:This makes it hard to write tests that are strict for values near zero while not being too strict for large values.
Proposed solution
I propose that
expect_near_relhas two tolerances relative (tol_rel) and minimal (tol_min). The final tolerance is then computed aswhere
x1andx2are the values we are comparing.The relative tolerance now increases smoothly when approaching zero as seen in this plot made with
tol_rel = 1e-8, tol_min = 1e-16.It is slightly unclear what should be the default value for
tol_min. Usingtol_min = tol_relmeans the tolerance stays the same or is increased (relative to the current implementation). Usingtol_min = tol_rel ^ 2means the tolerance stays the same or decreases. Values in between would make the tolerance sometimes increase sometimes decrease.Currently the default tolerance is
1e-8. Havingtol_min = 1e-8is quite lax, while havingtol_min = 1e-16is very strict (I can't make almost any slightly strong test pass withtol_min = 1e-16).I would currently prefer somehing in between, possibly
tol_min = max(tol_rel ^ 2,1e-14)(as1e-14is IMHO achievable for many functions).Current Version:
v3.1.0