Skip to content

Conversation

@tompng
Copy link
Member

@tompng tompng commented May 17, 2025

Increase BigMath.atan(x, prec) precision when |x|>0.5

BigMath.atan(x, prec) is calculated with precision=prec+BigDecimal.double_fig when |x| <= 0.5.
But this safe margin was not properly applied when |x| > 0.5.

BigMath.atan(BigDecimal('0.2'), 100) #=> accuracy was about 116 digits
BigMath.atan(BigDecimal('0.8'), 100) #=> accuracy was about 108 digits
BigMath.atan(BigDecimal('3'), 100) #=> accuracy was about 100 digits

@mrkn mrkn modified the milestones: v3.2, v3.3 May 29, 2025
BigMath.atan(x, prec) is calculated with precision=prec+BigDecimal.double_fig when |x| <= 0.5.
But this safe margin was not properly applied when |x| > 0.5.
@tompng tompng force-pushed the atan_precision_safe_factor branch from 9972f16 to fa9cba9 Compare June 7, 2025 20:12
@tompng tompng merged commit 8459a3c into ruby:master Jun 8, 2025
78 checks passed
@tompng tompng deleted the atan_precision_safe_factor branch June 8, 2025 06:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants