SPUG: Rounding error
Fred Morris
m3047 at m3047.net
Wed Nov 23 09:48:20 PST 2011
On Wednesday 23 November 2011 10:24, Amit Sett wrote:
> [...] The program works well except for some cases where it short
> changes a customer by 1 penny [...]
Try doing things in pennies, e.g. a dollar is 100 pennies.
Do you know what a repeating decimal is, and do you know what causes them?
Basically any time you divide 1 by a prime number not in your number base you
get a repeating decimal.
For instance in base 10 (2 * 5) if you divide by 3, 7, 11, 13, etc. you get a
repeating decimal.
In base 2, dividing by 3, 5, 7, 11, etc does the same thing. If you divide by
100 (2 * 2 * 5 * 5) you get a repeating decimal, or in other words there is
rounding error.
ObMathNote: 0.9999999... actually is provably equal to 1. ;-)
--
Fred
More information about the spug-list
mailing list