[Chicago-talk] addition weirdness
lembark at wrkhors.com
Thu Jan 29 10:29:44 CST 2004
> Any idea why this is happening?
IEEE number formats have a limit on scale. When you try
to add integers the .00000000 part eventually starts
rounding and you end up with a .999999 or .000001 [don't
quote me on the exact precision].
This also happens if you add a bunch of values in
tentns (i.e., /\d\.[0-9]/) since 1/10 is a repeating
fraction base two.
If this is a problem then
will give you integer precision up to your processor's
word limit; use <IForgot>::BigInt does pretty much what it
Steven Lembark 2930 W. Palmer
Workhorse Computing Chicago, IL 60647
+1 888 359 3508
More information about the Chicago-talk