lkml.org 
[lkml]   [1999]   [Feb]   [23]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
SubjectRe: [OFFTOPIC] Rounding problem
From
Date
In <99Feb23.101214gmt.28674@worm.brammer.net> Andrew Logan <aml@bsl.co.uk> writes:

> Hi all,
>
> I've found a worrying rounding error which is manifested by the
> following simple test program.

This is not at all linux-kernel related. It should not be posted here.

> func( (long) d3 ) -> 2400
> func( (long) (d1 * d2) ) -> 2399
> func1( d1 * d2 ) -> 2400

What you are seeing here is the result of fact that 2.4 can't be
represented exactly in the binary notation that floating point uses.
This is basically the same as not being able to exactly represent 1/3
in decimal notation (with a fixed number of digits).

The other thing you are seeing is that x86 floating point processors
hold "temporary" calculations in 80-bit format, while a double is only 64
bits. This makes serious numerical analysis a real pain in the ass,
and can make even simple programs like yours give surprising results.

This is really the fault of both Intel and gcc. Intel didn't design
the x86 to allow true 64-bit only math and uses 80-bit "temporaries"
as a default. Gcc leaves the 80-bits as a default, and there isn't any
"easy" way of switching to Intel's "almost IEEE754 64-bit" mode. The
only option that comes close to turning off this "feature" is the
-ffloat-store option, which causes massive slowdowns.

Note that simply changing the optimization levels can the results that
you see 'cause the "temporary 80-bit" values will be used at different
times.


> The second line of output manifests the rounding problem.
> I compiled and ran exactly the same program on Non-StopUX with the
> native C compiler and Win98 using M$ 32-bit compiler version
> 10.20.6166 which both produced the correct value on the second
> output line.

You probably got lucky that it produced "correct" results. The MS
compiler does have an option to uses Intel's "almost IEEE754 64-bit"
mode, but IIRC, it is off by default.


-wayne


--
Wayne Schlitt can not assert the truth of all statements in this
article and still be consistent.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:50    [W:0.035 / U:0.424 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site