lkml.org 
[lkml]   [1999]   [Feb]   [23]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [OFFTOPIC] Rounding problem
Andrew Logan wrote:
>
> Hi all,
>
> I've found a worrying rounding error which is manifested by the
> following simple test program.
> [ snip ]

Wow. One I can answer..

Here's the degenerate example:

double d1 = 2.4, d2 = 1000, d3 = 2400;

main()
{
if (d1 * d2 == d3)
printf("just lucky\n");

printf("%d\n", (long)(d1 * d2));
}

You'll find that the floating point representation of one or both of 2.4
and 1000 isn't exact, so the product of them comes to a tiny, tiny bit
less that 2400. The Pentium's hardware float to long conversion then
rounds down to 2399.

Note that this program doesn't say "just lucky". This is why floats
should never be compared for equality.

It works with 'cc -msoft-float'. Just lucky.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 13:50    [W:0.411 / U:0.500 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site