dbTalk Databases Forums  

[BUGS] Precision errors in float8 type casting (as of 7.3.2 and some earlier

mailing.database.pgsql-bugs mailing.database.pgsql-bugs


Discuss [BUGS] Precision errors in float8 type casting (as of 7.3.2 and some earlier in the mailing.database.pgsql-bugs forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Philip Edelbrock
 
Posts: n/a

Default [BUGS] Precision errors in float8 type casting (as of 7.3.2 and some earlier - 07-09-2003 , 10:58 PM








I ran into this oddity today and tested it on a few of our PosgreSQL
backends (all of which gave the same response):

phil=# select 3.85::float4*1;
?column?
------------------
3.84999990463257
(1 row)

phil=# select 3.85::float4*1::float8;
?column?
------------------
3.84999990463257

phil=# select (3.85::float4)::float8;
float8
------------------
3.84999990463257
(1 row)

(Or substitute 3.85 for any number with something other than 0 to the
right of the decimal point, or pull the same values from any table which
stores in float4/real format.)

Obviously, this is wrong and should return 3.85. We traced this down on
an ecom server which was shaving off pennies from some transactions
(because we truncate to the hundredths place instead of rounding what we
get back from the SQL backend).

The newest server we have is 7.3.2, so I haven't tried this on the
current 7.3.3 release.

phil=# select version();

version
---------------------------------------------------------------------------------------------------------
PostgreSQL 7.3.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.2
20020903 (Red Hat Linux 8.0 3.2-7)
(1 row)


Phil


---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.