dbTalk Databases Forums  

Question about significant figures

comp.database.oracle comp.database.oracle


Discuss Question about significant figures in the comp.database.oracle forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
James B. Millard
 
Posts: n/a

Default Question about significant figures - 07-16-2004 , 04:44 PM






I'm working on a query that has to provide values with a specified number of
significant figures. What I've done so far is to use:

select
test_number,
value,
sig_figs_number,
decode
(
sig_figs_number,
-1,
value,
0,
value,
round(value, sig_figs_number - trunc(ln(value)/ln(10)) - 1)
) formatted
from
result

Output of this looks like:

TEST_NUMBE VALUE SIG_FIGS_NUMBER FORMATTED
---------- ---------- --------------- ----------
130 3.14159 2 3.1
130 3.14159 2 3.1
131 3 3 3
131 3 3 3
132 3.14159 -1 3.14159
132 3.14159 -1 3.14159
133 3.14159 1 3
133 3.14159 1 3
134 3.14159 2 3.1
134 3.14159 2 3.1
135 314.59 1 300
135 314.59 1 300
136 314.59 0 314.59
136 314.59 0 314.59

The problem is the value of 3 -- it should be 3.00 and I don't know a good
way to do that.

The "-1" in the sig_figs_number column is a default and we don't want any
formatting or rounding of the result, same for zero, hence the decode.

Thanks!

Brad
--

************************************************** *********************
Brad Millard On-line ballistics for small arms...
jbm (AT) eskimo (DOT) com www.eskimo.com/~jbm
************************************************** *********************


Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.