dbTalk Databases Forums  

simple numeric calculation doent work

comp.databases.ms-sqlserver comp.databases.ms-sqlserver


Discuss simple numeric calculation doent work in the comp.databases.ms-sqlserver forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
NZDeveloper
 
Posts: n/a

Default simple numeric calculation doent work - 12-03-2009 , 06:55 PM






why in gods name doesnt this simple sql calculation not work in a view

98/912500 = 0.0001073972603

if I do a simple view with the following calcualted field:

SELECT 98 / 912500 AS aaa
FROM dbo.tbl_CA_Cases

I get 0??????

If I explicitly convert to decimal

SELECT CONVERT(decimal(38, 16), 98) AS v, CONVERT(decimal(38, 16),
912500) AS vt, CONVERT(decimal(38, 16), 98) / CONVERT(decimal(38, 16),
912500)
AS aaa, CONVERT(decimal(38, 16), CONVERT(decimal
(38, 16), 98) / CONVERT(decimal(38, 16), 912500)) AS bbb
FROM dbo.tbl_CA_Cases

I get 0.000107

Why is it only going to 6 decimal places????

If I do
SELECT 118993 * 98 / 912500 * 0.8468 AS aaa
FROM dbo.tbl_CA_Cases

I get 10.1616 where did SQL get this value from??? the correct
calculation is 10.82169939200

If I do
SELECT CONVERT(decimal(38, 16), 118993) * 98 / 912500 * 0.8468 AS
aaa
FROM dbo.tbl_CA_Cases

I get 10.821699392000 whcih is correct

What exactly is SQL calculating?????????????????

I'm assuming there is some logical explanation.....

Cheers

Grant

Reply With Quote
  #2  
Old   
Plamen Ratchev
 
Posts: n/a

Default Re: simple numeric calculation doent work - 12-03-2009 , 08:52 PM






In the first case you get 0 because you have integer division. In the other examples you get different results because
of the different precision used in the dividend and divisor. See in the following article details on how the resulting
precision and scale are calculated:
http://msdn.microsoft.com/en-us/library/ms190476.aspx

--
Plamen Ratchev
http://www.SQLStudio.com

Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.