Profile: WinXP Pro, FMP 6.0


I have a simple series of calculations:



A / B = C
D / E = F

C + F = G

Occasionally C + F does not equal G. It is off by 0.01

I always figured it was a rounding issue, but what I've discovered makes
no sense. Unless I forgot some basic math principles.

Filemaker rounds 2.08500 up to 2.09 when adjusting the number format from
5 to 2 decimals. This is fine.

FM will then take a number like 2.08496 and make it 2.085 when you change
the decimals from 5 to 3. All good. Then when you change the decimals
from 3 - 2 the number becomes 2.08. FM looks at the number as 2.0840
instead of 2.0849. Shouldn't FM round it up to 2.09?

2.08496 to 2.0850
2.08496 to 2.085
2.08496 to 2.08?

I guess what I'm saying, I see no consistency here.


Or do I need to go back to Math 101?

In article <Xns969B7968CC6E5wbsimonhotmailcom (AT) 209 (DOT) 242.86.10>, W Brent
Simon <wbsimonAThotmail.com> wrote:

Yep, you need to go back the math(s) class. FileMaker is correct. )

When you round a number to X decimal places you only look at the number
in the X+1 decimal place and round that digit up or down (possibly
causing a ripple effect if the next decimal place is a 9). The decimal
digits below X+1 are completely irrelevant to Round.

For your examples:

2.08496 rounded to four decimal places
the fifth decimal digit is a 6, so that is rounded UP
to 10 and because of the preceeding 9 actually bumps
up the digit before the 9 as well, giving 2.0850

2.08496 rounded to three decimal places
the fourth decimal digit is 9, so that is rounded UP
to 10, giving 2.085

2.08496 rounded to 2 decimal places
the third decimal digit is 4, so that is rounded DOWN
to 0, giving 2.08


Now, when you use two or more nested Round functions you'll get an
incorrect answer - the one you seem to be expecting.
ie.
Round(Round(2.08496, 3), 2) = Round(2.085, 2) = 2.09

This answer is mathematically incorrect, which depending on what scale
you're working in may or may not be important. For example, correctly
rounding 2.08496 to two decimal places gives 2.08, whereas your version
gives 2.09 - you're a whole 0.01 out, now 0.01mm could be nothing, but
0.01km is likely to be more important.

So, for your first example if:

C = Round(A / B, 3) and F = Round(D / E, 3)

then it depends on how G is calculated as to what answer it will give.

G = Round(C + F, 2) OR G = Round((A / B) + (D / E), 2)

can give different answers caused by the rounding error.

Normally you would use the full numbers for all calculations and only
round the final answer (in this case the second calculation is more
accurate) ... but it depends on what you're trying to do.



Helpful Harry
Hopefully helping harassed humans happily handle handiwork hardships ;o)

Helpful Harry <helpful_harry (AT) nom (DOT) de.plume.com> wrote in
news:230720051550408858%helpful_harry (AT) nom (DOT) de.plume.com:

Thanks once again Harry. I guess as I appoach 50, some things just just
get fuzzy or fade away and others just disappear altogether!

In article <Xns969E6FD0B12D8wbsimonhotmailcom (AT) 209 (DOT) 242.86.10>, W Brent
Simon <wbsimonAThotmail.com> wrote:
At 50 you're doing extremely well - most people these days seem to lose
ALL their intelligence and common sense as soon as they hit 15 or 16
years old. \


Helpful Harry
Hopefully helping harassed humans happily handle handiwork hardships ;o)