weird rounding error?

Hello. I'm getting the following odd "error" which I hope is just a display thing but really doesn't seem to be. I'm using R2016a on redhat linux.

>> format long

>> 177.5 - 170.6

ans = 6.900000000000006

>> 177.5 - 170.6 -6.9

ans = 5.329070518200751e-15

>> 1.5 - 0.6

ans = 0.900000000000000

>> 10.5 - 9.6

ans = 0.900000000000000

>> 100.5 - 99.6

ans = 0.900000000000006

>> (100.5 - 99.6) > 0.9

ans = 1

Answer 1

[David Goodmanson]

Hi Jason, this is just Computer Science 101. Since computers have only so many bits to describe floating point numbers, there are going to be differences like this (not errors), down at the last-bit level. Good that you had enough interest in the details to recognize this effect. See 'help eps'.

In IEEE 754 standard which Matlab uses, 64 bit floating point numbers have 53 bits allotted for the significant figure part. It's worthwhile to go to Wikipedia for a summary of IEEE 754.

[James Tursa]

E.g., some of the exact decimal conversions involved above:

>> num2strexact(177.5)
ans =
1.775e2
>> num2strexact(177.6)
ans =
1.77599999999999994315658113919198513031005859375e2
>> num2strexact(6.9)
ans =
6.9000000000000003552713678800500929355621337890625
>> num2strexact(177.5 - 170.6 -6.9)
ans =
5.3290705182007513940334320068359375e-15
>> num2strexact(1.5 - 0.6)
ans =
0.90000000000000002220446049250313080847263336181640625
>> num2strexact(10.5 - 9.6)
ans =
0.9000000000000003552713678800500929355621337890625
>> num2strexact(100.5 - 99.6)
ans =
0.900000000000005684341886080801486968994140625

Answer 2

[Jason]

While I am aware of the fact that we are representing decimals with a finite number of bits the following behavior certainly makes life miserable. I don't expect this type of thing with simple subtraction only to the tenths place that any generic calculator can handle.

>> (100.5 - 99.6) == 0.9

ans = 0

>> (10.5 - 9.6) == 0.9

ans = 0

>> (1.5 - 0.6) == 0.9

ans = 1

I guess I have to accept the answer if not the matlab behavior.

[James Tursa]

This is not "MATLAB" behavior, per se ... it is floating point arithmetic behavior that you would encounter on any machine using binary floating point to represent the numbers. You will run into it with other languages as well.

For calculators, any particular calculator might be using binary floating point with a different number of mantissa bits, or it might be using binary coded decimal. So yes the results would not necessarily be expected to compare to MATLAB.

[Roger Stafford]

@Jason: You should be aware of the fact that what you call “matlab behavior” is not really caused by matlab, but rather by the computer you are using to run matlab. Its floating point numbers are designed to use binary numbers, not decimal numbers, and as such it cannot represent most decimal fractions exactly. In particular it cannot represent your fractions, 99.6, 0.6, 0.9 exactly. (This is apparently due to the inefficiency in constructing electronic ten-state circuits, as opposed to two-state circuits, so you can lay the blame on computer engineers.) However, even on a truly decimal machine if you wanted to test, say, the equality

   3/14 + (3/14 + 15/14) == (3/14 + 3/14) + 15/14

it is likely the computer would give “false” as its answer, and thereby violate the associative law of addition. Try it on your favorite decimal calculator. The lesson to be learned here is that one should be very cautious in using the equality test for floating point numbers, since it requires exact equality, and even a one-bit difference in the least bit will fail the test.

Answer 3

[David Goodmanson]

Yes, it's interesting that for some tasks such as addition and subtraction of quantities with a fixed number of decimal digits, a $10 calculator does a better job in a sense than does expensive scientific software. But as you alluded, one is stuck with the rules of the game.