calculation error or double precision limit?
9 views (last 30 days)
Show older comments
I was wondering about the result when calculating following simple formula:
K>> 0.067 * 1e9
Following gives the correct result (integer):
K>> 0.067 * 1e8 * 1e1
Where is the difference in the calcualation?
Is it an error or a double precision limit?
Stephen23 on 17 Sep 2015
Edited: Stephen23 on 17 Sep 2015
You should download this and give it a try with the same examples as above:
Also have a look at the intermediate values, such as 0.067 * 1e8. This shows that there are a few digits that creep in after the end of the double precision limit (fifteen digits).
Find more on Data Type Conversion in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!Start Hunting!