top | item 27711568 (no title) slver | 4 years ago Excel rounds doubles to 15 digits for display and comparison. The exact precision of doubles is something like 15.6 digits, those remaining 0.6 digits causing some of those examples floating (heh) around. discuss order hn newest okl|4 years ago That depends https://randomascii.wordpress.com/2012/03/08/float-precision... slver|4 years ago A lot of these edge cases are about theoretical concerns like "how many digits we need in decimal to represent an exact IEEE binary float".In practice a double is 15.6 digits precise, which Excel rounds to 15 to eliminate some weirdness.In their documentation they do cite their number type as 15 digit precision type. Ergo that's the semantic they've settled on.
okl|4 years ago That depends https://randomascii.wordpress.com/2012/03/08/float-precision... slver|4 years ago A lot of these edge cases are about theoretical concerns like "how many digits we need in decimal to represent an exact IEEE binary float".In practice a double is 15.6 digits precise, which Excel rounds to 15 to eliminate some weirdness.In their documentation they do cite their number type as 15 digit precision type. Ergo that's the semantic they've settled on.
slver|4 years ago A lot of these edge cases are about theoretical concerns like "how many digits we need in decimal to represent an exact IEEE binary float".In practice a double is 15.6 digits precise, which Excel rounds to 15 to eliminate some weirdness.In their documentation they do cite their number type as 15 digit precision type. Ergo that's the semantic they've settled on.
okl|4 years ago
slver|4 years ago
In practice a double is 15.6 digits precise, which Excel rounds to 15 to eliminate some weirdness.
In their documentation they do cite their number type as 15 digit precision type. Ergo that's the semantic they've settled on.