cancel
Showing results for 
Search instead for 
Did you mean: 

Help Understanding Precision

ekallivrousis
New Contributor II

Can you please help me understand the below and why that number is converting to 99999 at end

 

q)\P 0
q)79347181.71
79347181.709999993


q)\P 16
q)79347181.71
79347181.70999999


q)\P 15
q)79347181.71
79347181.71

 

 

1 ACCEPTED SOLUTION

Jason_Fealy
New Contributor III
New Contributor III

That number cannot be represented exactly as a floating point number i.e. it has a infinitely repeating representation (if you look at the attached pic you'll see the binary representation is infinite). Therefore, by adjusting display precision, you'll see different outputs. Some numbers can be represented as a floating point number e.g.

 

 

q)\P 7
q)1.125 1.126
1.125 1.126
q)\P 0
q)1.125 1.126
1.125 1.1259999999999999

 

 

 repeating.JPG

https://www.exploringbinary.com/why-0-point-1-does-not-exist-in-floating-point/ 

View solution in original post

2 REPLIES 2

Jason_Fealy
New Contributor III
New Contributor III

That number cannot be represented exactly as a floating point number i.e. it has a infinitely repeating representation (if you look at the attached pic you'll see the binary representation is infinite). Therefore, by adjusting display precision, you'll see different outputs. Some numbers can be represented as a floating point number e.g.

 

 

q)\P 7
q)1.125 1.126
1.125 1.126
q)\P 0
q)1.125 1.126
1.125 1.1259999999999999

 

 

 repeating.JPG

https://www.exploringbinary.com/why-0-point-1-does-not-exist-in-floating-point/ 

View solution in original post

Thank you for explaining this Jason!