cancel
Showing results for
Did you mean:

## Help Understanding Precision New Contributor III

Can you please help me understand the below and why that number is converting to 99999 at end

q)\P 0
q)79347181.71
79347181.709999993

q)\P 16
q)79347181.71
79347181.70999999

q)\P 15
q)79347181.71
79347181.71

1 ACCEPTED SOLUTION New Contributor III

That number cannot be represented exactly as a floating point number i.e. it has a infinitely repeating representation (if you look at the attached pic you'll see the binary representation is infinite). Therefore, by adjusting display precision, you'll see different outputs. Some numbers can be represented as a floating point number e.g.

``````q)\P 7
q)1.125 1.126
1.125 1.126
q)\P 0
q)1.125 1.126
1.125 1.1259999999999999`````` 2 REPLIES 2 New Contributor III

That number cannot be represented exactly as a floating point number i.e. it has a infinitely repeating representation (if you look at the attached pic you'll see the binary representation is infinite). Therefore, by adjusting display precision, you'll see different outputs. Some numbers can be represented as a floating point number e.g.

``````q)\P 7
q)1.125 1.126
1.125 1.126
q)\P 0
q)1.125 1.126
1.125 1.1259999999999999``````  New Contributor III

Thank you for explaining this Jason! 