cancel
Showing results for
Did you mean:

## Help Understanding Precision New Contributor II

Can you please help me understand the below and why that number is converting to 99999 at end

q)\P 0
q)79347181.71
79347181.709999993

q)\P 16
q)79347181.71
79347181.70999999

q)\P 15
q)79347181.71
79347181.71

1 ACCEPTED SOLUTION  New Contributor III

That number cannot be represented exactly as a floating point number i.e. it has a infinitely repeating representation (if you look at the attached pic you'll see the binary representation is infinite). Therefore, by adjusting display precision, you'll see different outputs. Some numbers can be represented as a floating point number e.g.

``````q)\P 7
q)1.125 1.126
1.125 1.126
q)\P 0
q)1.125 1.126
1.125 1.1259999999999999`````` 2 REPLIES 2  New Contributor III

That number cannot be represented exactly as a floating point number i.e. it has a infinitely repeating representation (if you look at the attached pic you'll see the binary representation is infinite). Therefore, by adjusting display precision, you'll see different outputs. Some numbers can be represented as a floating point number e.g.

``````q)\P 7
q)1.125 1.126
1.125 1.126
q)\P 0
q)1.125 1.126
1.125 1.1259999999999999``````  New Contributor II

Thank you for explaining this Jason! 