Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- KX Community
- :
- Discussion Forums
- :
- q-bies
- :
- Re: Help Understanding Precision

Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page

Options

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

2021.08.25 10:44 AM

Can you please help me understand the below and why that number is converting to 99999 at end

q)\P 0

q)79347181.71

79347181.709999993

q)\P 16

q)79347181.71

79347181.70999999

q)\P 15

q)79347181.71

79347181.71

1 ACCEPTED SOLUTION

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

2021.08.25 12:19 PM - edited 2021.08.25 12:23 PM

That number cannot be represented exactly as a floating point number i.e. it has a infinitely repeating representation (if you look at the attached pic you'll see the binary representation is infinite). Therefore, by adjusting display precision, you'll see different outputs. Some numbers can be represented as a floating point number e.g.

```
q)\P 7
q)1.125 1.126
1.125 1.126
q)\P 0
q)1.125 1.126
1.125 1.1259999999999999
```

https://www.exploringbinary.com/why-0-point-1-does-not-exist-in-floating-point/

2 REPLIES 2

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

2021.08.25 12:19 PM - edited 2021.08.25 12:23 PM

That number cannot be represented exactly as a floating point number i.e. it has a infinitely repeating representation (if you look at the attached pic you'll see the binary representation is infinite). Therefore, by adjusting display precision, you'll see different outputs. Some numbers can be represented as a floating point number e.g.

```
q)\P 7
q)1.125 1.126
1.125 1.126
q)\P 0
q)1.125 1.126
1.125 1.1259999999999999
```

https://www.exploringbinary.com/why-0-point-1-does-not-exist-in-floating-point/

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

2021.09.02 11:44 AM

Thank you for explaining this Jason!

Main Office Contacts

**EMEA**

Tel: +44 (0)28 3025 2242

**AMERICAS**

Tel: +1 (212) 447 6700

**APAC**

Tel: +61 (0)2 9236 5700

Useful Information

Resources

Popular Links

Follow Us

KX. All Rights Reserved.

KX and kdb+ are registered trademarks of KX Systems, Inc., a subsidiary of FD Technologies plc.