How to query data in KDB that is too large to fit in memory?

[vivek_shende]

How do I setup my KDB server so that I can query data too large to fit into the memory? Aggregations or splayed tables won't be of help as the size of the minimum amount of data after applying all these techniques is also bigger than the total RAM size.

[Sam_McMillen]

For a partitioned database, the entire table doesn't need to fit in memory provided the query is formatted first by partition, then by the next most restrictive clause, and so on. Beyond that I think it would be necessary to rethink how the data is stored, depth of partitioning, etc.

[TerryLynch]

What is the actual constraint here - are you running the free/personal/32bit version of kdb and that's the limitation or are you running 64bit and are limited by available physical RAM on the machine? How much RAM are we talking about and how big is the data?

Terry

[vivek_shende]

It is actual limitation of the physical RAM. The VM I'm running has 12 GB of RAM. The actual available memory without the KDB process is around 10GB. The actual data is around 5-6 GB, but it is growing and I expect it to reach around 8GB in a year or two.

[Flying]

Is the data bieng partitioned?

[vivek_shende]

No, the data is not partitioned. It is historical data (monthly) of thousands of companies across multiple attributes. The queries on this data are usually based on a group criteria that filters and identifies comapnies that satisfy the criteria. The 'group criteria' isn't static and comes from user input. A criteria may end up selecting all the companies in the table or just a few. There is no criteria on the time and I need to get all the historical data once the companies are filtered. I don't know any partitioning strategy that could help in this case.

[cbiggs]

You could partition on enumerated integer, corresponding to the enumeration of the sym for each company. That would be viable unless you also have severe constraints on your inode. This would result in a table for each company. 

Callum Biggs | Data Scientist | Kx | +44 75 7627 4469 | cbiggs@kx.com

Kx is a division of First Derivatives

[hzadonis]

The solution is create HDB and RDB, then combine them together with Gateway.

在 2019年3月21日星期四 UTC+8下午8:02:44，vivek....@wissen.com写道：

How do I setup my KDB server so that I can query data too large to fit into the memory? Aggregations or splayed tables won't be of help as the size of the minimum amount of data after applying all these techniques is also bigger than
the total RAM size.