Showing results for 
Search instead for 
Did you mean: 

How to efficiently store historical data registered in memory(RDB).

New Contributor III


I am using Tickplant, but I wonder if there is an efficient way to store historical data in memory in the correct partition as historical data should be registered in real-time memory.
You are using a non-date time partition, and after you go down to the HDB, you are distributing historical data in that partition to each partition in a separate process.

Is there a way to do it efficiently and quickly?...


Moderator Moderator

Hi k-mambo,

Do you mean if the data in the RDB contains data that has e.g. timestamps from multiple different dates, you'd want to save those down to their respective date partitions?

If yes, you'd need to customize your RDB's .u.end function .

For example, instead of calling .Q.hdft (as default in r.q), you could instead write some logic to determine which dates you have for each table, then upsert on disk for each pair.

You'd also need to handle enumeration, sorting, compression (where applicable) and adding attributes on disk post operation.

To get a bit of an idea, check out some of the logic in w.q as mentioned here

New Contributor III

If you have a lot of days to save and it takes a lot of time, is the only way to increase disk performance?.

I recognize that it does not fit the KDB TICK concept that a lot of past data is registered at the current time.

Thank you.