cancel
Showing results for 
Search instead for 
Did you mean: 

How to efficiently store historical data registered in memory(RDB).

k-mambo
New Contributor III

Hello,

I am using Tickplant, but I wonder if there is an efficient way to store historical data in memory in the correct partition as historical data should be registered in real-time memory.
You are using a non-date time partition, and after you go down to the HDB, you are distributing historical data in that partition to each partition in a separate process.

Is there a way to do it efficiently and quickly?...

2 REPLIES 2

davidcrossey
Moderator Moderator
Moderator

Hi k-mambo,

Do you mean if the data in the RDB contains data that has e.g. timestamps from multiple different dates, you'd want to save those down to their respective date partitions?

If yes, you'd need to customize your RDB's .u.end function .

For example, instead of calling .Q.hdft (as default in r.q), you could instead write some logic to determine which dates you have for each table, then upsert on disk for each pair.

You'd also need to handle enumeration, sorting, compression (where applicable) and adding attributes on disk post operation.

To get a bit of an idea, check out some of the logic in w.q as mentioned here

k-mambo
New Contributor III

If you have a lot of days to save and it takes a lot of time, is the only way to increase disk performance?.

I recognize that it does not fit the KDB TICK concept that a lot of past data is registered at the current time.

Thank you.