Replay old log file to hdb

[joshmyzie2]

Hi,Say I have a standard kdb+/tick setup and for whatever reason mytickerplant and rdb are shut down before the end of the day and broughtback up after the new day has started.So, I will have a sym2015.07.09 file, but because it is now 2015.07.10and I missed the end-of-day hdb write, my 2015.07.09 data will never getwritten to the hdb.What's the recommended way to load the log file and save all of thetables to the hdb?Thanks,Josh

[pressjonny0]

Hi Josh

The usual way of doing it is to use -11! to replay the log file.  A basic script would be something like 

// load schema

\l schema.q

// define upd

upd:insert

// replay log file 

-11!`:schema2015.07.09

// save

.Q.hdpf[`::;`:hdb;2015.07.09;`sym]

This will read the full log file into memory.  So you will need to have RAM available. 

TorQ has a TP log replay script: 

https://github.com/AquaQAnalytics/TorQ/blob/master/code/processes/tickerlogreplay.q

which allows a bit more flexibility.  For example, you can replay the tp log file in chunks (so less memory usage), ignore certain tables, replay messages in a range, replay batches of log files etc. 

Thanks 

Jonny