What did -11!log do underlying

[Xiaocheng]

Hi Community,

I tried to replay a tickerplant file (~12GB), with -11!logfile.

After done, I tried .Q.w[] and saw iterms, used ~18GB, heap ~33GB.

Then I called .Q.gc[], used still ~18GB, heap ~18GB, 15GB space were freed after gc.

To better manage the memory used, my questions are:

1. What did -11!logfile do underlying, looks like it used extra 15GB mem, can we optimize this?

2. Is there any auto gc function/procedure, or only available for manual .Q.gc[] calls.

Thanks,

Xiaocheng

[leades]

Hello,

Great questions for the Community! Here is some more documentation about replaying log files that might be useful, https://code.kx.com/q/kb/replay-log/. I have reached out internally for a more detailed response.

Many Thanks

Luke

[leades · Accepted solution]

Hello,

Returning with some feedback; the -11! command itself is not consuming large amounts of memory. It simply reads each line of the log file and applies 'value' to it. So the memory will be consumed by what happens when value is applied. It's usually something simple like an insert or update to a table but it can be more complex leading to more memory usage.

Secondly, you can set garbage collection to immediate by using 

system"g 1"

It is to an extent automatic but it doesn't have the full effect as the manually invoked .Q.gc[]. Here is some documentation with more information; https://code.kx.com/q/basics/syscmds/#g-garbage-collection-mode and https://code.kx.com/q4m3/13_Commands_and_System_Variables/#13110-garbage-collection-g.

I hope this helps.

Many Thanks

Luke

[Xiaocheng]

Thanks!