cancel
Showing results for 
Search instead for 
Did you mean: 

What did -11!log do underlying

Xiaocheng
New Contributor II

Hi Community,

 

I tried to replay a tickerplant file (~12GB), with -11!logfile.

After done, I tried .Q.w[] and saw iterms, used ~18GB, heap ~33GB.

Then I called .Q.gc[], used still ~18GB, heap ~18GB, 15GB space were freed after gc.

To better manage the memory used, my questions are:

1. What did -11!logfile do underlying, looks like it used extra 15GB mem, can we optimize this?

2. Is there any auto gc function/procedure, or only available for manual .Q.gc[] calls.

 

Thanks,

Xiaocheng

 

1 ACCEPTED SOLUTION

Hello,

Returning with some feedback; the -11! command itself is not consuming large amounts of memory. It simply reads each line of the log file and applies 'value' to it. So the memory will be consumed by what happens when value is applied. It's usually something simple like an insert or update to a table but it can be more complex leading to more memory usage.

Secondly, you can set garbage collection to immediate by using 

system"g 1"

It is to an extent automatic but it doesn't have the full effect as the manually invoked .Q.gc[]. Here is some documentation with more information; https://code.kx.com/q/basics/syscmds/#g-garbage-collection-mode and https://code.kx.com/q4m3/13_Commands_and_System_Variables/#13110-garbage-collection-g.

I hope this helps.

Many Thanks

Luke

View solution in original post

3 REPLIES 3

leades
Moderator Moderator
Moderator

Hello,

Great questions for the Community! Here is some more documentation about replaying log files that might be useful, https://code.kx.com/q/kb/replay-log/. I have reached out internally for a more detailed response.

Many Thanks

Luke

Hello,

Returning with some feedback; the -11! command itself is not consuming large amounts of memory. It simply reads each line of the log file and applies 'value' to it. So the memory will be consumed by what happens when value is applied. It's usually something simple like an insert or update to a table but it can be more complex leading to more memory usage.

Secondly, you can set garbage collection to immediate by using 

system"g 1"

It is to an extent automatic but it doesn't have the full effect as the manually invoked .Q.gc[]. Here is some documentation with more information; https://code.kx.com/q/basics/syscmds/#g-garbage-collection-mode and https://code.kx.com/q4m3/13_Commands_and_System_Variables/#13110-garbage-collection-g.

I hope this helps.

Many Thanks

Luke

View solution in original post

Xiaocheng
New Contributor II

Thanks!