Orphan memory in KDB process using Rserve

[MuneishAdya]

Hi Everyone,

our team is one of the users of Rserve for integrating KDB with R. Recently we noticed that one of our processes working with a R server was bloating in memory. Oddly, the .Q.w did not show any bloating in memory and .Q.gc didnt release any memory either. However the OS view of the memory consumed by the process was high and different from KDB view of its own memory. 

We dug bit deeper and found that this was specifically happening in RSet function of the library.

 While we are unable to view the implementation of Rserve at https://code.kx.com/q/interfaces/r/#remote-r-rserve I believe we are missing de-allocating memory in Rset function. Can someone help in fixing this issue or provide more insight.

regards,

Muneish

[rocuinneagain]

1. Is it the embedR project that your are using? 
2. Are you on the latest release of the code? https://github.com/KxSystems/embedR/releases 
3. Is it possible share a small generic piece of code to reproduce the issue? 

[MuneishAdya]

Hi Rocuinneagain ,

1. I believe it is embedR project indeed.

2. yes we are using the latest code.

3. 

Rset:`rserve 2:(`rsset;2);

meminfo:{(5#system"w"), 1024*"J"$system "ps -eo size -h -q ",string .z.i};

m1:meminfo[];

RSet["temp";100000?100];

m2:meminfo[];

[rocuinneagain]

The src is available at https://github.com/KxSystems/embedR/blob/fa5101b64e15f9ba0aa5c20affc0cd041fb41bc0/src/rserver.c#L458

Have you tried calling

Rcmd "rm(temp)"
Rcmd "gc()"

q does not manage the memory for R - you must still delete variables and call garbage collection 

[MuneishAdya]

Currently we reassign the same variable with new data, will just calling gc resolve the bloating in that case?

[rocuinneagain]

You can delete R variables and run R garbage collection using Rcmd as above. 

[MuneishAdya]

The solution didnt work.

meminfo:{(5#system"w"), 1024*"J"$system "ps -eo size -h -q ",string .z.i};
localhost:5042>meminfo[]
3264247792 3690987520 3959422976 83886080000 0 5337485312
localhost:5042>Rset["mu";200#enlist 1000000?100]
75i
localhost:5042>meminfo[]
3264255600 3690987520 3959422976 83886080000 0 6937485312
localhost:5042>Rcmd"rm(mu)"
0i
localhost:5042>meminfo[]
3264261936 3690987520 3959422976 83886080000 0 6937485312
localhost:5042>Rcmd"gc()"
0i
localhost:5042>meminfo[]
3264266608 3690987520 3959422976 83886080000 0 6937485312
localhost:5042>Rcmd"gc()"
0i
localhost:5042>meminfo[]
3264268144 3690987520 3959422976 83886080000 0 6937485312

[rocuinneagain]

https://github.com/KxSystems/cookbook/tree/master/r/rserve

You are only looking at the memory usage of the q process.

With Rserve R is a separate process. You need to monitor it's memory usage also.

To free memory in q after creating a large object use .Q.gc[]. You can also start q with -g 1.

• https://code.kx.com/q/basics/cmdline/#-g-garbage-collection
• https://code.kx.com/q/ref/dotq/#gc-garbage-collect 

[MuneishAdya]

Also, need to add that we are using a file called as rserve.so , I believe its not part of embedR project.

[MuneishAdya]

Hi there

the memory bloating problem is with the kdb process and not the R process.

whenever the q process calls Rset there is orphan memory on the q process. 

Also, .Q.gc does not release any memory. The q process and OS have different view on the amount of memory being consumed by the q process .in the end the q process shuts down even though it thinks it has low memory foot print.