Understanding KDBs memory and disk utilization

[project_sifr]

I would like to confirm how KDB uses disk and memory resources.

Specifically I am after what happens when updating a single value in a KDB table (assuming the table is not already loaded into memory).

Does the whole splay table partition have to be loaded into memory first?

From my understanding the data is loaded into memory, updated in memory and written back to disk (if you want to write it to disk, otherwise it stays in memory).

Is my understanding correct?

I appreciate your responses.

[LamHinYan]

data on disk is append only

data in memory can be selectively updated

if you want to update a table on disk selectively, you have to load, update, re-write.

[SunDew]

Almost similar question - I have a huge splayed kdb table and limited memory on my production server. Now this table gets appended to daily, but on most occasions, my table reload process gets killed by the system admin due to "Out of memory" concerns on the server. Can someone clarify if this process loads the entire splayed table in the memory before re-writing, or is there any other way to simply append to a splayed table without loading it in memory?

[Rolf]

`:/path/to/splay/ upsert <new records>

note the trailing forward slash

very minimally referenced on the kx website
http://code.kx.com/wiki/JB:KdbplusForMortals/splayed_tables#1.2.4.1_Appending_Records

[SunDew]

Ohh.. right.

upsert should do the trick. 

Thank you 🙂

[LamHinYan]

If you go OOM, consider partitioning or segmenting your database. Each partition is limited to 1023KB. Partition size can be too large for mmap if you keep appending data to it.