ja voids ref-counter

[AE]

Hello,

Could someone please explain why it happens?

It is unclear why adding element to the vector voids reference counter.

Also strange, that it does not happen when ktn(KI, 1)

int main() {

  K l = ktn(KI, 0);

  int i = 10;

  r1(l);

  printf("%d\n", l->r); // 1

  ja(&l, &i);

  printf("%d\n", l->r); // 0

}

Thank you,

  Alexander.

[mousepolice]

> Could someone please explain why it happens?

reallocation?

> printf("%p %lld %d\n", l, l->n, l->r);

0x10e500000 0 1

0x10e500020 1 0

[charlie]

https://code.kx.com/q/interfaces/c-client-for-q/#managing-memory-and-reference-counting

Initialise the kdb+ memory system

Before calling any 'generator' functions in a standalone application, you must initialise the kdb+ internal memory system. (It is done automatically when you open a connection to other kdb+ processes.) Without making a connection, use khp("",-1);

[AE]

Charles,

I did it, but I lost it in the lines, but it did not change the ref-counter behavior.

Regards,

  Alexander.

[charlie]

khp("",-1) is essential to avoid further issues.

>From that page

"Be aware that if a reference count is >0, you should very likely not change the data stored in that object as it is being referenced by another piece of code which may not expect the change. In this case, create a new copy of the object, and change that."

I think we need to make this clearer in the docs that joins (ja,jk,..) assume a ref count of 0. I can't think of a scenario where you would want to suffer your observations - appending to a common object until a copy is forced at a capacity threshold.

[charlie]

ahah. It is already documented at

https://code.kx.com/q/interfaces/c-client-for-q/

To join

an atom to a list: K ja(K*,V*);
a string to a list: K js(K*,S);
another K object to a list: K jk(K*,K);
another K list to the first: K jv(K*,K);
The join functions assume there are no other references to the list, as the list may need to be reallocated during the call. 

We'll add this to https://code.kx.com/q/interfaces/capiref/#ja-join-value etc.

[AE]

Charles, Thank you for clarification,

and thank you to Antonio, who helped to investigate the behavior yesterday.

Looks like it is necessary to very carefully using RC of the struct and do some kind of control of relocation with manual copy of the rc (if necessary).

Regards,

  Alexander.

[AE]

Charles,

Probably, if I would apply copy-of-change logic than it seems more clear to me. Except one thing:

K top = knk(0, 0)

K v = ktn(KI, 0)

ja(&v, 1)

jk(&top, v)

// right now top has pointer to v and its value

ja(&v, 2)

ja(&v, 3)

ja(&v, 4)

ja(&v, 5)

// relocation happened -> top has pointer to v with 1,2,3,4 elements

But, in Q it pass values, and looks like there is internal copy/relocation after ja(&v, 2), or probably during jk().

Question: how to emulate the following Q behavior in the C-code above?:

v:6h$()

v,:1

l:()

l,:enlist v  // v copied here

v,:2          // ...  or here

Regards,

  Alexaner.

[charlie]

>Question: how to emulate the following Q behavior in the C-code above?:

v:6h$()

v,:1

l:()

l,:enlist v  // v undergoes r1(v) here

v,:2          // copy on write happens here

btw, you can observe ref counts with -16!x. Bear in mind that the act of reading x increments its ref count.

q)-16!0
1i
q)-16!0+1
0i

[AE]

Charles,

I mean that the copy-on-write does not happens in C-code, are there any easy way to make it in C-API, except manually?

BTW: Do I understand correct that the copy-on-write does not happen all the times?, because another way we would have a copy-on-write every ,:, which is quite heavy. And we would have copy-on-write every insertion a new row to a table, and the table does not loose the links to the vector.

Regards,

[AE]

Charles,

One more question:

    int i = 10;

    K top1 = ktn(0, 0);

    K v = ktn(KI, 0);

    jk(&top1, r1(v));

    printf("%p 0:%p v:%p\n", top1, kK(top1)[0], v);

    ja(&v, &i);

    printf("%p 0:%p v:%p\n", top1, kK(top1)[0], v);

0x7f9d17100020 0:0x7f9d17100010 v:0x7f9d17100010

0x7f9d17100020 0:0x7f9d17100010 v:0x7f9d17100040

We see here copy-on-change, everything looks ok, except that we lost initial top1 - how to know about it in C code and call r0 on it?

And more important question: this is part of table's structure, but looks like adding to the table do not cause copy-on-write every change + mixed list does not loose pointer to relocated structure - which is also not behavior of the example.

[charlie]

Any realloc inside a join is not due to ref count>0, it is due to reaching capacity thresholds.

Bottom line: do not call join functions with a ref count > 0.

I recommend you try a different approach that does not require sharing objects. E.g.

K top1=knk(1,ktn(KI,0));

ja(&kK(top1)[0], &i);

or build the vectors first and when complete, finally add them to top1.

[AE]

But vec's of the table always have ref count > 0. Does it mean that we cannot insert data into table?

Regards,

  Alexander.

[charlie]

no, they don't always have a ref count > 0.

Perhaps it is better you explain the problem you are trying to solve?

[AE]

I am trying to have table in C-code, which I can extend dynamically. And cleanup it from memory correct with all copies after usage.

Regards,

[charlie]

ok, so just create the table, ensure that nothing references the elements inside it directly other than the table itself, such that all elements in the table have ref count 0. e.g.

 J j=42;
 K names=ktn(KS,2);kS(names)[0]=ss("col1");kS(names)[1]=ss("col2");
 K t=xT(xD(names,knk(2,ktn(KS,0),ktn(KJ,0))));

Then, if you want to reference the table from some other code for whatever reason, use

 K t1=r1(t);

you can still append directly to the vectors:

 js(&kK((kK(t->k)[1]))[0],ss("row1"));
 ja(&kK((kK(t->k)[1]))[1],&j);

or if you like 

 K*col1=&kK((kK(t->k)[1]))[0];
 K*col2=&kK((kK(t->k)[1]))[1];
 js(col1,ss("row1"));
 ja(col2,&j);

when you have finished, just r0(t) and r0(t1).

If you want t1 to reference t and then also be able to append to t without updating t1 (i.e. copy on write), you will have to manually copy t and the contents into a new table prior to appending to t.

Alternatively, if you are writing a shared library to be loaded into kdb+, you can lean on q to do this via k(0,....).

hth,

Charlie

[AE]

Antonio, Thank you.

But I do not understand how it is possible to use r1 and r0 if you can loose it any moment.

Regards,

  Alexander.