cancel
Showing results for 
Search instead for 
Did you mean: 

ja voids ref-counter

AE
New Contributor
Hello,

Could someone please explain why it happens?

It is unclear why adding element to the vector voids reference counter.

Also strange, that it does not happen when ktn(KI, 1)

int main() {
  K l = ktn(KI, 0);
  int i = 10;
  r1(l);
  printf("%d\n"l->r); // 1
  ja(&l, &i);
  printf("%d\n"l->r); // 0
}

Thank you,
  Alexander.

16 REPLIES 16

mousepolice
New Contributor
> Could someone please explain why it happens?

reallocation?

> printf("%p %lld %d\n", l, l->n, l->r);
0x10e500000 0 1
0x10e500020 1 0

charlie
New Contributor II
New Contributor II

Initialise the kdb+ memory system

Before calling any 'generator' functions in a standalone application, you must initialise the kdb+ internal memory system. (It is done automatically when you open a connection to other kdb+ processes.) Without making a connection, use khp("",-1);


AE
New Contributor
Charles,
I did it, but I lost it in the lines, but it did not change the ref-counter behavior.

Regards,
  Alexander.

charlie
New Contributor II
New Contributor II
khp("",-1) is essential to avoid further issues.

>From that page

"Be aware that if a reference count is >0, you should very likely not change the data stored in that object as it is being referenced by another piece of code which may not expect the change. In this case, create a new copy of the object, and change that."

I think we need to make this clearer in the docs that joins (ja,jk,..) assume a ref count of 0. I can't think of a scenario where you would want to suffer your observations - appending to a common object until a copy is forced at a capacity threshold.

charlie
New Contributor II
New Contributor II
ahah. It is already documented at


To join

an atom to a list: K ja(K*,V*);
a string to a list: K js(K*,S);
another K object to a list: K jk(K*,K);
another K list to the first: K jv(K*,K);
The join functions assume there are no other references to the list, as the list may need to be reallocated during the call. 



AE
New Contributor
Charles, Thank you for clarification,

and thank you to Antonio, who helped to investigate the behavior yesterday.

Looks like it is necessary to very carefully using RC of the struct and do some kind of control of relocation with manual copy of the rc (if necessary).

Regards,
  Alexander.

AE
New Contributor
Charles,

Probably, if I would apply copy-of-change logic than it seems more clear to me. Except one thing:

K top = knk(0, 0)
K v = ktn(KI, 0)
ja(&v, 1)
jk(&top, v)
// right now top has pointer to v and its value
ja(&v, 2)
ja(&v, 3)
ja(&v, 4)
ja(&v, 5)
// relocation happened -> top has pointer to v with 1,2,3,4 elements

But, in Q it pass values, and looks like there is internal copy/relocation after ja(&v, 2), or probably during jk().

Question: how to emulate the following Q behavior in the C-code above?:
v:6h$()
v,:1
l:()
l,:enlist v  // v copied here
v,:2          // ...  or here

Regards,
  Alexaner.


charlie
New Contributor II
New Contributor II
>Question: how to emulate the following Q behavior in the C-code above?:
v:6h$()
v,:1
l:()
l,:enlist v  // v undergoes r1(v) here
v,:2          // copy on write happens here

btw, you can observe ref counts with -16!x. Bear in mind that the act of reading x increments its ref count.

q)-16!0
1i
q)-16!0+1
0i

AE
New Contributor
Charles,

I mean that the copy-on-write does not happens in C-code, are there any easy way to make it in C-API, except manually?

BTW: Do I understand correct that the copy-on-write does not happen all the times?, because another way we would have a copy-on-write every ,:, which is quite heavy. And we would have copy-on-write every insertion a new row to a table, and the table does not loose the links to the vector.

Regards,

AE
New Contributor
Charles,

One more question:
    int i = 10;
    K top1 = ktn(0, 0);
    K v = ktn(KI, 0);

    jk(&top1, r1(v));
    printf("%p 0:%p v:%p\n", top1, kK(top1)[0], v);

    ja(&v, &i);
    printf("%p 0:%p v:%p\n", top1, kK(top1)[0], v);

0x7f9d17100020 0:0x7f9d17100010 v:0x7f9d17100010
0x7f9d17100020 0:0x7f9d17100010 v:0x7f9d17100040

We see here copy-on-change, everything looks ok, except that we lost initial top1 - how to know about it in C code and call r0 on it?

And more important question: this is part of table's structure, but looks like adding to the table do not cause copy-on-write every change + mixed list does not loose pointer to relocated structure - which is also not behavior of the example.

charlie
New Contributor II
New Contributor II
Any realloc inside a join is not due to ref count>0, it is due to reaching capacity thresholds.
Bottom line: do not call join functions with a ref count > 0.
I recommend you try a different approach that does not require sharing objects. E.g.

K top1=knk(1,ktn(KI,0));
ja(&kK(top1)[0], &i);

or build the vectors first and when complete, finally add them to top1.

AE
New Contributor
But vec's of the table always have ref count > 0. Does it mean that we cannot insert data into table?

Regards,
  Alexander.

charlie
New Contributor II
New Contributor II
no, they don't always have a ref count > 0.
Perhaps it is better you explain the problem you are trying to solve?

AE
New Contributor
I am trying to have table in C-code, which I can extend dynamically. And cleanup it from memory correct with all copies after usage.

Regards,

charlie
New Contributor II
New Contributor II
ok, so just create the table, ensure that nothing references the elements inside it directly other than the table itself, such that all elements in the table have ref count 0. e.g.

 J j=42;
 K names=ktn(KS,2);kS(names)[0]=ss("col1");kS(names)[1]=ss("col2");
 K t=xT(xD(names,knk(2,ktn(KS,0),ktn(KJ,0))));

Then, if you want to reference the table from some other code for whatever reason, use

 K t1=r1(t);

you can still append directly to the vectors:

 js(&kK((kK(t->k)[1]))[0],ss("row1"));
 ja(&kK((kK(t->k)[1]))[1],&j);

or if you like 

 K*col1=&kK((kK(t->k)[1]))[0];
 K*col2=&kK((kK(t->k)[1]))[1];
 js(col1,ss("row1"));
 ja(col2,&j);

when you have finished, just r0(t) and r0(t1).

If you want t1 to reference t and then also be able to append to t without updating t1 (i.e. copy on write), you will have to manually copy t and the contents into a new table prior to appending to t.

Alternatively, if you are writing a shared library to be loaded into kdb+, you can lean on q to do this via k(0,....).

hth,
Charlie

AE
New Contributor
Antonio, Thank you.


But I do not understand how it is possible to use r1 and r0 if you can loose it any moment.

Regards,
  Alexander.