cancel
Showing results for 
Search instead for 
Did you mean: 

Efficient `over` with a large state? (in-place state updates)

My_Phone
New Contributor
How can I efficiently iterate over some values while making small updates to a large state vector (in my actual application it's a dictionary).

For example, if I use `over` it's very slow because the entire state vector is copied at each iteration:
q)n:prd 7#10
q)\ts {x[y]:rand 1f; x}/[n#1f;100?n]
2108 402655360

But if I were to use a global state (what I'm trying to avoid), it's much more efficient:
q)xx:n#1f
q)\ts {xx[x]:rand 1f}'[100?n]
0 2912

Thanks,
John
6 REPLIES 6

osvaldo_fuica
New Contributor
Hi John, 

You can store your large state vector in the global state and then update it in place.

q)n:prd 7#10
q)`xx set n#1f
q)\ts @[;;:;rand 1f]/[`xx;100?n]
0 1872

Best, 
Osvaldo


ajay
New Contributor II
No need to create the projection in this case


q)\ts @/[`xx;100?n;:;rand 1f]
0 1648



-Ajay

If you can generate the values you'd like to update to separately then it'd be easier to use replace as demonstrated in https://code.kx.com/q/ref/amend/#replacement

q)n:prd 7#10
q)v:n#1f  // the initial state of your vector
q)indx:100?n // the indices to update
q)nv:count[indx]?1f // new values
q)\ts @[v;indx;:;nv]
//11 134219232


q)\ts {@[x;y;:;rand 1f]}/[`xx;100?n]
0 2160
q)count distinct xx
101

Any other idea with over?

Osvaldo

Thanks for the suggestions, but I should have clarified two points earlier:

1. The actual calculation is path-dependent and non-trivial, so the solution should work for any arbitrary function of the current state and an element of the iterated vector.
2. I want to avoid updating globals because ultimately I want to be able to run this calculation many times in parallel with peach.  And AFAIK, it's not possible to use thread local globals directly from q.

I think you are not including the creation of the variable. when you make it a global. When you do your timings.
Making it hard to see what your really comparing and wanting to optimize.
Using global will reduce memory but dont think there is much difference in speed.

q)\ts {x[y]:rand 1f; x}/[n#1f;100?n]
1117 402655360
q)\ts xx:n#1f ; {x[y]:rand 1f; x}/[xx;100?n]
1168 268437936