.Q.fc vs peach

[newstudent2017]

Question1

This line from .Q.fc page is confusing. “….overhead of creating 100000 threads in peach significantly outweighs….”

Why is it creating overhead of 100k threads ?  I do not see 100k mentioned anywhere. Both peach and .Q.fc where started with -s 8.

Question 2

Why is peach timing more than just “f vec” ?

q)f:{2 xexp x}

q)vec:til 100000
q)\t f peach vec
88
q)\t f vec
5
q)\t .Q.fc[f]vec
12
q)\t f each vec
122
q)\\

Links:

https://code.kx.com/wiki/DotQ/DotQDotfc

https://code.kx.com/wiki/Reference/peach

[Ciaranaquaq]

Hi,

I can see where the confusion comes from, think of f applied to each number of vec as a job, in that case there are 100000 jobs that need assigned to the slave threads. 

The reason peach is slower is because q needs to assign these jobs to the various slaves. Since the job is so simple to carry out; the act of assigning another slave thread to do it takes more time than to just calculate the exponential.

Hope this answers your question.

[newstudent2017]

Well, .Q.fc uses slave threads "if possible" and peach becomes each without slave threads. 

Still do not get the difference 😞

[rossedens2]

Hi,

.Q.fc, a.k.a. on parallel cut, will break a vector argument into n parts (equal sized), and pass each part to a slave for execution.

peach is slightly different, each value of the vector is passed to the slave thread and the value returned, which as mentioned before will maximize data transfer overhead. If you had 2 slaves for example, the first slave would be assigned items 0,2,4,6 etc and the other would be given 1,3,5,7 etc..

Along with this, more information and useful examples can be found at the following links: 

Links

https://code.kx.com/q/wp/multi_threading_in_kdb_performance_optimisations_and_use_cases.pdf

http://www.timestored.com/kdb-guides/parallel-peach/ 

Hope this helps,

Ross

[Jason_Fealy]

Yep,

We can make peach behave similarly to .Q.fc by splitting up the vector to match the number of slaves

q)\s

4i

q)f:{2 xexp x}

q)vec:til 100000

q)

q)\t .Q.fc[f;vec]

7

q)\t raze f peach vec

63

q)\t raze f peach 4 0N#vec

11

q)\t raze f peach 4 0N#vec

4

q)\t .Q.fc[f;vec]

4

q)(raze f peach 4 0N#vec)~(.Q.fc[f;vec])

1b

q)

q).Q.fc

k){[f;x]$[(#x)&1<n:"j"$."\\s";,/f':(n;0N)#x;f x]}

Jason

[seanlukemurphy]

Hey,

Just to add to Ross's answer:

Using an example of:

f:{x*x}

f peach til 10

With peach, item 0 would be assigned to thread #0 and item one to thread #1 - once a thread completes it's task, it receives the next task (e.g. thread #0 gets 2 and so on)

on the other hand with .Q.fc, 0,2,4,6,8 are all sent to thread #0 immediately, while 1,3,5,7,9 are all sent to thread #1 - this means that there only needs to be one IPC message from main thread to each slave thread, and one from each slave thread back to main thread (total 4 msgs in this example), rather than one message in both directions for every single item (20 msgs in this example).

Hope this helps,

Seán
On Wednesday, January 24, 2018 at 3:46:37 PM UTC, Science Student wrote:

Question1

This line from .Q.fc page is confusing. “….overhead of creating 100000 threads in peach significantly outweighs….”

Why is it creating overhead of 100k threads ?  I do not see 100k mentioned anywhere. Both peach and .Q.fc where started with -s 8.

Question 2

Why is peach timing more than just “f vec” ?

q)f:{2 xexp x}

q)vec:til 100000
q)\t f peach vec
88
q)\t f vec
5
q)\t .Q.fc[f]vec
12
q)\t f each vec
122
q)\\

Links:

https://code.kx.com/wiki/DotQ/DotQDotfc

https://code.kx.com/wiki/Reference/peach

[newstudent2017]

Great explanation !

Thanks everyone.

@Sean,

One clarification, in case of .Q.fc, I believe there is no interleaving of data right ? i.e. thread 0 will get 0,1,2,3,4 and thread 1 will get 5,6,7,8,9 . Wanted your thoughts on it.