2021.10.11 08:55 PM
Hi,
A bit puzzled here... Say I have a table comprised of 2 columns, with the first one made of variable length lists, and the second showing a binary:
t:([] a:{x?50} each 2+100000?100; b:100000?0b)
I want to apply a specific function to each list found in t`a, depending on the binary t`b. In this example, let's simply use <first> and <last> as dependant functions.
Below are 3 possible syntax versions and their time/memory consumption. I would have expected the <scan> version to beat the 2 others by a substantial factor, at least on timing (with +/- impact on memory). The reality seems to be the opposite:
Cond$ with Each
q)\ts {$[x`b; last x`a; first x`a]} each t
40 3697968
Cond? with Each within function
q)\ts {?[x`b; last each x`a; first each x`a]}t
16 4746672
Scan
q)\ts ({?[y`b; last; first] y`a}\)[::;t]
62 12746416
Is there an issue with my wording of the scan version (4x slower, 2.5x memory hungry), or is the second one the actual optimal solution? As usual, your insight is highly appreciated. Thx.
2021.10.12 02:57 AM - edited 2021.10.12 08:18 AM
This is not really a suitable application of scan. Scan is an accumulator where it is useful when the calculation of a subsequent calculation depends of the result of the previous calculation. This has an overhead as the calculation is computed item by item and each result must be passed back in to the next calculation. As you never use the variable 'x' inside the scan it is an indication it is not the best use-case. This blog has some visualisations which aim to show how scan functions internally.
Your second version is fastest as it operates on 'x`b' as a vector rather than inside 'each'.
One other possible variation is shown below:
q)\ts {?[x`b; last each x`a; first each x`a]}t
16 4746672
q)\ts {((first;last) x`b)@'x`a}t
7 4746640
It's goal is to avoid calculating 'last each' and 'first each' for every row.
Instead it uses each both (') to apply first or last after it is known which function is needed.
2021.10.12 02:57 AM - edited 2021.10.12 08:18 AM
This is not really a suitable application of scan. Scan is an accumulator where it is useful when the calculation of a subsequent calculation depends of the result of the previous calculation. This has an overhead as the calculation is computed item by item and each result must be passed back in to the next calculation. As you never use the variable 'x' inside the scan it is an indication it is not the best use-case. This blog has some visualisations which aim to show how scan functions internally.
Your second version is fastest as it operates on 'x`b' as a vector rather than inside 'each'.
One other possible variation is shown below:
q)\ts {?[x`b; last each x`a; first each x`a]}t
16 4746672
q)\ts {((first;last) x`b)@'x`a}t
7 4746640
It's goal is to avoid calculating 'last each' and 'first each' for every row.
Instead it uses each both (') to apply first or last after it is known which function is needed.
2021.10.12 08:04 AM
Thank you for both the explanation and the more efficient <each both> alternative... Best.
EMEA
Tel: +44 (0)28 3025 2242
AMERICAS
Tel: +1 (212) 447 6700
APAC
Tel: +61 (0)2 9236 5700
KX. All Rights Reserved.
KX and kdb+ are registered trademarks of KX Systems, Inc., a subsidiary of FD Technologies plc.