Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- KX Community
- :
- Discussion Forums
- :
- kdb+ and q
- :
- Re: Timing/Memory surprise on 3 versions of <Condi...

Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page

Options

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

2021.10.11 08:55 PM

Hi,

A bit puzzled here... Say I have a table comprised of 2 columns, with the first one made of variable length lists, and the second showing a binary:

t:([] a:{x?50} each 2+100000?100; b:100000?0b)

I want to apply a specific function to each list found in t`a, depending on the binary t`b. In this example, let's simply use <first> and <last> as dependant functions.

Below are 3 possible syntax versions and their time/memory consumption. I would have expected the <scan> version to beat the 2 others by a substantial factor, at least on timing (with +/- impact on memory). The reality seems to be the opposite:

Cond$ with Each

q)\ts {$[x`b; last x`a; first x`a]} each t

40 3697968

Cond? with Each within function

q)\ts {?[x`b; last each x`a; first each x`a]}t

16 4746672

Scan

q)\ts ({?[y`b; last; first] y`a}\)[::;t]

62 12746416

Is there an issue with my wording of the scan version (4x slower, 2.5x memory hungry), or is the second one the actual optimal solution? As usual, your insight is highly appreciated. Thx.

1 ACCEPTED SOLUTION

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

2021.10.12 02:57 AM - edited 2021.10.12 08:18 AM

This is not really a suitable application of scan. Scan is an accumulator where it is useful when the calculation of a subsequent calculation depends of the result of the previous calculation. This has an overhead as the calculation is computed item by item and each result must be passed back in to the next calculation. As you never use the variable 'x' inside the scan it is an indication it is not the best use-case. This blog has some visualisations which aim to show how scan functions internally.

Your second version is fastest as it operates on 'x`b' as a vector rather than inside 'each'.

One other possible variation is shown below:

```
q)\ts {?[x`b; last each x`a; first each x`a]}t
16 4746672
q)\ts {((first;last) x`b)@'x`a}t
7 4746640
```

It's goal is to avoid calculating 'last each' and 'first each' for every row.

Instead it uses each both (') to apply first or last after it is known which function is needed.

2 REPLIES 2

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

2021.10.12 02:57 AM - edited 2021.10.12 08:18 AM

This is not really a suitable application of scan. Scan is an accumulator where it is useful when the calculation of a subsequent calculation depends of the result of the previous calculation. This has an overhead as the calculation is computed item by item and each result must be passed back in to the next calculation. As you never use the variable 'x' inside the scan it is an indication it is not the best use-case. This blog has some visualisations which aim to show how scan functions internally.

Your second version is fastest as it operates on 'x`b' as a vector rather than inside 'each'.

One other possible variation is shown below:

```
q)\ts {?[x`b; last each x`a; first each x`a]}t
16 4746672
q)\ts {((first;last) x`b)@'x`a}t
7 4746640
```

It's goal is to avoid calculating 'last each' and 'first each' for every row.

Instead it uses each both (') to apply first or last after it is known which function is needed.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

2021.10.12 08:04 AM

Thank you for both the explanation and the more efficient <each both> alternative... Best.

Main Office Contacts

**EMEA**

Tel: +44 (0)28 3025 2242

**AMERICAS**

Tel: +1 (212) 447 6700

**APAC**

Tel: +61 (0)2 9236 5700

Useful Information

Resources

KX. All Rights Reserved.

KX and kdb+ are registered trademarks of KX Systems, Inc., a subsidiary of FD Technologies plc.