cancel
Showing results for 
Search instead for 
Did you mean: 

kdb+3.5 2017.03.15 Production Release

charlie
New Contributor II
New Contributor II
kdb+ v3.5 has now reached production status and the 32bit version is available on kx.com. Licensed customers should obtain the 64bit version from their company representatives.

kdb+ v3.5 Highlights

Enhanced Debugger
In kdb+ v3.5, the debugger has been extended to include the backtrace of the q call stack, including the current line being executed, the filename, line and character offset of code, with a visual indicator (caret) pointing to the operator which failed. The operator and arguments may be captured programmatically for further propagation in error reporting. Backtraces may also be printed at any point by inserting the .Q.bt[] command in your code. Please see http://code.kx.com/qref/reference/debug for further details.

Concurrent Memory Allocator
kdb+ v3.5 has an improved memory allocator which allows memory to be used across threads without the overhead of serialization, hence the use-cases for (multithreaded) peach now expand to include large result sets.

Socket Sharding
kdb+ v3.5 introduces a new feature that enables the use of the SO_REUSEPORT socket option, which is available in newer versions of many operating systems, including Linux (kernel version 3.9 and later). This socket option allows multiple sockets (kdb+ processes) to listen on the same IP address and port combination. The kernel then load-balances incoming connections across the processes.

When the SO_REUSEPORT option is not enabled, a single kdb+ process receives incoming connections on the socket.

With the SO_REUSEPORT option enabled, there can be multiple processes listening on an IP address and port combination. The kernel determines which available socket listener (and by implication, which process) gets the connection. This can reduce lock contention between processes accepting new connections, and improve performance on multicore systems. However, it can also mean that when a process is stalled by a blocking operation, the block affects not only connections that the process has already accepted, but also connection requests that the kernel has assigned to the process since it became blocked.

To enable the SO_REUSEPORT socket option, include the new reuseport parameter (rp) to the listen directive for the \p command, or -p cmd line arg. e.g.
 q)\p rp,5000

Use cases include coarse load-balancing and HA/failover.

N.B. when using socket sharding (e.g. -p rp,5000) the unix domain socket (uds) is not active; this is deliberate and not expected to change.

Improved sort performance
kdb+ uses a hybrid sort, selecting the algorithm it deems best for the data type, size and domain of the input. With kdb+ v3.5, this has been tweaked to significantly improve the sort performance of certain distributions, typically those including a null. e.g.
 q)a:@[10000001?100000;0;:;0N];system"t iasc a" / 5x faster than v3.4

Improved search performance
kdb+ v3.5 significantly improves the performance of bin, find, distinct and various joins for large inputs, particularly for multi-column input. The larger the data set, the better the performance improvement compared to previous versions. e.g.
 q)nn:166*n:60000;v1:50?v2:neg[100]?`2;t1:`c1`c2`c3#n?t2:([]c1:`g#nn?v1;c2:nn?v1;c3:nn?v2;val:nn?100);system"ts t1 lj 3!t2" / 100x faster than v3.4
 q)a:-1234567890 123456789,100000?10;b:1000?a;system each("ts:100 distinct a";"ts:1000 a?b") / 30% faster than v3.4

NUCs - Not Upwardly Compatible
Although we have tried to make the process of upgrading seamless, please pay attention to the following NUCs to consider whether they impact your particular installation:

added ujf (new keyword) which mimics the behaviour of uj from v2.x, i.e. that it fills from lhs. e.g.
 q)([a:1 2 3]b:2 3 7;c:10 20 30;d:"WEC")~([a:1 2]b:2 3;c:5 7;d:"WE")ujf([a:1 2 3]b:2 3 7;c:10 20 30;d:"  C")

constants limit in lambdas reduced from 96 to 95; could cause existing user code to throw 'constants error. e.g.
 q)value raze"{",(string[10+til 96],\:";"),"}"

now uses abstract namespace for unix domain sockets on linux to avoid file permission issues in /tmp.
N.B. hence 3.5 cannot connect to 3.4 using uds. e.g.
 q)hopen`:unix://5000

comments no longer stripped from the function text by the tokenizer (-4!x); they can be stripped explicitly from the -4! result with
 q){x where not(1<count each x)&x[;0]in" /\t\n"} -4!"/a comment\n{2+ 3; /another comment\n3\n\t/yet another\n \n} /and one more"

the structure of the result of value lambda, e.g. value {x+y}, is:
(bytecode;parameters;locals;(namespace,globals);constants[0];...;constants[n];m;n;f;l;s)
where
m: bytecode to source position map, -1 if position unknown
n: fully qualified (with namespace) function name as a string, set on first global assignment, with @ appended for inner lambdas. () if n/a
f: full path to the file where the function originated from, "" if n/a
l: line number in said file, -1 if n/a
s: source code
this structure is subject to change.

Suggested upgrade process
Even though we have run a wide range of tests on kdb+v3.5, and various customers have been kind enough to repeatedly run their own tests during the last few months of development, users who wish to upgrade to v3.5 should run their own tests on their own data and code/queries before promoting to production usage.
8 REPLIES 8

Flying
New Contributor III
Does this new version require use to rebuild DLLs written using C/C#/etc. from source? If so, where can we find the updated headers/libraries?

charlie
New Contributor II
New Contributor II
no, dlls/drivers which worked with 3.x will work with 3.5 without change

Flying
New Contributor III
hmm... With the new 32-bit binary, I have had two occasions where calling .Q.gc[] hangs (in a program that makes heavy use of custom functions from a DLL) this far. Guess I'll need to try and narrow it down further and see if I can create a minimalistic test case, then.

Flying
New Contributor III
Ok, here is one (hopefully) minimalistic sample of .Q.gc[] hanging once -s is enabled (TEST.zip attached).

My test environment is a WIndows 10 Home on Lenovo X260.

kdb+ version tested:
KDB+ 3.5 2017.03.15 Copyright (C) 1993-2017 Kx Systems
w32/ 4()core 4095MB test.user thinkpad-2 10.0.0.127 NONEXPIRE

Welcome to kdb+ 32bit edition
For support please see http://groups.google.com/d/forum/personal-kdbplus
Tutorials can be found at http://code.kx.com/wiki/Tutorials
To exit, type \\

To reproduce the problem (hangs at .Q.gc[]😞
q TEST -s 4

Whereas the following doesn't hang when multithreading is not enabled:
q TEST

charlie
New Contributor II
New Contributor II
thanks! this seems to be a windows only issue. We're looking into it.

charlie
New Contributor II
New Contributor II
there's a new release of the windows builds available which fixes that issue.
thanks for reporting it.

Flying
New Contributor III
May I know where this new build is hosted? Downloading again from kx.com gave me the same binary that exhibits the problem.

charlie
New Contributor II
New Contributor II
​it seems the server was caching the previous release. Please try downloading again.

thanks​