The perils of benchmarking

[joelr1]

Mime-Version: 1.0 (Apple Message framework v919.2)Subject: The perils of benchmarkingDate: Sun, 6 Apr 2008 15:38:18 +0100X-Mailer: Apple Mail (2.919.2)Suppose we had a list of 10000000phone numbers and wanted to takejust the first 8 digits of each.We could create the list like this:q)l:10000000 10#99?"0123456789"Here, 99?... creates a vector of 99random characters from the set [0-9]and 10#... takes 10 characters fromeach generated "phone number".10000000 ... repeats it that many times.A simple 8#l will just gives us the first8 elements of the list which is not whatwe want.q)f1:{x@\:til 8}q)f2:{8#/:x}q)f3:{8#'x}The above 3 solutions will give us thefirst 8 digits of every phone numberin the list.Which solution is the fastest, though?The list is very very long specificallyto make benchmarking easier.q)\t f1 l906\t here gives us execution time for f1in milliseconds. We proceed to time f2and f3.q)\t f2 l738q)\t f3 l738Looks like f1 is much slower but is it?Lets run the benchmark several times...q)\t f1 l624q)\t f2 l735 -joel--wagerlabs.com

[sa]

C:\q>qKDB+ 2.4 2008.03.13 Copyright (C) 1993-2008 Kx Systemsw32/ 2()core 2046MB sa jax 172.16.6.139 EXPIRE 2009.01.01 sa@nsl.com#42171q)\t 8#'1000 10000#0156q)\t 8#'1000 10000#062q)the best way to benchmark is to take averages over several runs:C:\q>qKDB+ 2.4 2008.03.13 Copyright (C) 1993-2008 Kx Systemsw32/ 2()core 2046MB sa jax 172.16.6.139 EXPIRE 2009.01.01 sa@nsl.com#42171q)\t do[10;8#'1000 10000#0]640q)On Apr 6, 9:38�am, Joel Reymont wrote:> Suppose we had a list of 10000000> phone numbers and wanted to take> just the first 8 digits of each.>> We could create the list like this:>> q)l:10000000 10#99?"0123456789">> Here, 99?... creates a vector of 99> random characters from the set [0-9]> and 10#... takes 10 characters from> each generated "phone number".>> 10000000 ... repeats it that many times.>> A simple 8#l will just gives us the first> 8 elements of the list which is not what> we want.>> q)f1:{x@\:til 8}> q)f2:{8#/:x}> q)f3:{8#'x}>> The above 3 solutions will give us the> first 8 digits of every phone number> in the list.>> Which solution is the fastest, though?>> The list is very very long specifically> to make benchmarking easier.>> q)\t f1 l> 906>> \t here gives us execution time for f1> in milliseconds. We proceed to time f2> and f3.>> q)\t f2 l> 738> q)\t f3 l> 738>> Looks like f1 is much slower but is it?>> Lets run the benchmark several times...>> q)\t f1 l> 624> q)\t f2 l> 735>> � � � � -joel>> --> wagerlabs.com

[joelr1]

X-Mailer: Apple Mail (2.919.2)Stevan,I agree with you but f1is still the fastest!q)\t do[10;f1 l]7749q)\t do[10;f2 l]8906q)\t do[10;f3 l]8894On Apr 6, 2008, at 4:07 PM, sa wrote:>> the best way to benchmark is to take averages over several runs:>> C:\q>q> KDB+ 2.4 2008.03.13 Copyright (C) 1993-2008 Kx Systems> w32/ 2()core 2046MB sa jax 172.16.6.139 EXPIRE 2009.01.01 sa@nsl.com> #42171>> q)\t do[10;8#'1000 10000#0]> 640> q)--wagerlabs.com

[Christian_Langr]

X-Mailer: Apple Mail (2.919.2)X-HELO-Warning: Remote host 213.182.238.93 incorrectly presented itself as [90.0.0.65]X-Scan-Signature: 7c4c3be5d1cc838e04794291f81be8da> Which solution is the fastest, though?depending on what you'll do to the data afterwards, you can also flip l:q)l0:flip lthen:q)\t l0[til 8]0;-)