cancel
Showing results for 
Search instead for 
Did you mean: 

mdev function

vortexsbkny
New Contributor
Hi,

Is mdev supposed to calculate a standard deviation? I compared the output with Excel stdev 
and it does not match. Please see my example

// input 
>t:153.839996, 154.869995, 154.449997, 155.529999, 156.110001, 154.979996, 153.979996, 156.770004, 158.289993, 158.110001, 158.850006

// calling the function
>5 mdev t
>0 0.5149995 0.4228734 0.6158083 0.7962429 0.5755155 0.755753 0.9553374

// window is 5
VALUE MOVING_DEV
153.84
154.87
154.45
155.53
156.11 0.890226668
154.98 0.643445886
153.98 0.844957543
156.77 1.068099648
158.29 1.655786759
158.11 1.904109448
158.85 1.955251178
11 REPLIES 11

Kevin_Smyth
New Contributor

Hi gekaprog,

The mdev function in kdb+ calculates the population standard deviation, in Excel you have the sample deviation (SDDEV.S) rather than the population one (SDDEV.P).

Multiply your result in kdb+ by the square root of (n-1)/n to get the Excel result.

Thanks

Kevin,

Thank you

For the sake of other readers, I checked and the multiplier comes out of equation sqrt(1/n-1)=x*sqrt(1/n). then x=sqrt(n/n-1). 

Out of curiosity, I checked and noticed sqrt(5%4) is not multiplying the list through correctly. Why?

t:153.84 154.87 154.45 155.53 156.11 154.98 153.98 156.77 158.29 158.11 158.85
t2:5 mdev t

sqrt(5%4)*t2   // prints wrong results
1.118034f*t2   // prints correct results, matching Excel stdev.s

Thanks,
Eugene

This is a right-to-left execution problem. If you rearrange, it works:

q)t2*sqrt(5%4)
0 0.5757875 0.4727872 0.6884936 0.8902247 0.6434439 0.8449556 1.068096 1.655787 1.904109 1.955249
q)1.118034f*t2
0 0.5757875 0.4727872 0.6884936 0.8902247 0.6434439 0.8449556 1.068096 1.655787 1.904109 1.955249

Hope that helps
Jonathon




sqrt(5%4)*t2   // prints wrong results
1.118034f*t2   // prints correct results, matching Excel stdev.s

you should use [] to execute functions with params, so  sqrt[5%4]*t2  is correct.

with  sqrt(5%4)*t2   you   multiply t2 with (5%4)  and then you do the square root of the result.

Markus, this is very helpful. Clearly, not how I expected an interpreter to parse expression sqrt(5%4)*t2. odd

q/k handles many things quite different then the usual programming languages.
You really have to accept this and learn the language (-> http://code.kx.com/wiki/JB:QforMortals )

but I promise: it's worth! Once you grabbed the concepts of k/q you will love its simplicity and power.

Thanks, it's extremely
powerful and fast.

As for my problem, I've noticed another discrepancy - 0n in the result set and
decided against using mdev[].

t:22.89276 22.89276 22.89276 22.89276 22.87285 22.82308 22.84299
q)3 mdev t
0 0 0n 0n 0.00938425 0.02930163 0.02045185 0.00938425 0.02930163 0.1048127

i see no nulls with mdev:

q)x:22.89276 22.89276 22.89276 22.89276 22.87285 22.82308 22.84299
q)3 mdev x
0 0 0 0 0.009385664 0.02930446 0.02045342

i note that your t has 7 values, but your 3 mdev t has 10 values.  
your 3 mdev t is not a result from the t you present, but may be a result of another list you have generated that does contain 0n.

---------- Forwarded message ----------
From: gekaprog <vortexsbkny@gmail.com>
Date: 27 November 2016 at 08:10
Subject: Re: [personal kdb+] mdev function
To: Kdb+ Personal Developers <personal-kdbplus@googlegroups.com>


Thanks, it's extremely
powerful and fast.

As for my problem, I've noticed another discrepancy - 0n in the result set and
decided against using mdev[].

t:22.89276 22.89276 22.89276 22.89276 22.87285 22.82308 22.84299
q)3 mdev t
0 0 0n 0n 0.00938425 0.02930163 0.02045185 0.00938425 0.02930163 0.1048127

Sorry, that was a bad screen copy. Here is another try. 0n would make sense if q engine 
couldn't calculate a value due to division by zero.

q)t5:22.89276 22.89276 22.89276 22.89276 22.87285 22.82308 22.84299
q)3 mdev t5
0 0 0 0 0.009385664 0.02930446 0.02045342
q)2 mdev t5
0 0 0n 0 0.009955 0.024885 0.009955

> nulls in mdev result

q)t5:22.89276 22.89276 22.89276 22.89276 22.87285 22.82308 22.84299
q)2 mdev t5
0 0 0n 0 0.00995500001 0.024885 0.00995499999

the 0n occurs due to sqrt of a negative number:

q)mdev
k){sqrt mavg[x;y*y]-m*m:mavg[x;y:"f"$y]}
q){mavg[x;y*y]-m*m:mavg[x;y:"f"$y]}[2;t5]
0 0 -2.273737e-13 0 9.910203e-05 0.0006192632 9.910202e-05

to correct the result, you can fill:

q)0^2 mdev t5
0 0 0 0 0.00995500001 0.024885 0.00995499999


vortexsbkny
New Contributor


On Monday, November 21, 2016 at 4:22:01 AM UTC-5, gekaprog wrote:
Hi,

Is mdev supposed to calculate a standard deviation? I compared the output with Excel stdev 
and it does not match. Please see my example

// input 
>t:153.839996, 154.869995, 154.449997, 155.529999, 156.110001, 154.979996, 153.979996, 156.770004, 158.289993, 158.110001, 158.850006

// calling the function
>5 mdev t
>0 0.5149995 0.4228734 0.6158083 0.7962429 0.5755155 0.755753 0.9553374

// window is 5
VALUE MOVING_DEV
153.84
154.87
154.45
155.53
156.11 0.890226668
154.98 0.643445886
153.98 0.844957543
156.77 1.068099648
158.29 1.655786759
158.11 1.904109448
158.85 1.955251178