issue with cached buckets

[Mike1]

Received: by 10.150.202.8 with SMTP id z8mr1471042ybf.29.1234272667358; Tue,
10 Feb 2009 05:31:07 -0800 (PST)
Date: Tue, 10 Feb 2009 05:31:07 -0800 (PST)
X-IP: 199.89.103.10
User-Agent: G2/1.0
X-Google-Token: NlimpgwAAAC0VkKc7xLhrkHQNFyktTlR
X-HTTP-UserAgent: Mozilla/4.0 (compatible; MSIE 7.0; MSBrowserIE7; Windows NT
5.1; msie6xpv1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 2.0.50727;
.NET CLR 3.0.04506.30; MS-RTC LM 8; .NET CLR 3.0.04506.648; .NET CLR
3.5.21022; MSIE6ENV21; MSIE6ENV21),gzip(gfe),gzip(gfe)
X-HTTP-Via: 1.1 piinbcgw01
Message-ID: <36677e72-1e27-4e91-bd9c-f7c434aa5eef@m40g2000yqh.googlegroups.com>
Subject: issue with cached buckets
From: Mike
To: "Kdb+ Personal Developers"
X-Google-Approved: charlie@kx.com via web at 2009-02-10 13:35:10

Hi,
I am new to kdb and trying to create cached bucket using Arthur script
but it seems like it has problem. any help in this regard is greatly
appreciated. Is there any other way to cache derived data?

It works fine for the first query but if i change date or ticker for
subsequent query, it fails with below error message.

here is the script

/ get fiveminbars data for date&sym: takes a few milliseconds per
date&sym
g:{[d;s]select low:min price,open:first price,close:last
price,high:max price,
volume:sum size,vwap:size wavg price by 5 xbar time.minute from
trade where date=d,sym=s}

/ cache for every date/sym combination
c:([]date:"d"$();sym:"s"$())!()

/ check for cached otherwise cache and return
f:{[d;s]$[type r:c x:(d;s);r;c[x]:g[d;s]]}


d:2008.10.02
s:`GE
f[d;s]


so if i change parameter to like this for second query

d:2008.10.03 or s:`IBM. it will fail with below error

k){$[99h=@y;sublist[x;!y]!sublist[x;. y];~0>@x;$[.Q.qp y;.Q.ind[y];y]i
+!"i"$x[1]&(#y)-i:*x;abs[x]<#y;x#y;y]}
'length
!
+(,`minute)!,09:30 09:35 09:40 09:45 09:50 09:55 10:00 10:05 10:10
10:15 10:2..
+`low`open`close`high`volume`vwap!(`real$();`real$();`real$();`real$
();`int$(..


Thanks
Mike

[peter_durkan]

Hi Mike,

This works fine for me after making the following change.

/ check for cached otherwise cache and return

f:{[d;s]$[count r:c x:(d;s);r;c[x]:g[d;s]]}

This uses count instead of type as once the cache has been populated with one result then the return value of indexing the cache dictionary with a key that does not exist will be an empty table rather than an empty list causing the first test in the conditional to always be true.


Peter

[Mike1]

Hi Peter,Thanks for the quick response but somehow I am still getting the sameerror if i change s or d to some new value.Am I missing something?q)f{[d;s]$[count r:c x:(d;s);r;c[x]:g[d;s]]}q)show f[d;s]k){$[99h=@y;sublist[x;!y]!sublist[x;. y];~0>@x;$[.Q.qp y;.Q.ind[y];y]i+!"i"$x[1]&(#y)-i:*x;abs[x]<#y;x#y;y]}'length!+(,`minute)!,09:30 09:35 09:40 09:45 09:50 09:55 10:00 10:05 10:1010:15 10:2..+`low`open`close`high`volume`vwap!(`real$();`real$();`real$();`real$();`int$(..q))\MikeOn Feb 10, 8:51�am, Peter Durkan wrote:> Hi Mike,>> This works fine for me after making the following change.>> / check for cached otherwise cache and return> f:{[d;s]$[count r:c x:(d;s);r;c[x]:g[d;s]]}>> This uses count instead of type as once the cache has been populated with> one result then the return value of indexing the cache dictionary with a key> that does not exist will be an empty table rather than an empty list causing> the first test in the conditional to always be true.>> Peter>>>> On Tue, Feb 10, 2009 at 8:31 AM, Mike wrote:>> > Hi,> > I am new to kdb and trying to create cached bucket using Arthur script> > but it seems like it has problem. any help in this regard is greatly> > appreciated. Is there any other way to cache derived data?>> > It works fine for the first query but if i change date or ticker for> > subsequent query, it fails with below error message.>> > here is the script>> > / get fiveminbars data for date&sym: takes a few milliseconds per> > date&sym> > g:{[d;s]select low:min price,open:first price,close:last> > price,high:max price,> > �volume:sum size,vwap:size wavg price by 5 xbar time.minute from> > trade where date=d,sym=s}>> > / cache for every date/sym combination> > c:([]date:"d"$();sym:"s"$())!()>> > / check for cached otherwise cache and return> > f:{[d;s]$[type r:c x:(d;s);r;c[x]:g[d;s]]}>> > d:2008.10.02> > s:`GE> > f[d;s]>> > so if i change parameter to like this for second query>> > d:2008.10.03 or s:`IBM. it will fail with below error>> > k){$[99h=@y;sublist[x;!y]!sublist[x;. y];~0>@x;$[.Q.qp y;.Q.ind[y];y]i> > +!"i"$x[1]&(#y)-i:*x;abs[x]<#y;x#y;y]}> > 'length> > !> > +(,`minute)!,09:30 09:35 09:40 09:45 09:50 09:55 10:00 10:05 10:10> > 10:15 10:2..> > +`low`open`close`high`volume`vwap!(`real$();`real$();`real$();`real$> > ();`int$(..>> > Thanks> > Mike- Hide quoted text ->> - Show quoted text -

[peter_durkan]

What version of kdb+ are you running.

What is .z.K and .z.k

I don't think the function f is failing as your error seems to be in the q function sublist.

What happens when you just call f[d;s] without the show?
What does c look like after each call?

[Mike1]

Thanks Peter, You are absolutely right, I had old version, downloadedlatest version and your solution worked like a charm. I was just goingcrazy. Thanks for saving my hairs:)Thanks so muchMikeOn Feb 10, 9:41�am, Peter Durkan wrote:> What version of kdb+ are you running.>> What is .z.K and .z.k>> I don't think the function f is failing as your error seems to be in the q> function sublist.>> What happens when you just call f[d;s] without the show?> What does c look like after each call?>>>> On Tue, Feb 10, 2009 at 9:21 AM, Mike wrote:>> > Hi Peter,> > Thanks for the quick response but somehow I am still getting the same> > error if i change s or d to some new value.Am I missing something?>> > q)f> > {[d;s]$[count r:c x:(d;s);r;c[x]:g[d;s]]}>> > q)show f[d;s]> > k){$[99h=@y;sublist[x;!y]!sublist[x;. y];~0>@x;$[.Q.qp y;.Q.ind[y];y]i> > +!"i"$x[1]&(#y)-i:*x;abs[x]<#y;x#y;y]}> > 'length> > !> > +(,`minute)!,09:30 09:35 09:40 09:45 09:50 09:55 10:00 10:05 10:10> > 10:15 10:2..> > +`low`open`close`high`volume`vwap!(`real$();`real$();`real$();`real$> > ();`int$(..> > q))\>> > Mike>> > On Feb 10, 8:51 am, Peter Durkan wrote:> > > Hi Mike,>> > > This works fine for me after making the following change.>> > > / check for cached otherwise cache and return> > > f:{[d;s]$[count r:c x:(d;s);r;c[x]:g[d;s]]}>> > > This uses count instead of type as once the cache has been populated with> > > one result then the return value of indexing the cache dictionary with a> > key> > > that does not exist will be an empty table rather than an empty list> > causing> > > the first test in the conditional to always be true.>> > > Peter>> > > On Tue, Feb 10, 2009 at 8:31 AM, Mike wrote:>> > > > Hi,> > > > I am new to kdb and trying to create cached bucket using Arthur script> > > > but it seems like it has problem. any help in this regard is greatly> > > > appreciated. Is there any other way to cache derived data?>> > > > It works fine for the first query but if i change date or ticker for> > > > subsequent query, it fails with below error message.>> > > > here is the script>> > > > / get fiveminbars data for date&sym: takes a few milliseconds per> > > > date&sym> > > > g:{[d;s]select low:min price,open:first price,close:last> > > > price,high:max price,> > > > �volume:sum size,vwap:size wavg price by 5 xbar time.minute from> > > > trade where date=d,sym=s}>> > > > / cache for every date/sym combination> > > > c:([]date:"d"$();sym:"s"$())!()>> > > > / check for cached otherwise cache and return> > > > f:{[d;s]$[type r:c x:(d;s);r;c[x]:g[d;s]]}>> > > > d:2008.10.02> > > > s:`GE> > > > f[d;s]>> > > > so if i change parameter to like this for second query>> > > > d:2008.10.03 or s:`IBM. it will fail with below error>> > > > k){$[99h=@y;sublist[x;!y]!sublist[x;. y];~0>@x;$[.Q.qp y;.Q.ind[y];y]i> > > > +!"i"$x[1]&(#y)-i:*x;abs[x]<#y;x#y;y]}> > > > 'length> > > > !> > > > +(,`minute)!,09:30 09:35 09:40 09:45 09:50 09:55 10:00 10:05 10:10> > > > 10:15 10:2..> > > > +`low`open`close`high`volume`vwap!(`real$();`real$();`real$();`real$> > > > ();`int$(..>> > > > Thanks> > > > Mike- Hide quoted text ->> > > - Show quoted text -- Hide quoted text ->> - Show quoted text -