Running user defined aggregation on partitioned tables

[lzl]

Hello

I want to run a user defined aggregating function getPercentile (to get say 99% or 50% or 1% percentile of a list of value) on a partitioned table. From this link map-reduce , only few types of aggregation can KDB decompose into map step and reduce step (Although med is listed, it will throw a part error if being executed across partitions). Since I got lots of data in each partition, I don't want to load them all into memory and apply the aggregation, so is there any way to apply user specific map-reduce for aggregation on partitioned tables? 

Thanks! 

[darrenwsun · Accepted solution]

To my knowledge, the best way you can do is to fetch the data into memory and apply the custom aggregate function to the retrieved data, like the below

update percentile:getPercentile price from select sym, price from trade where date>=.z.d-7  // and other filters

Certain function like sum and prd can reduce in a memory-effective way as they don't need to keep the original values from each partition. However med or your getPercentile function cannot.