Using python inside kdb+; anyone would like to participate for code review, writing documentation and test cases?

[kuentang]

Hi all,

My tool to execute python code inside kdb+ like q/rserver is ready. Unfortunately documentation and test cases are missing. If anyone would like to participate/volunteer please drop me an email and I can give you access to my github repository then.

And a talk is scheduled for June in London during python meetup to showcase this tool.

Thanks,

Kim

--
Kim Tang

[Alexander_Belop]

On Tuesday, April 12, 2016 at 3:41:37 AM UTC-4, kuentang wrote:

My tool to execute python code inside kdb+ like q/rserver is ready.

Why are you reinventing the wheel?  Doesn't PyQ do what you need already?

For example,

q)p)print "Hello word!"

Hello word!

See pyq.enlnt.com for details.

[kuentang]

Hi Sasha,

>>> Why are you reinventing the wheel?  Doesn't PyQ do what you need already?

I do miss that PyQ is able to execute python inside q.

cat p.k

PYVER:"2.7"

SOEXT:".so\000"

PSO:`p

lib:"libpython",PYVER,SOEXT

PYTHON:"python.py"

`QVER setenv$_.Q.k

\d .p

@[`.[`PSO]2:(`i;1);`. `lib]

e:{e0 x,"\000";}

\d .

p)import sys, os

p)from pyq import q

p)sys.executable = str(q.PYTHON)

p)script = str(q(".z.f"))

p)sys.argv = [script] + [str(a) for a in  q(".z.x")]

p)sys.modules['__main__'].__file__ = script

p)sys.path.insert(0, os.path.dirname(script))

p)del sys, os, script

It looks like that pyq is just using one function to execute python inside q and unfortunately I was...

How is it possible to get results from a python calculation back in q using pyq?

I can see that you can use sd0/sd1 from the _k module and does it help?

cat test_sd.q

p)import os

p)import sys

p)from pyq import _k

p)def f(d):

    x = os.read(d, 1)

    print(x.decode())

    sys.stdout.flush()

    _k.sd0(d)

p)r,w = os.pipe()

p)_k.sd1(r, None)

p)os.write(w, b'X')

Nevertheless the module that I have invented is able to execute the following q code:

cat plot_cv_predict.q

n) "Plotting Cross-Validated Predictions"

/ loading required libraries from python

n)from sklearn import datasets

from sklearn.cross_validation import cross_val_predict

from sklearn import linear_model

import matplotlib.pyplot as plt

/ execute the function in python

n) lr = linear_model.LinearRegression()

boston = datasets.load_boston()

/ get result back from python

y:neval"boston.target"

/ # cross_val_predict returns an array of the same size as `y` where each entry

/ # is a prediction obtained by cross validated:

/ pass y back to python using backtick

n) predicted = cross_val_predict(lr, boston.data, `y, cv=10)

n) fig, ax = plt.subplots()

n) ax.scatter(y, predicted)

n) ax.plot([y.min(), y.max()], [y.min(), y.max()], 'k--', lw=4)

n) ax.set_xlabel('Measured')

n) ax.set_ylabel('Predicted')

n) plt.show()

qpredicted:neval "predicted"

(::)t:([]a:y;b:qpredicted)

n) fig = plt.figure()

n) `t.plot(x='a',y='b')

n) plt.show()

/ a table in q can be passed as panda in python

neval"str(type(t))"

/ "<class 'pandas.core.frame.DataFrame'>"

/ pandas can be retrieved as table from python

t~neval"t"

/ 1b

Kim

Von: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] Im Auftrag von Sasha
Gesendet: Montag, 13. Juni 2016 21:41
An: Kdb+ Personal Developers
Betreff: [personal kdb+] Re: Using python inside kdb+; anyone would like to participate for code review, writing documentation and test cases?

[Alexander_Belop]

On Wednesday, June 15, 2016 at 4:08:48 AM UTC-4, kuentang wrote:

How is it possible to get results from a python calculation back in q using pyq?

It is often not a good idea to call python from q.  I consider Python to be a higher level language where you have classes, modules and other features useful for organizing the high level of your application,  whereas q is best at the data access level.  The only valid reason to call Python from q is in a top level callback.  It is mostly because of this philosophy that I have never perfected passing Python results to q.  Nevertheless, there are several ways to work around this limitation.

1.  You can assign the results of python computation to a q global inside Python:

q)p)q.x = 20 + 22

q)x

42

2. If you define a Python function that takes instances of class K and returns an instance of class K, you can expose it to q by simply assigning it to a q global:

q)p)def f(x, y): return x + y

q)p)q.f = f

q)f(20;22)

42

(Note that the exposed function is monadic taking a list of arguments.) Such function can be used freely in q callbacks.

[kuentang]

Thanks for sharing.

>>> I consider Python to be a higher level language where you have classes, modules and other features useful for organizing the high level of your application,  whereas q is best at the data access level.

In my opinion q is good both for data access and implementing an application (see tick+ for example).

Kim

Von: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] Im Auftrag von Sasha
Gesendet: Sonntag, 26. Juni 2016 20:59
An: Kdb+ Personal Developers
Betreff: Re: [personal kdb+] Re: Using python inside kdb+; anyone would like to participate for code review, writing documentation and test cases?

[Alexander_Belop]

On Sunday, June 26, 2016 at 6:21:16 PM UTC-4, kuentang wrote:

>>> I consider Python to be a higher level language where you have classes, modules and other features useful for organizing the high level of your application,  whereas q is best at the data access level.

In my opinion q is good both for data access and implementing an application (see tick+ for example).

I agree, but if your app is written in q, why do you need Python? 🙂

Seriously, though the main idea in PyQ is to expose kdb+ data objects as first class Python objects.

Consider this example (in IPython):

In [5]: t = %q ([]n:til 10;r:10?1f)

In [6]: t

Out[6]: 

n r        

-----------

0 0.3927524

1 0.5170911

2 0.5159796

3 0.4066642

4 0.1780839

5 0.3017723

6 0.785033 

7 0.5347096

8 0.7111716

9 0.411597

In [7]: np.log10(t.r, out=np.asarray(t.r))

Out[7]: 

array([-0.40588117, -0.28643292, -0.28736743, -0.39076407, -0.74937544,

       -0.52032069, -0.10511208, -0.27188201, -0.14802557, -0.38552782])

In [8]: t

Out[8]: 

n r         

------------

0 -0.4058812

1 -0.2864329

2 -0.2873674

3 -0.3907641

4 -0.7493754

5 -0.5203207

6 -0.1051121

7 -0.271882 

8 -0.1480256

9 -0.3855278

In [9]: t.reverse[-3:]

Out[9]: 

n r         

------------

2 -0.2873674

1 -0.2864329

0 -0.4058812

Here, the table t is created in q, but is manipulated entirely in Python.  PyQ code can be easily understood by a Python programmer with a minimal knowledge of kdb+ concepts.

Note that calling a numpy function on q data (see In[7] above) could be arranged from q as well

q)t:([]n:til 10;r:10?1f)

q)t

n r         

------------

0 0.4931835 

1 0.5785203 

2 0.08388858

3 0.1959907 

4 0.375638  

5 0.6137452 

6 0.5294808 

7 0.6916099 

8 0.2296615 

9 0.6919531 

q)p)import numpy as np

q)p)np.log10(q.t.r, out=np.asarray(q.t.r))

q)t

n r         

------------

0 -0.3069915

1 -0.2376814

2 -1.076297 

3 -0.7077645

4 -0.4252305

5 -0.2120119

6 -0.2761498

7 -0.1601388

8 -0.6389117

9 -0.1599234

In this case, however there is no need to return a Python object to q because the result is written directly to a kdb+ table.

On the other hand, if we implement an automatic Python to q translation, something like "p)np.log10(q.t.r, out=np.asarray(q.t.r))"

would be wasteful because an extra copy of t.r will be created just to be returned to q and discarded there.

[kuentang]

>>>I agree, but if your app is written in q, why do you need Python? 🙂

Unfortunately there are not that many scientific libraries compare to python (scikit), r or matlab in q available. And you don’t want to spend 1 – 2 weeks to implement say svm. In this case you just trigger a function in python.

>>> On the other hand, if we implement an automatic Python to q translation, something like "p)np.log10(q.t.r, out=np.asarray(q.t.r))" would be wasteful because an extra copy of t.r will be created just to be returned to q and discarded there.

Yes, it is wasteful and there is unfortunately no workaround about it. L I think this is the price we need to pay.

Kim

Von: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] Im Auftrag von Sasha
Gesendet: Montag, 27. Juni 2016 03:22
An: Kdb+ Personal Developers
Betreff: Re: [personal kdb+] Re: Using python inside kdb+; anyone would like to participate for code review, writing documentation and test cases?

[Alexander_Belop]

On Monday, June 27, 2016 at 2:50:55 AM UTC-4, kuentang wrote:

>>> On the other hand, if we implement an automatic Python to q translation, something like "p)np.log10(q.t.r, out=np.asarray(q.t.r))" would be wasteful because an extra copy of t.r will be created just to be returned to q and discarded there.

Yes, it is wasteful and there is unfortunately no workaround about it. 

Maybe a trailing ; can signify that return value should be suppressed.   

[carfield1]

On Monday, June 27, 2016 at 6:21:16 AM UTC+8, kuentang wrote:

Thanks for sharing.

>>> I consider Python to be a higher level language where you have classes, modules and other features useful for organizing the high level of your application,  whereas q is best at the data access level.

In my opinion q is good both for data access and implementing an application (see tick+ for example).

Kim

 Yes, opinion is difference by difference people, and each language have their edge, q is simple and concise, but in the meantime it miss some features, say for quite a other language, if there is error, there is exception come with stacktrace. However, in q usually there is just an error word

[kuentang]

>>>  Yes, opinion is difference by difference people, and each language have their edge, q is simple and concise,

True true and opinion will change or advance in time. At the beginning I thought that kdb+ is like excel but in console. J But after say 3 months my opinion has changed. And after reading this article https://kx.com/_media-coverage/0809-AutoTrader-sm.pdf my opinion has changed again.

>>> but in the meantime it miss some features, say for quite a other language, if there is error, there is exception come with stacktrace. However, in q usually there is just an error word

Check Aaron Davies’s presentation about “dequeing bugs” http://www.q-ist.com/2013/03/my-kdb-user-meeting-presentation.html . It will show how to debug an exception efficiently.

Funny enough is that I wanted something similar that is able to trace back an exception in runtime.  And instead asking around I come up with my own library which is called bt, short for behaviour tag engine: https://github.com/kimtang/btlib-bt . The pattern is stolen from Chris Granger’s blog: http://www.chris-granger.com/2013/01/24/the-ide-as-data/

Btw I will be in hkg next month. If you have time for a coffee or tea just let me know.

Kim

Von: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] Im Auftrag von Carfield Yim
Gesendet: Donnerstag, 30. Juni 2016 17:05
An: Kdb+ Personal Developers
Betreff: Re: [personal kdb+] Re: Using python inside kdb+; anyone would like to participate for code review, writing documentation and test cases?

On Monday, June 27, 2016 at 6:21:16 AM UTC+8, kuentang wrote:

Thanks for sharing.

>>> I consider Python to be a higher level language where you have classes, modules and other features useful for organizing the high level of your application,  whereas q is best at the data access level.

In my opinion q is good both for data access and implementing an application (see tick+ for example).

Kim

 Yes, opinion is difference by difference people, and each language have their edge, q is simple and concise, but in the meantime it miss some features, say for quite a other language, if there is error, there is exception come with stacktrace. However, in q usually there is just an error word

--
You received this message because you are subscribed to the Google Groups "Kdb+ Personal Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to personal-kdbplus+unsubscribe@googlegroups.com.
To post to this group, send email to personal-kdbplus@googlegroups.com.
Visit this group at https://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

[Jiahe_Xi]

Hi Kim, 

I would like to particiate in the test. Can you drop me the link to jiahe.xi@gmail.com please? It's exactly what i was looking for - i want to call the python's statsmodel module in q for some stats functions.

best,

robert