Debugging embedPy library extension called extendPy

[Kovacsi]

Hi all

   I try to hack and learn how to debug python library loaded to q kdb+ context.

I use michaelsimonelli/qoinbase-q: Q client for the Coinbase Pro API (github.com) as a sample to 

learn from. 

   While hacking I discovered Interesting insights:

1. embedPy doesn't work with conda python 3.9 on win 10 x64 bit.

2. embedPy is working with base python environment - never with conda virtual env

so you must erase all python 3.9 base environments on windows machine to make it work.

3. embedPy is working with conda python 3.8.10 as base env.

4. A python package that work well with embedPy on linux will probably fail on windows

example:

qcoinbase fails to view and use the PublicClient module

(py38q) E:\workpy38\qoinbase-q>%QCMD% C:\Users\srv\.conda\envs\py38q\q\packages\qoinbase-q\cbpro.q
KDB+ 4.0 2021.11.04 Copyright (C) 1993-2021 Kx Systems
w64/ 16(16)core 16160MB  EXPIRE 2022.12.02 

2022.02.22T09:15:31.316 [Py] Imported module 'builtins'
2022.02.22T09:15:31.316 [Py] Mapped functions (builtins; list, next, vars, str) in `.py context
2022.02.22T09:15:31.450 [Py] Imported module 'cbpro'
2022.02.22T09:15:31.528 [Py] Reflected module 'cbpro' to `cbpro context
.

q)key `.cbpro
`AuthenticatedClient`CBProAuth`OrderBook`PublicClient`WebsocketClient

q)cb:`.cbpro.PublicClient[]
'cxt
[4] C:\Users\srv\.conda\envs\py38q\q\extend.q:181: .py.reflection.priv.context@:{.py.reflection.priv[y;`cxt][x;z]}

seams `ctx is not recognized.

I am searching for someone to help me hack the path and understand how to make it work on windows

My hacking goal is to understand how the context ``cxt parameter flows in to the module.

Does anyone like to join hacking?

Ori

[LeahS]

Hello Ori, 

Thanks for sharing your findings here.

Just going to highlight some embedPy documentation that may be useful:

KX embedPy - kdb+ and q

KX embedPy user guide 

 KX embedPy integration  

KX embedPy FAQ  

KX embedPy Reference  

Please let me know if these help or not. 

Also welcome any feedback from fellow community contributors. 

Kind regards,

Leah 

[Kovacsi]

Thanks Leah

   Actually I am searching for more technical details about tracing into code

There is no explanation in any of embedPy documentation how to hack and trace

in to the program flow and view the contents of objects  of python and q on runtime.

Ori

[LeahS]

Hello Ori, 

Thanks for your response. 

Points 1,2 and 3 are correct observations. embedPy is not supported on Python 3.9 for Windows or within virtual environments for a number of technical reasons.

Regarding point 4, the requirements are stated in the GitHub and only Linux is supported as of now: https://github.com/michaelsimonelli/qoinbase-q#requirements. Also, the function that is failing is not from embedPy directly, but rather it's from the developer's extension of embedPy, extendPy, and we don't control or support that: https://github.com/michaelsimonelli/qoinbase-q#extendpy

If Windows is supported in the future, we will communicate this with you. 

Kind regards,

Leah  

[Kovacsi]

Thanks Leah

    Can you advise other web socket connector sample to wort on crypto real time data RDB on kdb+?

I understand you doesn't put resources to support this specific project. do you have KX internal sample project to learn how to use Python and q kdb inside q process (with reflections from python to q objects)

with web socket connection?

Thanks

Ori

[LeahS]

Hello Ori, 

Thanks for your response. 

Yes, to make it clear for everyone interested, extendPy is an embedPy library extension that KX does not control nor support. 

I believe you are in the process of debugging and the Github page in question will be updated by the author with/when there are relevant updates.

Regarding another web socket connector, I welcome feedback from others here on that. 

Kind regards,

Leah