Reading server-sent events from HTTP request

[Clu]

Hello,

A broker I want to integrate has their tick system sending prices using an event stream (see: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#Sending_events_from_the_server ). It is a long running HTTP request that output every tick is in a separate line delimited by two line breaks.

echo 'data: {"pair": "EUR/USD", "buy": "1.123", "sell": "1.122", "time": 1440365747319 }';
echo "\n\n";

I was wondering if there is away (and should be done) in q or should I put the request and parsing on the Python side?

Thank you

[FutureInSound]

Clu,

If it does come in as such, assume you can use read0 on it then maybe something like this might work (I don't know what format the time is in...):

q)quote😞[] sym:`$();buy:`float$(); sell:`float$(); time:`long$());
q)quote insert "SFFJ"$ssr[;"[\"\" }';]";""] each last each ":" vs/: "," vs first read0 httpRequest
,0
q)quote
sym     buy   sell  time
---------------------------------
EUR/USD 1.123 1.122 1440365747319

[Clu]

Thanks Zak,

The problem is that it's one long streaming HTTP request with all the ticks.

I tried:

txt: `:http://example.com "GET /stream HTTP/1.1\r\nHost:example.com\r\nAuthorization: Basic reqsAuthRandomHash\r\nCache-Control: no-cache\r\nAccept: text/event-stream\r\n\r\n";
txt

Since the request doesn't finish / connection doesn't close, kdb seems to stay on this line and just keeps going forever.

[LamHinYan]

write something that consumes the stream and prints the payload to stdout. "wget -O -" may work. i havent your http streaming server to test it here. below please find the code that streams payload to stdout.

$ seq 3 | perl -ne 'print "data: {\"pair\": \"EUR/USD\", \"buy\": \"$.\", \"sell\": \"1.122\", \"time\": 1440365747319 }\n"' | ncat -klp 6000

$ ncat localhost 6000 | perl -ne 'BEGIN {print ".z.pi:{show .j.k 6_x};\n"}; print' | q

Welcome to kdb+ 32bit edition

For support please see http://groups.google.com/d/forum/personal-kdbplus

Tutorials can be found at http://code.kx.com/wiki/Tutorials

To exit, type \\

To remove this startup msg, edit q.q

close: No error

pair| "EUR/USD"

buy | ,"1"

sell| "1.122"

time| 1.440366e+012

pair| "EUR/USD"

buy | ,"2"

sell| "1.122"

time| 1.440366e+012

pair| "EUR/USD"

buy | ,"3"

sell| "1.122"

time| 1.440366e+012

[DonovanHide]

Consuming SSE would indeed be a useful feature. if anyone wants a live feed to test against, there is one here:

curl https://bitcoinity.org/ev/markets/markets_bitstamp_USD

Cheers,

Donovan.

[LamHinYan]

q).j.k "{\"pair\": \"EUR/USD\", \"buy\": \"1.123\", \"sell\": \"1.122\", \"time\": 1440365747319 }"

pair| "EUR/USD"

buy | "1.123"

sell| "1.122"

time| 1.440366e+012

[tjcelaya]

Clu,

I believe keeping the connection open in .z.ph would hold up the rest of the process since kdb+ is single-threaded by default so it might be simplest to handle the long-lived connection from a separate process.

You can test this by starting up a Q process with an open port, running \sleep 100 (or some sufficiently long interval) and attempting to access it through HTTP or sending a message to it from another process. Either method should hang while Q is executing the sleep command.

[tjcelaya]

Clu,

I believe keeping the connection open in .z.ph would hold up the rest of the process since kdb+ is single-threaded by default so it might be simplest to handle the long-lived connection from a separate process.

You can test this by starting up a Q process with an open port, running\sleep 100 (or some sufficiently long interval) and attempting to access it through HTTP or sending a message to it from another process. Either method should hang while Q is executing the sleep command.

(Apologies in advance if this shows up as a double-post.)