cancel
Showing results for 
Search instead for 
Did you mean: 

Question about storing large amounts of data per second

xiaomuwing
New Contributor
There are 1024 real-time data per second, how to save al these 1024 floats into a partition table per second? The insert string is out of length......

Thanks ALL!
5 REPLIES 5

TerryLynch
New Contributor II
Not quite sure I understand the problem: are you saying that you have 1024 live/realtime messages per second and you want to write them every second to a partitioned table on disk? Or that you want to insert them into an in-memory table?

Can you give examples of what the records looks like?

not into in-mem table but a partitioned table, because of the real-time data is sensitive and have to flush them to disk immediately.  
Thank u for response!

在2021年4月6日星期二 UTC+8 下午11:58:28<TerryLynch> 写道:
Not quite sure I understand the problem: are you saying that you have 1024 live/realtime messages per second and you want to write them every second to a partitioned table on disk? Or that you want to insert them into an in-memory table?

Can you give examples of what the records looks like?

dotsch
New Contributor
You have quite a few options to connect to kdb.

Andras

Thank u for response , I'm not quite sure but you mean send parameters by bytes stream ? or what.....

在2021年4月7日星期三 UTC+8 上午12:25:29<dot...@gmail.com> 写道:
You have quite a few options to connect to kdb.

Andras

I mean sending data through the q ipc protocol, yes.
I guess you used http get and you run into some limit of the url size? You can also split up the message btw, but I recommend using the ipc protocol.

András