qJava and Publisher

[bardiya]

I'm trying to run the publisher posted on

https://github.com/exxeleron/qJava/blob/master/src/sample/java/Publisher.java

and I'm getting a very generic error

Press <ENTER> to close application
com.exxeleron.qjava.QException: .u.upd
    at com.exxeleron.qjava.QReader.readError(QReader.java:409)
    at com.exxeleron.qjava.QReader.readObject(QReader.java:189)
    at com.exxeleron.qjava.QReader.read(QReader.java:99)
    at com.exxeleron.qjava.QBasicConnection.sync(QBasicConnection.java:159)
    at PublisherTask.run(Publisher.java:74)
    at java.lang.Thread.run(Thread.java:744)

Anyone seen this? I wish there it has more detail. My Q connection is running on port 5001 and seems to be functioning.

[charlie]

it probably expects the remote process to have the function .u.upd defined, which publishes the data to subcribers.

See

http://code.kx.com/wiki/Startingkdbplus/tick

To get you going, and just print the received data on the kdb+ console, define it as

q).u.upd:{0N!(x;y);}

[bardiya]

That worked! Just a follow up question. Since the ticker subscriber is too complicated for what I want to do, how would you define the u.upd function to write to disk every time it's called. My file is

q)select from `:dsPricing
id date| open close high low volume
-------| --------------------------

Thanks.

[charlie]

if you have no symbols in the data, then upsert to the splayed table

.u.upd:{`:dsPricing/ upsert y;}

but this is not suitable for frequent updates; for that you should append in memory, and flush to disk at suitable intervals - e.g. 30 seconds.

[bardiya]

It's so frustrating when I have no Q skills ( I'm taking some courses when the budget gets approved!).
This is what I'm trying to insert

InfoCode:1,Date:Wed Sep 10 00:00:00 EDT 2014,Open_:0.0,close:0.0,High:0.0,Low:0.0,Volume:1.0
com.exxeleron.qjava.QException: type
    at com.exxeleron.qjava.QReader.readError(QReader.java:409)
    at com.exxeleron.qjava.QReader.readObject(QReader.java:189)
    at com.exxeleron.qjava.QReader.read(QReader.java:99)
    at com.exxeleron.qjava.QBasicConnection.sync(QBasicConnection.java:159)
    at PublisherTask.run(Publisher.java:84)
    at java.lang.Thread.run(Thread.java:744)

and my 'dsPricing looks like this

q)select from `:dsPricing
id date| open close high low volume
-------| --------------------------

private Object[] getData(List < DataStream > prices) {

    /*final Object[] data = new Object[] {new int[prices.size()], new QDate[prices.size()],
                                        new float[prices.size()], new float[prices.size()],
                                        new float[prices.size()],new float[prices.size()],
                                        new int[prices.size()]
                                               
    };*/
    final Object[] data = new Object[] {
        new int[1], new QDate[1],
            new float[1], new float[1],
            new float[1], new float[1],
            new int[1]

    };
    for (int i = 0; i < 1; i++) {
        System.out.println("InfoCode:" + prices.get(i).getInfoCode() + ",Date:" + prices.get(i).getMarketDate() + ",Open_:" + prices.get(i).getOpen_() + ",close:" + prices.get(i).getClose_() + ",High:" + prices.get(i).getHigh() + ",Low:" + prices.get(i).getLow() + ",Volume:" + prices.get(i).getVolume());
        ((int[]) data[0])[i] = prices.get(i).getInfoCode();
        ((QDate[]) data[1])[i] = new QDate(prices.get(i).getMarketDate());
        ((float[]) data[2])[i] = (float) prices.get(i).getOpen_();
        ((float[]) data[3])[i] = (float) prices.get(i).getClose_();
        ((float[]) data[4])[i] = (float) prices.get(i).getHigh();
        ((float[]) data[5])[i] = (float) prices.get(i).getLow();
        ((int[]) data[6])[i] = (int) prices.get(i).getVolume();
    }

    return data;
}

[charlie]

if you want to insert more than one row, you'll need to do it a little differently

.u.upd:{`:dsPricing upsert flip `id`date`open`close`high`low`volume!y;}

but as we discussed in another thread, you should really append to splayed tables instead.

[bardiya]

Thanks for all your help ( since now I know the CTO for the company is helping me!). I think you're right that I'm not doing this correctly and still getting the error
nfoCode:1,Date:Thu Feb 17 00:00:00 EST 1994,Open:73.0,close:73.0,High:76.0,Low:73.0,Volume:2223000.0
InfoCode:1,Date:Wed Feb 16 00:00:00 EST 1994,Open:76.0,close:76.0,High:77.899994,Low:75.0,Volume:3167000.0
com.exxeleron.qjava.QException: type
    at com.exxeleron.qjava.QReader.readError(QReader.java:409)
    at com.exxeleron.qjava.QReader.readObject(QReader.java:189)
    at com.exxeleron.qjava.QReader.read(QReader.java:99)
    at com.exxeleron.qjava.QBasicConnection.sync(QBasicConnection.java:159)
    at PublisherTask.run(Publisher.java:84)
    at java.lang.Thread.run(Thread.java:744)

I was wondering if there is a tutorial out there for Java that I can read some historical data for about 200,000 securities and insert that into a file ( obviously not enough memory). How should I approach this?
I assume the SQL query needs to be done in Java.

[charlie]

> read some historical data for about 200,000 

Are you extracting that from another database using java (and their SQL)? Which database vendor? If you have a jdbc driver for it, and the SQL is fairly simple, you could consider http://code.kx.com/wiki/Babel

I'm not familiar with the devnet qjava, but with the Kx http://kx.com/q/c/kx/c.java it looks like this (maybe you can tell if there are type diffs here).

public static void main(String[]args){

  try{

    c c=new c("",5001);

    c.k("addData",new Object[]{new int[]{1,1},

                               new Date[]{Date.valueOf("1994-2-17"),Date.valueOf("1994-2-16")},

                               new double[]{73.,76.},

                               new double[]{73.,76.},

                               new double[]{76.,77.899994},

                               new double[]{73.,75.},

                               new int[]{2223000,3167000}});

    c.close();

  }

  catch(Exception e){

    e.printStackTrace();

  }

}

And then in kdb+, create an empty table on disk, and define our upsert function

q)`:dsPricing set ([id:`int$(); date:`date$()] open:`float$();close:`float$();high:`float$();low:`float$();volume:`int$())

`:dsPricing

q)addData:{`:dsPricing upsert flip `id`date`open`close`high`low`volume!x;}

q)select from `:dsPricing

id date| open close high low volume

-------| --------------------------

then run the java program

$java kx/c

then confirm data is present

q)select from `:dsPricing

id date      | open close high     low volume 

-------------| -------------------------------

1  1994.02.17| 73   73    76       73  2223000

1  1994.02.16| 76   76    77.89999 75  3167000

With 32bit kdb+, you'll probably want to run with the -g 1 cmd line option to try to keep mem usage low.

[charlie]

and with that as a splayed table - just remove the key, and append / to the tablename

q)`:dsPricing/ set ([]id:`int$(); date:`date$(); open:`float$();close:`float$();high:`float$();low:`float$();volume:`int$())

`:dsPricing/

q)addData:{`:dsPricing/ upsert flip `id`date`open`close`high`low`volume!x;}

q)select from `:dsPricing

id date       open close high     low volume 

---------------------------------------------

1  1994.02.17 73   73    76       73  2223000

1  1994.02.16 76   76    77.89999 75  3167000

[maciej_lach]

There are some minor differences between Kx c.java and DEVnet qJava in this example, namely:

• q date type is represented in qJava as instances of QDate class,
• QConnection.sync( String query, Object... parameters ) method uses Java varargs. In case your q function takes only a single array parameter, the array in qJava has to be explicitly cast to Object.
  E.g.: q.sync("addData", (Object) data);
  

Please take a look at sample code:

public static void main( final String[] args ) throws IOException {
        final QConnection q = new QBasicConnection("localhost", 5001, "", "");

        try {
            q.open();

            Object[] data = new Object[] { new int[] { 1, 1 },
                                           new QDate[] { QDate.fromString("1994.02.17"), QDate.fromString("1994.02.16") },
                                           new double[] { 73., 76. },
                                           new double[] { 73., 76. },
                                           new double[] { 76., 77.899994 },
                                           new double[] { 73., 75. },
                                           new int[] { 2223000, 3167000 } };
            q.sync("addData", (Object) data);
        } catch ( final Exception e ) {
            System.err.println(e);
        } finally {
            q.close();
        }
    }

Best regards

Maciej Lach