cancel
Showing results for 
Search instead for 
Did you mean: 

ArrowKDB

kxnoob
New Contributor

I have tried installing ArrowKDB from source. 

 

I followed the installation instructions and the instructions to build worked properly.

 

git clone https://github.com/KxSystems/arrowkdb.git
cd arrowkdb
mkdir build cd build cmake --build . --config Release
cmake --build . --config Release --target install
 I had to change somethings in the config and some environment variables to make it works but it builds successfully and it builds successfully.

$ cmake --build . --config Release --target install
[100%] Built target arrowkdb
Install the project...
-- Install configuration: "Release"
-- Installing: <>/q/l64/arrowkdb.so
-- Set runtime path of " <>/q/l64/arrowkdb.so" to ""
-- Up-to-date: <>/q
-- Installing:<>/q/arrowkdb.q
 
However, when I try to load arrowkdb I get the following:

q arrowkdb.q
KDB+ 4.0 Cloud Edition 2022.09.30 Copyright (C) 1993-2022 Kx Systems
l64/ 32()core 253885MB <> EXPIRE 2024.03.14 <> KXCO KXCE <>

'<>/q/l64/arrowkdb.so: undefined symbol: _ZNK5arrow9ArraySpan24UnionMayHaveLogicalNullsEv
[2] <>/q/arrowkdb.q:6: dt.boolean:`arrowkdb 2:(`boolean;1);

 

Can someone point me to what might have gone wrong in the build process? I am building against arrow 9. Using the release version did not work. 

1 ACCEPTED SOLUTION

nmcdonnell
New Contributor II
New Contributor II

Arrowkdb should work with earlier libarrow versions, say 5.0 (or maybe even 3.0) since your are building from source.

What is your distro?

If you're able to install libarrow.so.800 you should also be able to install libparquet.so.800.  It'll be two separate yum commands similar to https://github.com/KxSystems/arrowkdb#linux (although that's apt on ubuntu)

See https://arrow.apache.org/install/#c-and-glib-c-packages-for-debian-gnulinux-ubuntu-and-centos:

Something like this from centos 7:

sudo yum install -y epel-release || sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1).noarch.rpm
sudo yum install -y https://apache.jfrog.io/artifactory/arrow/centos/$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1)/apache-arrow-release-latest.rpm
sudo yum install -y --enablerepo=epel arrow-devel # For C++
sudo yum install -y --enablerepo=epel parquet-devel # For Apache Parquet C++

 

View solution in original post

5 REPLIES 5

nmcdonnell
New Contributor II
New Contributor II

Hi,

I suspect you're missing the libarrow dependencies.
Could run `ldd $QHOME/l64/arrowkdb.so`?
It should return something like:

ldd arrowkdb.so
linux-vdso.so.1 (0x00007ffed21f9000)
libarrow.so.900 => /lib/x86_64-linux-gnu/libarrow.so.900 (0x00007f92d5ded000)
libparquet.so.900 => /lib/x86_64-linux-gnu/libparquet.so.900 (0x00007f92d5ac2000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f92d58e0000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f92d58c5000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f92d56d3000)
libbrotlienc.so.1 => /lib/x86_64-linux-gnu/libbrotlienc.so.1 (0x00007f92d564a000)
libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007f92d563a000)
libutf8proc.so.2 => /lib/x86_64-linux-gnu/libutf8proc.so.2 (0x00007f92d55ed000)
libre2.so.5 => /lib/x86_64-linux-gnu/libre2.so.5 (0x00007f92d557c000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f92d5576000)
libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f92d529f000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f92d528c000)
liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f92d5269000)
libsnappy.so.1 => /lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007f92d525e000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f92d5242000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f92d5199000)
libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f92d5107000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f92d4fb8000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f92d4f93000)
/lib64/ld-linux-x86-64.so.2 (0x00007f92d7c87000)
libthrift-0.13.0.so => /lib/x86_64-linux-gnu/libthrift-0.13.0.so (0x00007f92d4edc000)
libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007f92d4eb9000)
libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f92d4e90000)
libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f92d4e6f000)
librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f92d4e4d000)
libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4 (0x00007f92d4ddf000)
libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f92d4dcc000)
libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f92d4d39000)
libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f92d4cec000)
libldap_r-2.4.so.2 => /lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f92d4c96000)
liblber-2.4.so.2 => /lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f92d4c83000)
libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f92d4b01000)
libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f92d492b000)
libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 (0x00007f92d48f4000)
libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 (0x00007f92d48ba000)
libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f92d4834000)
libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f92d4757000)
libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f92d4726000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f92d471f000)
libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f92d4710000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f92d46f2000)
libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f92d46d5000)
libgssapi.so.3 => /lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f92d4690000)
libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f92d455a000)
libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f92d4544000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f92d453b000)
libheimntlm.so.0 => /lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f92d452f000)
libkrb5.so.26 => /lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f92d449c000)
libasn1.so.8 => /lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f92d43f5000)
libhcrypto.so.4 => /lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f92d43bd000)
libroken.so.18 => /lib/x86_64-linux-gnu/libroken.so.18 (0x00007f92d43a2000)
libffi.so.7 => /lib/x86_64-linux-gnu/libffi.so.7 (0x00007f92d4396000)
libwind.so.0 => /lib/x86_64-linux-gnu/libwind.so.0 (0x00007f92d436c000)
libheimbase.so.1 => /lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f92d435a000)
libhx509.so.5 => /lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f92d430c000)
libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f92d41e3000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f92d41a6000)

 

Hello, thank you for your response!

I got the following output.

 

$ ldd $QHOME/l64/arrowkdb.so
        linux-vdso.so.1 (0x00007ffd3e3fc000)
        libarrow.so.1200 => <>/miniconda3/lib/python3.9/site-packages/pyarrow/libarrow.so.1200 (0x00007fbaf4ee9000)
        libparquet.so.1200 => <>/miniconda3/lib/python3.9/site-packages/pyarrow/libparquet.so.1200 (0x00007fbaf4760000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fbaf43cb000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fbaf4049000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fbaf3e31000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fbaf3a6c000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fbaf3868000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fbaf3660000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fbaf3440000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fbaf7c8a000)

nmcdonnell
New Contributor II
New Contributor II

Well as you can see it's trying to use libarrow 12 and it's picked up the miniconda version used by pyarrow.  I'm not sure that will work, even if this is what you intended.  Conda uses a sandboxed version of libstdc++ which isn't compatible with the system version.

Do you have a system (not conda) install of libarrow 9?

> I had to change somethings in the config and some environment variables to make it works but it builds successfully

What exactly did you do?

 

kxnoob
New Contributor

> I had to change somethings in the config and some environment variables to make it works but it builds successfully

Yeah - I had to change the CMakeLists to point to that libarrow12 (ARROW_LIBRARY and PARQUET_LIBRARY). I tried pointing to a libarrow9 version in a different miniconda environment. I also had to force it to use C++ 17.

 

Unfortunately, I am not able to install libarrow 9 on my system outside of conda. The most my system is able to yum install is libarrow8. It installs  libarrow.so.800 to /usr/lib64 but not a libparquet.so.800.

Do you have any suggestions for any workarounds to this? I'm not an expert on CMake or building C++ packages and whatnot. 

nmcdonnell
New Contributor II
New Contributor II

Arrowkdb should work with earlier libarrow versions, say 5.0 (or maybe even 3.0) since your are building from source.

What is your distro?

If you're able to install libarrow.so.800 you should also be able to install libparquet.so.800.  It'll be two separate yum commands similar to https://github.com/KxSystems/arrowkdb#linux (although that's apt on ubuntu)

See https://arrow.apache.org/install/#c-and-glib-c-packages-for-debian-gnulinux-ubuntu-and-centos:

Something like this from centos 7:

sudo yum install -y epel-release || sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1).noarch.rpm
sudo yum install -y https://apache.jfrog.io/artifactory/arrow/centos/$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1)/apache-arrow-release-latest.rpm
sudo yum install -y --enablerepo=epel arrow-devel # For C++
sudo yum install -y --enablerepo=epel parquet-devel # For Apache Parquet C++