Skip to content
This repository has been archived by the owner on Feb 17, 2023. It is now read-only.

Commit

Permalink
SKYHOOK-207: [Java] Add optional Java and JNI builds to deploy_skyhoo…
Browse files Browse the repository at this point in the history
…k.sh script
  • Loading branch information
JayjeetAtGithub committed Jul 17, 2021
1 parent 728b52e commit c80de1c
Show file tree
Hide file tree
Showing 3 changed files with 40 additions and 8 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ where mon1, mon2, osd1, etc. are the internal hostnames of the nodes.
3. Build and install the SkyhookDM CLS plugins and PyArrow (with Rados Parquet extensions):

```bash
./deploy_skyhook.sh osd1,osd2,osd3 arrow-master true true
./deploy_skyhook.sh osd1,osd2,osd3
```
This will build the CLS plugins as shared libraries and deploy them to the OSD nodes.

Expand Down
2 changes: 1 addition & 1 deletion cpp/src/arrow/adapters/arrow-rados-cls/docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ where mon1, mon2, osd1, etc. are the internal hostnames of the nodes.

2. Build and install the SkyhookDM CLS plugins and PyArrow (with Rados Parquet extensions) using [this](../scripts/deploy_skyhook.sh) script. For example,
```bash
./deploy_skyhook.sh osd1,osd2,osd3 arrow-master
./deploy_skyhook.sh osd1,osd2,osd3
```
This will build the CLS plugins as shared libraries and deploy them to the OSD nodes.

Expand Down
44 changes: 38 additions & 6 deletions cpp/src/arrow/adapters/arrow-rados-cls/scripts/deploy_skyhook.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,32 +19,57 @@

set -eu

if [[ $# -lt 2 ]] ; then
echo "./deploy_skyhook.sh [nodes] [skyhook-branch] [deploy cls libs] [build python bindings]"
if [[ $# -lt 1 ]] ; then
echo "./deploy_skyhook.sh [nodes] [branch] [deploy CLS libs] [build python] [build java] [nproc]"
exit 1
fi

NODES=$1
BRANCH=$2
BRANCH=${2:-arrow-master}
DEPLOY_CLS_LIBS=${3:-true}
BUILD_PYTHON_BINDINGS=${4:-true}
BUILD_JAVA_BINDINGS=${5:-false}
NPROC=${6:-4}

IFS=',' read -ra NODE_LIST <<< "$NODES"; unset IFS

apt update
apt install -y python3 python3-pip python3-venv python3-numpy cmake libradospp-dev rados-objclass-dev
apt install -y python3 \
python3-pip \
python3-venv \
python3-numpy \
cmake \
libradospp-dev \
rados-objclass-dev \
llvm

if [ ! -d "/tmp/arrow" ]; then
git clone https://github.com/uccross/arrow /tmp/arrow
fi

cd /tmp/arrow
git submodule update --init --recursive
git checkout $BRANCH
mkdir -p cpp/release
cd cpp/release

cmake -DARROW_CLS=ON -DARROW_PARQUET=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_ZLIB=ON -DARROW_BUILD_EXAMPLES=ON -DPARQUET_BUILD_EXAMPLES=ON -DARROW_PYTHON=ON -DARROW_DATASET=ON -DARROW_CSV=ON -DARROW_WITH_LZ4=ON -DARROW_WITH_ZSTD=ON ..
make -j4 install
cmake -DARROW_CLS=ON \
-DARROW_PARQUET=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_BUILD_EXAMPLES=ON \
-DPARQUET_BUILD_EXAMPLES=ON \
-DARROW_PYTHON=ON \
-DARROW_ORC=ON \
-DARROW_JAVA=ON \
-DARROW_JNI=ON \
-DARROW_DATASET=ON \
-DARROW_CSV=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_ZSTD=ON \
..

make -j${NPROC} install

if [[ "${BUILD_PYTHON_BINDINGS}" == "true" ]]; then
export WORKDIR=${WORKDIR:-$HOME}
Expand Down Expand Up @@ -77,5 +102,12 @@ if [[ "${DEPLOY_CLS_LIBS}" == "true" ]]; then
done
fi

if [[ "${BUILD_JAVA_BINDINGS}" == "true" ]]; then
apt install -y default-jdk maven
cd /tmp/arrow/java
mvn clean install
mvn clean install -P arrow-jni -pl format,memory,vector -am -Darrow.cpp.build.dir=/tmp/arrow/cpp/build/release -Dmaven.test.skip=true -Dcheckstyle.skip -Dos.detected.name=linux -Dos.detected.arch=x86_64 -Dos.detected.classifier=linux-x86_64
fi

export LD_LIBRARY_PATH=/usr/local/lib
cp /usr/local/lib/libparq* /usr/lib/

0 comments on commit c80de1c

Please sign in to comment.