Skip to content

Commit

Permalink
fixed some minor issues
Browse files Browse the repository at this point in the history
  • Loading branch information
michaelkamprath committed Feb 9, 2020
1 parent 9e10e00 commit 86ee08b
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 4 deletions.
9 changes: 6 additions & 3 deletions spark-qfs-swarm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,11 @@ This set up depends on have a GlusterFS volume mounted at `/mnt/gfs` and a norma
* `/mnt/data/qfs/checkpoint` - Where the QFS metaserver will store the fulesystem check points. This actually only needs to exist on the master node.
* `/mnt/data/spark` - The local working directory for spark

You can adjust these as you see fit, but be sure to update the mounts specified in `deploy-spark-qfs-swarm.yml`.
You can adjust these as you see fit, but be sure to update the mounts specified in `deploy-spark-qfs-swarm.yml`. Then build the docker images from in this project's directory:

```
./build-images.sh
```

Before the first time you run this cluster, you will need to initialize the QFS file system. Do so by launching a qfs-master container on the master node:
```
Expand All @@ -31,9 +35,8 @@ qfs -mkdir /history/spark-event
exit
```

Finally, to start up the Spark cluster in your Docker swarm, `cd` into this project's directory and:
Finally, to start up the Spark cluster in your Docker swarm:
```
./build-images.sh
docker stack deploy -c deploy-spark-qfs-swarm.yml spark
```

Expand Down
2 changes: 1 addition & 1 deletion spark-qfs-swarm/worker-node/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ RUN mkdir -p /data/qfs/ \
# SPARK
ENV SPARK_PACKAGE spark-${SPARK_VERSION}-bin-hadoop${HADOOP_MINOR_VERSION}
ENV SPARK_HOME /usr/spark-${SPARK_VERSION}
ENV SPARK_DIST_CLASSPATH="$QFS_HOME/lib/hadoop-$HADOOP_VERSION-qfs-$QFS_VERSION.jar:$QFS_HOME/lib/qfs-access-$QFS_VERSION"
ENV SPARK_DIST_CLASSPATH="$QFS_HOME/lib/hadoop-$HADOOP_VERSION-qfs-$QFS_VERSION.jar:$QFS_HOME/lib/qfs-access-$QFS_VERSION.jar"
ENV HADOOP_CONF_DIR=${SPARK_HOME}/conf/
ENV PATH $PATH:${SPARK_HOME}/bin
ARG SPARK_DOWNLOAD_URL="https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/${SPARK_PACKAGE}.tgz"
Expand Down

0 comments on commit 86ee08b

Please sign in to comment.