Skip to content

Commit

Permalink
updated QFS configuration for replication
Browse files Browse the repository at this point in the history
By default, QFS saves files to the cluster using Reed-Solomon encoding. If you have
a cluster with smaller than 9 chunck servers, then can be come problematice from a
file resilience perspective (RS needs 9 chubks to reliably encode a file). This change
instead make QFS write all files with a replication factor of 2, meaning only 2
nodes are needed to reliably save a file. It's wort noting that RS encoding only
requires 1.5x the file size to save the file, and file relication of 2 requires 2x.
  • Loading branch information
michaelkamprath committed Feb 6, 2020
1 parent 7f5a0fa commit a956d82
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 1 deletion.
2 changes: 1 addition & 1 deletion spark-qfs-swarm/qfs-master/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ COPY start-qfs-master.sh /
USER spark
# create some useful bash aliases for when at bash shell prompt of this image
RUN echo 'export PATH=$PATH:$QFS_HOME/bin/:$QFS_HOME/bin/tools/' >> ~/.bash_aliases \
&& echo 'alias qfs="qfs -fs qfs://qfs-master:20000"' >> ~/.bash_aliases \
&& echo 'alias qfs="qfs -fs qfs://qfs-master:20000 -D fs.trash.minPathDepth=2 -D fs.createParams=2"' >> ~/.bash_aliases \
&& echo 'alias cptoqfs="cptoqfs -s qfs-master -p 20000"' >> ~/.bash_aliases \
&& echo 'alias cpfromqfs="cpfromqfs -s qfs-master -p 20000"' >> ~/.bash_aliases \
&& echo 'alias qfsshell="qfsshell -s qfs-master -p 20000"' >> ~/.bash_aliases \
Expand Down
9 changes: 9 additions & 0 deletions spark-qfs-swarm/worker-node/spark-conf/core-site.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,13 @@
<name>fs.qfs.metaServerPort</name>
<value>20000</value>
</property>
<property>
<!-- This parameter makes it so that all files written from spark are
created with a replication of 2 rather than using the defaault
Reed-Solomon encoding. If you have a cluster with 9 or greater
nodes, remove this option to favor the Reed-Solomon encoding.
-->
<name>fs.qfs.createParams</name>
<value>2</value>
</property>
</configuration>

0 comments on commit a956d82

Please sign in to comment.