Friday, February 17, 2017

How to upgrade Spark to 2.1.0

Step to upgrade Spark 2.1.0:

1. download required tag from https://github.com/apache/spark/tags and extract to spark folder
2. Go to spark folder and run below command
./dev/make-distribution.sh --name custom-spark --tgz -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.8 -Dhbase.version=1.0.0-cdh5.4.8 -Dflume.version=1.5.0-cdh5.4.8 -Dzookeeper.version=3.4.5-cdh5.4.8 -Phive -Phive-thriftserver

3. output of above command is spark-2.1.0-bin-custom-spark.tgz
- copy spark-2.1.0-bin-custom-spark.tgz to /usr/lib/spark/ and extract and then delete spark-2.1.0-bin-custom-spark.tgz
scp -r root@:/spark_2.1.0/spark-2.1.0/shekhar/spark-2.1.0-bin-custom-spark.tgz /usr/lib/spark/.
tar xvzf spark-2.1.0-bin-custom-spark.tgz
rm -r spark-2.1.0-bin-custom-spark.tgz

(for backup) hdfs dfs -get /user/spark/share/lib /root/shekhar (spark:spark)

4. create soft likes
ln -s /var/run/spark/work /usr/lib/spark/work
ln -s /etc/spark/conf /etc/alternatives/spark-conf
ln -s /usr/lib/spark/jars /usr/lib/spark/lib

5. update /usr/lib/spark/bin/spark-submit
vi /usr/lib/spark/bin/spark-submit
export HADOOP_CONF_DIR=/etc/hadoop/conf
export YARN_CONF_DIR=/etc/yarn/conf

6. run below command
chmod -R 777 /tmp/hive
export JAVA_HOME=/usr/

No comments: