Skip to content
This repository has been archived by the owner on Nov 23, 2017. It is now read-only.

Use Hadoop 2.6 instead of 2.4 #88

Open
wants to merge 2 commits into
base: branch-1.6
Choose a base branch
from
Open

Use Hadoop 2.6 instead of 2.4 #88

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7a0f890
use hadoop 2.6
Feb 2, 2017
e4a6420
use hadoop 2.6
Feb 2, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ephemeral-hdfs/init.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,11 @@ case "$HADOOP_MAJOR_VERSION" in
cp /root/hadoop-native/* /root/ephemeral-hdfs/lib/native/
;;
yarn)
wget http://s3.amazonaws.com/spark-related-packages/hadoop-2.4.0.tar.gz
wget http://s3.amazonaws.com/spark-related-packages/hadoop-2.6.0.tar.gz
echo "Unpacking Hadoop"
tar xvzf hadoop-*.tar.gz > /tmp/spark-ec2_hadoop.log
rm hadoop-*.tar.gz
mv hadoop-2.4.0/ ephemeral-hdfs/
mv hadoop-2.6.0/ ephemeral-hdfs/

# Have single conf dir
rm -rf /root/ephemeral-hdfs/etc/hadoop/
Expand Down
4 changes: 2 additions & 2 deletions persistent-hdfs/init.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,11 @@ case "$HADOOP_MAJOR_VERSION" in
cp /root/hadoop-native/* /root/persistent-hdfs/lib/native/
;;
yarn)
wget http://s3.amazonaws.com/spark-related-packages/hadoop-2.4.0.tar.gz
wget http://s3.amazonaws.com/spark-related-packages/hadoop-2.6.0.tar.gz
echo "Unpacking Hadoop"
tar xvzf hadoop-*.tar.gz > /tmp/spark-ec2_hadoop.log
rm hadoop-*.tar.gz
mv hadoop-2.4.0/ persistent-hdfs/
mv hadoop-2.6.0/ persistent-hdfs/

# Have single conf dir
rm -rf /root/persistent-hdfs/etc/hadoop/
Expand Down
2 changes: 1 addition & 1 deletion spark/init.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ else
elif [[ "$HADOOP_MAJOR_VERSION" == "2" ]]; then
wget http://s3.amazonaws.com/spark-related-packages/spark-$SPARK_VERSION-bin-cdh4.tgz
else
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do older versions like 1.2.1 have a hadoop-2.6 artifact ? or was it not supported till recently ?

wget http://s3.amazonaws.com/spark-related-packages/spark-$SPARK_VERSION-bin-hadoop2.4.tgz
wget http://s3.amazonaws.com/spark-related-packages/spark-$SPARK_VERSION-bin-hadoop2.6.tgz
fi
if [ $? != 0 ]; then
echo "ERROR: Unknown Spark version"
Expand Down