What is Apache Hadoop?
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
Apache Hadoop includes these modules:
- Hadoop Common: The common utilities that support the other Hadoop modules.
- Hadoop Distributed File System (HDFS): A distributed file system that provides high-throughput access to application data.
- Hadoop YARN: A framework for job scheduling and cluster resource management.
- Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.
|Version||Release Notes||Release Date|
|2.7.3||Click here - 2.7.3||2016-01-25|
|2.6.4||Click here - 2.6.4||2016-02-11|
|2.7.2||Click here - 2.7.2||2016-01-25|
|2.6.3||Click here - 2.6.3||2015-12-17|
|2.6.2||Click here - 2.6.2||2015-10-28|
|2.7.1||Click here - 2.7.1||2015-07-06|
Hadoop overview and HDFS
Hadoop is an open-source software framework for storage and large-scale processing of data-sets in a distributed computing environment.
It is sponsored by Apache Software Foundation.
It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.
- Hadoop was created by Doug Cutting and Mike Cafarella in 2005.
- Cutting, who was working at Yahoo! at the time, named it after his son's toy elephant.
- It was originally developed to support distribution for the search engine project.
Hadoop Distributed File System (HDFS): A distributed file system that provides high-throughput access to application data.
Hadoop MapReduce: A software framework for distributed processing of large data sets on compute clusters.
Suitable for applications with large data sets.
Can be built out of commodity hardware.
HDFS cluster consists of a single Namenode, a master server that manages the file system namespace and regulates access to files by clients.
The DataNodes manage storage attached to the nodes that they run on.
HDFS exposes a file system namespace and allows user data to be stored in files.
A file is split into one or more blocks and set of blocks are stored in DataNodes.
DataNodes: serves read, write requests, performs block creation, deletion, and replication upon instruction from Namenode.
HDFS is designed to store very large files across machines in a large cluster.
Each file is a sequence of blocks.
All blocks in the file except the last are of the same size.
Blocks are replicated for fault tolerance.
The Namenode receives a Heartbeat and a BlockReport from each DataNode in the cluster.
BlockReport contains all the blocks on a Datanode.
Common commands used:-
ls Usage: hadoop fs –ls Path(dir/file path to list).
Cat Usage: hadoop fs -cat PathOfFileToView
Link for hadoop shell commands:- https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/FileSystemShell.html
Installation of Hadoop on ubuntu
Creating Hadoop User:
sudo addgroup hadoop
Adding a user:
sudo adduser --ingroup hadoop hduser001
su -hduser001 ssh-keygen -t rsa -P "" cat .ssh/id rsa.pub >> .ssh/authorized_keys
Note: If you get errors [bash: .ssh/authorized_keys: No such file or directory] whilst writing the authorized key. Check here.
Add hadoop user to sudoer's list:
sudo adduser hduser001 sudo
sudo add-apt-repository ppa:hadoop-ubuntu/stable sudo apt-get install hadoop
Installation or Setup on Linux
A Pseudo Distributed Cluster Setup Procedure
Install JDK1.7 and set JAVA_HOME environment variable.
Create a new user as "hadoop".
Setup password-less SSH login to its own account
su - hadoop ssh-keygen << Press ENTER for all prompts >> cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 0600 ~/.ssh/authorized_keys
Verify by performing
Disable IPV6 by editing
/etc/sysctl.confwith the followings:
net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1
Check that using
(should return 1)
Installation and Configuration:
Download required version of Hadoop from Apache archives using
cd /opt/hadoop/ wget http:/addresstoarchive/hadoop-2.x.x/xxxxx.gz tar -xvf hadoop-2.x.x.gz mv hadoop-2.x.x.gz hadoop (or) ln -s hadoop-2.x.x.gz hadoop chown -R hadoop:hadoop hadoop
.kshrcbased on your shell with below environment variables
export HADOOP_PREFIX=/opt/hadoop/hadoop export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop export JAVA_HOME=/java/home/path export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin:$JAVA_HOME/bin
$HADOOP_HOME/etc/hadoopdirectory edit below files
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:8020</value> </property> </configuration>
mapred-site.xmlfrom its template
cp mapred-site.xml.template mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
<configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>localhost</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:///home/hadoop/hdfs/namenode</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:///home/hadoop/hdfs/datanode</value> </property> </configuration>
Create the parent folder to store the hadoop data
mkdir -p /home/hadoop/hdfs
Format NameNode (cleans up the directory and creates necessary meta files)
hdfs namenode -format
Start all services:
start-dfs.sh && start-yarn.sh mr-jobhistory-server.sh start historyserver
Instead use start-all.sh (deprecated).
Check all running java processes
Namenode Web Interface: http://localhost:50070/
Resource manager Web Interface: http://localhost:8088/
To stop daemons(services):
stop-dfs.sh && stop-yarn.sh mr-jobhistory-daemon.sh stop historyserver
Instead use stop-all.sh (deprecated).