Hadoop Distributed File System:

HDFS

Richa
Assistant Professor
CSE Dept
Chandigarh University

University Institute of Engineering (UIE)

CONTENTS

Introduction To HDFS
Features of HDFS/ Hadoop
Hadoop Daemons
HDFS Architecture/Design

University Institute of Engineering (UIE)

 INTRODUCTION
HDFS is an acronym for Hadoop Distributed File System.
It is one of the main component of Hadoop Architecture.
Hadoop is ideal for storing large amount of data and it uses
HDFS as its storage system.
It lets you connect nodes contained within clusters over which
data files are distributed.
Anyone can access and store the datafilee as one seamless file
system.
The accessing of data is done in a streaming manner i.e
applications are executed directly using MapReduce.

University Institute of Engineering (UIE)

INTRODUCTION CONTD..
HDFS is an Apache Foundation Project and also a subproject
of Apache Hadoop Project.
It is a Distributed File System that is designed to run on
Commodity Hardware.

University Institute of Engineering (UIE)

Commodity Hardware
It is the hardware that are affordable, easy to obtain, inexpensive or
widely available.
Its set up involves desktop computers that can run operating system
like: Windows, Linux and DOS.
Most of the commodity hardware or servers used in the production of
Hadoop clusters have an average ratio of disk space memory.
The servers are not designed specially as a pert of distributed storage
and processing but have been appropriated for this role in Hadoop.
Chance of node failure across the large cluster is high.
HDFS is designed to carry out working without interruption to the
user.

University Institute of Engineering (UIE)

HDFS is different from other DFS
HDFS has many similarities with the other distributed file system
but it is different as follows:
1. HDFS is Write Once and Read Many model
-It means that the file needs not to be changed.
-By this it: Relaxes the concurrency control requirements
Simplifies data coherency
Enables high throughput
Concurrency Control: ensures correct result for the operation and also
getting the result quickly.

University Institute of Engineering (UIE)

Data Coherency: data is same across the network or the data on all
the servers and to all the clients is synchronised .
High Throughput: the maximum rate at which the data can be
processed and accessed.

2. HDFS Viewpoint
-Locate processing logic near data instead of moving data to the
application.

3. It restricts one writer at one time for data writing. Bytes are
always appended at the end of the stream in order.

University Institute of Engineering (UIE)

Features of HDFS
1. Cost Effective System
-It does not require any expensive or specialised hardware in order
to implement.
-It can be implemented on the simple hardware i.e Commodity
Hardware which is inexpensive

2. Large cluster of nodes

- Thousands of nodes or machines are inter connected with each
other.
- This offers more computing power and huge storage systems to the
client.

University Institute of Engineering (UIE)

3. Parallel Processing
-HDFS supports parallel processing
-Data can be processed simultaneously across all the nodes within
the cluster.
-It saves time.

4. Distributed Data
-HDFS takes care of splitting and distributing the data across all the
nodes within a cluster
-It also replicates the data over the entire cluster

University Institute of Engineering (UIE)

5. Automatic Fail Over Management
- In case any of the node within the cluster fails, this framework
would replace that node with another node.
- It also replicates all the configuration setting and the data from the
failed machine to the newly replicated machine
- Admin needs not to worry about all this once Automatic Failover
Management has been properly configured on the cluster.

6. Data Locality Optimisation

- Instead of moving the PBs of data to processing logic, the
processing logic(Software) will be moved to the place where the
PBs of data resides for processing.

University Institute of Engineering (UIE)

7. Heterogenous Cluster
- A Hadoop cluster is made up of several nodes.
- Heterogenous Cluster is a cluster within which each node can be
from a different vendor and each node could be running at a
different version on a different operating system.

8. Scalability
- It is the ability of adding or removing the nodes or the hardware
components to or from the cluster.
- We can add or remove a node from a cluster without effecting the
cluster operation.
- Each individual hardware components such as RAM or hard disks
can be added or removed from the cluster.

University Institute of Engineering (UIE)

9. Fault tolerance:
-It is the working strength of a system in unfavourable condition.
HDFS is highly Fault Tolerant as here the data is divided into
blocks and the blocks are further stored on different DataNodes
within a cluster.
-So whenever any machine goes down, then client can easily access
their data from other machines which contain the replica of that
copy.

10. High Availability:

-HDFS is a highly available file system, data gets replicated among
the nodes in the HDFS cluster by creating a replica of the blocks on
the other slaves present in HDFS cluster.

University Institute of Engineering (UIE)

-Hence whenever a user wants to access his data, they can access
their data from the slaves which contains its blocks and which is
available on the nearest node in the cluster.

11. Data Reliability:

- Replication makes HDFS a reliable storage system.
- HDFS divides the data into blocks and these blocks are stored on
nodes present in HDFS cluster. It stores data reliably by creating a
replica of each and every block present on the nodes present in the
cluster and hence provides fault tolerance facility and makes HDFS
reliable.

University Institute of Engineering (UIE)

12. Replication:
-Data Replication is one of the most important and unique features
of Hadoop HDFS. In HDFS replication of data is done to solve the
problem of data loss in unfavourable conditions like crashing of a
node, hardware failure, and so on.
-As data is replicated across a number of machines in the cluster by
creating blocks. The process of replication is maintained at regular
intervals of time by HDFS and HDFS keeps creating replicas of
user data on different machines present in the cluster.
-So whenever any machine in the cluster gets crashed, the user can
access their data from other machines which contain the blocks of
that data. Hence there is no possibility of losing of user data.

University Institute of Engineering (UIE)

Hadoop Daemons
HDFS is responsible for storing huge amount of data on the cluster
in Hadoop.
To understand how HDFS achieves all this, we need to first
understand the HDFS Daemons.

Daemons???????????
-Daemons in computing term is a process that runs in the
background.

So Hadoop is having five such daemons that runs in the background

to store this huge amount of data and process it.

University Institute of Engineering (UIE)

 Five Hadoop Daemons are:

1. NameNode
2. DataNode
3. SecondaryNameNode
4. JobTracker
5. TaskTracker

Each daemon runs separately in its own JVM.

Basically first three are the daemons of HDFS and last two
are the daemons of MapReduce.

University Institute of Engineering (UIE)

HDFS Design/ HDFS Architecture
HDFS is a fundamental or building block of Hadoop.
Its design is based on Master-Slave Model.

Master-Slave Model
A cluster is having thousands of nodes interconnected with
each other.
There is one Master node within the cluster that is also
referred to as Head of the cluster.
All other nodes are the Slave nodes or also referred to as the
Worker nodes.

University Institute of Engineering (UIE)

Description of Architecture/Design
NameNode
Cluster
Metadata Block Location

fsimage & edit log

DN1 DN4

DN2 DN5

DN3 DN6
Rack1 Rack2

University Institute of Engineering (UIE)

Description
It is having a single Active NameNode and multiple
DataNode
At any given point of time, there will be only one active
NameNode.
NameNode and DataNode are basically the server
machines.
Together they form a cluster of machines as shown in the
figure.

University Institute of Engineering (UIE)

1. NameNode: Master Node
NameNode is the master of the cluster.
It manages the file system namespace i.e it stores the
Metadata of files and directories of the filesystem.
Also it stores the location of all the blocks.
As shown in the figure, we can say that the NameNode
includes the following :
A. Metadata
B. Block Location
C. fsimage and Edit log

University Institute of Engineering (UIE)

A.Metadata
Metadata means data about data.
Here it contains the following:
File name abc.txt

File Size 300MB, 150 MB

No. of Blocks 3

Block ID [1, 2, 3]

User Hduser

Group Hadoop

Permission rw

Replication 3

Block size 128MB, 64MB

University Institute of Engineering (UIE)

A. Metadata contd..
This metadata is stored on both the main memory as well as the
hard disks.
Why main memory?????
-It is stored on the main memory to serve the client request.
- If the client wants to access the file abc.txt, then client should
contact the NameNode.
- NameNode will look for metadata in the main memory and
serve the client request accordingly.
- It may allow access or deny it according to the file permission.
- So, for fast access of data in HDFS, metadata is stored in the
main memory.

University Institute of Engineering (UIE)

Why hard disks??????
-If the NameNode crashes then whole data in the main memory is
lost, this is when metadata stored on the hard disk comes to
rescue.
- Only metadata is recovered using this not the block location
- Basically this NameNode stores the metadata, it does not stores
the actual data.

Where and how the actual data is stored??????

- The actual data in HDFS is stored as blocks.
- Blocks in HDFS is same as blocks in the traditional file system
with block size 4K or 8K.
University Institute of Engineering (UIE)
 When the file is to be stored in HDFS, it is first divided into Blocks
and the blocks are stored on the DataNodes.
So, here comes the role of DataNode.

University Institute of Engineering (UIE)

2. DataNode: Slave Node
DataNode acts as workhorses of HDFS.
The actual data is stored in the DataNode.
Files are divided into blocks and then these blocks are stored in different
DataNodes across the cluster.
for e.g: If the file size is 150 MB and the block size is 64MB(default)
then:

64 MB 64 MB 22 MB

Block ID 1 Block ID 2 Block ID 3

64+64+22=150 MB and no. of blocks are 3.

University Institute of Engineering (UIE)
After dividing Blocks are stored one by one.
While it is being stored, it is also replicated and stored in different
DataNodes as shown:

64MB 64MB 64MB 22MB

DN1 DN4

64MB 64MB 64MB 22MB

DN2 DN5

22MB
DN3 DN6

Rack1 Rack2

University Institute of Engineering (UIE)

 Block ID 1 is of 64 MB, it replicates and stored in DN1, the second
and third replica of Block ID 1 is stored in DN4 and DN5
respectively.
Similarly Block ID 2 is of 64 MB, and it is stored in DN2, DN4 and
DN5.
Block ID 3 is only 22 MB, and is stored in DN3, DN4 and DN5.
Last Block Size<=Block Size
All it depends on the file size.

University Institute of Engineering (UIE)

Why Blocks are Replicated???
Replication makes HDFS a reliable storage system.
If blocks are lost or corrupted, replicas of these blocks are available
on different DataNodes.
A. e.g1 If block ID 1 of abc.txt file is lost or gets corrupted, then
replicas of this block is available in DN4 and DN5.
B. e.g2 If DataNode itself down, replicas of all the blocks in that
node would be available in different DataNodes
-If DN5 is down, replicas are available in DN1, DN2, DN3 and
DN4.
DataNode does not have any information about the files but they do
have the information of the blocks.

University Institute of Engineering (UIE)

B. Block Location :NameNode
When DataNode start up it will send Block Report to the NameNode
Block Report????
-DataNode will scan through its hard disk, generates a list of all the
blocks on it and sends this report to the NameNode. This report is
referred to as Block Report.
-This report is send periodically and contains the following:

Block ID Generation Size

Stamp
1 x 64 MB
2 y 64 MB
3 z 22 MB

-Generation stamp is used for append operations.

University Institute of Engineering (UIE)

 Using this block report from all the DataNodes, NameNode use
Block ID to Location Mapping.
This mapping is known as Block Location.
In Metadata the filename is mapped with the Block IDs and in
Block Location the Block IDs are mapped with the DataNodes
address.
As a result of this, for a given file NameNode knows on which
DataNodes the Blocks are located.
Block ID DataNode
1 DN1, DN4, DN5
2 DN2, DN4, DN5
3 DN3, DN4, DN5

University Institute of Engineering (UIE)

 So, the NameNode knows that the blocks of abc.txt file is
stored on DN1, DN2, DN3, DN4 and DN5.
Using Metadata and Block Location, NameNode has a
complete knowledge of the entire cluster.

University Institute of Engineering (UIE)

HeartBeat Signal
It handles coordination.
It is used by the NameNode to check whether the DataNode is alive/
active or not.
HeartBeat signal is send from the DataNode to the NameNode after
every 3 seconds.
If the DataNode is down, the NameNode would not receive the
HeartBeat signal from that DataNode.
- In this case, the NameNode will wait for 5-10 minutes, after that
it will consider that DataNode as Out Of Service(OOS).
It carries the information like: Total Disk Space, Used Space, Free
Space and Data Transfer in Progress.

University Institute of Engineering (UIE)

C. fsimage and edit log: NameNode
fsimage represents the complete snapshot of the entire HDFS.
Its format is very efficient to read.
The complete file system image means the complete details of
files and blocks on HDFS and also includes:
A. Namespace image: metadata
B. Edit Log
Any modification done on HDFS is recorded in edit log.
Edit log is nothing but the log of activities on HDFS performed
by the clients.
It keeps on piling up and grows as the activities on the HDFS
keeps on happening.

University Institute of Engineering (UIE)

 e.g Let us consider copying of abc.txt file on HDFS. This
operation is recorded in edit log and the data stored on the
main memory is also updated. as a result client always sees
the updated fs namespace.
Each operation in the edit log is called the record or
transaction.
Each transaction is identified by the transaction ID.
Transaction ID is incremented sequentially.
Now consider if the NameNode crashes, then the whole file
system would go down, this is also known as Single Point Of
Failure(SPOF).

University Institute of Engineering (UIE)

 Due to SPOF, all operation done till now are erased from the main
memory.
So when the NameNode restarts, fsimage is first loaded into the
main memory.
By doing this NameNode will restore all the previous state but to
restore the recent operation, it is done by replaying all the
operation from current edit log.
There are many problems with this edit log as overtime edit log
will accumulate lot of records and the size become very very large
and in extreme cases it can fill up the disk space of NameNode.
Also replaying lot of records from edit log would take a lot of time,
as a result NameNodes boot up time is increased.
This is when Secondary NameNode comes into pictures.

University Institute of Engineering (UIE)

 3. Secondary NameNode
Secondary NameNode does not function like NameNode. If
NameNode crashes then Secondary NameNode is used as the
Master Node of the cluster is a wrong statement.
It does not replace the NameNode.
The main function of Secondary NameNode is to combine the
fsimage and the edit log to produce a new fsimage(compressed)
so that NameNodes main memory does not fills up due to ever
increasing edit log.
Also it creates checkpoints of the NameNode. Also known as
Checkpoint Node.
Basically Secondary NameNode is a Java program that just
merges the fsimage and the edit log and creates checkpoints.

University Institute of Engineering (UIE)

 As this merger is complex too so Secondary NameNode should
be running on a good hardware configuration as it requires
good computing resources.
Due to merging, less time will be taken to replay all the records
and because of this the boot up time of the NameNode is also
reduced.
Creating new fsimage is too complex. Sometimes it takes more
time, during this time client access to HDFS is restricted.
At this point checkpoint comes, now NameNode does not
restricts access to the client, it will be active all the time.
And Checkpointing is triggered after 1 hour or 1 million
transactions in the edit log whatever is early.

University Institute of Engineering (UIE)

4. JobTracker: Master Process
JobTracker is basically a MapReduce Daemon.
It is a Master Process.
Each cluster can have a single JobTracker and multiple
TaskTrackers.
The foremost function of the JobTracker is Resource
Management that is:
1. tracking the TaskTrackers
2. tracking its progress
3. fault tolerance
JobTracker creates and run the Job on the NameNode and
allocates the job to the TaskTracker.

University Institute of Engineering (UIE)

 Whenever the client submits the job to the JobTracker, it divides
the job and splits it into the tasks.
After this the JobTracker decides what tasks should run on
which worker node.
The process of assigning tasks to the worker nodes is referred to
as Task scheduling.
JobTracker is responsible for monitoring the tasks assigned to
the worker node.
The JobTracker carries out the communication between the
client and the TaskTracker by making use of Remote
Procedure Calls(RPC).

University Institute of Engineering (UIE)

 RPC can be considered as a language that is used by the
processes to communicate with each other.
JobTracker keeps track of all the jobs and the associated tasks
within the main memory.
The memory requirement is huge, it depends on the number of
tasks and tasks differs from one job to another.

University Institute of Engineering (UIE)

5. TaskTracker: Slave Process
TaskTracker is again a MapReduce Daemon.
It is a Slave Process.
Within a cluster there can be multiple TaskTracker.
It is the responsibility of the TaskTracker to execute all the tasks
theatre assigned by the JobTracker.
Each DataNode/SlaveNode can have a single TaskTracker
running under them.
Within each TaskTracker there are number of Map and reduce
slots.
These slots are referred to as a Task slots.

University Institute of Engineering (UIE)

 The number of Map and Reduce slots determine how many Map
and Reduce task can be executed simultaneously.
The TaskTracker is pre-configured with the number of slots
indicating the number of tasks it can accept.
When a JobTracker tries to schedule a task, it looks for an empty
slot in the TaskTracker running on the same server which hosts
the DataNode, where the data for that task resides. If not
found, it looks for the machine in the same rack.
TaskTracker sends the HeartBeat signal to the JobTracker after
every 3 seconds and if the JobTracker doesn't receive this
signal, it will consider that TaskTracker as in dead state.

University Institute of Engineering (UIE)

 Cluster
MapReduce Layer
JobTracker

HDFS Layer NameNode

Map-
Reduce
TaskTracker TaskTracker
Layer

HDFS DataNode DataNode

Layer

University Institute of Engineering (UIE)

 THANK YOU

University Institute of Engineering (UIE)