Hadoop Tutorial

Hadoop Tutorial for Big Data Fanatics – The Best way of Learning Hadoop

Hadoop Tutorial – One of the most searched terms on the internet today. Do you know the reason? It is because Hadoop is that the major part or framework of big data.

If you don’t know anything about Big Data then you are in major trouble. But don’t worry I have something for you which is completely FREE – 520+ Big Data Tutorials. This free tutorial series will make you a master of big data in only a few weeks. Also, I have explained a little about Big Data in this blog.

“Hadoop may be a technology to store massive datasets on a cluster of cheap machines during a distributed manner”.

Hadoop was developed by Doug Cutting and Mike Cafarella.

Doug Cutting’s kid named Hadoop to at least one of his toy that was a yellow elephant. Doug then used the name for his open source project because it had been easy to spell, pronounce, and not used elsewhere.

Interesting, right?

Hadoop Tutorial

Now, let us begin our interesting Hadoop tutorial with the basic introduction to Big Data.

What is Big Data?

Big Data refers to the datasets overlarge and sophisticated for traditional systems to store and process. the main problems faced by Big Data majorly falls under three Vs. they’re volume, velocity, and variety.

Do you know – Every minute we send 204 million emails, generate 1.8 million Facebook likes, send 278 thousand Tweets, and up-load 200,000 photos to Facebook.

Volume: the data is getting generated so as of Tera to petabytes. the most important contributor to knowledge is social media. as an example, Facebook generates 500 TB of knowledge a day. Twitter generates 8TB of knowledge daily.

Velocity: Every enterprise has its requirement of the time-frame within which they need process data. Many use cases like MasterCard fraud detection have only a couple of seconds to process the data in real-time and detect fraud. Hence there’s a requirement of the framework which is capable of high-speed data computations.

Variety: Also the info from various sources have varied formats like text, XML, images, audio, video, etc. Hence the big Data technology should have the potential of performing analytics on a spread of knowledge.

Why Hadoop is Invented?

Let us discuss the inadequacies of the traditional approach which led to the invention of Hadoop –

1. Storage for Large Datasets

The conventional RDBMS is incapable of storing huge amounts of Data. The cost of data storage is available RDBMS is very high. As it incurs the cost of hardware and software both.

2. Handling data in different formats

The RDBMS is capable of storing and manipulating data in a structured format. But in the real world, we have to deal with data in a structured, unstructured, and semi-structured format.

 3. Data getting generated with high speed:

The data in creeping out in the order of tera to petabytes daily. Hence we need a system to process data in real-time within a few seconds. The traditional RDBMS fails to provide real-time processing at great speeds.

What is Hadoop?

Hadoop is that the solution to above Big Data problems. it’s the technology to store massive datasets on a cluster of cheap machines during a distributed manner. Not only this it provides Big Data analytics through distributed computing framework.

Hadoop is an open-source software developed as a project by Apache Software Foundation. Doug Cutting created Hadoop. within the year 2008 Yahoo gave Hadoop to Apache Software Foundation. Since then two versions of Hadoop have come. Version 1.0 within the year 2011 and version 2.0.6 within the year 2013. Hadoop comes in different flavors like Cloudera, IBM BigInsight, MapR, and Hortonworks.

Prerequisites to Learn Hadoop

• Familiarity with some basic Linux Command – Hadoop is set up over Linux Operating System preferable Ubuntu. So one must know certain basic Linux commands. These commands are for uploading the file in HDFS, downloading the file from HDFS, and so on.
• Basic Java concepts – Folks want to learn Hadoop can get started in Hadoop while simultaneously grasping basic concepts of Java. We can write map and reduce functions in Hadoop using additional languages too. And these are Python, Perl, C, Ruby, etc. This is possible via streaming API. It supports reading from standard input and writing to standard output. Hadoop also has high-level abstraction tools like Pig and Hive which do not require knowledge of Java.

Hadoop consists of three core components –

• Hadoop Distributed filing system is that the storage layer of Hadoop.

• Map-Reduce is that the processing layer of Hadoop.

• YARN is that the resource management layer of Hadoop.

Core Components of Hadoop

Let us understand these Hadoop components in detail.

1. HDFS

Short for Hadoop Distributed File System provides for distributed storage for Hadoop. HDFS has a master-slave topology.

Master is a high-end machine whereas slaves are inexpensive computers. The Big Data files get separated into the number of blocks. Apache Hadoop stores these blocks in a distributed fashion on the cluster of slave nodes. On the master, we have metadata stored.

HDFS has two daemons running for it. They are :

NameNode : NameNode performs following functions –

•  NameNode Daemon runs on the master machine.
•  It is responsible for maintaining, monitoring, and managing DataNodes.
• NameNode records the metadata of the files like the location of blocks, file size, permission, hierarchy, etc.
• Namenode captures all the changes to the metadata like deletion, creation, and renaming of the file in edit logs.
• It regularly receives heartbeat and block reports from the DataNodes.

DataNode: The various functions of DataNode are as follows –

• DataNode runs on the slave machine.
• It stores the actual business data.
• It serves the read-write request from the user.
• DataNode does the groundwork of creating, replicating, and deleting the blocks on the command of NameNode.
• After every 3 seconds, by default, it sends a heartbeat to NameNode reporting the health of HDFS.

Erasure Coding in HDFS

Till Hadoop 2.x replication is that the only method for providing fault tolerance. Hadoop 3.0 introduces another method called erasure coding. Erasure coding provides an equivalent level of fault tolerance but with lower storage overhead.

Erasure coding is typically utilized in RAID (Redundant Array of Inexpensive Disks) kind of storage. RAID provides erasure coding via striping. In this, it divides the data into smaller units like bit/byte/block and stores the consecutive units on different disks. Hadoop calculates parity bits for every one of those cells (units). We call this process as encoding. In the event of loss of certain cells, Hadoop computes these by decoding. Decoding may be a process during which lost cells gets recovered from remaining original and parity cells.

Erasure coding is usually used for warm or cold data which undergo less frequent I/O access. The replication factor of the Erasure coded file is usually one. we cannot change it by -setrep command. Under erasure coding storage overhead isn’t quite 50%.

Under conventional Hadoop storage replication factor of three is the default. It means 6 blocks will get replicated into 6*3 i.e. 18 blocks. this provides a storage overhead of 200%. As against this in Erasure coding technique there are 6 data blocks and three parity blocks. this provides a storage overhead of fifty.

The File System Namespace

HDFS supports hierarchical file organization. One can create, remove, move, or rename a file. NameNode maintains the file system Namespace. NameNode records the changes in the Namespace. It also stores the replication factor of the file.

2. MapReduce 

It is the data processing layer of Hadoop. It processes data in two phases.

They are:-

Map Phase- This phase applies business logic to the info. The input data gets converted into key-value pairs.                                                                              

Reduce Phase- This phase takes as input the output of the Map Phase. It applies aggregation based on the key of the key-value pairs.

Map-Reduce works in the following way:

• he client specifies the file for input to the Map function. It splits it into tuples
• Map function defines key and value from the input file. The output of the map function is this key-value pair.
• MapReduce framework sorts the key-value pair from map function.
• The framework merges the tuples having the same key together.
• The reducers get these merged key-value pairs as input.
• Reducer applies aggregate functions on a key-value pair.
• The output from the reducer gets written to HDFS.

3. YARN

Short for Yet Another Resource Locator has the following components:-

Resource Manager

• Resource Manager runs on the master node.
• Resource Manager knows where the location of slaves (Rack Awareness).
• Resource Manager is aware of how much resources each slave has.
• Resource Scheduler is one of the important service run by the Resource Manager.
• Resource Scheduler decides how the resources get assigned to various tasks.
• Application Manager is one more service run by Resource Manager.
• Application Manager negotiates the first container for an application.
• Resource Manager keeps track of the heartbeats from the Node Manager.

Node Manager

•  It runs on slave machines.
• It manages containers. Containers are nothing but a fraction of Node Manager’s resource capacity
•  The node manager monitors the resource utilization of each container.
• It sends a heartbeat to the Resource Manager.

Job Submitter

The application startup process is as follows:-

• The client submits the job to the Resource Manager.
• Resource Manager contacts Resource Scheduler and allocates container.
• Immediately Resource Manager contacts the relevant Node Manager to launch the container.
• Container runs Application Master.

The basic impression of YARN was to split the task of resource management and job scheduling. Job submitter has one global Resource Manager and per-application Application Master. An application can be either one job or Directed Acyclic Graph of jobs.

The Resource Manager’s job is to assign resources to various competing applications. Node Manager runs on the slave nodes. It is responsible for containers, monitoring resource utilization, and informing about the same to Resource Manager.

The job of the Application master is to negotiate resources from the Resource Manager. It also works with NodeManager to execute and monitor the tasks.

Why Hadoop?

Let us now understand why Big Data Hadoop is extremely popular, why Apache Hadoop captures quite 90% of the big data market.

Apache Hadoop isn’t only a storage system but also a platform for data storage also as processing. it’s scalable (as we will add more nodes on the fly), Fault-tolerant (Even if nodes go down, data processed by another node).

Following characteristics of Hadoop make it a singular platform:

• Flexibility to store and mine any sort of data whether it’s structured, semi-structured, or unstructured. it’s not bounded by one schema.
• Excels at processing data of complex nature. Its scale-out architecture divides workloads across many nodes. Another added benefit is that its flexible file-system eliminates ETL bottlenecks.
• Scales economically, as conversed it can deploy on commodity hardware. aside from this, its open-source nature secures against vendor lock.

What is Hadoop Architecture?

After understanding what’s Apache Hadoop, allow us to now understand the Hadoop Architecture intimately.

Hadoop works in master-slave fashion. There is a master node and there are n numbers of slave nodes where n are often 1000s. Master manages, maintains, and monitors the slaves while slaves are the particular worker nodes. In Hadoop architecture, the Master should be deployed on good configuration hardware, not just commodity hardware. As it is the heart of the Hadoop cluster.

Master stores the metadata (data about data) while slaves are the nodes that store the info. Distributedly data stores in the cluster. The client connects with the master node to perform any task. Now during this Hadoop tutorial for beginners, we’ll discuss different features of Hadoop intimately.

Hadoop Features

Here are the top Hadoop features that make it popular –

1. Reliability

In the Hadoop cluster, if any node goes down, it’ll not disable the entire cluster. As an alternative, another node will take the place of the failed node. Hadoop cluster will continue functioning as nothing went on. Hadoop has a built-in fault tolerance feature.

2. Scalable

Hadoop gets integrated with cloud-based service. If you are installing Hadoop on the cloud you need not worry about scalability. You can easily acquire more hardware and expand your Hadoop cluster within minutes.

3. Economical

Hadoop gets deployed on commodity hardware which is reasonable machines. This makes Hadoop very economical. Also as Hadoop is an open system software there’s no cost of license too.

4. Distributed Processing

In Hadoop, any job submitted by the client gets divided into the number of sub-tasks. These sub-tasks are independent of each other. Hence they execute in parallel giving high throughput.

5. Distributed Storage

Hadoop splits each file into the number of blocks. These blocks of data get stored distributed on the cluster of machines.

6. Fault Tolerance

Hadoop replicates every block of file repeatedly counting on the replication factor. The replication factor is 3 by default. In Hadoop suppose any node goes down then the data thereon node gets recovered. This is because this copy of the data would be available on other nodes due to replication. Hadoop is fault-tolerant…

Hadoop Flavors

This section of the Hadoop Tutorial talks about the assorted flavors of Hadoop.

• Apache – Vanilla flavor, in this the actual code is residing in Apache repositories.
• Hortonworks – Popular distribution in the industry.
• Cloudera – It is the most popular in the industry.
• MapR – It has rewritten HDFS and its HDFS is faster as compared to others.
• IBM – Proprietary distribution is known as Big Insights.

All the databases have provided local connectivity with Hadoop for fast data transfer. Because, to transfer data from Oracle to Hadoop, you would like a connector.

All flavors are almost the same and if you know one, you can easily work on other flavors as well.

Hadoop Future Scope

There is going to be a lot of investment in the Big Data industry in the coming years. According to a report by FORBES, 90% of global organizations will be investing in Big Data technology. Hence the demand for Hadoop resources will also grow. Learning Apache Hadoop will give you faster growth in your career. It also tends to increase your pay package.

There is a lot of gap between the supply and demand of Big Data professionals. The skill in Big Data technologies continues to be in high demand. This is because companies grow as they try to get the most out of their data. Hence, their salary package is quite high as compared to professionals in other technology.

The managing director of Dice has said that Hadoop jobs have seen a 64% increase from the previous year. It is evident that Hadoop is ruling the Big Data market and its future is bright. The demand for Big Data Analytics professionals is ever increasing. As it is a known fact that data is nothing without the power to analyze it.

Summary – Hadoop Tutorial

On concluding this Hadoop tutorial, we can say that Apache Hadoop is the most familiar and powerful big data tool. Big Data stores huge volumes of data in the distributed manner and processes the data in parallel on a cluster of nodes. Apache Hadoop provides the world’s most reliable storage layer- HDFS. MapReduce is the batch processing engine and the Resource management layer is YARN.

On summarizing this Hadoop Tutorial, I want to give you a quick revision of all the topics we have discussed

• The concept of Big Data
• Reason for Hadoop’s Invention
• Prerequisites to learn Hadoop
• Introduction to Hadoop
• Core components of Hadoop
• Why Hadoop
• Hadoop Architecture
• Features of Hadoop
• Hadoop Flavours
• Future Scope of Hadoop

We hope this Hadoop Tutorial helped you.