README.md

Learning how to tame the Big Data with Hadoop and related technologies

Table of Contents

• Hadoop
  • Installation
  • Overview of Hadoop Ecosystem
  • HDFS
    • Login using Ambari
    • Login using Putty
    • Admin access for Ambari
• Map Reduce
  • Architecture
  • Handling Failures
  • Map Reduce Example
  • Running on HortonWorks Sandbox
  • Map Reduce Challenge 01
  • Map Reduce Challenge 02
• Pig
  • Pig Latin Example
  • Running the script using Ambari
  • Pig Latin: Diving Deeper
  • Pig Challenge
• Spark
  • Why is it scalable
  • Components of Spark
  • Resilient Distributed Dataset (RDD)
    • Creating RDDs
    • Transforming RDDs
    • RDD Actions
  • Spark Example
  • Running the Spark script
  • Spark Challenge 01
• Spark SQL
  • Shell access for Spark SQL
  • User defined functions
  • Spark SQL Example
  • Running Spark SQL
  • Spark SQL Challenge 01
• Using MLLib in Spark 2.0
  • Making a recommendation model
• Hive
  • Why we use Hive
  • Where to not use Hive
  • HiveQL
    • HiveQL Example
    • How Hive works
    • HiveQL Challenge 01
• Sqoop
  • What is Sqoop
  • Transfer data to MySQL DB using Sqoop
• HBase
  • HBase Architecture
  • HBase Data Model
  • How to access HBase
  • Populate HBase using Python REST API
  • Integrate Pig with HBase
• Cassandra DB
  • So why Cassandra
  • What is CQL
  • Cassandra and Spark
  • Installing Cassandra
  • Populate Cassandra DB with Spark
• Mongo DB
  • Does it have a schema
  • Lets learn Mongo terminology
  • How does it replicate
    • Something we should know about Replica sets
  • Sharding
    • Things to know about Sharding
  • Mongo Extra knowledge
  • How to use MongoDB with Spark
  • Example queries

Hadoop

• Hadoop is an open source software platform for distributed storage and distributed processing of very large datasets on computer clusters built from commodity hardware
• Why Hadoop?
  • Data is too big
  • Vertical scaling isn't an option
    • Disk seek times
    • Hardware failures
    • Processing times
  • Horizontal scaling is linear
  • You can do much more instead of just batch processing

Installation

• Download Virtual Box from https://www.virtualbox.org/
• Download image of Hadoop to run on Virtual Box
  • (Horton Works Data Platform) HDP 2.5 Sandbox is preffered because it boots up faster than new versions
    • Download from https://hortonworks.com/downloads/#sandbox
• Import the image into Virtual Box
• Once you bootup, you will have CentOS instance that has Hadoop up and running
• We can use CLI, it also has browser interface
  • Ambari is available to easily navigate and manage different systems on Hadoop
  • Goto http://localhost:8888
• Launch Dashboard and login to Ambari
  • Username: maria_dev
  • Password: maria_dev

• Trouble shooting

  • Enable virtualization in your BIOS
  • Disable Hyper-V acceleration in Windows