Philip D. Coyne

Hadoop

What do I try to do:   Experiment using Hadoop in analyze large quantiy of data for scienitifc and text mining application.  I will start with installation and configuring Single Node Hadoop, load data needed, using Pig and Hive to transform and load data  onto HDFS.

Single Node

Pre-requisite software tools

  • OS:  Ubuntu 14.01.1 on 64-bit machine with 16GB RAM
  • JVM 1.7 (bundled with Ubuntu 14.01.1)
  • Hadoop 2.6.0 download
  • WinSCP to copy file from Windows to Ubuntu environemnt
  • putty if on Windows; or totalTerminal on Mac OS

Prep the Environment

  • Assume that when Ubuntu is installed, the hostname is ubuntuboninc and a user boninc is created.   From Windows, use putty to ssh into unbutuboninc; from Mac OS use total terminal applicaiton:
    • ssh boninc@ubuntuboninc
  • Add hadoop user, named hdUser
    • sudo useradd -m hdUser
  • Add password for hdUser
    • sudo passwd hdUser (enter passowrd when system asks)
  • Change hdUser shell to use bash shell
    •  sudo chsh -s /bin/bash hdUser
  • Allow hdUser to be able for sudo

    • sudo adduser -g admin hdUser

Hadoop Installation (Installation directory: /usr/local/hadoop-2.6.0)

  • On ubuntuboninc machine, in /home/hdUser, create intall directory
    • mkdir install
  • Use WinSCP to copy hadoop 2.6.0 tar file to ubuntuboninc machine, directory /home/hdUser/install
  • Uncompress hadoop tar to /user/local
    • cd /usr/local
    • tar -xzf /home/hdUser/install/hadoop-2.6.0.tar.gz /usr/local/hadoop-2.6.0
  • Create a group name hadoop
    • sudo groupadd hadoop
  • Change hadoop files to hdUser and group hadoop
    • sudo chown -R /usr/local/hadoop-2.6.0

Prepare Hadoop Environment Variables

  • Use an editor to edit the file:
    • hadoop-env.sh
    • in directory: /usr/local/hadoop-2.6.0/etc/hadoop
  • Ensure the set the following two environment variables:
    • export JAVA_HOME = ${JAVA_HOME}  # this is set in /home/hdUser/.bashrc
    • export HADOOP_PREFIX=/usr/local/hadoop-2.6.0
Test:
  • chmod +x /usr/local/etc/hadoop/hadoop-env.sh
  • /usr/local/etc/hadoop/hadoop-env.sh ## execute the environement shell
  • Try the following command
    • $ bin/hadoop
If the above environment variables are set up correctly, the above command will display the usage docummentation for hadoop script.

HDFS Configuration

<TBA>

Multi Nodes

<To be Added>