Sunday, March 5, 2017

Given a binary tree, print all root-to-leaf paths

For the below example tree, all root-to-leaf paths are:
10 –> 8 –> 3
10 –> 8 –> 5
10 –> 2 –> 2


         10

     8            2

3        5    2


Sol:

Algorithm:
Use a path array path[] to store current root to leaf path. Traverse from root to all leaves in top-down fashion. While traversing, store data of all nodes in current path in array path[]. When we reach a leaf node, print the path array.

/*Given a binary tree, print out all of its root-to-leaf
 paths, one per line. Uses a recursive helper to do the work.*/
void printPaths(TreeNode T)
{
  int path[1000];
  printPathsRecur(T, path, 0);
}
 
/* Recursive helper function -- given a node, and an array containing
 the path from the root node up to but not including this node,
 print out all the root-leaf paths.*/
void printPathsRecur(TreeNode T, int path[], int pathLen)
{
  if (node==NULL)
    return;
 
  /* append this node to the path array */
  path[pathLen] = T.data;
  pathLen++;
 
  /* it's a leaf, so print the path that led to here  */
  if (T.left==NULL && T.right==NULL)
  {
    printArray(path, pathLen);
  }
  else
  {
    /* otherwise try both subtrees */
    printPathsRecur(T.left, path, pathLen);
    printPathsRecur(T.right, path, pathLen);
  }
}


Print Ancestors of a given node in Binary Tree

Given a Binary Tree and a key, write a function that prints all the ancestors of the key in the given binary tree.
For example, if the given tree is following Binary Tree and key is 7, then your function should print 4, 2 and 1.
             1
            /   \
          2      3
        /  \
      4     5
     /
    7

Sol:

bool printAncestors(TreeNode T, int target)
{
  if (T == NULL)
     return false;
 
  if (T.data == target)
     return true;
 
  /* If target is present in either left or right sub tree of this node then print this node */
  if ( printAncestors(T.left, target) || printAncestors(T.right, target) )
  {
      System.out.println(T.data + " ");
      return true;
  } 
  return false;
}

Time Complexity: O(n) where n is the number of nodes in the given Binary Tree.

Saturday, March 4, 2017

Populate Inorder Successor for all nodes

Given a Binary Tree where each node has following structure, write a function to populate next pointer for all nodes. The next pointer for every node should be set to point to inorder successor.

Sol:

Public void populateInorderSuccessor(TreeNode T)
{
   TreeNode next = NULL;

   populateInorderSuccessorRecur(T, next);
}

Private void populateInoderSuccessorRecur(TreeNode T, TreeNode next)
{

   if(T != NULL ) {
         populateInoderSuccessorRecur(T.right, next);

          // Set the next as previously visited node in reverse Inorder
         T.next = next;

         // Change the prev for subsequent node
         next = T;

         populateInoderSuccessorRecur(T.left, next);
   }
}

Time Complexity: O(n)

Friday, March 3, 2017

Spark Installation on Linux

Here are the steps to install Hadoop 2.x on Linux machine

Step 1: Install Java

Check Java installation on your machine

$ java -version 

java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

If you don’t have Java installed on your system, use below link to install the java.
https://www.java.com/en/download/help/linux_x64_install.xml

Step 2: Install Scala

Check Scala installation on your machine

$ scala -version
scala: command not found 
2.1 Download and Install Scala

$ cd /opt/hadoop
$ wget http://downloads.lightbend.com/scala/2.12.1/scala-2.12.1.tgz
$ tar -xzf scala-2.12.1.tgz
$ mv scala-2.12.1 scala
$ chown -R hadoop scala

2.2 Set PATH

$ export SCALA_HOME=/opt/hadoop/scala
$ export PATH=$SCALA_HOME/bin:$PATH

2.3 Verify Scala Installation

$ scala -version
Scala code runner version 2.12.1 -- Copyright 2002-2016, LAMP/EPFL and Lightbend, Inc.
$
Step 3: Install Spark

Download and install Spark


cd /opt/hadoop
$ wget http://d3kbcqa49mib13.cloudfront.net/spark-2.1.0-bin-hadoop2.6.tgz
$ tar -xzf spark-2.1.0-bin-hadoop2.6.tgz
$ mv spark-2.1.0-bin-hadoop2.6 spark
$ chown -R hadoop spark
Set PATH

$ export SPARK_HOME=/opt/hadoop/spark
$ export PATH=$SPARK_HOME/bin:$PATH
Step 3: Verify Spark Installation 

Run spark-shell

$ spark-shell


Exit scala shell

scala> :q
$ 

Small files problem in Hadoop?

Hadoop works better with a small number of large files and not with large number of small files. Large number of small files take up lots of memory on the Namenode. Each small file generates a map task and hence there are too many such map task with insufficient input. Storing and transforming small size file in HDFS creates an overhead to map reduce program which greatly affects the performance of Namenode. Default size of HDFS block is 128 MB. The files whose size is smaller than the default block size in HDFS are termed as small files.

Some of the possible solutions:

1. Merge small files in the same directory into large one and accordingly build index for each small file to enhance storage efficiency of small files and reduce burden on Namenode caused by metadata.

2. Modify InputFormat class. InputFormat will be modified in such a way that multiple files are combined into a single split. So, the map task will get more input to process, unlike existing system. As a result, the time required to process large number of small files will be minimized. In addition, multiple reducers will be used for taking advantage of parallelism.

Incremental Updates in Apache Hadoop or Hive


1. Keep the Master Data. This is one-time, initial load of the data.

2. Load the Delta Data (newly added and updated data) into HDFS at different location.

3. Merge the data: Join the master and delta data together on the business key field(s).

4. Compact the data: After the merge you will have one or more records for each business key. Keep the most recently updated data for each business key.

5. Write the Data to a Temporary Output: Since most Hadoop jobs (e.g. MapReduce) cannot overwrite an existing directory, you must write the compacted data to a temporary output location.

6. Overwrite the original master data with temporary output.


High Level:

  • Ingest. Complete (base_table) table movement followed by Change (incremental_table) records only.
  • Reconcile. Creating a Single View of Base + Change records (reconcile_view) to reflect the most up-to-date record set.
  • Compact. Creating a Reporting table (reporting_table) from the reconciled view.
  • Purge. Replacing the Base table with Reporting table contents and deleting any previously processed Change records before the next Data Ingestion cycle.

Extract data ( nested columns ) from JSON without specifying schema using PIG

How to extract required data from JSON without specifying schema using PIG?

Sample Json Data:

----------
{"autopolicy": {"policy_holder_name": "someone", "policy_num": "20141012", "is_active": true, "vehicle": {"brand": {"model": "Lexus", "year": 2012}, "vin": "RANDOM123", "price": 23450.50}}}
------

Pig Latin Script to extract the data
-----

REGISTER '/apps/opt/hadoop/pig/lib/elephantbird/json-simple-1.1.1.jar'
REGISTER '/apps/opt/hadoop/pig/lib/elephantbird/elephant-bird-pig-4.3.jar'
REGISTER '/apps/opt/hadoop/pig/lib/elephantbird/elephant-bird-hadoop-compat-4.3.jar'

data = LOAD '/data/json/test1.json' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad');
vehiclemodel= FOREACH data GENERATE $0#'autopolicy'#'vehicle'#'brand'#'model' as model;
dump vehiclemodel;

-----