Difference between GROUP and COGROUP in Pig

GROUP

Groups the data in one or more relations.

Note: The GROUP and COGROUP operators are identical. Both operators work with one or more relations. For readability GROUP is used in statements involving one relation and COGROUP is used in statements involving two or more relations. You can COGROUP up to but no more than 127 relations at a time.

Syntax

alias = GROUP alias { ALL | BY expression} [, alias ALL | BY expression …] [USING 'collected' | 'merge'] [PARTITION BY partitioner] [PARALLEL n];

Example:

X = COGROUP A BY owner, B BY friend2;

DESCRIBE X;
X: {group: chararray,A: {owner: chararray,pet: chararray},B: {friend1: chararray,friend2: chararray}}

How to write UDF in Pig

While writing UDF’s using Java, we can create and use the following three types of functions −

• Filter Functions − The filter functions are used as conditions in filter statements. These functions accept a Pig value as input and return a Boolean value.

• Eval Functions − The Eval functions are used in FOREACH-GENERATE statements. These functions accept a Pig value as input and return a Pig result.

• Algebraic Functions − The Algebraic functions act on inner bags in a FOREACHGENERATE statement. These functions are used to perform full MapReduce operations on an inner bag.

Eval/Filter Functions

• Each UDF must extend the EvalFunc (or) FilterFunc class
• Provide implementation to exec() function
• Create a jar file with UDF class
• Register jar file in pig script
• Define and use it 

Extend EvalFunc Class and implement exec() function

public class SimpleUDF extends EvalFunc<String>{
   
   public String exec(Tuple input) throws IOException {

      .........
     }
   } 

OR

Extend FilterFunc Class and implement exec() function

public class SimpleUDF extends FilterFunc {

    @Override
    public Boolean exec(Tuple input) throws IOException {
       ....
    }
} 

Register jar file

REGISTER '/$PIG_HOME/simpleUDF.jar'

Define the alias for simpleUDF as shown below.

DEFINE simpleUDF simpleUDF();

Use it in pig

grunt> Upper_case = FOREACH emp_data GENERATE simpleUDF(name);

How to store Pig output into Hive table

There are two approaches explained below with 'Employee' table example to store pig output into hive table. (Prerequisite is that hive table should be already created)
A =  LOAD 'EMPLOYEE.txt' USING PigStorage(',') AS(EMP_NUM:int,EMP_NAME:chararray,EMP_PHONE:int);



Approach 1: Using HCatalog
// dump pig result to Hive using Hcatalog 
store A into 'Empdb.employee' using org.apache.hive.hcatalog.pig.HCatStorer();
(or)

Approach 2: Using HDFS physical location
// dump pig result to external hive warehouse location
STORE A INTO 'hdfs://<<nmhost>>:<<port>>/user/hive/warehouse/Empdb/employee/' USING PigStorage(',')

How to read data stored in Hive table using Pig

Use HCatLoader to load Hive table data using Pig

A = LOAD 'hivedb.hivetable' using org.apache.hive.hcatalog.pig.HCatLoader();

-- Load table 'sample_07'
sample_07 = LOAD 'sample_07' USING org.apache.hcatalog.pig.HCatLoader();
 
-- Compute the average salary of the table
salaries = GROUP sample_07 ALL;
out = FOREACH salaries GENERATE AVG(sample_07.salary);
DUMP out;

Similar to HCatLoader, use HCatStorer to update the table, e.g.:


STORE alias INTO 'sample_07' USING org.apache.hcatalog.pig.HCatStorer();

Apache Pig Interview Questions

1. Can you name complex data types in Pig Latin?
Ans:

Map: Set of key-value pairs; keys must be character arrays, but values may be any type. Ex: ['a'#'pomegranate']
Tuple: Sequence of fields of any type. Ex: (1,'pomegranate')
Bag: Unordered collection of tuples, possibly with duplicates. Ex: {(1,'pomegranate'),(2)}

Apache Pig Installation on Linux

Apache Pig Installation with Hadoop on Linux

Step 1: Install JAVA and Hadoop

Apache Hive required java 6 or later version. We also need to install hadoop first before installing apache hive on our system. Use below links to install them.

Check Java installation on your machine

# java -version 

java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

If you don’t have Java installed on your system, use below link to install the java.

https://www.java.com/en/download/help/linux_x64_install.xml

Refer below article for Hadoop installation.
http://hadoopnalgos.blogspot.in/2016/04/hadoop-installation-on-linux.html

Step 2: Download Pig 

After configuring hadoop successfully on your linux system. lets start hive setup. First download latest hive source code and extract archive using following commands.

$ wget http://apache.claz.org/pig/pig-0.15.0/pig-0.15.0.tar.gz 
$ tar -xvf pig-0.15.0.tar.gz
$ mv pig-0.15.0 pig

Step 3: Setup Environment Variables

After extracting hive archive file, switch to hadoop user and setup following environment variables.

$ export PIG_HOME=/scratch/vchennar/hadoop/pig

$  export PATH=$PATH:$PIG_HOME/bin

Step 4: Start Pig

Try the following command, to get a list of Pig commands:

$ pig -help 

Try the following command, to start the Grunt shell:

$ pig  

Run Modes

Pig has two run modes or exectypes:

• Local Mode - To run Pig in local mode, you need access to a single machine.
• Mapreduce Mode - To run Pig in mapreduce mode, you need access to a Hadoop cluster and HDFS installation. Pig will automatically allocate and deallocate a 15-node cluster.

You can run the Grunt shell, Pig scripts, or embedded programs using either mode.

Grunt Shell

Use Pig's interactive shell, Grunt, to enter pig commands manually. See the Sample Code for instructions about the passwd file used in the example.

You can also run or execute script files from the Grunt shell. See the run and exec commands.

Local Mode

$ pig -x local

Mapreduce Mode

$ pig
or
$ pig -x mapreduce

For either mode, the Grunt shell is invoked and you can enter commands at the prompt. The results are displayed to your terminal screen (if DUMP is used) or to a file (if STORE is used).

grunt> A = load 'passwd' using PigStorage(':'); 
grunt> B = foreach A generate $0 as id; 
grunt> dump B; 
grunt> store B; 

Refer Apache Pig Documentation for further details : https://pig.apache.org/docs/r0.7.0/setup.html