Oracle to Hive Data Type Conversions

Table 8-1 Oracle to Hive Data Type Conversions

Oracle Data Type	Hive Data Type
NUMBER	INT when the scale is 0 and the precision is less than 10 BIGINT when the scale is 0 and the precision is less than 19 DECIMAL when the scale is greater than 0 or the precision is greater than 19
BINARY_DOUBLE	DOUBLE
BINARY_FLOAT	FLOAT
CHAR NCHAR	CHAR
VARCHAR2 NVARCHAR2	VARCHAR
DATE	TIMESTAMP
TIMESTAMP	TIMESTAMP
TIMESTAMPTZFoot 1  TIMESTAMPLTZ	Unsupported
RAW	BINARY

Reference : https://docs.oracle.com/cd/E63064_01/doc.42/e63062/copy2bda.htm#BIGUG76752

How Much RAM does my Linux machine have?

Ans:

The best way to find out RAM information such as amount of free and used memory in the system is to open a terminal and type the following command:

$ free -m

OR

$ free -mt

How to create thread safe classes in Java

Thread safety in java is the process to make our program safe to use in multi threaded environment, there are different ways through which we can make our program thread safe.

1) Synchronization is the easiest and most widely used tool for thread safety in java.

2) Immutable objects are by default thread safe because there state can not be modified once created. For example, String class is immutable in java which inherently thread-safe.

3) Use of Atomic Wrapper classes from java.util.concurrent.atomic package. Atomic operations in Java are thread-safe. For example AtomicInteger.

4) Use of locks from java.util.concurrent.locks package.

5) Using volatile keyword with variables to make every thread read the data from memory, not read from thread cache. 

6) Using thread safe classes for thread safety. Example of thread-safe class in Java: Vector, Hashtable, ConcurrentHashMap, String etc.


7)  Locking is one way of achieving thread-safety in Java.

Create a binary tree with minimal height

Q: Given a sorted (increasing order) array, write an algorithm to create a binary tree with minimal height.

Ans:

Public TreeNode addToTree(int arr[], int start, int end)
{
     if (start > end) {
         return null;
     }
    int mid = (start + end) / 2;
    TreeNode T = new TreeNode(arr[mid]);
     T.left = addToTree(arr, start, mid - 1);
     T.right = addToTree(arr, mid + 1, end);
     return T;
}


Public TreeNode createBST(int array[]) {
    return addToTree(array, 0, array.length - 1);
}

Spark Streaming

Overview

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Kinesis, or TCP sockets, and can be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. In fact, you can apply Spark’s machine learning and graph processing algorithms on data streams.

Internally, it works as follows. Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.

Spark Streaming provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data. DStreams can be created either from input data streams from sources such as Kafka, Flume, and Kinesis, or by applying high-level operations on other DStreams. Internally, a DStream is represented as a sequence of RDDs.

What is Druid?

Druid is a high-performance, column-oriented, distributed data store.

Druid is an open-source analytics data store designed for business intelligence (OLAP) queries on event data. Druid provides low latency (real-time) data ingestion, flexible data exploration, and fast data aggregation. Existing Druid deployments have scaled to trillions of events and petabytes of data. Druid is most commonly used to power user-facing analytic applications.

Ref: http://druid.io/druid.html

Map Side Join

Case 1: Map Side Join:

Assume you have

1 table with 1TB and 1 table with 1MB

Assume as well that we have 50 nodes. ( I think he only copies datasets for the distributed cache once per node and not once per task might be wrong )

So MapSide Join:

You have to copy the small table to every node: 50x1MB = 50MB of data are copied across the network

Shuffle Join:

Both tables need to be copied once i.e. 1TB + 1MB = 1.00001 TB will be copied over the network.

MapSide Join is much better than Shuffle join in this case.

Hive supports MAPJOINs, which are well suited for this scenario – at least for dimensions small enough to fit in memory. Before release 0.11, a MAPJOIN could be invoked either through an optimizer hint:

select /*+ MAPJOIN(time_dim) */ count(*) from
store_sales join time_dim on (ss_sold_time_sk = t_time_sk)

or via auto join conversion:

set hive.auto.convert.join=true;
select count(*) from
store_sales join time_dim on (ss_sold_time_sk = t_time_sk)

The default value for hive.auto.convert.join was false in Hive 0.10.0.  Hive 0.11.0 changed the default to true (HIVE-3297). Note that hive-default.xml.template incorrectly gives the default as false in Hive 0.11.0 through 0.13.1.                                                                                                                                               

Case 2: Shuffle Join

Assume you have 1 table with 1TB and one table with 500GB:

MapSide Join:

500GB table needs to be copied to 50 nodes for 50 * 500 = 25 TB of data being copied over the network

Shuffle Join:

1.5TB of data need to be copied over the network.

So in this case MapSide join is much worse than shuffle join.