How Much RAM does my Linux machine have?

Ans:

The best way to find out RAM information such as amount of free and used memory in the system is to open a terminal and type the following command:

$ free -m

OR

$ free -mt

How to create thread safe classes in Java

Thread safety in java is the process to make our program safe to use in multi threaded environment, there are different ways through which we can make our program thread safe.

1) Synchronization is the easiest and most widely used tool for thread safety in java.

2) Immutable objects are by default thread safe because there state can not be modified once created. For example, String class is immutable in java which inherently thread-safe.

3) Use of Atomic Wrapper classes from java.util.concurrent.atomic package. Atomic operations in Java are thread-safe. For example AtomicInteger.

4) Use of locks from java.util.concurrent.locks package.

5) Using volatile keyword with variables to make every thread read the data from memory, not read from thread cache. 

6) Using thread safe classes for thread safety. Example of thread-safe class in Java: Vector, Hashtable, ConcurrentHashMap, String etc.


7)  Locking is one way of achieving thread-safety in Java.

Create a binary tree with minimal height

Q: Given a sorted (increasing order) array, write an algorithm to create a binary tree with minimal height.

Ans:

Public TreeNode addToTree(int arr[], int start, int end)
{
     if (start > end) {
         return null;
     }
    int mid = (start + end) / 2;
    TreeNode T = new TreeNode(arr[mid]);
     T.left = addToTree(arr, start, mid - 1);
     T.right = addToTree(arr, mid + 1, end);
     return T;
}


Public TreeNode createBST(int array[]) {
    return addToTree(array, 0, array.length - 1);
}

Spark Streaming

Overview

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Kinesis, or TCP sockets, and can be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. In fact, you can apply Spark’s machine learning and graph processing algorithms on data streams.

Internally, it works as follows. Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.

Spark Streaming provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data. DStreams can be created either from input data streams from sources such as Kafka, Flume, and Kinesis, or by applying high-level operations on other DStreams. Internally, a DStream is represented as a sequence of RDDs.

What is Druid?

Druid is a high-performance, column-oriented, distributed data store.

Druid is an open-source analytics data store designed for business intelligence (OLAP) queries on event data. Druid provides low latency (real-time) data ingestion, flexible data exploration, and fast data aggregation. Existing Druid deployments have scaled to trillions of events and petabytes of data. Druid is most commonly used to power user-facing analytic applications.

Ref: http://druid.io/druid.html

Map Side Join

Case 1: Map Side Join:

Assume you have

1 table with 1TB and 1 table with 1MB

Assume as well that we have 50 nodes. ( I think he only copies datasets for the distributed cache once per node and not once per task might be wrong )

So MapSide Join:

You have to copy the small table to every node: 50x1MB = 50MB of data are copied across the network

Shuffle Join:

Both tables need to be copied once i.e. 1TB + 1MB = 1.00001 TB will be copied over the network.

MapSide Join is much better than Shuffle join in this case.

Hive supports MAPJOINs, which are well suited for this scenario – at least for dimensions small enough to fit in memory. Before release 0.11, a MAPJOIN could be invoked either through an optimizer hint:

select /*+ MAPJOIN(time_dim) */ count(*) from
store_sales join time_dim on (ss_sold_time_sk = t_time_sk)

or via auto join conversion:

set hive.auto.convert.join=true;
select count(*) from
store_sales join time_dim on (ss_sold_time_sk = t_time_sk)

The default value for hive.auto.convert.join was false in Hive 0.10.0.  Hive 0.11.0 changed the default to true (HIVE-3297). Note that hive-default.xml.template incorrectly gives the default as false in Hive 0.11.0 through 0.13.1.                                                                                                                                               

Case 2: Shuffle Join

Assume you have 1 table with 1TB and one table with 500GB:

MapSide Join:

500GB table needs to be copied to 50 nodes for 50 * 500 = 25 TB of data being copied over the network

Shuffle Join:

1.5TB of data need to be copied over the network.

So in this case MapSide join is much worse than shuffle join. 

Replicated Join in MapReduce:

The Replicated Join is a Map-Side Join. The joining is done in mappers, and no reducer is even needed for this operation. So in a sense it should be faster join, but only if certain requirements are met.

The main idea in the Replicated Join is to cache all data from the smaller dataset into the mappers which process splits of the bigger dataset.

In our example, our Donations sequence file is 4-blocks long, but our Projects sequence file is only 1-block long. The aim is to cache the Projects data in each of the 4 Donations mappers. This way, each mapper has all the Project records in memory and can use them to join its Donation records from the input split.

When using a Replicated join, the results are grouped by the bigger dataset records. The mapper outputs still contain the same records as their respective input splits.

Limitations: There are 2 important limitations using the Replicated Join:

1. The smallest dataset has to fit into memory.
2. Only an Inner Join or a Left Join can be performed.