Performance of Delta Vs Parquet file formats

spark.sql("set spark.databricks.delta.autoCompact.enabled = true")

spark.sql("set spark.databricks.delta.optimizeWrite.enabled = true")

OPTIMIZE the Databricks Delta table   

     
display(spark.sql("OPTIMIZE flights ZORDER BY (DayofWeek)"))

The query over the Databricks Delta table runs much faster after OPTIMIZE is run. 

How much faster the query runs can depend on the configuration of the cluster you are 

running on, however should be 5-10X faster compared to the standard table.

References:

https://docs.databricks.com/_static/notebooks/delta/optimize-scala.html

https://docs.databricks.com/delta/optimizations/index.html#compaction-bin-packing

https://databricks.com/blog/2018/07/31/processing-petabytes-of-data-in-seconds-with-databricks-delta.html

Hadoop Performance Tuning

Various possible way to improve the performance of Hadoop

1. Compress input and output data : Compression of files saves storage space on HDFS and also improve speed of transfer.

Some of compression techniques are Snappy, bzip2, and lz4 (splittable) and gzip, deflate and lzo (non-splittable)

2. Adjust spill records (try to minimize disk spills) and sorting buffer

3. Try to implement combiner if the aggregate operations follows associative and commutative rule.

4.  Consider reducing replication factor

5. Adjust number of map tasks, reduce tasks and memory.

6. Using Skewed joins

Some times the data being processed might have some skewness - meaning 80% of the data is going to a single reducer. If there is a huge amount of data for a single key, then one of the reducer will be held up with processing majority of the data –this is when Skewed join comes to the rescue. Skewed join computes a histogram to find out which key is dominant and then data is split based on its various reducers to achieve optimal performance.

7. Speculative Execution

The performance of MapReduce jobs is seriously impacted when tasks take a long time to finish execution. Speculative execution is a common approach to solve this problem by backing up slow tasks on alternate machines. Setting the configuration parameters ‘mapreduce.map.tasks.speculative.execution’ and ‘mapreduce.reduce.tasks.speculative.execution’ to true will enable speculative execution so that the job execution time is reduced if the task progress is slow due to memory unavailability.

8. Use Distribute cache