Lockers puzzle

Puzzle: There are one hundred closed lockers in a hallway. A man begins by opening all one hundred lockers. Next, he closes every second locker. Then he goes to every third locker and closes it if it is open or opens it if it is closed (e.g., he toggles every third locker). After his one hundredth pass in the hallway, in which he toggles only locker number one hundred, how many lockers are open? 

Ans:

For which rounds is a door toggled (open or closed)?

A door n is toggled once for each factor of n, including itself and 1. That is, door 15 is toggled on round 1, 3, 5, and 15.

When would a door be left open?
A door is left open if the number of factors (x) is odd. You can think about this by
pairing factors o as an open and a close. If there’s one remaining, the door will be open. 

When would x be odd?
x is odd if n is a perfect square. Here’s why: pair n’s factors by their complements. For example, if n is 36, the factors are (1, 36), (2, 18), (3, 12), (4, 9), (6, 6). Note that (6, 6) only contributes 1 factor, thus giving n an odd number of factors.

How many perfect squares are there?
There are 10 perfect squares. You could count them (1, 4, 9, 16, 25, 36, 49, 64, 81, 100), or you could simply realize that you can take the numbers 1 through 10 and square them (1*1, 2*2, 3*3, ..., 10*10).

Therefore, there are 10 lockers open. 

Egg drop puzzle

Puzzle: There is a building of 100 floors. If an egg drops from the Nth floor or above it will break. If it’s dropped from any floor below, it will not break. You’re given 2 eggs. Find N, while minimizing the number of drops for the worst case. 

Ans: Observation: Regardless of how we drop Egg1, Egg2 must do a linear search. i.e., if Egg1 breaks between floor 10 and 15, we have to check every floor in between with the Egg2

The Approach:
A First Try: Suppose we drop an egg from the 10th floor, then the 20th, ...

• »  If the first egg breaks on the first drop (Floor 10), then we have at most 10 drops total.
  
• »  If the first egg breaks on the last drop (Floor 100), then we have at most 19 drops total ( floors 10, 20, ...,90, 100, then 91 through 99).
  
• »  That’s pretty good, but all we’ve considered is the absolute worst case. We should do some “load balancing” to make those two cases more even.
  Goal: Create a system for dropping Egg1 so that the most drops required is consistent, whether Egg1 breaks on the rst drop or the last drop.
  

1. A perfectly load balanced system would be one in which Drops of Egg1 + Drops of Egg2 is always the same, regardless of where Egg1 broke.
   
2. For that to be the case, since each drop of Egg1 takes one more step, Egg2 is allowed one fewer step.
   
3. We must, therefore, reduce the number of steps potentially required by Egg2 by one drop each time. For example, if Egg1 is dropped on Floor 20 and then Floor 30, Egg2 is potentially required to take 9 steps. When we drop Egg1 again, we must reduce potential Egg2 steps to only 8. That is, we must drop Egg1 at floor 39.
   
4. We know, therefore, Egg1 must start at Floor X, then go up by X-1 floors, then X-2, ..., until it gets to 100.
   
5. Solve for X+(X-1)+(X-2)+...+1 = 100. X(X+1)/2 = 100 -> X = 14
   

We go to Floor 14, then 27, then 39, ... This takes 14 steps maximum. 

Magical Hat

Puzzle: A bunch of men are on an island. A genie comes down and gathers everyone to-gather and places a magical hat on some people’s heads (i.e., at least one person has a hat). The hat is magical: it can be seen by other people, but not by the wearer of the hat himself. To remove the hat, those (and only those who have a hat) must dunk themselves underwater at exactly midnight. If there are n people and c hats, how long does it take the men to remove the hats? The men cannot tell each other (in any way) that they have a hat. 

Ans:

so let’s simplify it by looking at specific cases. 

Case c = 1: Exactly one man is wearing a hat.

Assuming all the men are intelligent, the man with the hat should look around and realize that no one else is wearing a hat. Since the genie said that at least one person is wearing a hat, he must conclude that he is wearing a hat. Therefore, he would be able to remove it that night.

Case c = 2: Exactly two men are wearing hats.
The two men with hats see one hat, and are unsure whether c = 1 or c = 2. They know, from the previous case, that if c = 1, the hats would be removed on Night #1. Therefore, if the other man still has a hat, he must deduce that c = 2, which means that he has a hat. Both men would then remove the hats on Night #2

Case General: If c = 3, then each man is unsure whether c = 2 or 3. If it were 2, the hats would be removed on Night #2. If they are not, they must deduce that c = 3, and therefore they have a hat. We can follow this logic for c = 4, 5, ... 

There are c hats so it will take c nights to remove all of them. 

Facade Design Pattern

Facade design pattern provides a simplified interface to a larger body of code. 

The facade pattern is typically used when

• a simple interface is required to access a complex system,
• a system is very complex or difficult to understand,
• an entry point is needed to each level of layered software, or
• the abstractions and implementations of a subsystem are tightly coupled.

Purpose of marker interfaces in Java

Marker interface is an empty interface (no field or methods). Examples of marker interface are Serializable, Clonnable, Remote and ThreadSafe interfaces. 

Note: Marker interface is used to inform the JVM that the classes implementing them will have some special behavior.

 In java we have following four major marker interfaces:

• Searilizable interface

• Cloneable interface

• Remote interface

• ThreadSafe interface

public interface Serializable 
{
  // nothing here
}

Q. Marker interface won't contain any method then how the objects will get that special ability?

Ans. JVM is responsible to provide required ability in marker interfaces

Design a parking lot

• Vehicle
  • size of vehicle (small, medium, large)
  • status of vehicle (run or parked)
  • findSlot()
  • park()
  • leave()
• Car, Bike, Bus, Truck... extends Vehicle
• Slot
• size of slot
• status (available or not)
• occupy()
• release()
• ParkingLot
  • hold slots in parking lot
  • totalSlots()
  • occupiedSlots()
  • availableSlots()

Spark and Hive Useful Commands

Q. How to submit pyspark script to Spark in Cluster YARN mode?

spark-submit --master yarn --deploy-mode cluster <Config_Options> <SparkScript> <PARAMETERS>

Ex:

spark-submit --master yarn --deploy-mode cluster --driver-memory 5G --conf spark.yarn.executor.extraClassPath=./ --conf spark.scheduler.mode=fair --conf spark.yarn.maxAppAttempts=1  --files /home/hadoop/hive-site.xml,s3://test-emr-bin/qa/test.config s3://test-emr-bin/qa/aggregates/emr_scripts/spark/test_aggregate_spark.py QA aggregates-qa aggregates/consume/ QA_COMM QA_AGGREGATES


Q. How to generate a hive query output to a file with columns separated by | and sorted?

hive -f src.hql > output; cat output | sed 's/\t/|/g' | sort > source;

Q. How to do you compare source and target with hive query results?

hive -f target.hql > output; cat output | sed 's/\t/|/g' | sort > target;

Q. diff source target

cat source | head -1

cat target | head -1


Q: How to update partitions with  msck repair table?

msck repair table test_daily_aggregate

Q How to show partitions

show partitions test_daily_aggregate;

Q. How to run a task in Airflow scheduler?

airflow run -i -f <DAG_NAME> <TASK_NAME> <Schedule_Date> <AIRFLOW_SCHEDULER>

airflow run -i -f demand_aggregate spark_abc_incremental_aggregate 2017-04-17 abcsla

Q. How to compile a new DAG?

airflow list_dags -sd test_dag.py 

Q. How to push a DAG?

   $ push_dag abc_data

Q. How to list Airflow dags

   $ airflow list_dags

Q. How do you pass arguments to Hive Query?

hive -f s3://abc-emr-bin/qa/pos/emr_scripts/hive/abc_transaction_lookup_stg.hive -hivevar HIVEDATABASE_STG=QA_STG_ABC -hivevar HIVEDATABASE=QA_ABC -hivevar DATE_RANGE=2017-02-10

Q. How to know size of folder in S3?

$ aws s3 ls --summarize --human-readable --recursive s3://bi-manage/dev/bi/test/global_test_daily_snapshot 

Q. How to know list of files in S3?

aws s3 ls s3://test/bi/global_test_daily_snapshot/geo_part=USA/transfer_local_snapshot_date_part=2017-05-01/ --recursive

Q. How to copy a file from local machine to EMR cluster

scp -i /Users/test/Desktop/rsa_keys/EMR.cer test.txt hadoop@10.1.1.120:/home/hadoop/

Q. How to get YARN logs

YARN logs :

---------

$ yarn logs -applicationId application_1495046596880_0037

$ yarn logs -applicationId <Application_ID>

Q. How to enable debug logging in Hive?

hive --hiveconf hive.root.logger=DEBUG,console

Q. How to know IP Address?

$ ifconfig

Q. How to add jar file to Spark (pyspark)

pyspark --jars UDFs-1.0-SNAPSHOT-jar-with-dependencies.jar;

Q. How to change execution engine in hive from Tez to mr

set hive.execution.engine=mr;

Q. How to create permanent function:

CREATE FUNCTION qa_TestFunc AS "com.org.nbac.AbcTest" using JAR 's3://emr-bin/qa/data-test/lib/UDFs-1.0-SNAPSHOT-jar-with-dependencies.jar';  

Q. How to run a java class file in a jar:

java -cp s3://emr-bin/dev/data-protection/lib/UDFs-1.0-SNAPSHOT-jar-with-dependencies.jar com.org.nbac.AbcTest.class

Q. How to source env file

$ source source.env 

$ source /Users/bac/Features/path.evn

Q. How to Build UDF Jar file using maven

$ mvn clean install -Dmaven.test.skip

Q. Basic Unit Testing of UDF in Hive:

ADD JAR  s3://-emr-bin/dev/data-test-/lib/UDFs-1.0-SNAPSHOT-jar-with-dependencies.jar;

CREATE TEMPORARY FUNCTION Encrypt AS "com.abc.test.Encrypt";

CREATE TEMPORARY FUNCTION EncryptSignature AS "com.abc.test.EncryptSignature";

CREATE TEMPORARY FUNCTION Decrypt AS "com.abc.test.Decrypt";

SELECT Encrypt('abc', 'xyz', "dev_test");

Q. Basic Unit Testing of UDF in Spark:

spark.sql('CREATE TEMPORARY FUNCTION Encrypt AS "com.abc.test.Encrypt"')

spark.sql('CREATE TEMPORARY FUNCTION EncryptSignature AS "com.abc.test.EncryptSignature"')

spark.sql('CREATE TEMPORARY FUNCTION Decrypt AS "com.abc.test.Decrypt"')

spark.sql('select source_id,EncryptSignature(source_id, current_status, "dev_activity_status") as sourceid_signature from stg_inactivity')

spark-submit --master yarn --deploy-mode cluster --executor-cores 5 --num-executors 90 --driver-memory 5GB --executor-memory 10G --conf spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 --conf fs.s3n.multipart.uploads.enabled=true --conf spark.dynamicAllocation.enabled=false --conf spark.yarn.executor.memoryOverhead=5120 --jars s3://test-emr-bin/dev/test-protection/lib/Test_UDFs-1.0-SNAPSHOT-jar-with-dependencies.jar test_encrypt.py

Q. How to add partitions in Hive

ALTER TABLE dev_test ADD  PARTITION 's3n://bi-managed/dev/test/process_date=2017-05-31'

Q. Insert Overwrite table in Hive

INSERT OVERWRITE TABLE dev_list

SELECT member_id, email_addr

FROM dev_test

;

Q. How to create an external table in Hive

DROP TABLE IF EXISTS dev_list;

CREATE EXTERNAL TABLE `dev_list`(

  `source_id` string, 

  `sourceid_signature` string, 

  `source_cd` string, 

  `status` string)

STORED AS PARQUET

LOCATION

  's3://bi-managed/prod/test/data/2017-06-12'

;

Q. How to submit a job in Spark 

spark-submit --jars s3://emr-bin/dev/common/emr_scripts/jars/UDFs-1.0-SNAPSHOT-jar-with-dependencies.jar pysark_job.py bi-managed/prod bi-managed/prod 2017-06-12

Q. How to set JAVA compiler in Maven to JDK 1.7 

-------------

<properties>

    <maven.compiler.source>1.7</maven.compiler.source>

    <maven.compiler.target>1.7</maven.compiler.target>

  </properties>

Ref: https://maven.apache.org/plugins/maven-compiler-plugin/examples/set-compiler-source-and-target.html

--------------------------

Q. How to get avro schema from given avro data file

java -jar ~/Downloads/avro-tools-1.7.4.jar getschema part-r-00000-c7fc5e23-842e-4723-86e1-c069afdeb7c1.avro > chn_pos_clean_stage.avsc

Q. How is use of coalesce() function in Spark SQL

public static Column coalesce(Column... e)

Returns the first column that is not null, or null if all inputs are null.

For example, coalesce(a, b, c) will return a if a is not null, or b if a is null and b is not null, or c if both a and b are null but c is not null.

Q. What is use of concat_ws() function in Spark SQL

public static Column concat_ws(java.lang.String sep, Column... exprs)

Concatenates multiple input string columns together into a single string column, using the given separator.