Lambdas in Python

Lambdas are one line functions. They are also known as anonymous functions in some other languages.

Blueprint

lambda argument: manipulate(argument)

Example

add = lambda x, y: x + y

print(add(3, 5))
# Output: 8

List sorting

a = [(1, 2), (4, 1), (9, 10), (13, -3)]
a.sort(key=lambda x: x[1])

print a
# Output: [(13, -3), (4, 1), (1, 2), (9, 10)]

Filter with Lambdas

number_list = range(-5, 5)
less_than_zero = list(filter(lambda x: x < 0, number_list))
print less_than_zero

# Output: [-5, -4, -3, -2, -1]

Reduce with Lambdas

from functools import reduce
product = reduce((lambda x, y: x * y), [1, 2, 3, 4])
# Output: 24

PySpark Basic Commands

rddRead.first() : Return the first element from the dataset.

rddRead.take(5) : Return the first n lines from the dataset and display them on the console.

rddRead.count() : Return number of lines in a RDDt
table.columns : Display columns in a table
rdd.distinct().count() : Count distinct records

TableName.columns : Show list of columns on that table

Python Basic Commands

print("Hello World!")

O/P: Hello World!

# Single line Comment

''' 

Multi line comments

blah blah blah

'''

name = "Venkat"  # variable name with string value

print(name)

O/P: Venkat

# Data Types : Numbers, Strings, Lists, Tuples, and Dictionaries

name = 15 # variable name with number value

# Operators : + - * / %  ** //

print "10 + 5 =",10+5

print "10 - 5 =", 10-5

print "10 * 5 =", 10*5

print "10 / 5 =", 10/5

print "10 ** 5 =", 10**5

print "10 // 5 =", 10//5

O/P:

10 + 5 = 15

10 - 5 = 5

10 * 5 = 50

10 / 5 = 2

10 ** 5 = 100000

10 // 5 = 2

# Print 5 new lines
print "\n" * 5

# Lists
fruits_list = ["mango", "orange", "apple", "banana"]
# print first fruit
print fruits_list[0]
#print first fruit to third fruit
print fruits_list[1:3]

other_events = ["pick up kids", "wash car", "check balance"]
# List of lists
to_do_list = [fruits_list, other_events]
print to_do_list
#print third item of second list
print to_do_list[1][2]

# Append item to a list
fruits_list.append("guava")
print fruits_list

# Insert item to list at specified index
fruits_list.insert(1, "avocado")
print fruits_list

# Remove item from list
fruits_list.remove("guava")

#  Sort list
fruits_list.sort()
print fruits_list

# Reverse list
fruits_list.reverse()
print fruits_list

# Delete item from list
del fruits_list[3]
print fruits_list

O/P:
mango
['orange', 'apple']
check balance
['mango', 'orange', 'apple', 'banana', 'guava']
['mango', 'avocado', 'orange', 'apple', 'banana', 'guava']
['apple', 'avocado', 'banana', 'mango', 'orange']
['orange', 'mango', 'banana', 'avocado', 'apple']
['orange', 'mango', 'banana', 'apple']
['pick up kids', 'wash car', 'check balance', 'orange', 'mango', 'banana', 'apple']
7
wash car
apple

# Tuples : Sequence of immutable Python objects.
'''The tuples cannot be changed unlike lists and tuples use parentheses (), whereas lists use square brackets [].
'''
new_tuple = (1,2,3,4,5)
# Convert tuple to list
new_list = list(new_tuple)
print new_list
# Convert list to tuple
new_tuple2 = tuple(new_list)
print new_tuple2

O/P:
[1, 2, 3, 4, 5]
(1, 2, 3, 4, 5)

# Dictionary : Key and Values pairs represented in {key1:value1, key2:val2}
'''
Keys are unique within a dictionary while values may not be.The values of a dictionary can be of any type, but the keys must be of an immutable data type such as strings, numbers, or tuples.

'''

String Format

'{2}, {1}, {0}'.format('a', 'b', 'c')
O/P:
'c, b, a'

'{0},{1},{0}'.format('abra', 'cad')   # arguments' indices can be repeated
O/P:
'abra,cad,abra'

'Coordinates: {latitude}, {longitude}'.format(latitude='37.24N', longitude='-115.81W')
O/P:
'Coordinates: 37.24N, -115.81W'

coord = {'latitude': '37.24N', 'longitude': '-115.81W'}
 'Coordinates: {latitude}, {longitude}'.format(**coord)
O/P:
'Coordinates: 37.24N, -115.81W'

Strip()
str.strip([chars]);
The method strip() returns a copy of the string in which all chars have been stripped from the beginning and the end of the string (default whitespace characters).

How to run Scala or Python scripts in Spark

How to run Python script in Spark?
$ spark-submit test.py

How to run scala script in Spark?
$ spark-shell -i test.scala