PySpark Essential Training

Course Summary: In this course, you will learn Apache Spark framework and its components. Pyspark is an interactive layer of Spark built on python. You can leverage all Spark capabilities through Pyspark. During the training, We demonstrate how to build your data products over spark using Spark streaming, Spark RDDs, Spark SQL, Spark MLIB, Kafka and Flume.We also discuss in depth architecture of Spark and differences between Map Reduce and Spark.

1. Introduction to Spark
   1. Origins of Spark
   2. Understanding Spark
   3. Where Spark Shines
   4. Introduction to Notebooks

1. Spark Architecture and components/Installation
   1. Overview of Spark Components
   2. Spark Vs Hadoop
   3. Challenges Spark addresses
   4. Installation
   5. Create and Configure Spark Cluster
   6. Performance benchmarking – How Spark is faster than Hadoop

1. Working with RDDs – Execution on Spark Engine (Behind the scenes)
   1. What is RDD
   2. Spark transformations in RDD
   3. Actions in RDD
   4. Loading and Saving Data in RDD
   5. RDD – key value pair
   6. Broadcast Variables

1. Memory Management/Fault Tolerance/Lazy evaluation
   1. RDD fault tolerance
   2. In Memory computing
   3. Lazy evaluation and its advantages

1. Dataframes – aggregate/filter/sort/transform (Actions/Transformations)
   1. Introduction to Actions (take/collect/reduce/reduceByKey/foreach/histogram etc)
   2. Introduction to Transformations(aggregate/.sql/joins/.distinct/temporary table creation etc)
   3. Creating DataFrames
   4. Specifying schema for a dataframe
   5. Interacting and transforming dataframes

1. Introduction to Pyspark
   1. Apache Spark Stack
   2. Newest capabilities of Pyspark
   3. Spark Execution process

1. Pyspark SQL and Dataframes
   1. Spark SQL architecture
   2. Interacting with RDDs and converting objects to DataFrames
   3. SQL context in Spark SQL
   4. Performance Tuning
   5. Data processing with Spark Dataframes and UDFs
   6. Select/Filter/aggregate/sort/presenting the data

1. Apache Kafka and Flume
   1. Introduction to Kafka and Flume
   2. Creating and configuring Kafka Cluster
   3. Kafka architecture
   4. Basic Kafka operations

1. Pyspark Streaming
   1. Introduction to Spark Streaming
   2. Transformations using Dstreams
   3. Receiver based approach and Direct approach
   4. Streaming context setup
   5. Querying streaming data

1. Using Mlib in Spark for Machine Learning

                  10.1 Introduction to machine learning with Spark

                  10.2 Preparing data for Machine Learning

                  10.3 Building a linear regression model

                  10.4 Evaluating a linear regression model

                  10.5 visualizing a linear regression model

Hands-on :

1.  Install Spark and Build Spark Applications
2. How to use Dataframes, functions
3. Load Data in RDDs, RDD transformations
4. RDD actions and functions, partitioning
5. Spark SQL, Spark-Hive Integration, Loading data and querying using Spark SQL
6. Create ETL pipelines using Pyspark
7. Setting up Kafka Cluster, Kafka - Flume Integration
8. Spark – Flume Integration
9. Applying ML models and ML workflow utilities