[Activity] Analyzing Apache Log files with Structured Streaming

November 6, 2019

Back to: Streaming Big Data with Spark Streaming, Scala, and Spark 3!

Intro to Structured Streaming in Spark 2

Getting Started
- Tip: Apply for a Twitter Developer Account now!
- Download the Course Materials
- Introduction, and Getting Set Up
- [Activity] Stream Live Tweets with Spark Streaming!
A Crash Course in Scala
- [Activity] Scala Basics: Part 1
- [Exercise] Scala Basics: Part 2
- [Exercise] Flow Control in Scala
- [Exercise] Functions in Scala
- [Excercise] Data Structures in Scala
- Introduction, and Getting Set Up
Spark Streaming Concepts
- Introduction to Spark
- The Resilient Distributed Dataset (RDD)
- [Activity] RDD’s in action: simple word count application
- Introduction to Spark Streaming
- [Activity] Revisiting the PrintTweets application
- Windowing: Aggregating data over longer time spans
- Fault Tolerance in Spark Streaming
Spark Streaming Examples with Twitter
- [Exercise] Saving Tweets to Disk
- [Exercise] Tracking the Average Tweet Length
- [Exercise] Tracking the Most Popular Hashtags
Spark Streaming Examples with Clickstream / Apache Access Log Data
- [Exercise] Tracking the Top URL’s Requested
- [Exercise] Alarming on Log Errors
- [Exercise] Integrating Spark Streaming with Spark SQL
- Intro to Structured Streaming in Spark 2
- [Activity] Analyzing Apache Log files with Structured Streaming
Integrating with Other Systems
- Integrating with Apache Kafka
- Integrating with Apache Flume
- Integrating with Amazon Kinesis
- [Activity] Writing Custom Data Receivers
- Integrating with Cassandra
Advanced Spark Streaming Examples
- [Exercise] Stateful Information in Spark Streams
- [Activity] Streaming K-Means Clustering
- [Activity] Streaming Linear Regression
Spark Streaming in Production
- [Activity] Running with spark-submit
- [Activity] Packaging your code with SBT
- Running on a real Hadoop cluster with EMR
- Troubleshooting and Tuning Spark Jobs
You Made It!
- Learning More
- Continue your Learning Journey!

[Activity] Analyzing Apache Log files with Structured Streaming

[Activity] Analyzing Apache Log files with Structured Streaming

Intro to Structured Streaming in Spark 2

Integrating with Apache Kafka