Machine Learning Using Spark.ML and Python

Learn how to use Spark.ML and Python for machine learning.

Machine Learning Using Spark.ML and Python

Learn how to use Spark.ML and Python for machine learning.
Share the link to this page
One-time Purchase
List Price:  $99.99
You save:  $30
List Price:  €103.18
You save:  €30.95
List Price:  £92.08
You save:  £27.62
List Price:  CA$135.88
You save:  CA$40.76
List Price:  A$152.70
You save:  A$45.81
List Price:  S$142.97
You save:  S$42.89
List Price:  HK$784.88
You save:  HK$235.48
CHF 68.66
List Price:  CHF 98.09
You save:  CHF 29.43
NOK kr745.17
List Price:  NOK kr1,064.58
You save:  NOK kr319.40
DKK kr537.04
List Price:  DKK kr767.23
You save:  DKK kr230.19
List Price:  NZ$174.15
You save:  NZ$52.25
List Price:  د.إ367.27
You save:  د.إ110.19
List Price:  ৳10,374.02
You save:  ৳3,112.51
List Price:  ₹8,123.71
You save:  ₹2,437.35
List Price:  RM457.80
You save:  RM137.35
List Price:  ₦43,720.26
You save:  ₦13,117.39
List Price:  ₨24,073.58
You save:  ₨7,222.79
List Price:  ฿3,751.62
You save:  ฿1,125.59
List Price:  ₺1,841.38
You save:  ₺552.47
List Price:  B$526.42
You save:  B$157.94
List Price:  R1,792.44
You save:  R537.78
List Price:  Лв201.73
You save:  Лв60.52
List Price:  ₩142,261.77
You save:  ₩42,682.80
List Price:  ₪350.88
You save:  ₪105.27
List Price:  ₱5,874.41
You save:  ₱1,762.50
List Price:  ¥14,336.21
You save:  ¥4,301.29
List Price:  MX$2,019.75
You save:  MX$605.98
List Price:  QR364.06
You save:  QR109.23
List Price:  P1,336.27
You save:  P400.92
List Price:  KSh12,063.79
You save:  KSh3,619.50
List Price:  E£1,948.80
You save:  E£584.70
List Price:  ብር5,315.72
You save:  ብር1,594.87
List Price:  Kz43,111.11
You save:  Kz12,934.62
List Price:  CLP$95,056.69
You save:  CLP$28,519.86
List Price:  CN¥712.76
You save:  CN¥213.85
List Price:  RD$5,358.56
You save:  RD$1,607.73
List Price:  DA14,060.82
You save:  DA4,218.66
List Price:  FJ$228.66
You save:  FJ$68.60
List Price:  Q788.76
You save:  Q236.65
List Price:  GY$21,051.61
You save:  GY$6,316.11
ISK kr10,084.15
List Price:  ISK kr14,406.55
You save:  ISK kr4,322.40
List Price:  DH1,091.42
You save:  DH327.45
List Price:  L1,936.86
You save:  L581.11
List Price:  ден6,258.45
You save:  ден1,877.72
List Price:  MOP$813.54
You save:  MOP$244.08
List Price:  N$1,794.82
You save:  N$538.50
List Price:  C$3,616.33
You save:  C$1,085
List Price:  रु13,023.52
You save:  रु3,907.44
List Price:  S/391.09
You save:  S/117.34
List Price:  K352.46
You save:  K105.75
List Price:  SAR376.18
You save:  SAR112.86
List Price:  ZK1,609.93
You save:  ZK483.02
List Price:  L509.80
You save:  L152.95
List Price:  Kč2,539.94
You save:  Kč762.06
List Price:  Ft41,913.60
You save:  Ft12,575.34
SEK kr790.75
List Price:  SEK kr1,129.69
You save:  SEK kr338.94
List Price:  ARS$14,547.29
You save:  ARS$4,364.62
List Price:  Bs695.29
You save:  Bs208.60
List Price:  COP$439,225.38
You save:  COP$131,780.79
List Price:  ₡63,460.25
You save:  ₡19,039.97
List Price:  L2,473.75
You save:  L742.20
List Price:  ₲706,052.14
You save:  ₲211,836.82
List Price:  $U4,110.84
You save:  $U1,233.37

About the Class

You've heard how practically useful classification is for learning about your users. You are keen to try it for yourself. However, you've been unable to find any realistic user classification datasets. Such data is very difficult to find in published data sources.

You're familiar with Python and Apache Spark. You may have done some machine learning using Spark.ML. You have done parts of an end-to-end machine learning task, but there are other parts that you have yet to learn. 

You are good at some aspects of machine learning but less confident at others. You would like to work through a user classification task from beginning to end, starting from raw transactional user log data.

You will use a realistic user classification dataset that closely emulates the type of data one might encounter in a production setting. You will create a machine learning feature set from raw user log data and use it to create a classification model. You won't stop there - you will tune this model and improve it using hyperparameter tuning and data selection.

Spark combines the power of distributed computing with the ease of use of Python and SQL. Level up today.

What will you learn in this course?

  • How to train a classification model on high-dimensional data
  • How to do sensitivity analysis on a trained model.
  • How to assess the accuracy of a trained model.
  • How to improve model accuracy by tuning model hyperparameters.
  • How to improve model accuracy by judiciously selecting the data.
  • How to automate model tuning.
  • How to determine whether a dataset lends itself to classification. This entails analyzing the data and determining whether a classifier can have any chance of attaining a useful level of prediction accuracy. What is the power-law analysis and how to do it on a classification dataset?
  • How to analyze raw user log data.  In practice, data is not served up in an easy-to-use form. It typically is stored in a format that is easy to store. This is because being able to store the data efficiently often outweighs the need to be able to access that data in the future. 
  • How to do machine learning from end-to-end. This has several stages: (a) loading a raw file, (b) putting the data into a format suitable for analysis and for feature set creation, (c) creating the feature data, for example vectorizing the data, properly handling missing values, and reducing the dimensionality, (d) training the machine learning model, (e) assessing the performance of the model, (e) tuning the model.
  • Extracting, Transforming, and Selecting (ETS).  How to convert raw user log data to a tabular format that is most useful for training a classifier. The data may not be provided in a tabular format.  A machine learning engineer often needs to create a table to represent the data. They will also be responsible for identifying and resolving issues with the data that would otherwise hamper the accuracy of the model fit to the data.

What are the learning objectives?

The learner will complete two machine learning projects from beginning-to-end.

  • For the first project, the learner will be given a dataset that initially may seem inscrutable. However, they are soon able to extract useful information from the data. The learner will be able to do this using both interactive analyses as well as automated methods.
  • For the second project, the learner will predict the last item in a sequence. This technique is generally applicable to several types of recommendation systems.

The learner will know how to improve the accuracy of a seemingly marginally useful model. The learner will do a pre-deployment analysis. A machine learning task does not end when the model is trained. Calculating the overall model accuracy is not always sufficient. Assessing the model accuracy under different constraints on the input data is important to a successful deployment. This is because a model can be more or less accurate in some regions of the input space compared to other regions of the input space.

Here are some of the specific concepts that are covered:

  • What is the first thing to do with a messy comma-delimited log file
  • How to load raw data from a file and convert it into a tabular format 
  • What is the power-law analysis and how to use it on a dataset to determine whether that dataset is suitable for classification?
  • How to create feature data for training a logistic regression classifier.
  • How to train a logistic regression classifier on text data. 
  • How data selection can be used to optimize model accuracy, and the tradeoff between coverage and accuracy.
  • How to use sensitivity analysis to determine how well the model performs in different regions of the input space.

What technologies, packages, and functions will students use?

  • Bash
  • Python
  • Matplotlib
  • Pyplot
  • Apache Spark
  • Spark SQL
  • Spark.ML

All libraries used in this course are contained in the pyspark package. This means that we will not need to install additional packages. All modules are either part of the standard python library or imported from the pyspark module.

What terms and jargon will be defined?

Here is a list of technical terms, jargon, and acronyms that will be used in the course:

Feature engineering, machine learning, model fitting, classification, logistic regression, logistic classification, power law, Extract Transform and Select (ETS), pipeline, cross-validation, hyperparameters, feature engineering, feature sets, vectorizer, vocabulary, area under the curve (AUC), data selection, grid search, automated model tuning, Dataframe, Spark.ML, StructType, StructField, CountVectorizer, 

What concepts will be taught?

Active versus Casual is a common way to characterize users.  Active users can have many different behavioral characteristics than casual users. How can you ascertain this from the data?

Sanity checks and sensitivity analysis are pro tips familiar to every seasoned machine learning engineer.  How can you quickly determine whether your model is unbiased? How can you quickly determine how your model will perform on various segments of your user base? 

Prediction Accuracy vs Coverage. Students are usually taught to calculate prediction accuracy by averaging over the entire sample space. One can surface opportunities to improve prediction accuracy by segmenting the space. One can also determine what is the effective coverage of a model. The coverage is determined by the portion of novel data for which the prediction accuracy exceeds a desired threshold. A model might do extremely well for certain data, but be no better than a random guess on other data.

When presented with a large amount of raw data that belongs to two different classes, the analyst might initially believe that there is no discernible statistical difference between the two classes of data. However, if you know how to look more carefully, you can determine differences that can be identified.  Moreover, you can determine whether these effects are strong enough to automate a classification task using machine learning.

What pro tips are taught?

  • Learners may be inclined to use too much data when training a machine learning model. They might give short shrift to dimensionality reduction.
  • Learners may be inclined to impose their own preconceptions on the data, rather than automating analysis and letting the data speak for itself.
  • Learners might be aware of how to tune hyperparameters, but they might not be aware of how data selection can be used to optimize model performance.
  • Learners might not know how to use simple forms of sanity checks and sensitivity analysis to more quickly ascertain whether their model is valid.

What datasets will be used?

Two datasets are used. One is for guessing user demographic class membership from usage log data. The second is a text corpus. 

The usage of the log dataset is special.  This dataset closely emulates data commonly found in production settings, but it is difficult to find in published data sources. The dataset has been carefully created to emulate real demographic data.  Such data is rarely if ever published. One reason is to protect proprietary secrets. Another is to avoid privacy leaks.

This dataset looks and behaves very much like real data. This allows practicing real techniques that are easily transferrable to proprietary data you might encounter in a production setting. 

The data will be transformed several times along the way for the various stages of the task. So, while the entire course is based on a single dataset, the data will be manipulated such that it can seem like a completely different dataset at various stages along the way.

Although the dataset closely emulates the statistical qualities of a real-world demographic dataset, rather than use real-world labels we instead use a comical hypothetical scenario. However, make no mistake, the data is very real in that it captures very realistic qualities of data encountered in a production setting.

This approach has some benefits.  Firstly, it prevents us from being biased by unwarranted assumptions. You may encounter a dataset and make assumptions about it based on your own experience. Instead, you should be data-driven.

Using this hypothetical scenario also emphasizes the generality of this technique. This approach can be used to guess the gender, political affiliation, retiree vs teenager, homeowner vs renter, cancerous vs healthy, hotdog vs non-hotdog, just to name a few.

Additionally, this allows us to have a bit of levity along the way.

The primary dataset labels its two classes as "rabbit" and "duck".  At first, they look alike, but when the learner knows what to look for, they are easy to tell apart.

The text corpus dataset is a standard dataset commonly used to teach or demonstrate text processing. In our case, we are going to use it to demonstrate sequence prediction.  The text corpus is large enough that it poses realistic constraints on our algorithms. It is small enough that training can still be done quickly. In a production situation, the actual text corpus may be orders of magnitude larger. However, the underlying concepts taught here are applicable.

We use the text corpus as a stand-in for user session data.  Examples of this include song identifiers, topic ids, hashtags, URLs, and any other type of identifier that is consumed in a sequential manner. The task is predicting the last item in a sequence. 

We could have instead created a dataset that closely mimics the statistical characteristics of a production session log dataset.  However, using a text corpus has some advantages. Working with text makes this more intuitive. It is easier to understand what is going on when working with sequences of tokens that correspond to sentences of words rather than opaque identifiers.


Mark Plutowski

Data Professional
Share the instructor profile
Has 20 years experience in data-driven analysis and machine learning development. Mark has a Ph.D in Computer Science from UCSD and a Master of Science in Electrical and Computer Engineering from USC. Mark worked at IBM, Sony, and Netflix generating 29 patents. Mark is an experienced educator, having published courseware on multiple online...

Class Requirements

  • You should have SQL installed on your PC/Mac
  • You should have knowledge of python lambdas
  • You should have Pyspark and Python installed on your PC/Mac

What's Included

Language: English
Level: Intermediate
Skills: Data Analysis, Python, Pyplot, Apache Spark, Matplotlib, Machine Learning, Spark SQL, Spark.ML
Age groups: All ages
Duration: 3 hours 12 minutes
17 Videos
4 Documents
This class has not been saved

Sign Up


Share with friends, get 20% off
Invite your friends to LearnDesk learning marketplace. For each purchase they make, you get 20% off (upto $10) on your next purchase.