WEBVTT

00:00.720 --> 00:04.500
-: So in July of 2016, Spark released Spark 2.0.

00:04.500 --> 00:06.600
And let's talk about what's new and what new capabilities

00:06.600 --> 00:08.223
exist in ML Lib now.

00:08.223 --> 00:11.130
So, the main thing with Spark 2.0 is that they're moving

00:11.130 --> 00:14.400
people more and more toward data frames and data sets.

00:14.400 --> 00:16.740
Data sets and data frames are kinda used interchangeably,

00:16.740 --> 00:19.950
sometimes, technically, a data frame is a data set of

00:19.950 --> 00:21.360
row objects.

00:21.360 --> 00:24.690
They're kinda like RDDs but the only difference is that

00:24.690 --> 00:27.360
where an as RDD just contains unstructured data,

00:27.360 --> 00:30.150
every row of an RDD can contain pretty much anything,

00:30.150 --> 00:32.490
a data set has a defined schema to it.

00:32.490 --> 00:35.490
So a data set knows ahead of time exactly what columns

00:35.490 --> 00:38.880
of information exist in each row of that little RDD, of

00:38.880 --> 00:41.490
that data set, and what types those are.

00:41.490 --> 00:43.980
So because it knows about the actual structure of that

00:43.980 --> 00:45.450
data set ahead of time,

00:45.450 --> 00:47.730
it can optimize things more efficiently.

00:47.730 --> 00:50.250
It also lets us think of the contents of this data set

00:50.250 --> 00:53.190
as little mini database. Well, actually a very big database

00:53.190 --> 00:54.390
if it's on a cluster, right?

00:54.390 --> 00:56.850
And that means we can do things like issue sequel queries

00:56.850 --> 01:00.030
on it. So this creates a higher level API, with which we

01:00.030 --> 01:03.150
can query and analyze large, massive data sets on

01:03.150 --> 01:05.220
a Spark cluster. So it's pretty cool stuff.

01:05.220 --> 01:07.830
It's faster, it has more opportunities for optimization,

01:07.830 --> 01:10.290
and it has a higher level API that's often easier to

01:10.290 --> 01:13.504
work with. Now going forward in Spark 2.0,

01:13.504 --> 01:16.800
ML Lib is pushing data frames as its primary API, so this

01:16.800 --> 01:18.090
is kinda the way of the future here.

01:18.090 --> 01:20.283
So let's take a look at how it works.

01:21.180 --> 01:23.640
So I've gone ahead and opened up the Spark linear regression

01:23.640 --> 01:26.940
dot py file. Let's walk through it a little bit here.

01:26.940 --> 01:30.210
So you see for one thing, we're using ML instead of ML Lib.

01:30.210 --> 01:32.550
And that's the new data frame-based APIs in there.

01:32.550 --> 01:35.220
So, for, in this example, what we're gonna do is implement

01:35.220 --> 01:38.910
linear regression. And linear regression is just a way of

01:38.910 --> 01:41.160
fitting a line to a set of data. So what we're gonna

01:41.160 --> 01:44.280
do in this exercise is take a bunch of fabricated data

01:44.280 --> 01:46.950
that we have in two dimensions and try to fit a line

01:46.950 --> 01:49.350
to it with a linear model. And what we're gonna do is

01:49.350 --> 01:52.410
actually separate our data into two sets: one for building

01:52.410 --> 01:54.990
the model and one for evaluating the model, and we'll

01:54.990 --> 01:58.080
compare how well this linear model does to actually

01:58.080 --> 02:00.180
predicting real values.

02:00.180 --> 02:02.820
So to do that, first of all, in Spark 2, if you're gonna

02:02.820 --> 02:05.250
be doing stuff with the Spark SQL interface and using data

02:05.250 --> 02:08.460
sets, you gotta be using a Spark session object instead of a

02:08.460 --> 02:10.800
Spark context. So to set one up, you're gonna do something

02:10.800 --> 02:13.110
like this. You can say Spark, that's gonna be the name

02:13.110 --> 02:16.560
of our Spark session, builder config, now this bit

02:16.560 --> 02:19.740
is only necessary on Windows in Spark 2.0. It kinda

02:19.740 --> 02:22.350
works around a little bug that they have, to be honest.

02:22.350 --> 02:25.020
So if you're on Windows, make sure you have a C temp folder

02:25.020 --> 02:27.840
if you wanna run this. Go create that now if you need to.

02:27.840 --> 02:30.000
If you're not on Windows, you can delete that whole bit

02:30.000 --> 02:33.030
here that I've highlighted, okay, cause it's not gonna be

02:33.030 --> 02:36.330
necessary and it won't work. Give it an app name,

02:36.330 --> 02:39.570
and get or create. Now, this is interesting because

02:39.570 --> 02:41.940
once you create a Spark session, if it terminates

02:41.940 --> 02:43.890
unexpectedly, it can actually recover from that

02:43.890 --> 02:46.530
next time you run it. So, if we have a checkpoint

02:46.530 --> 02:48.840
directory, it can actually restart where it left off

02:48.840 --> 02:51.870
using get or create. All right, now we're gonna use

02:51.870 --> 02:54.720
this regression.txt file that I've included with the

02:54.720 --> 02:57.150
course materials, and all that is is a text file that

02:57.150 --> 03:00.570
has comma delimited value of two columns, and they're just

03:00.570 --> 03:04.830
two columns of, more or less, randomly linearly correlated

03:04.830 --> 03:06.360
data, and it can represent whatever you want.

03:06.360 --> 03:09.090
Let's imagine that it represents heights and weights,

03:09.090 --> 03:12.390
for example. So, the first column may represent heights,

03:12.390 --> 03:15.060
the second column might represent weights.

03:15.060 --> 03:17.190
So, in the lingo of machine learning,

03:17.190 --> 03:19.470
we talk about labels and features,

03:19.470 --> 03:21.420
where labels are usually the thing that you're trying to

03:21.420 --> 03:23.940
predict, and features are a set of known attributes

03:23.940 --> 03:26.610
of the data that you use to make a prediction from.

03:26.610 --> 03:29.400
So, in this example, maybe heights are the labels

03:29.400 --> 03:32.160
and the features are the weights. Maybe we're trying to

03:32.160 --> 03:34.650
predict heights based on your weight.

03:34.650 --> 03:36.330
It can be anything, it doesn't matter.

03:36.330 --> 03:41.010
It's all normalized down to like, data between -1 and 1, so

03:41.010 --> 03:43.350
there's no real meaning to the scale of the data anywhere.

03:43.350 --> 03:45.870
You can pretend it means anything you want, really.

03:45.870 --> 03:48.240
So to do this, to use this with ML Lib, we need to

03:48.240 --> 03:50.880
transform our data into the format it expects. So the first

03:50.880 --> 03:53.100
thing we're gonna do is split this data up with this map

03:53.100 --> 03:57.810
function, that just splits each line into two distinct

03:57.810 --> 04:01.530
values in a list. Then we're gonna map that to the format

04:01.530 --> 04:02.730
that ML Lib expects.

04:02.730 --> 04:05.730
So that's gonna be a floating point label.

04:05.730 --> 04:07.980
And then a dense vector of the feature data.

04:07.980 --> 04:10.320
Now in this case, we only have one bit of the feature data,

04:10.320 --> 04:12.960
the weight, so we have a vector that just has one thing

04:12.960 --> 04:16.110
in it. But even if it's just one thing, the ML Lib

04:16.110 --> 04:19.020
linear regression model requires a dense vector there.

04:19.020 --> 04:22.350
Okay, this is like a label point in the older API.

04:22.350 --> 04:25.020
But we have to kinda do it the hard way here.

04:25.020 --> 04:27.930
Now next we need to actually assign names to these columns.

04:27.930 --> 04:31.740
So here's the syntax for that, we're gonna tell ML Lib

04:31.740 --> 04:35.910
that these two columns in my resulting RDD here actually

04:35.910 --> 04:39.540
correspond to the label and the features and then I can

04:39.540 --> 04:42.570
convert that RDD to a data frame object.

04:42.570 --> 04:44.940
So at this point, I have an actual data frame, or

04:44.940 --> 04:47.880
if you will, a data set that contains two columns,

04:47.880 --> 04:51.000
label and features, where the label is a floating point

04:51.000 --> 04:54.780
height, and the features column is a dense vector of

04:54.780 --> 04:58.140
floating point weights. And that is the format required

04:58.140 --> 05:01.410
by ML Lib, and ML Lib can be pretty picky about this stuff

05:01.410 --> 05:04.230
so it's important that you pay attention to these formats.

05:04.230 --> 05:06.180
All right, now like I said, we're going to split our data

05:06.180 --> 05:09.210
in half, so we're gonna do a 50/50 split between

05:09.210 --> 05:11.640
training data and test data, so this returns back

05:11.640 --> 05:14.130
two data frames; one that I'm actually gonna use to

05:14.130 --> 05:16.470
create my model, and one that I'm gonna use to evaluate

05:16.470 --> 05:17.910
my model.

05:17.910 --> 05:20.340
I will next create my actual linear regression model

05:20.340 --> 05:22.560
with a few standard parameters here that I've set.

05:22.560 --> 05:24.900
We're gonna call that lir, linear regression.

05:24.900 --> 05:27.450
And then I will fit that model to the set of data that

05:27.450 --> 05:30.420
I've held aside for training, the training data frame.

05:30.420 --> 05:32.310
And that gives me back a model that I can use to make

05:32.310 --> 05:33.780
predictions from.

05:33.780 --> 05:37.554
So let's go ahead and do that. I will call model.transform.

05:37.554 --> 05:42.554
with testdf. And what that's gonna do is predict the heights

05:42.900 --> 05:46.020
based on the weights in my testing data. Okay, so the

05:46.020 --> 05:48.450
testing data set, I actually have the known labels,

05:48.450 --> 05:50.940
the actual correct heights, and this is going to add a

05:50.940 --> 05:53.940
new column to that data frame called predictions, that

05:53.940 --> 05:56.850
has the predictive values based on the linear model.

05:56.850 --> 05:58.560
I'm gonna do a couple things with this, I'm gonna

05:58.560 --> 06:00.870
cache the results and now I can just

06:00.870 --> 06:02.580
extract that and compare them together.

06:02.580 --> 06:06.090
So, let's pull out the prediction column just using

06:06.090 --> 06:09.000
dot select, just like you would in SQL. And then I'm

06:09.000 --> 06:11.700
gonna actually transform that data frame and pull out

06:11.700 --> 06:14.523
the RDD from it, and use that to map it to just a

06:14.523 --> 06:19.230
plain old RDD full of floating point heights and what, uh,

06:19.230 --> 06:21.870
uh, floating point heights in this case, right.

06:21.870 --> 06:24.330
So these are the predicted heights, and then we're going

06:24.330 --> 06:27.570
to get to the actual heights from the label column, and

06:27.570 --> 06:29.340
then we can zip them back together and just print them

06:29.340 --> 06:32.280
out side by side and see how well it does.

06:32.280 --> 06:34.920
Now, mind you, this is kind of a convoluted way of doing it.

06:34.920 --> 06:37.440
I did it this way to be more consistent with the previous

06:37.440 --> 06:39.653
example, but a simpler approach would actually just be to

06:39.653 --> 06:42.990
select prediction and label together into a single RDD

06:42.990 --> 06:45.990
that maps out those two columns together, and then

06:45.990 --> 06:48.690
I don't have to zip them up. But either way works.

06:48.690 --> 06:50.340
So let's see if it works.

06:50.340 --> 06:53.889
-: So to run this, again, we'll open up an Anaconda prompt.

06:53.889 --> 06:54.722
Do do do.

06:56.100 --> 06:58.683
And we will cd to our course material folder.

07:01.350 --> 07:04.050
And let's type in spark dash submit.

07:04.050 --> 07:08.013
Spark linear regression dot py, and let it do its thing.

07:12.870 --> 07:15.180
So again, we're using the new ML interface that's data

07:15.180 --> 07:17.670
frame based. And by the way, if we're using Scala

07:17.670 --> 07:19.950
instead of Python, you'd probably be using data sets

07:19.950 --> 07:22.800
instead of data frames, just different terminology there.

07:27.720 --> 07:30.240
And there we have the results. So we have here, each, uh

07:30.240 --> 07:33.120
result here is the predictive value followed by the

07:33.120 --> 07:35.760
actual value. And you can see that by and large, they're

07:35.760 --> 07:38.880
pretty close. I mean, it's not perfect, but, uh

07:38.880 --> 07:41.340
the model at least did something reasonable.

07:41.340 --> 07:42.900
So that's it, that's cool.

07:42.900 --> 07:44.970
So, yeah, we just did a linear regression, uh,

07:44.970 --> 07:49.650
using Spark's new ML API, well relatively new, er, 2016

07:49.650 --> 07:51.930
it's, uh, you know, several years have passed since then,

07:51.930 --> 07:54.000
but this is still kinda the way of the future that they're

07:54.000 --> 07:57.660
pushing people toward. Again, RDDs still work, but, um,

07:57.660 --> 08:00.930
the data frame-based API is sort of where they're

08:00.930 --> 08:02.310
concentrating their efforts these days.

08:02.310 --> 08:04.350
So it's worth understanding it.

08:04.350 --> 08:06.903
There you have it, ML in action, in Spark.
