WEBVTT 00:01.219 --> 00:02.052 Narrator: So, so far we've seen the power 00:02.052 --> 00:03.900 of just using a simple multilayer perceptron 00:03.900 --> 00:05.970 to solve a wide variety of problems. 00:05.970 --> 00:07.260 But you can take things up a notch. 00:07.260 --> 00:10.110 You can arrange more complicated neural networks together 00:10.110 --> 00:11.970 and do more complicated problems with them. 00:11.970 --> 00:13.590 So let's start by talking about 00:13.590 --> 00:16.533 convolutional neural networks or CNNs for short. 00:17.400 --> 00:19.410 Usually you hear about CNNs in the context 00:19.410 --> 00:22.830 of image analysis, and their whole point is to find things 00:22.830 --> 00:24.660 in your data that might not be exactly 00:24.660 --> 00:26.250 where you expect it to be. 00:26.250 --> 00:29.670 So technically we call this feature-location invariant. 00:29.670 --> 00:31.800 That means that if you're looking for some pattern 00:31.800 --> 00:33.150 or some feature in your data, 00:33.150 --> 00:35.850 but you don't know where exactly it might be in your data, 00:35.850 --> 00:37.650 a CNN can scan your data 00:37.650 --> 00:40.080 and find those patterns for you wherever they might be. 00:40.080 --> 00:41.580 So for example, in this picture here, 00:41.580 --> 00:43.740 that stop sign could be anywhere in the image 00:43.740 --> 00:46.200 and a CNN is able to find that stop sign 00:46.200 --> 00:47.730 no matter where it might be. 00:47.730 --> 00:50.190 Now, it's not just limited to image analysis, 00:50.190 --> 00:51.990 it can also be used for any sort of problem 00:51.990 --> 00:53.460 where you don't know where the features 00:53.460 --> 00:55.710 you are might be located within your data 00:55.710 --> 00:58.800 and machine translation or natural language processing tasks 00:58.800 --> 00:59.940 come to mind for that. 00:59.940 --> 01:02.220 You don't necessarily know where the noun or the verb 01:02.220 --> 01:04.380 or a phrase that you care about might be 01:04.380 --> 01:06.690 in some paragraph or sentence that you're analyzing, 01:06.690 --> 01:08.940 but a CNN can find it and pick it out for you. 01:08.940 --> 01:11.820 Sentiment analysis's another application of CNNs, 01:11.820 --> 01:14.280 you might not know exactly where a phrase might be 01:14.280 --> 01:16.530 that indicates some happy sentiment 01:16.530 --> 01:18.270 or some frustrated sentiment, 01:18.270 --> 01:20.130 or whatever you might be looking for, 01:20.130 --> 01:23.010 but a CNN can scan your data and pluck it out 01:23.010 --> 01:24.330 and you'll see that the idea behind it 01:24.330 --> 01:26.220 isn't really as complicated as it sounds. 01:26.220 --> 01:28.290 This is another example of using fancy words 01:28.290 --> 01:31.410 to make things sound more complicated than they really are. 01:31.410 --> 01:32.460 So how do they work? 01:32.460 --> 01:35.010 Well, CNN's convolutional neural networks are inspired 01:35.010 --> 01:37.140 by the biology of your visual cortex. 01:37.140 --> 01:40.110 It takes cues from how your brain actually processes images 01:40.110 --> 01:41.880 from your retina, and it's pretty cool 01:41.880 --> 01:43.620 and it's also another example 01:43.620 --> 01:45.570 of interesting emergent behavior. 01:45.570 --> 01:49.170 So the way your eyes work is that individual groups 01:49.170 --> 01:52.950 of neurons service a specific part of your field of vision. 01:52.950 --> 01:55.470 So we call these local receptive fields. 01:55.470 --> 01:57.270 They are just groups of neurons that respond only to 01:57.270 --> 01:59.174 a part of what your eyes see. 01:59.174 --> 02:01.950 It's sub samples, the image coming in from your retinas 02:01.950 --> 02:03.750 and just has specialized groups 02:03.750 --> 02:06.600 of neurons for processing specific parts of the field 02:06.600 --> 02:08.940 of view that you see with your eyes. 02:08.940 --> 02:11.100 Now, these little areas overlap each other 02:11.100 --> 02:12.720 to cover your entire visual field 02:12.720 --> 02:14.700 and this is called convolution. 02:14.700 --> 02:16.440 Convolution is just a fancy word of saying 02:16.440 --> 02:18.930 I'm going to break up this data into little chunks 02:18.930 --> 02:20.850 and process those chunks individually 02:20.850 --> 02:22.650 and then they'll assemble a bigger picture 02:22.650 --> 02:24.900 of what you're seeing higher up in the chain. 02:25.740 --> 02:27.900 So the way it works within your brain 02:27.900 --> 02:29.250 is that you have many layers, 02:29.250 --> 02:32.490 it is a deep neural network that identifies various 02:32.490 --> 02:34.560 complexities of features, if you will. 02:34.560 --> 02:37.860 So the first layer that you go into from your convolutional 02:37.860 --> 02:40.860 neural network inside your head might just identify 02:40.860 --> 02:43.410 horizontal lines or lines at different angles 02:43.410 --> 02:47.610 or you know, specific kinds of edges, we call these filters, 02:47.610 --> 02:49.410 and that might feed into a layer above them 02:49.410 --> 02:51.510 that would then assemble those lines 02:51.510 --> 02:54.330 that identified at the lower level into shapes 02:54.330 --> 02:56.520 and maybe there's a layer above that that would be able to 02:56.520 --> 02:58.740 recognize objects based on the patterns 02:58.740 --> 03:00.180 of shapes that you see 03:00.180 --> 03:01.680 and then if you're dealing with color images, 03:01.680 --> 03:03.330 we have to multiply everything by three 03:03.330 --> 03:05.580 because you actually have specialized cells 03:05.580 --> 03:08.490 within your retina for detecting red, green, and blue light 03:08.490 --> 03:10.080 and we need to assemble those together as well. 03:10.080 --> 03:12.450 Those each get processed individually too. 03:12.450 --> 03:14.100 So that's all a CNN is. 03:14.100 --> 03:17.220 It is just taking a source image or source data 03:17.220 --> 03:19.170 of any sort, really, breaking it up 03:19.170 --> 03:21.810 into little chunks called convolutions 03:21.810 --> 03:24.210 and then we assemble those and look for patterns 03:24.210 --> 03:26.130 at increasingly higher complexities, 03:26.130 --> 03:28.773 at higher levels in your neural network. 03:30.810 --> 03:31.890 So how does your brain know 03:31.890 --> 03:33.360 that you're looking at a stop sign there? 03:33.360 --> 03:36.150 Let's talk about this in more colloquial language, 03:36.150 --> 03:37.020 if you will. 03:37.020 --> 03:39.990 So like we said, you have individual local receptive fields 03:39.990 --> 03:41.310 that are responsible for processing 03:41.310 --> 03:43.500 specific parts of what you see 03:43.500 --> 03:45.657 and those local receptive fields are scanning your image 03:45.657 --> 03:48.390 and they overlap with each other looking for edges. 03:48.390 --> 03:51.060 You might notice that your brain is very sensitive 03:51.060 --> 03:53.250 to contrast and edges that it sees in the world. 03:53.250 --> 03:54.660 Those tend to catch your attention, right? 03:54.660 --> 03:57.450 That's why the letters on this slide catch your attention 03:57.450 --> 03:59.070 because there's high contrast 03:59.070 --> 04:01.650 between the letters and the white background behind them. 04:01.650 --> 04:04.110 So at a very low level, you're picking up the edges 04:04.110 --> 04:04.943 of that stop sign 04:04.943 --> 04:07.260 and the edges of the letters on the stop sign. 04:07.260 --> 04:09.450 Now, a higher level might take those edges 04:09.450 --> 04:11.760 and recognize the shape of that stop sign says, 04:11.760 --> 04:13.410 Oh, there's an octagon there. 04:13.410 --> 04:14.970 That means something special to me 04:14.970 --> 04:16.440 or those letters form the word stop. 04:16.440 --> 04:18.420 That means something special to me too 04:18.420 --> 04:19.830 and ultimately, that will get matched 04:19.830 --> 04:22.290 against whatever classification pattern your brain has 04:22.290 --> 04:23.580 of a stop sign. 04:23.580 --> 04:27.480 So no matter which receptive field picked up that stop sign, 04:27.480 --> 04:30.660 at some layer, it will be recognized as a stop sign 04:30.660 --> 04:33.120 and furthermore, because you're processing data in color, 04:33.120 --> 04:34.200 it can also use the information 04:34.200 --> 04:37.770 that this stop sign is red and further use that to aid 04:37.770 --> 04:40.710 in its classification of what this object really is. 04:40.710 --> 04:42.840 So somewhere in your head there's a neural network 04:42.840 --> 04:46.410 that says, Hey if I see edges arranged in an octagon pattern 04:46.410 --> 04:49.470 that has a lot of red in it and says, stop in the middle, 04:49.470 --> 04:51.990 that means I should probably hit the brakes on my car 04:51.990 --> 04:54.450 and it's some even higher level where your brain 04:54.450 --> 04:57.210 is actually doing higher reasoning that's what happened. 04:57.210 --> 04:59.010 There's a wire that says, Hey, 04:59.010 --> 05:00.300 there's a stop sign coming up here. 05:00.300 --> 05:01.530 I better hit the brakes in my car 05:01.530 --> 05:03.060 and if you've been driving long enough 05:03.060 --> 05:05.555 you don't even really think about it anymore, do you? 05:05.555 --> 05:06.388 Like, it's almost hardwired 05:06.388 --> 05:08.040 and that literally may be the case. 05:08.040 --> 05:10.359 Anyway, a convolution neural network, 05:10.359 --> 05:11.880 an artificial convolutional neural network 05:11.880 --> 05:13.833 works the same way, same exact idea. 05:15.540 --> 05:18.336 So how do you build a CNN with Keras? 05:18.336 --> 05:19.560 You know, obviously you probably don't wanna do this 05:19.560 --> 05:20.940 at the low level TensorFlow layer. 05:20.940 --> 05:23.850 You can, but CNNs get pretty complicated. 05:23.850 --> 05:27.180 A higher level library like Keras becomes essential. 05:27.180 --> 05:28.740 First of all, you need to make sure your source data 05:28.740 --> 05:30.000 is of the appropriate dimensions, 05:30.000 --> 05:31.980 of the appropriate shape, if you will 05:31.980 --> 05:34.830 and you are going to be preserving the actual 2D structure 05:34.830 --> 05:36.630 of an image if you're dealing with image data here. 05:36.630 --> 05:38.970 So the shape of your data might be the width times 05:38.970 --> 05:41.280 the length times the number of color channels 05:41.280 --> 05:43.410 and by color channels, I mean, if it's a black 05:43.410 --> 05:46.260 and white image, there's only one color, black and white. 05:46.260 --> 05:47.490 So you'd only have one color channel 05:47.490 --> 05:49.200 for a a gray scale image. 05:49.200 --> 05:51.720 But if it's a color image, you'd have three color channels, 05:51.720 --> 05:54.137 one for red, one for green, and one for blue 05:54.137 --> 05:55.140 because you can create any color 05:55.140 --> 05:58.140 by combining red, green, and blue together, okay. 05:58.140 --> 06:00.840 Now, there are some specialized types of layers in Keras 06:00.840 --> 06:02.070 that you use when you're dealing with 06:02.070 --> 06:03.900 convolutional neural networks. 06:03.900 --> 06:06.240 For example, there's the Conv2D layer type, 06:06.240 --> 06:08.940 that does the actual convolution on a 2D image 06:08.940 --> 06:11.910 and again, convolution is just breaking up that image 06:11.910 --> 06:14.100 into little subfields that overlap each other 06:14.100 --> 06:15.840 for individual processing. 06:15.840 --> 06:19.290 There's also a Conv1D and a Conv3D layer available as well. 06:19.290 --> 06:20.940 You don't have to use CNNs with images. 06:20.940 --> 06:22.860 Like we said, it can also be used with text data, 06:22.860 --> 06:24.390 for example, that might be an example 06:24.390 --> 06:26.160 of one dimensional data 06:26.160 --> 06:29.190 and it's also a Conv3D layers available as well, 06:29.190 --> 06:31.470 if you're dealing with 3D volumetric data of some sort. 06:31.470 --> 06:33.810 So the lot of possibilities there. 06:33.810 --> 06:35.880 Another specialized layer and Keras for CNNs 06:35.880 --> 06:37.380 is MaxPooling2d. 06:37.380 --> 06:39.960 Obviously, there's a 1D and 3D variant of that as well. 06:39.960 --> 06:41.970 The idea of that is just to reduce the size 06:41.970 --> 06:43.110 of your data down. 06:43.110 --> 06:45.540 So if I take just the maximum value scene 06:45.540 --> 06:47.610 in a given block of an image, 06:47.610 --> 06:50.040 and reduce the two layer down to those maximum values, 06:50.040 --> 06:52.830 it's just a way of shrinking the images in such a way 06:52.830 --> 06:55.290 that it can reduce the processing load on the CNN. 06:55.290 --> 06:58.890 As you'll see, processing CNNs is very computing intensive 06:58.890 --> 07:01.050 and the more you can do to reduce the work you have to do, 07:01.050 --> 07:02.480 the better. 07:02.480 --> 07:04.350 So if you have more data in your image than you need, 07:04.350 --> 07:07.320 a MaxPooling2D layer can be useful for distilling that down 07:07.320 --> 07:10.260 to the the bare essence of what you need to analyze. 07:10.260 --> 07:11.970 Finally, at some point, you need to feed this data 07:11.970 --> 07:14.820 into a flat layer of neurons, right? 07:14.820 --> 07:17.820 At some point it's just gonna go into a perceptron, 07:17.820 --> 07:21.000 and at this stage we need to flatten that 2D layer 07:21.000 --> 07:23.070 into a 1D layer so we can just pass it 07:23.070 --> 07:25.260 into a layer of neurons and from that point, 07:25.260 --> 07:28.050 it just looks like any other multilevel perception. 07:28.050 --> 07:30.600 So the magic of CNNs really happens at a lower level. 07:30.600 --> 07:31.890 You know, ultimately it gets converted 07:31.890 --> 07:33.330 into what looks like the same types 07:33.330 --> 07:35.880 of multilayer perceptrons that we've been using before, 07:35.880 --> 07:38.640 the magic happens and actually processing your data, 07:38.640 --> 07:40.740 involving it and reducing it down 07:40.740 --> 07:42.270 to something that's manageable. 07:42.270 --> 07:44.580 So typical usage of image processing 07:44.580 --> 07:45.900 with a CNN would look like this. 07:45.900 --> 07:48.300 You might start with a con 2D layer that does 07:48.300 --> 07:50.490 the actual convolution of your image data. 07:50.490 --> 07:52.740 You might follow that up with a MaxPooling2D layer 07:52.740 --> 07:55.800 on top of that, that distills that image down, 07:55.800 --> 07:58.170 just shrinks the amount of data that you have to deal with. 07:58.170 --> 08:00.300 You might then do a dropout layer on top of that, 08:00.300 --> 08:03.060 which just prevents overfit like we talked about before. 08:03.060 --> 08:05.220 And at that point you might apply a flattened layer 08:05.220 --> 08:08.610 to actually be able to feed that data into a perceptron 08:08.610 --> 08:10.470 and that's where a dense layer might come into play. 08:10.470 --> 08:14.681 So a dense layer in Keras is just a perceptron, really. 08:14.681 --> 08:16.410 You know, it's a layer of a hidden layer of neurons. 08:16.410 --> 08:18.060 From there, you might do another dropout pass 08:18.060 --> 08:21.240 to further prevent overfitting, and finally do a soft max 08:21.240 --> 08:24.150 to choose the final classification that comes out of 08:24.150 --> 08:25.150 your neural network. 08:27.390 --> 08:30.300 Now, like I said, CNNs are compute intensive. 08:30.300 --> 08:32.490 They are very heavy in your CPU, your GPU 08:32.490 --> 08:35.610 and your memory requirements, shuffling all that data around 08:35.610 --> 08:38.760 and convolving it adds up really, really fast 08:38.760 --> 08:39.780 and beyond that, there's a lot 08:39.780 --> 08:42.210 of what we call hyper parameters, a lot of different knobs 08:42.210 --> 08:44.580 and dials that you can adjust on CNNs. 08:44.580 --> 08:46.320 So in addition to the usual stuff you can tune 08:46.320 --> 08:48.240 like the topology of your neural network, 08:48.240 --> 08:51.000 or what optimizer you use, or what loss function you use, 08:51.000 --> 08:53.250 or what activation function you use, 08:53.250 --> 08:55.470 there's also choices to make about the kernel sizes. 08:55.470 --> 08:58.290 What is the area that you actually convolve across? 08:58.290 --> 08:59.340 How many layers do you have? 08:59.340 --> 09:00.660 How many units do you have? 09:00.660 --> 09:02.535 How much pooling do you do 09:02.535 --> 09:03.368 when you're reducing the image down? 09:03.368 --> 09:04.890 There's a lot of variance here. 09:04.890 --> 09:06.930 There's almost an infinite amount of possibilities here 09:06.930 --> 09:08.820 for configuring a CNN 09:08.820 --> 09:11.430 and often just obtaining the data to train your CNN 09:11.430 --> 09:13.110 with is the hardest part. 09:13.110 --> 09:15.540 So for example if you own a Tesla, 09:15.540 --> 09:17.760 that's actually taking pictures of the world around you 09:17.760 --> 09:19.470 and the road around you and all the street signs 09:19.470 --> 09:21.300 and traffic lights as you drive 09:21.300 --> 09:23.070 and every night it sends all those images 09:23.070 --> 09:25.050 back to some data server somewhere 09:25.050 --> 09:27.330 so Tesla can actually run training 09:27.330 --> 09:29.160 on its own neural networks based on that data. 09:29.160 --> 09:32.400 So if you slam on the brakes while you're driving a Tesla, 09:32.400 --> 09:35.190 at night, that information's gonna be fed into, you know, 09:35.190 --> 09:36.330 a big data center somewhere 09:36.330 --> 09:38.250 and Tesla's gonna crunch on that and say, Hey, 09:38.250 --> 09:40.590 is there a pattern here to be learned of what I saw 09:40.590 --> 09:42.180 from the cameras from the car, 09:42.180 --> 09:43.410 that means you should slam 09:43.410 --> 09:46.380 on the brakes in the case of a self-driving car 09:46.380 --> 09:48.240 and if you think about the scope of that problem, 09:48.240 --> 09:50.190 just the sheer magnitude of processing 09:50.190 --> 09:52.950 and obtaining and analyzing all that data, 09:52.950 --> 09:55.300 that becomes very challenging in and of itself. 09:57.000 --> 09:59.820 Now, fortunately the problem tuning the parameters 09:59.820 --> 10:01.830 doesn't have to be as hard as I described it to be. 10:01.830 --> 10:03.510 There are specialized architectures 10:03.510 --> 10:05.130 of convolutional neural networks that do 10:05.130 --> 10:06.240 some of that work for you. 10:06.240 --> 10:08.499 So the lot of research goes 10:08.499 --> 10:10.890 into trying to find the optimal topologies 10:10.890 --> 10:14.250 and parameters for a CNN for a given type of problem 10:14.250 --> 10:15.180 and you can just think of this 10:15.180 --> 10:16.950 as like a library you can draw from. 10:16.950 --> 10:18.180 So for example, 10:18.180 --> 10:20.340 there's the LeNet-5 architecture that you can use, 10:20.340 --> 10:23.460 that's suitable for handwriting recognition in particular. 10:23.460 --> 10:25.170 There's also one called AlexNet, 10:25.170 --> 10:27.330 which is appropriate for image classification. 10:27.330 --> 10:29.760 It's a deeper neural network than LeNet, you know, 10:29.760 --> 10:32.460 so in the example we talked about on the previous slides, 10:32.460 --> 10:34.170 we only had a single hidden layer 10:34.170 --> 10:36.390 but you can have as many as you want, really just a matter 10:36.390 --> 10:39.270 of how much computational power you have available. 10:39.270 --> 10:40.830 There's also something called GoogLeNet, 10:40.830 --> 10:42.720 you can probably guess who came up with that. 10:42.720 --> 10:45.030 It's even deeper, but it has better performance 10:45.030 --> 10:48.060 because it introduces this concept called inception modules. 10:48.060 --> 10:50.550 They basically group convolution layers together 10:50.550 --> 10:53.940 and that's a useful optimization for how it all works. 10:53.940 --> 10:57.720 Finally, the most sophisticated one today is called ResNet, 10:57.720 --> 10:59.520 that stands for residual network. 10:59.520 --> 11:01.200 It's an even deeper neural network 11:01.200 --> 11:02.760 but it maintains performance 11:02.760 --> 11:04.020 by what's called skip connections. 11:04.020 --> 11:06.960 So it has special connections between the layers 11:06.960 --> 11:09.480 of the perceptron to further accelerate things. 11:09.480 --> 11:11.516 So it's sort of like builds upon 11:11.516 --> 11:13.950 the fundamental architecture of a neural network 11:13.950 --> 11:17.040 to optimize its performance and as you'll see, 11:17.040 --> 11:21.120 CNNs can be very demanding on performance. 11:21.120 --> 11:22.320 So with that, let's give it a shot. 11:22.320 --> 11:25.320 Let's actually use a CNN and see if we can do a better job 11:25.320 --> 11:28.043 at image classification than we've done before using one.