WEBVTT 00:00.000 --> 00:02.580 Announcer: So a big part of feature engineering 00:02.580 --> 00:04.650 is imputation of missing data. 00:04.650 --> 00:05.580 What do you do when your data 00:05.580 --> 00:07.906 has missing data elements in it? 00:07.906 --> 00:09.660 This is what happens in the real world. 00:09.660 --> 00:11.070 For every observation you have, 00:11.070 --> 00:12.780 there's going to be some missing data points, 00:12.780 --> 00:14.130 more than likely. 00:14.130 --> 00:17.100 Well, a simple solution is just called mean replacement. 00:17.100 --> 00:20.010 The idea is that if you have a missing attributor, 00:20.010 --> 00:22.530 or feature within one of the rows of your data 00:22.530 --> 00:25.350 just replace it with the mean from the entire column. 00:25.350 --> 00:27.840 And remember, we're talking about columns, not rows here. 00:27.840 --> 00:29.310 You want to take the mean 00:29.310 --> 00:32.190 of all the other observations of that same feature. 00:32.190 --> 00:33.480 Doesn't really make sense to take the mean 00:33.480 --> 00:35.490 of all the other features for that row, right? 00:35.490 --> 00:37.410 So mean replacement is all about 00:37.410 --> 00:39.090 taking the mean of that column 00:39.090 --> 00:42.120 and replacing all the empty values with that mean. 00:42.120 --> 00:43.680 So it's fast and it's easy. 00:43.680 --> 00:46.050 Those are some of the positives of this approach. 00:46.050 --> 00:47.430 It also doesn't affect the mean, 00:47.430 --> 00:49.410 or the sample size of your overall data set, 00:49.410 --> 00:51.990 because you're just replacing missing data with the mean. 00:51.990 --> 00:53.190 It won't affect the overall mean 00:53.190 --> 00:55.830 of the entire data set, which can be nice. 00:55.830 --> 00:58.860 Now, one nuance is that if you have a lot of outliers 00:58.860 --> 00:59.693 in your dataset, 00:59.693 --> 01:00.930 which is also something you have to deal with 01:00.930 --> 01:02.280 when preparing your data, 01:02.280 --> 01:04.410 you might find that median is actually a better choice 01:04.410 --> 01:05.340 than the mean. 01:05.340 --> 01:07.290 So if you have a data set of a bunch of people 01:07.290 --> 01:09.450 and maybe one of those columns is income 01:09.450 --> 01:11.160 and some people don't report their income 01:11.160 --> 01:14.100 because they think it's sensitive, you know, 01:14.100 --> 01:15.600 you might have your mean skewed 01:15.600 --> 01:17.910 by a bunch of millionaires or billionaires in your dataset. 01:17.910 --> 01:19.710 So, if you do mean imputation 01:19.710 --> 01:22.380 and that's sort of a situation where you have outliers 01:22.380 --> 01:25.950 you might end up with an overly high or overly low value 01:25.950 --> 01:27.150 that you're using for replacement. 01:27.150 --> 01:29.760 So if you do have outliers that are skewing your mean 01:29.760 --> 01:31.800 you might wanna think about using median instead. 01:31.800 --> 01:34.800 That will be less sensitive to those outliers. 01:34.800 --> 01:35.820 But generally speaking 01:35.820 --> 01:39.120 it's not the best choice for imputation. 01:39.120 --> 01:41.460 First of all, it only works on the column level. 01:41.460 --> 01:44.010 So if there are correlations between other features 01:44.010 --> 01:44.843 in your data set, 01:44.843 --> 01:46.830 it's not going to pick up on those. 01:46.830 --> 01:48.930 So, you know, if there is a relationship between, 01:48.930 --> 01:50.550 say age and income, 01:50.550 --> 01:52.680 that relationship is gonna be totally missed. 01:52.680 --> 01:54.540 So you couldn't end up saying that, 01:54.540 --> 01:56.547 a 10 year old is making, you know, 01:56.547 --> 02:00.000 $50,000 a year because that's the mean of your data set. 02:00.000 --> 02:01.770 But it really doesn't make sense, right? 02:01.770 --> 02:04.230 I mean a 15 year old wouldn't be making that much money yet. 02:04.230 --> 02:06.990 So it's a very naive approach from that standpoint. 02:06.990 --> 02:08.130 The other issue is that 02:08.130 --> 02:11.640 you can't really use it on categorical features. 02:11.640 --> 02:12.780 You know, how do you take the mean 02:12.780 --> 02:14.730 of a categorical piece of data, 02:14.730 --> 02:16.440 that just doesn't make sense, right? 02:16.440 --> 02:19.830 Although you could use the most frequent value that appears 02:19.830 --> 02:22.080 you know, the most commonly seen category 02:22.080 --> 02:23.790 would be a reasonable thing to do in that case, 02:23.790 --> 02:25.766 it's sort of in the same spirit 02:25.766 --> 02:26.599 as mean replacement, 02:26.599 --> 02:27.600 but not really the same thing. 02:27.600 --> 02:30.900 Overall though, it's not going to be a very accurate method. 02:30.900 --> 02:33.990 It's a very ham-handed attempt at doing imputation. 02:33.990 --> 02:36.000 So, although it's quick and easy 02:36.000 --> 02:38.190 and has some advantages. 02:38.190 --> 02:40.170 In practice if someone's asking you say, 02:40.170 --> 02:41.670 on a certification exam, 02:41.670 --> 02:44.130 what's the best way to do data amputation? 02:44.130 --> 02:46.263 Mean replacement probably, isn't it? 02:48.450 --> 02:50.940 It's also probably not just dropping the missing rows. 02:50.940 --> 02:52.800 Although as we've seen sometimes 02:52.800 --> 02:54.990 that's a reasonable thing to do 02:54.990 --> 02:56.970 if you do have enough data such 02:56.970 --> 02:58.830 that dropping a few rows doesn't matter. 02:58.830 --> 03:00.090 You know, if you don't have too many rows 03:00.090 --> 03:01.500 that contain missing data, 03:01.500 --> 03:03.661 well it doesn't sound that unreasonable. 03:03.661 --> 03:06.090 The other thing too is that you wanna make sure 03:06.090 --> 03:08.250 that dropping the rosette have missing data 03:08.250 --> 03:11.460 doesn't bias your data set in some way. 03:11.460 --> 03:13.140 What if there's an actual relationship 03:13.140 --> 03:15.150 between which rows are missing data 03:15.150 --> 03:18.150 and some other attribute of those observations? 03:18.150 --> 03:21.750 For example, let's say that we're looking at income again, 03:21.750 --> 03:23.340 there might be a situation 03:23.340 --> 03:25.620 where people that have very high or very low incomes 03:25.620 --> 03:27.600 are more likely to not report it. 03:27.600 --> 03:30.630 So by removing or dropping all of those observations 03:30.630 --> 03:32.190 you're actually removing a lot of people 03:32.190 --> 03:34.980 that have very high or low incomes from your model. 03:34.980 --> 03:36.390 And that might have a very bad effect 03:36.390 --> 03:38.760 on the accuracy of the model you end up with. 03:38.760 --> 03:41.550 So you wanna make sure that if you are going to drop data 03:41.550 --> 03:43.320 that it's not going to bias the data set 03:43.320 --> 03:46.200 in some way as as a byproduct, right? 03:46.200 --> 03:47.910 So it's a very quick and easy thing to do. 03:47.910 --> 03:49.530 Probably the quickest and easiest thing to do. 03:49.530 --> 03:52.680 You can literally do this in one line of code in Python, 03:52.680 --> 03:55.830 but it's probably never going to be the best approach. 03:55.830 --> 03:58.050 So again, if an exam is asking you, 03:58.050 --> 04:00.540 what's the best way to impute missing data? 04:00.540 --> 04:03.000 Dropping data probably is not the right answer. 04:03.000 --> 04:05.070 Almost anything is going to be better. 04:05.070 --> 04:07.410 Maybe you could just substitute a similar field, right? 04:07.410 --> 04:09.930 I mean that would also be a simple way of doing it. 04:09.930 --> 04:12.660 For example, I might have a data set 04:12.660 --> 04:15.780 of customer reviews on movies, right? 04:15.780 --> 04:17.250 Maybe if I have a review summary 04:17.250 --> 04:19.050 and a full text review as well, 04:19.050 --> 04:21.120 it would make more sense to just take the review summary 04:21.120 --> 04:22.560 and copy that into the full text 04:22.560 --> 04:25.770 for people who left the full text blank as an example. 04:25.770 --> 04:28.110 So almost anything is better than just dropping data. 04:28.110 --> 04:31.080 But you know, in the real world 04:31.080 --> 04:32.970 if you're just trying to do something quick and dirty 04:32.970 --> 04:34.740 and sort of like start experimenting 04:34.740 --> 04:36.540 with some data just to start playing with it, 04:36.540 --> 04:38.190 it can be a reasonable thing to do. 04:38.190 --> 04:39.330 I just wouldn't leave that 04:39.330 --> 04:41.130 in place for production necessarily. 04:42.750 --> 04:45.810 The thing you probably really want to do in production is 04:45.810 --> 04:47.490 by using machine learning itself 04:47.490 --> 04:48.870 to impute your missing data 04:48.870 --> 04:50.190 into your machine learning training. 04:50.190 --> 04:52.650 So, it's a kind of a meta thing. 04:52.650 --> 04:54.330 There are different ways of doing this. 04:54.330 --> 04:57.570 One is called KNN that stands for K-Nearest Neighbors. 04:57.570 --> 04:59.340 And if you have any experience with machine learning 04:59.340 --> 05:01.080 you probably know what that is already. 05:01.080 --> 05:03.360 The general idea is to find the K, 05:03.360 --> 05:04.410 where K is some number 05:04.410 --> 05:05.820 of the most similar rows 05:05.820 --> 05:06.990 to the ones that you're looking at 05:06.990 --> 05:09.510 that has missing data and just average together 05:09.510 --> 05:11.970 the values from those most similar rows. 05:11.970 --> 05:13.320 So you can imagine having some sort of 05:13.320 --> 05:14.940 a distance metric between each row. 05:14.940 --> 05:17.520 Maybe it's just the euclidean distance between 05:17.520 --> 05:19.290 the normalized features within each row, 05:19.290 --> 05:20.640 or something like that. 05:20.640 --> 05:23.760 And if you find the, say 10 nearest rows 05:23.760 --> 05:25.650 that are most similar to the one that's missing data, 05:25.650 --> 05:27.720 you can just take the average of that feature 05:27.720 --> 05:29.460 from those 10 most similar rows 05:29.460 --> 05:31.770 and impute the value from that. 05:31.770 --> 05:33.810 So that takes advantage of relationships between 05:33.810 --> 05:35.400 the other features of your data set, 05:35.400 --> 05:37.290 which is a good thing. 05:37.290 --> 05:38.670 One problem with it though is that, 05:38.670 --> 05:39.630 that idea assumes 05:39.630 --> 05:41.670 that you have numerical data that you're trying to impute 05:41.670 --> 05:42.900 and not categorical data. 05:42.900 --> 05:45.030 It's tough to take the average of a category 05:45.030 --> 05:47.910 but there are ways of doing that with techniques 05:47.910 --> 05:49.560 like Hamming distance. 05:49.560 --> 05:51.810 But KNN is generally a better fit 05:51.810 --> 05:54.060 for numerical data not categorical. 05:54.060 --> 05:55.470 If you have categorical data, 05:55.470 --> 05:56.910 you're probably better served by 05:56.910 --> 05:59.430 actually developing a deep learning model. 05:59.430 --> 06:03.150 Neural networks are great at doing categorization problems. 06:03.150 --> 06:04.350 So the idea to be 06:04.350 --> 06:06.150 to actually build a machine learning model 06:06.150 --> 06:08.940 to impute the data for your machine learning model, right? 06:08.940 --> 06:11.696 It's a kind of a cycle there, 06:11.696 --> 06:13.950 and that works really well for categorical data. 06:13.950 --> 06:14.970 It really well. 06:14.970 --> 06:17.460 It's tough to be deep learning these days, 06:17.460 --> 06:18.990 however of course it is complicated. 06:18.990 --> 06:21.940 There's a lot of code involved and a lot of tuning as well. 06:22.890 --> 06:24.290 But it's hard to beat the results 06:24.290 --> 06:26.250 if you actually have a deep learning model 06:26.250 --> 06:28.980 that tries to predict what a missing feature is 06:28.980 --> 06:31.350 based on other features in your data set. 06:31.350 --> 06:32.700 That's gonna work out, 06:32.700 --> 06:33.533 a lot of work, 06:33.533 --> 06:34.890 a lot of computational effort, 06:34.890 --> 06:37.020 but it's going to give you the best results. 06:37.020 --> 06:39.030 You can also just do a multiple regression 06:39.030 --> 06:41.130 on the other features that are in your data set. 06:41.130 --> 06:43.320 That's also a totally reasonable thing to do. 06:43.320 --> 06:45.240 And through regressions you can find linear, 06:45.240 --> 06:47.940 or non-linear relationships between your missing feature 06:47.940 --> 06:50.160 and the other features that are in your data set. 06:50.160 --> 06:52.200 And there is a very advanced technique 06:52.200 --> 06:53.700 along these lines called MICE, 06:53.700 --> 06:56.970 which stands for, Multiple Imputation by Chained Equations. 06:56.970 --> 06:57.900 It's kind of the state of the art 06:57.900 --> 07:00.120 in this space right now for impeding missing data. 07:00.120 --> 07:01.440 So, alright. 07:01.440 --> 07:03.690 And finally, probably the best way to deal 07:03.690 --> 07:05.580 with missing data is to just get more data. 07:05.580 --> 07:07.920 So if you have a bunch of rows that have missing data 07:07.920 --> 07:09.570 maybe you just have to try harder to get more 07:09.570 --> 07:10.770 complete data from people. 07:10.770 --> 07:13.860 And it's hard to be just getting more real data, 07:13.860 --> 07:15.900 so that you can just not have to worry 07:15.900 --> 07:17.730 about all the rows that have missing data. 07:17.730 --> 07:20.280 Again, you wanna be careful that if you are dropping data 07:20.280 --> 07:22.140 that you're not biasing your data set in some way. 07:22.140 --> 07:24.510 But really the best way to deal 07:24.510 --> 07:26.880 with not having enough data is to just get more of it. 07:26.880 --> 07:28.260 Sometimes you just have to go back 07:28.260 --> 07:30.120 and figure out where that data came from 07:30.120 --> 07:32.460 and collect more better quality data. 07:32.460 --> 07:35.190 So the better quality data you have going into your system 07:35.190 --> 07:37.080 the better the results you will get. 07:37.080 --> 07:40.260 And while imputation techniques are a way to cover up issues 07:40.260 --> 07:41.790 where you just don't have enough data 07:41.790 --> 07:43.380 and you can't get more of it, 07:43.380 --> 07:44.970 it's always a good idea to just get more 07:44.970 --> 07:46.323 and better data if you can.