WEBVTT 00:00.600 --> 00:04.300 All right we're going to talk about the Hadoop ecosystem at a very high level now and it's got to be 00:04.300 --> 00:08.470 a little bit overwhelming I've got to warn you there's a lot of technologies here but bear with me. 00:08.500 --> 00:13.300 You know all I really want you to do is get some exposure to these terms and these technologies so they 00:13.300 --> 00:16.630 don't seem as foreign when we dive into them in more depth. 00:16.840 --> 00:19.750 And you know this can be a very valuable lecture in and of itself. 00:19.750 --> 00:23.950 So if you watch it a couple of times it might be a good idea because this is all you really need to 00:23.950 --> 00:28.810 really understand what all these cryptic names in the Hadoop ecosystem really mean and what everything 00:28.810 --> 00:30.070 is for at a very high level. 00:30.070 --> 00:34.750 So let's dive in and actually uncover the secrets of Hadoop. 00:35.170 --> 00:35.430 All right. 00:35.430 --> 00:38.610 So let's go into some detail about the major components in Hadoop. 00:38.650 --> 00:43.480 I just want to kind of briefly touch on all these different technologies and we're gonna go into a lot 00:43.480 --> 00:47.860 more depth. That's what the rest of this course is all about just deep diving into each one of these and 00:48.250 --> 00:49.780 giving you some examples of using them. 00:49.780 --> 00:55.270 So I split things up here into three general areas here we have what I call the core Hadoop ecosystem 00:55.270 --> 00:58.860 which is just things built on the Hadoop platform directly. 00:58.990 --> 01:01.980 And then we have some ancillary systems that we'll talk about as well. 01:02.020 --> 01:06.740 There's a lot of different ways of organizing these systems and this is just what makes sense to me. 01:06.760 --> 01:11.500 There are a lot of complex interdependencies between these systems so there's really no right way 01:11.500 --> 01:12.910 to represent these relationships. 01:12.910 --> 01:17.590 But what I'm trying to do here is show you graphically the things that build on top of each other within 01:17.590 --> 01:18.410 Hadoop. 01:18.550 --> 01:21.440 Now the pink things here are the things that are part of Hadoop itself. 01:21.460 --> 01:25.720 Everything else is sort of add on projects that have come out over time that integrate with it you can 01:25.720 --> 01:27.330 solve specific problems. 01:27.580 --> 01:33.930 So let's start at the base of it all which is HDFS that stands for the Hadoop distributed file system. 01:33.970 --> 01:39.160 So remember we talked about GFS. HDFS is the Hadoop version of that and that is the system that 01:39.160 --> 01:45.010 allows us to distribute the storage of big data across our cluster of computers so it makes all of the 01:45.010 --> 01:48.980 hard drives on our cluster look like one giant file system. 01:49.300 --> 01:52.690 And not only that it actually maintains redundant copies of that data. 01:52.690 --> 01:58.100 So if one of your computers happens to randomly burst into flames and melts into a puddle of silicon 01:58.590 --> 02:04.090 it happens - it can actually recover from that and it will back itself up to a backup copy that it had 02:04.090 --> 02:05.620 of that data automatically. 02:05.680 --> 02:07.250 It's like you'll never even know anything happened. 02:07.300 --> 02:09.610 So that's the power of HDFS. 02:09.610 --> 02:16.120 That is the data storage the distributed data storage piece of Hadoop now sitting on top of HDFS we 02:16.120 --> 02:20.350 have YARN and that stands for yet another resource negotiator. 02:20.350 --> 02:23.180 So we talked about the data storage part of Hadoop. 02:23.230 --> 02:27.790 And there's also the data processing part of Hadoop. YARN is where the data processing starts to come 02:27.790 --> 02:28.360 into play. 02:28.360 --> 02:33.980 So yarn is basically the system that manages the resources on your computing cluster. 02:34.000 --> 02:40.360 It's what decides what gets to run tasks when what notes are available for extra work which notes are 02:40.390 --> 02:45.880 not which ones are available which ones are not available so it's kind of the the heartbeat that keeps 02:45.880 --> 02:47.890 your cluster going. 02:47.890 --> 02:53.380 Now given that we have this resource negotiator we can build interesting applications on top of that. 02:53.590 --> 02:59.860 And one of them is map reduce which again is a piece of Hadoop proper and map reduce at a very high 02:59.860 --> 03:07.840 level is just a programming metaphor or programming model that allows you to process your data across 03:07.840 --> 03:08.920 an entire cluster. 03:08.950 --> 03:10.320 And let's break that down. 03:10.330 --> 03:12.920 It consists of mappers and reducers. 03:12.940 --> 03:17.050 These are both different scripts that you might write or different functions if you will when you're 03:17.050 --> 03:22.660 writing a map reduce program. Mappers have the ability to transform your data in parallel across your 03:22.660 --> 03:25.240 entire computing cluster in a very efficient manner. 03:25.240 --> 03:30.340 And reducers are what aggregate that data together and it may sound like a very simple model and it 03:30.340 --> 03:35.140 is but it's actually very versatile and we'll see later on that there are some very creative ways you 03:35.140 --> 03:39.100 can put mappers and reducers together to solve very complex problems. 03:39.100 --> 03:42.940 Now originally map reduce and YARN were kind of the same thing in Hadoop. They got split out recently and 03:42.940 --> 03:48.390 that's enabled other applications to be built on top of yarn that solve the same problem as map reduce 03:48.400 --> 03:49.690 but in a more efficient manner. 03:49.720 --> 03:50.770 Talk about that a bit. 03:52.320 --> 03:55.490 And then sitting on top of mapreduce we have technologies such as pig. 03:55.770 --> 04:00.600 So if you don't want to write Java or python map reduce code and you're more familiar with a scripting 04:00.600 --> 04:07.710 language that has sort of a SQL style syntax Pig is for you so Pig is a very high level programming 04:07.830 --> 04:13.440 API that allows you to write simple scripts that look a lot like SQL in some cases that allow you 04:13.440 --> 04:19.500 to chain together queries and get complex answers but without actually writing Python or Java code 04:19.500 --> 04:24.750 in the process so pig will actually transform that script into something that will run on map reduce 04:25.080 --> 04:30.750 which in turn goes through yarn and HDFS to actually process and get the data that it needs to get the 04:30.750 --> 04:32.970 answer you want. That's pig. 04:33.210 --> 04:36.670 Just a high level scripting language that sits on top of map reduce. 04:37.410 --> 04:40.730 Let's zoom out a little bit here and a reminder of where everything fits together. 04:40.740 --> 04:47.400 Talk about hive next which also sits on top of map reduce and it solves a similar problem to pig but 04:47.400 --> 04:49.290 it really more directly 04:49.290 --> 04:56.550 looks like a SQL database so hive is a way of actually taking SQL queries and making this distributed 04:56.550 --> 05:00.470 data that's just really sitting on your file system somewhere look like a SQL database. 05:00.570 --> 05:03.510 So for all intents and purposes it's just like a database. 05:03.510 --> 05:08.590 You can even connect to it through a shell client or ODBC or what have you. 05:08.670 --> 05:13.410 And actually execute SQL queries on the data that's stored on your Hadoop cluster even though it's 05:13.410 --> 05:15.590 not really a relational database under the hood. 05:15.780 --> 05:17.560 So if you're familiar with SQL 05:17.970 --> 05:22.350 Hive might be a very useful API - useful interface for you to use. 05:22.640 --> 05:25.430 Zoom out a little bit here and again get our bearings here. 05:25.430 --> 05:32.370 We'll talk next about Ambari and Apache Ambari is basically this thing that sits on top of everything 05:32.500 --> 05:37.020 and it just gives you a view of your cluster and lets you visualize what's running on your cluster What 05:37.020 --> 05:41.910 systems are using how much resources and also has some views in it that allow you to actually do things 05:41.910 --> 05:48.060 like execute hive queries or import databases into hive or execute Pig queries and things like that. 05:48.060 --> 05:54.510 So Ambari is what sits on top of all this and lets you have a view into the actual state of your cluster 05:54.510 --> 05:56.480 and the applications that are running on it. 05:56.490 --> 06:00.980 Now there are other technologies that do this for you and Ambari is what Hortonworks uses. 06:00.990 --> 06:04.470 There are competing distributions of Hadoop stacks out there, 06:04.590 --> 06:09.270 Hortonworks being one of them. Other ones include Cloudera and MapR but for Hortonworks they use 06:09.270 --> 06:12.070 Anbari. 06:12.060 --> 06:14.920 So let's go over here. 06:15.160 --> 06:21.100 Mesos so Mesos isn't really part of Hadoop proper but I'm including it here because it's basically 06:21.100 --> 06:22.580 an alternative to yarn. 06:22.600 --> 06:28.300 So it too is a resource negotiator remember YARN is yet another resource negotiator Mesos is another 06:28.300 --> 06:33.400 one they basically solve the same problems in different ways there are of course pros and cons to using 06:33.400 --> 06:34.870 each one that we'll talk about later on. 06:34.870 --> 06:39.140 But Mesos is another potential way of managing the resources on your cluster. 06:39.160 --> 06:42.310 And there are ways of getting Mesos and YARN to work together if you need to as well. 06:43.590 --> 06:47.970 And we bring up mesos because we're gonna talk about Spark which I think is one of the most exciting 06:47.970 --> 06:53.550 technologies in the Hadoop ecosystem this is sitting at the same level of map reduce in that it sits 06:53.550 --> 07:00.560 on top of yarn or Mesos it can go either way to actually run queries on your data and like MapReduce it 07:00.570 --> 07:05.490 requires some programming and need to actually write your SPARK scripts using either Python or Java 07:05.490 --> 07:08.460 or the Scala programming language Scala being preferred. 07:08.580 --> 07:14.310 But SPARK is kind of where it's at right now it is extremely fast it's under a lot of active development 07:14.310 --> 07:19.210 right now so Spark's a very exciting technology right now and a very powerful technology. 07:19.500 --> 07:26.460 So if you need to very quickly and efficiently and reliably process data on your cluster SPARK is a 07:26.460 --> 07:27.610 really good choice for that. 07:27.630 --> 07:32.850 And it's also very versatile it can do things like handle SQL queries that can do machine learning 07:32.850 --> 07:35.850 across an entire cluster of information. 07:36.000 --> 07:40.260 It can actually handle streaming data in real time and all sorts of other cool stuff. 07:40.260 --> 07:43.620 So I'm very excited teaching more about SPARK later in this course. 07:45.720 --> 07:51.460 Moving on Tez similar to spark in that it also uses some of the same techniques as SPARK notably with 07:51.460 --> 07:57.210 something that's called a directed acyclic graph and this gives Tez a leg up on what map reduce does 07:57.220 --> 08:04.230 because it can produce more optimal plans for actually executing queries. Tez is usually used in conjunction 08:04.230 --> 08:05.690 with Hive to accelerate it. 08:05.910 --> 08:11.400 So we remember we looked at hive earlier that kind of sat on top of map reduce but it can also sit on 08:11.400 --> 08:12.430 top of Tez. 08:12.570 --> 08:17.820 So you have an option there high through Tez can often be faster than high through map reduce. 08:17.820 --> 08:23.220 Both different means of optimizing queries to get a efficient answer from your cluster. 08:24.530 --> 08:27.150 We talk about HBase. 08:27.500 --> 08:32.930 So HBase kind of sits off to the side and it's a way of exposing the data on your cluster to transactional 08:32.930 --> 08:36.470 platforms so HBase is what we call a NoSQL database. 08:36.470 --> 08:41.450 It is a columnar data store and you might have heard that term before it's basically a really really 08:41.450 --> 08:48.110 fast database meant for very large transaction rates so it's appropriate for example for hitting from 08:48.110 --> 08:54.620 a web application hitting from a Web site doing all types of transactions so HBase can actually expose 08:54.620 --> 08:59.750 the data that's stored on your cluster and maybe that data was transformed in some way by spark or map 08:59.750 --> 09:00.980 reduce or something else. 09:01.250 --> 09:05.690 And it provides a very fast way of exposing those results to other systems. 09:07.220 --> 09:12.060 And what else can we talk about let's go over here and talk about Apache's storm. 09:12.470 --> 09:15.650 Storm is basically a way of processing streaming data. 09:15.680 --> 09:21.560 So if you have streaming data from say sensors or web logs you can actually process that in real time 09:21.560 --> 09:26.390 using storm and spark streaming solves the same problem. 09:26.390 --> 09:28.070 Storm just does it in a slightly different way. 09:28.070 --> 09:34.610 So Apache storm's made for processing streaming data quickly in real time so it doesn't have to be a batch 09:34.610 --> 09:35.570 thing anymore. 09:35.750 --> 09:41.120 You can actually update your machine learning models or transform data into a database all in real time 09:41.150 --> 09:42.760 as it comes in. 09:42.800 --> 09:43.780 Pretty cool stuff. 09:44.820 --> 09:51.690 Let's go over here to Oozie. Oozie is just a way of scheduling jobs on your cluster. 09:51.690 --> 09:56.640 So if you have a task that needs to happen on your Hadoop cluster that involves many different steps 09:56.640 --> 09:58.350 and maybe many different systems. 09:58.580 --> 10:04.290 Oozie's a way of scheduling all of these things together into jobs that can be run on some sort of schedule. 10:04.320 --> 10:09.300 So when you have more complicated operations that require loading data into hive and then integrating 10:09.300 --> 10:16.380 that with Pig and maybe querying it with SPARK and then transforming the results into HBase Oozie can manage 10:16.380 --> 10:21.790 that all for you and make sure that it runs reliably on a consistent basis. 10:21.870 --> 10:26.250 Moving over here a bit zookeeper also sits alongside all of these technologies. 10:26.250 --> 10:29.940 It's basically a technology for coordinating everything on your cluster. 10:29.940 --> 10:35.130 So it's it's the technology that can be used for keeping track of which nodes are up which nodes are 10:35.130 --> 10:41.790 down. It's a very reliable way of just kind of keeping track of shared states across your cluster that 10:41.790 --> 10:47.130 different applications can use and many many of these applications we've talked about rely on zookeeper 10:47.430 --> 10:51.650 to actually maintain reliable and consistent performance across the cluster. 10:51.660 --> 10:53.770 Even when a node randomly goes down. 10:53.850 --> 10:59.200 So zookeeper can be used for example for keeping track of who the current master node is or keeping 10:59.200 --> 11:01.190 track of who's up who's down what have you. 11:01.190 --> 11:04.830 And it's really more more extensible than that even. 11:04.830 --> 11:07.020 But we'll talk about that later. 11:07.200 --> 11:11.260 Over here there's also systems that are just focused on the problem of data ingestion. 11:11.280 --> 11:16.270 So how do we actually get data into your cluster and onto HDFS from external sources. 11:16.500 --> 11:22.740 Sqoop for example is a way of actually tying your Hadoop database into a relational database. 11:22.740 --> 11:29.970 Anything that can talk to ODBC or JDBC can be transformed by Sqoop into your 11:30.030 --> 11:37.150 HDFS file system so Sqoop is basically a connector between Hadoop and your legacy databases. Flume. 11:37.270 --> 11:44.160 It's a way of actually transporting Web logs at a very large scale and very reliably to your cluster. 11:44.160 --> 11:48.960 So let's say you have a fleet of web servers Flume can actually listen to the web logs coming in from 11:48.960 --> 11:55.020 those web servers in real time and publish them into your cluster in real time for processing by something 11:55.020 --> 11:56.940 like storm or spark streaming. 11:57.090 --> 12:00.840 Kafka solves a similar problem although it's a little bit more general purpose. 12:00.840 --> 12:07.140 It can basically collect data of any sort from a cluster of PCs from a cluster of web servers or whatever 12:07.140 --> 12:09.930 it is and broadcast that into your Hadoop cluster as well. 12:09.930 --> 12:13.840 So those are all three technologies that solve the problem of data ingestion. 12:15.590 --> 12:16.050 All right. 12:16.050 --> 12:18.480 Moving on now. 12:18.690 --> 12:21.780 Your data might be exposed or stored in other places too. 12:21.840 --> 12:23.780 So let's talk about those as well. 12:23.810 --> 12:25.910 HBase would also fit into this category. 12:25.920 --> 12:29.340 But since HBase is really part of the Hadoop stack itself. 12:29.340 --> 12:32.040 I left it off of this little collection here. 12:32.040 --> 12:36.430 MySQL of course or any SQL database is something you might be integrating with your cluster or 12:36.450 --> 12:41.370 you can not only import data from Sqoop into your cluster You can also export it to MySQL as well 12:41.370 --> 12:49.080 so a lot of these technologies like Spark have the ability to write to any JDBC or ODBC database and 12:49.080 --> 12:52.880 you can store and retrieve your results from a central database. 12:52.890 --> 12:54.410 If you're so inclined. 12:54.480 --> 12:59.880 Cassandra like HBase and also MongoDB are both also columnar data stores and there are also good choices 12:59.880 --> 13:05.010 for exposing your data for real time usage to say a web application. 13:05.010 --> 13:10.890 So you definitely want some sort of layer like this like Cassandra or MongoDB sitting between real time 13:10.890 --> 13:12.770 applications and or cluster. 13:12.810 --> 13:14.190 We'll talk about those and a lot more depth. 13:14.190 --> 13:17.290 Both are very popular choices for vending 13:17.670 --> 13:22.570 simple key value data store is at very large transaction rates. 13:22.590 --> 13:30.710 MySQL Cassandra MongoDB all external databases that might integrate with your cluster and there 13:30.710 --> 13:33.950 are also several query engines that sit on top of your head you cluster. 13:33.950 --> 13:39.890 So if you want to actually interactively enter SQL queries or whatever you can do that using these 13:39.890 --> 13:40.870 technologies again. 13:40.880 --> 13:44.460 Things don't always fit neatly into different circles here. 13:44.580 --> 13:47.440 If you remember Hive that actually is a similar thing as well. 13:47.450 --> 13:52.130 But again since Hive is more tightly integrated into Hadoop I chose to leave it out of this particular 13:52.130 --> 13:55.470 circle but it too is a way of querying your data. 13:56.000 --> 13:56.780 Apache Drill. 13:56.780 --> 13:57.560 Pretty cool stuff. 13:57.560 --> 14:03.380 It actually allows you to write SQL queries that will work across a wide range of NoSQL databases 14:03.410 --> 14:09.150 potentially so they can actually talk to your HBase database and maybe your Cassandra and your MongoDB 14:09.160 --> 14:14.570 database as well and tie those results all together and allow you to write queries across all those 14:14.570 --> 14:18.750 disparate data stores and bring them all back together when you're done. 14:18.750 --> 14:19.480 Hue. 14:19.610 --> 14:24.670 Also a way of interactively creating queries that works well with hive and hbase. 14:24.950 --> 14:30.410 Actually for Cloudera it kind of takes the role of Ambari as sort of the thing that sits on top of 14:30.410 --> 14:36.180 everything and lets you visualize and execute queries on the Hadoop cluster as a whole. 14:37.300 --> 14:43.210 Apache Phoenix kind of similar to drill it lets you do SQL style queries across the entire range 14:43.210 --> 14:48.850 of data storage technologies you might have but it takes it one step further it actually gives you ACID 14:48.850 --> 14:50.370 guarantees and OLTP. 14:50.410 --> 15:00.040 So you can actually make your not SQL Hadoop data store look a lot like a relational data store 15:00.040 --> 15:03.350 in a relational database with all the guarantees that come with that. 15:04.030 --> 15:08.720 And finally presto yet another way to execute queries across your entire cluster. 15:08.930 --> 15:11.080 These all solve you know kind of the same problem. 15:11.080 --> 15:16.690 Zeppelin is just another angle on it that takes more of a notebook type approach to the UI and how you 15:16.690 --> 15:18.910 actually interact with the cluster. 15:19.300 --> 15:24.100 But at the end of the day those are all ways of actually executing queries and extracting meaning from 15:24.670 --> 15:28.300 your cluster without necessarily writing programs to do it. 15:28.300 --> 15:31.240 So thats the world of Hadoop in a nutshell. 15:31.500 --> 15:36.430 A lot of stuff there but hey that's all there is to it. We're gonna spend the rest of this course diving into each one 15:36.430 --> 15:36.730 of these. 15:36.730 --> 15:41.770 One at a the time and doing some actual exercises and activities with them so you get to understand them 15:41.770 --> 15:42.970 more in more depth. 15:42.970 --> 15:48.340 But for now those are the buzzwords and hopefully when you hear these terms and these technologies they 15:48.340 --> 15:50.500 dont seem quite so foreign to you anymore. 15:50.800 --> 15:52.490 So welcome to the world of Hadoop. 15:52.510 --> 15:56.700 Let's move on and dive into some more detail. 15:56.700 --> 16:00.780 All right that was a lot of stuff right there and you know I understand you to go back and watch it 16:00.780 --> 16:01.150 again. 16:01.170 --> 16:04.650 There's a lot of information there but don't worry we're going to dive into each and every one of these 16:04.650 --> 16:07.320 technologies in a lot more depth later on in the course. 16:07.320 --> 16:12.060 I just wanted to kind of hit you with all this right up front so you could know just enough to be dangerous 16:12.480 --> 16:16.410 and know enough about these technologies to actually know what people are talking about when they're 16:16.410 --> 16:17.480 talking about Hadoop. 16:17.490 --> 16:19.740 I mean really that's half of the battle here. 16:19.740 --> 16:21.690 None of these things are really that complicated. 16:21.720 --> 16:26.550 To be honest you know all of these things are actually fairly easy to use and understand Hadoop is only 16:26.550 --> 16:31.230 really hard to grasp because there are so many different technologies and a lot of them do the same 16:31.230 --> 16:31.840 thing. 16:32.100 --> 16:35.850 But if you can understand what they are all for and how to choose between them and how to put them together 16:35.850 --> 16:37.760 to actually solve real business problems. 16:37.980 --> 16:40.250 Well that's what they pay people the big bucks for. 16:40.320 --> 16:43.710 So let's move on and actually start to dive into some more depth. 16:43.710 --> 16:46.030 Let's start with HDFS and see how that works.