WEBVTT 00:01.710 --> 00:03.660 -: So in this section we'll introduce GraphX, 00:03.660 --> 00:04.493 which is an API 00:04.493 --> 00:06.300 for dealing with graphs of information 00:06.300 --> 00:07.920 kind of like our network of superheroes 00:07.920 --> 00:09.300 earlier in the course. 00:09.300 --> 00:12.870 GraphX is kind of a neglected stepchild of Spark, 00:12.870 --> 00:13.703 to be honest. 00:13.703 --> 00:15.150 It hasn't seen a whole lot of development lately. 00:15.150 --> 00:17.941 And it's still stuck on the old RDD API. 00:17.941 --> 00:19.290 There is a newer version 00:19.290 --> 00:21.390 of it in development called GraphFrames 00:21.390 --> 00:23.310 but it's not ready for primetime yet. 00:23.310 --> 00:25.230 So for now, just for completeness 00:25.230 --> 00:27.900 I'm going to talk about GraphX in its current form, 00:27.900 --> 00:29.460 which does use the RDD interface 00:29.460 --> 00:31.350 but you'll see it's still quite powerful. 00:31.350 --> 00:34.050 And using it does feel a little bit more like SQL at times. 00:34.050 --> 00:35.220 So let's dive in 00:35.220 --> 00:37.710 and see how GraphX can help you solve massive 00:37.710 --> 00:40.263 distributed problems using graphs of information. 00:41.400 --> 00:43.410 Let's dive briefly into the world of GraphX, 00:43.410 --> 00:45.990 the last core component of Spark itself. 00:45.990 --> 00:48.180 And when we talk about GraphX, we're not talking 00:48.180 --> 00:50.881 about line graphs or charts or anything like that. 00:50.881 --> 00:53.490 We're talking about graphs in the computer science sense. 00:53.490 --> 00:55.530 So for example, our social network 00:55.530 --> 00:57.480 of superheroes that we saw earlier in the course, 00:57.480 --> 01:00.150 that's an example of the graph we're talking about 01:00.150 --> 01:01.650 where we have vertices 01:01.650 --> 01:04.350 that in that case represent individual superheroes 01:04.350 --> 01:06.570 and edges between those vertices that represent 01:06.570 --> 01:08.504 relationships between them. 01:08.504 --> 01:10.200 GraphX is kind of cool 01:10.200 --> 01:12.720 but it's really only useful for some specific things. 01:12.720 --> 01:13.860 So by itself 01:13.860 --> 01:15.810 it can't actually answer the questions that we 01:15.810 --> 01:16.860 were answering in the code 01:16.860 --> 01:19.560 that we wrote for analyzing our superhero network. 01:19.560 --> 01:22.110 But it can do things like measuring connectedness, 01:22.110 --> 01:25.110 degree distribution, average path length, triangle counts, 01:25.110 --> 01:27.990 sort of these high level measures of the graph as a whole. 01:27.990 --> 01:28.950 It can do things like count 01:28.950 --> 01:30.360 up all the triangles in the graph 01:30.360 --> 01:32.760 and apply the PageRank algorithm to them even. 01:32.760 --> 01:34.920 So I think that's really the driving force 01:34.920 --> 01:36.570 behind GraphX itself. 01:36.570 --> 01:38.910 You know, it's most useful for implementing something 01:38.910 --> 01:41.395 like PageRank, which is obviously an important use case. 01:41.395 --> 01:43.980 And that's also a problem that involves massive scale, 01:43.980 --> 01:46.470 of course, where the power of Spark comes in handy. 01:46.470 --> 01:48.980 So it's kind of made for that more than anything else. 01:48.980 --> 01:51.450 You can also do things like joining graphs together 01:51.450 --> 01:53.340 and transforming those graphs very quickly 01:53.340 --> 01:55.020 in a distributed manner. 01:55.020 --> 01:57.450 But for things like our degrees of separation example 01:57.450 --> 01:58.350 where we're trying to figure out 01:58.350 --> 01:59.850 how many degrees of separation 01:59.850 --> 02:02.610 a superhero is from Spider-Man, you're not gonna find 02:02.610 --> 02:05.100 built in support for operations like that. 02:05.100 --> 02:07.380 However, it does support the Pregel API 02:07.380 --> 02:08.880 for traversing a graph, 02:08.880 --> 02:11.010 and that allows you to write your own code 02:11.010 --> 02:13.200 and develop your own algorithms that sort of live 02:13.200 --> 02:14.490 within GraphX. 02:14.490 --> 02:16.020 And that gives you the flexibility to 02:16.020 --> 02:18.630 do those more complicated things that you might dream up. 02:18.630 --> 02:19.770 They just don't come out of the box. 02:19.770 --> 02:21.780 You have to think through it and think creatively 02:21.780 --> 02:24.210 as we had to do when we did this using DataFrames 02:24.210 --> 02:25.923 or using RDDs. 02:27.000 --> 02:29.490 So GraphX introduces a couple of new data types, 02:29.490 --> 02:32.100 the VertexRDD and the EdgeRDD, 02:32.100 --> 02:34.140 as well as the Edge data type. 02:34.140 --> 02:36.240 And that's how we represent, you know, the vertices 02:36.240 --> 02:38.670 and the edges between them that make up a graph. 02:38.670 --> 02:40.640 Now GraphX is a little bit of a holdout. 02:40.640 --> 02:44.610 It's still written on the RDD API, even though Spark itself 02:44.610 --> 02:46.770 has been trying to migrate more and more toward DataFrames 02:46.770 --> 02:48.180 and data sets. 02:48.180 --> 02:51.630 Quite honestly, GraphX has kind of fallen by the wayside. 02:51.630 --> 02:54.270 It hasn't really seen a lot of active development yet. 02:54.270 --> 02:56.520 It's still a core piece of Spark itself 02:56.520 --> 02:58.710 so I'm covering it here, but you're seeing it 02:58.710 --> 03:00.660 more and and more being replaced by other things 03:00.660 --> 03:02.550 or just not being used much at all. 03:02.550 --> 03:04.170 The real world, it turns out, 03:04.170 --> 03:06.030 doesn't really have that much use for GraphX 03:06.030 --> 03:08.940 and as a result it's kind of been neglected in all honesty. 03:08.940 --> 03:11.160 So GraphX is still built on RDDs, 03:11.160 --> 03:12.930 yet another reason to learn RDDs. 03:12.930 --> 03:16.200 There is an alternative package called GraphFrames 03:16.200 --> 03:18.630 that is built on the new DataFrame API, 03:18.630 --> 03:20.610 but it's not really released yet. 03:20.610 --> 03:23.280 It's at version like 0.8 last time I checked. 03:23.280 --> 03:27.057 So at some point we might see GraphFrames replace GraphX 03:27.057 --> 03:30.510 and Spark, but for now we have the RDD based GraphX 03:30.510 --> 03:31.740 to work with. 03:31.740 --> 03:33.450 And you'll find that GraphX code looks a lot 03:33.450 --> 03:35.790 like any other RDD Spark code for the most part. 03:35.790 --> 03:38.190 And actually once you have a graph built, 03:38.190 --> 03:39.420 dealing with it looks a little awful lot 03:39.420 --> 03:43.470 like Spark SQL anyways, so it's not as bad as it sounds. 03:43.470 --> 03:45.630 Creating a VertexRDD is pretty straightforward, 03:45.630 --> 03:47.700 really you just have to return a tuple that includes 03:47.700 --> 03:48.600 a vertex ID, 03:48.600 --> 03:51.900 some unique numerical identifier as its first field 03:51.900 --> 03:54.510 and whatever data you wanna associate with that. 03:54.510 --> 03:56.190 In this little snippet of code here that we're 03:56.190 --> 03:57.270 taking from our example, 03:57.270 --> 03:59.880 you're seeing that we're actually wrapping that in an Option 03:59.880 --> 04:02.280 and that's how we deal with null values in Scala. 04:02.280 --> 04:03.720 We didn't really talk about that before. 04:03.720 --> 04:06.690 But you see that we're defining an option 04:06.690 --> 04:07.980 of a vertex ID in a string, 04:07.980 --> 04:10.890 and that means that we have the option of returning nothing. 04:10.890 --> 04:13.140 So you'll see that in the case where we have a valid result 04:13.140 --> 04:16.530 we're returning Some with a tuple that consists 04:16.530 --> 04:20.310 of a vertex ID and the data associated with that vertex. 04:20.310 --> 04:23.580 But if we have an invalid entry there, we return None. 04:23.580 --> 04:25.500 And that just means there is no results. 04:25.500 --> 04:28.140 It's basically the the Scala equivalent of null. 04:28.140 --> 04:30.330 And this is useful because if you're calling flatmap 04:30.330 --> 04:32.820 on an RDD and your function returns none, 04:32.820 --> 04:34.590 that just gets discarded and that's okay. 04:34.590 --> 04:36.330 So that's the way of dealing with the case 04:36.330 --> 04:39.870 of not returning anything out of a flatmap operation. 04:39.870 --> 04:40.800 In the case of our data 04:40.800 --> 04:43.410 we do have some data lines that are invalid. 04:43.410 --> 04:45.000 It turns out that any hero ID 04:45.000 --> 04:47.370 above 6486 is not a real character 04:47.370 --> 04:49.140 so we need to discard those 04:49.140 --> 04:51.840 and that's the case where we return none in this case. 04:53.100 --> 04:54.960 Creating an edge is also pretty straightforward. 04:54.960 --> 04:57.750 All you do is create an Edge object containing a list 04:57.750 --> 04:59.100 of the nodes that it connects. 04:59.100 --> 05:01.860 So in this example here you can see that we're 05:01.860 --> 05:02.693 creating an edge 05:02.693 --> 05:06.390 between a given hero ID that starts the beginning of a line 05:06.390 --> 05:09.150 in our data file for the Marvel superhero data set 05:09.150 --> 05:11.010 and we constructed a new edge between that 05:11.010 --> 05:14.550 and every superhero that that hero is connected to, 05:14.550 --> 05:16.020 defined by that line of text. 05:16.020 --> 05:18.030 So an edge, very straightforward, 05:18.030 --> 05:20.460 it's just an Edge object that consists 05:20.460 --> 05:23.370 of two vertex IDs and some additional information 05:23.370 --> 05:25.310 that you might wanna associate with that as well. 05:25.310 --> 05:27.750 Pretty straightforward again. 05:27.750 --> 05:30.330 And to construct a graph, again that's straightforward. 05:30.330 --> 05:34.050 You just construct a Graph object and you construct it 05:34.050 --> 05:36.360 with the list of vertices that you want to have in it, 05:36.360 --> 05:38.460 the list of edges between those vertices, 05:38.460 --> 05:40.110 and that's pretty much it. 05:40.110 --> 05:42.060 You'll probably want to cache that graph 05:42.060 --> 05:43.560 because you're probably gonna wanna do a bunch 05:43.560 --> 05:44.880 of operations on it. 05:44.880 --> 05:45.780 And by caching it 05:45.780 --> 05:47.520 that ensures that it will remain in memory 05:47.520 --> 05:51.390 which can help Spark optimize things when you do stuff 05:51.390 --> 05:53.160 with that graph later on. 05:53.160 --> 05:55.410 So doing stuff with a graph is also pretty straightforward 05:55.410 --> 05:56.700 once you have it constructed. 05:56.700 --> 05:58.200 Although sometimes it's useful to 05:58.200 --> 06:00.120 see some sample code to start from. 06:00.120 --> 06:02.010 For example, if you want to take the top 10 06:02.010 --> 06:05.580 most connected heroes, we could call graph.degrees 06:05.580 --> 06:08.430 to get those degrees of connectedness and join those 06:08.430 --> 06:10.410 with the vertices themselves. 06:10.410 --> 06:11.520 Sort the given results 06:11.520 --> 06:13.140 by the field that corresponds to the number 06:13.140 --> 06:16.110 of connections that they have in descending order. 06:16.110 --> 06:17.340 Take the top 10 and you're done. 06:17.340 --> 06:18.900 So that's an easy way to figure out 06:18.900 --> 06:21.600 who the most connected superheroes are in just one line 06:21.600 --> 06:23.760 of code once you've constructed that graph object. 06:23.760 --> 06:25.290 So a little bit easier. 06:25.290 --> 06:27.570 The syntax is a little bit harder to follow here. 06:27.570 --> 06:29.550 So you know, it's a little, the jury's out, I think, 06:29.550 --> 06:31.770 on whether that's actually a more straightforward way 06:31.770 --> 06:33.540 of coding it, but it works. 06:33.540 --> 06:35.661 So that's one way of using GraphX. 06:35.661 --> 06:37.020 But as we said before 06:37.020 --> 06:39.990 it's a lot more flexible when we introduce the Pregel API 06:39.990 --> 06:41.130 on top of GraphX. 06:41.130 --> 06:43.680 So let's talk about that next and how we can actually 06:43.680 --> 06:46.740 extend GraphX to duplicate the results that we got 06:46.740 --> 06:50.133 in our degrees of separation example earlier in the course.