WEBVTT

00:00.680 --> 00:05.210
Let's let's talk about a very popular technique in the world of generative AI retrieval, augmented

00:05.210 --> 00:09.230
generation, or Rag for short, or as I like to call it, cheat to win.

00:09.230 --> 00:10.640
You'll see why in a moment.

00:10.670 --> 00:12.290
Retrieval augmented generation.

00:12.290 --> 00:15.740
You can think of it as sort of an open book exam for large language models.

00:15.740 --> 00:20.960
So the idea is that you take the prompt coming in and you augment that prompt by doing some sort of

00:20.960 --> 00:26.810
a lookup in some external database, and you include the results of that lookup in the prompt itself.

00:26.810 --> 00:31.880
So for example, let's say that somebody is asking what did the president say in his speech yesterday?

00:31.910 --> 00:36.050
Maybe your large language model wasn't trained up to date with stuff that happened just yesterday.

00:36.050 --> 00:40.580
So you might have some external data store that contains information about current events.

00:40.580 --> 00:43.220
So you're going to go off and do a query on that database.

00:43.220 --> 00:48.320
It's probably a vector database, but we'll get into that in a moment that just says, okay, um, here

00:48.320 --> 00:53.450
are documents that are relevant to the topic of what the president said in his speech yesterday.

00:53.470 --> 00:56.950
and it will take those results and just embed them within the prompt itself.

00:56.950 --> 01:01.630
For example, it might modify that prompt to say, what did the president say in his speech yesterday?

01:01.630 --> 01:05.080
Consider the text of the following news article in your response last night.

01:05.080 --> 01:06.730
President, whoever it is, said whatever.

01:06.730 --> 01:07.270
Right.

01:07.270 --> 01:13.030
So again, you're sort of doing a database lookup, an old school database retrieval there and then

01:13.030 --> 01:17.110
just punching it through a large language model to kind of rephrase that in natural language.

01:17.110 --> 01:19.460
So that's why I call it Cheat to Win.

01:19.460 --> 01:24.350
It's really an old school database lookup, but you're just putting the result through a large language

01:24.350 --> 01:31.640
model to both phrase it in a more natural manner, but also to gain whatever input you might have from

01:31.640 --> 01:33.170
the LMS training itself.

01:33.170 --> 01:37.850
But kind of it's just a database lookup and you're just punching it through generative AI.

01:37.880 --> 01:39.230
So you can call it generative AI.

01:39.230 --> 01:41.990
I mean, at the end of the day, that's kind of what's going on.

01:42.730 --> 01:45.400
So there are pros and cons to this this approach.

01:45.520 --> 01:50.620
First of all, this can be a faster and cheaper way, at least initially, to incorporate new or proprietary

01:50.620 --> 01:53.230
information into your application compared to fine tuning.

01:53.230 --> 01:55.930
Like we talked about, fine tuning a model can.

01:55.930 --> 01:59.500
Well, it involves actually training this huge model and that can be costly.

01:59.500 --> 02:03.820
Whereas with the retrieval augmented generation, you're just doing a relatively inexpensive database

02:03.820 --> 02:06.840
lookup and incorporating that into your prompt at runtime.

02:06.840 --> 02:10.560
That's very inexpensive, at least on a one shot basis.

02:11.220 --> 02:15.360
And the other thing is that if you want to update and add new data, that's just a matter of adding

02:15.360 --> 02:16.620
new data into that database.

02:16.620 --> 02:20.940
So you're pulling information in from so you don't have to go and fine tune your model again every day

02:20.940 --> 02:22.710
to get that most up to date information.

02:22.710 --> 02:26.160
It's a much simpler process of updating your info.

02:26.190 --> 02:30.000
This also takes advantage of something we call semantic search in many cases.

02:30.000 --> 02:35.250
So if your external database is something we call a vector store, talk about that in a moment.

02:35.280 --> 02:40.350
You can say you're doing semantic search and actually leveraging some pieces of AI in that database

02:40.350 --> 02:42.090
retrieval process as well.

02:42.090 --> 02:43.500
So that's pretty cool.

02:43.500 --> 02:45.540
It's not just a traditional database lookup.

02:45.720 --> 02:50.400
Rag is often presented as a solution to the hallucination problem as well, although it's not a complete

02:50.400 --> 02:55.080
solution really, but llms tend to hallucinate when you ask them about things they don't know about

02:55.080 --> 02:56.930
that just aren't in their training data to begin with.

02:56.930 --> 03:02.120
That's when it starts to make stuff up, usually, and by providing it contextual information that is

03:02.120 --> 03:07.640
retrieved from some data store with an authoritative set of data and placing that into the prompt itself,

03:07.640 --> 03:11.870
it's going to be a little bit less likely to make stuff up because, well, you handed it the answer

03:11.870 --> 03:12.590
already, right?

03:12.590 --> 03:16.220
So that's one way to increase the trust of these generative AI systems.

03:16.220 --> 03:20.550
And if your boss asks you for AI powered search, well, this is an easy way to do it because that's

03:20.550 --> 03:21.420
literally what it is.

03:21.420 --> 03:24.780
You're doing a traditional search and pumping it through AI.

03:24.810 --> 03:26.280
Hey, you got AI search now.

03:26.310 --> 03:31.020
There are cons, however, so I would argue that you have just made the world's most overcomplicated

03:31.020 --> 03:32.160
search engine here.

03:32.160 --> 03:37.140
You know, you're just taking that search result and pushing it through this very complex and energy

03:37.140 --> 03:39.900
intensive LLM at the end of the day.

03:40.230 --> 03:41.640
Now there are ways to mitigate that.

03:41.640 --> 03:42.180
To be fair.

03:42.180 --> 03:44.300
You know, you might not need a large language model.

03:44.300 --> 03:47.720
Maybe you can put that result through a smaller language model that is more energy efficient.

03:47.720 --> 03:52.520
That might be a good idea for saving resources and having less of an ecological impact.

03:52.520 --> 03:57.020
If you're just relying on the information from your data retrieval, and not so much the information

03:57.020 --> 04:01.130
that the model was trained on, maybe you can get away with not really having a complicated language

04:01.130 --> 04:02.540
model at the end of the day.

04:02.810 --> 04:07.820
Also, Rag is very sensitive to the prompt templates that you use to incorporate your data into it.

04:07.820 --> 04:12.340
So how you retrieve that data, how you incorporate that into the prompt, and the wording that you

04:12.340 --> 04:13.480
use to do so.

04:13.480 --> 04:17.080
It can really, uh, affect the quality of the results that you get.

04:17.200 --> 04:20.350
And at the end of the day, any LLM is going to be non-deterministic.

04:20.350 --> 04:22.240
So these models are changing all the time.

04:22.240 --> 04:24.280
And the way you ask for things will change.

04:24.280 --> 04:29.140
So it's very hard to test these things because well, you might get a different answer every time that

04:29.140 --> 04:29.620
you ask.

04:29.620 --> 04:33.310
And that's a difficult thing from transparency.

04:33.730 --> 04:37.120
And it can still hallucinate even though you're feeding it information.

04:37.120 --> 04:38.830
It can still make stuff up if you let it.

04:38.830 --> 04:42.790
So if the information you gave it isn't really relevant to the question at hand, that can definitely

04:42.790 --> 04:43.450
happen.

04:43.450 --> 04:48.970
You know, if I were to ask, for example, in that previous slide, what did the president say yesterday?

04:48.970 --> 04:50.440
The word yesterday?

04:50.740 --> 04:54.760
Um, it's not necessarily going to expand that into a specific date when it does that query for the

04:54.760 --> 04:55.390
news articles.

04:55.390 --> 04:55.690
Right.

04:55.690 --> 05:01.120
So it doesn't really know what yesterday is necessarily that database lookup might fail miserably,

05:01.120 --> 05:05.650
and I might get a bunch of irrelevant information instead that I'm feeding back into my prompt.

05:05.650 --> 05:11.980
So if that retrieval stage is not as relevant as you would hope, or maybe you don't even have that

05:11.980 --> 05:16.510
information in the data store to begin with, you can still have a situation where the system will hallucinate

05:16.510 --> 05:18.730
because, well, it doesn't know the answer.

05:18.730 --> 05:22.560
And the tendency of these systems when they don't know the answer is to make stuff up.

05:22.560 --> 05:23.520
Unfortunately.

05:24.360 --> 05:29.220
And just to reiterate that same point, this is all very sensitive to how relevant the information you

05:29.220 --> 05:30.030
retrieve is.

05:30.030 --> 05:35.490
So a fine point of building a good retrieval augmented generation system is making sure that the results

05:35.490 --> 05:40.470
you're getting back from that external data store are bits of text that really are relevant to the prompt

05:40.470 --> 05:42.750
to the question that was being posed by the end user.

05:42.750 --> 05:47.760
And that's harder than it sounds just to make this more concrete with an example, let's say I want

05:47.760 --> 05:49.860
to build a system that's good at winning at jeopardy.

05:49.860 --> 05:55.440
So, for example, we might have a prompt that says in 1986, Mexico scored as the first country to

05:55.440 --> 05:57.870
host this international sports competition twice.

05:57.930 --> 06:05.220
So maybe we go into what's called a retriever module there, where first we encode that query in a way

06:05.220 --> 06:06.510
that we can do a database lookup.

06:06.510 --> 06:12.230
So if we're using a vector store, that might be mean transforming that question into a vector that

06:12.230 --> 06:14.210
can be used in our vector database lookup.

06:14.210 --> 06:17.900
So maybe I have a database of past jeopardy questions and answers.

06:17.900 --> 06:23.660
And I can compare semantically the similarity between that question that I'm feeding in about Mexico's

06:23.660 --> 06:29.540
sports competitions, and see what in my database matches up semantically to that question.

06:29.540 --> 06:34.520
That might generate back a list of documents that came back from that database that are questions or,

06:34.520 --> 06:35.270
I guess, in jeopardy.

06:35.270 --> 06:36.410
You would call them answers, right?

06:36.410 --> 06:39.770
Because it's backwards that are relevant to that particular case.

06:39.770 --> 06:46.190
And then I would include those examples of correct answers or questions or whatever you want to call

06:46.190 --> 06:46.670
them.

06:46.790 --> 06:47.870
I shouldn't have used jeopardy!

06:47.870 --> 06:51.440
It's such a confusing example because you ask an answer and get a question back, but you know what

06:51.440 --> 06:52.250
I mean, right?

06:52.250 --> 06:57.110
We're taking this question coming in, finding similar answers, injecting them into the prompt that

06:57.110 --> 07:02.510
goes into the generator, which is your model, and out comes the other end, hopefully the World Cup.

07:02.990 --> 07:07.100
So basically we're handing the generator potential answers from some external database there.

07:07.100 --> 07:07.340
Right.

07:07.340 --> 07:11.840
And we're kind of feeding it the answer potentially, or at least things that are similar to the answer

07:11.840 --> 07:13.130
as part of the prompt.

07:14.630 --> 07:17.240
So what does that database look like exactly.

07:17.240 --> 07:17.510
Really.

07:17.510 --> 07:19.430
You can use any database you want to.

07:19.520 --> 07:22.010
A very popular choice lately is a graph database.

07:22.010 --> 07:26.680
We call that graph rag, something like Neo4j's or Neptune or something like that.

07:26.680 --> 07:32.200
And this can be useful where you have a knowledge graph where you have known relationships between items

07:32.200 --> 07:34.810
within that database that are semantically similar.

07:35.170 --> 07:39.160
Um, it might also be useful for things like product recommendations or relationships between items.

07:39.160 --> 07:44.710
So I might have a graph database that kind of tells me what items are similar to each other, what products

07:44.710 --> 07:49.640
are similar to each other, and that might be a good way of querying what might be some relevant products

07:49.640 --> 07:54.710
to include in that prompt as somebody asking for recommendations for some other product.

07:55.250 --> 08:00.650
Or you could just use old school search, you know, using Tfidf that stands for term frequency, inverse

08:00.650 --> 08:05.180
document frequency, which is just an algorithm for doing old school text search in a text database.

08:05.180 --> 08:07.610
So OpenSearch could be used for that.

08:07.790 --> 08:11.180
But more commonly these days we use something called a vector database.

08:11.420 --> 08:15.560
There are also some fancy tricks you can do, where you actually try to get the underlying model to

08:15.560 --> 08:20.450
provide structured queries and extract information for you, again using a graph database.

08:20.660 --> 08:24.110
There is a there are some examples out there if you want to get into what that looks like.

08:24.110 --> 08:27.830
Graph rag is hot, although it's not really something I think the exam is going to cover.

08:27.830 --> 08:32.600
It's more of a newer thing, but pretty much every example you find of retrieval augmented generation

08:32.600 --> 08:39.150
uses what we call a vector database and within Amazon, OpenSearch can function as a vector database,

08:39.150 --> 08:41.100
so can Memorydb now as well.

08:41.100 --> 08:43.560
So those are how you would probably do this.

08:43.560 --> 08:45.870
Let's review what we mean by a vector database.

08:45.870 --> 08:50.130
So to understand a vector database first you need to understand what an embedding is.

08:50.130 --> 08:54.360
And an embedding vector is just a big vector associated with your data.

08:54.360 --> 08:58.380
And you can think of that as just a point in a large multidimensional space.

08:58.380 --> 09:02.950
So we're visualizing this on the right here as two dimensions x and y.

09:02.950 --> 09:07.810
But in reality a real embedding vector might have thousands of dimensions or at least hundreds, right.

09:07.810 --> 09:13.750
So at the end of the day, though, it's just a point in this multidimensional space, and each point

09:13.750 --> 09:16.960
indicates how similar something is to each other semantically.

09:16.960 --> 09:22.630
So in this example here the words potato and rhubarb might get encoded to tokens that have embeddings

09:22.630 --> 09:27.390
that are kind of close together in this space because they're both root vegetables, I guess.

09:27.420 --> 09:32.790
And similarly, the Starship Enterprise, Spaceballs, The Orville, these are all science fiction TV

09:32.790 --> 09:33.930
shows or movies.

09:33.930 --> 09:38.130
Maybe those are all clustered together semantically as well, somewhere within this embedding space.

09:38.130 --> 09:45.030
So embeddings are just converting words or phrases or sentences or whatever it is to these large vectors,

09:45.030 --> 09:50.910
where the space between those vectors indicates how similar they are to in each other in meaning.

09:50.910 --> 09:52.410
How does it know what meaning is?

09:52.410 --> 09:56.910
Well, it learns this just like any other, uh, machine learning system or deep learning system would,

09:56.910 --> 09:58.410
you know, just you have to train it.

09:58.590 --> 10:02.850
Um, so to compute these embeddings, you need to use an embedding base model.

10:02.850 --> 10:04.470
Titan provides this for example.

10:04.470 --> 10:06.990
So in bedrock this is very easy to do.

10:07.020 --> 10:10.650
You can just say hey I want to feed this data into my embedding base model.

10:10.650 --> 10:12.060
Give me back embedding vectors.

10:12.060 --> 10:14.300
And you know it's a couple of lines of code basically.

10:14.300 --> 10:16.280
And you can do that in a batch fashion as well.

10:16.280 --> 10:21.290
So it's very quick, very cheap and very efficient to create these embedding models.

10:21.560 --> 10:26.270
If you think back to how these large language models work, down at the very bottom of the diagram of

10:26.270 --> 10:28.820
how the transformer model works was an embedding layer.

10:28.820 --> 10:31.370
And it's just a fundamental piece of this technology.

10:31.370 --> 10:35.750
So we're just reusing that for this purpose of creating embeddings for the things that we're searching

10:35.750 --> 10:36.410
for.

10:36.680 --> 10:38.560
So what do we do with these embedding vectors?

10:38.560 --> 10:39.670
Once we've computed them?

10:39.670 --> 10:44.230
Well, we store them in a vector database because, well, where else would you put a vector, right.

10:44.260 --> 10:49.120
All of vector database does is store your data alongside their computed embedding vectors.

10:49.120 --> 10:54.220
So we can do a search based on things that are close to each other in this vector space, and retrieve

10:54.220 --> 10:55.810
the original information from that.

10:55.840 --> 11:02.180
Now, it's important to point out that this is a fairly expensive process in terms of space and computational

11:02.180 --> 11:03.020
capacity.

11:03.050 --> 11:06.740
Think about it like, you know, every one of these little words is going to have an embedding vector

11:06.740 --> 11:09.290
that could be thousands of dimensions in size.

11:09.290 --> 11:13.430
So in many cases, that embedding vector is going to take up more space than the thing you're storing

11:13.430 --> 11:14.420
in the first place.

11:14.420 --> 11:18.200
So these things can get very large, very fast in terms of space.

11:18.200 --> 11:24.560
And also the problem of finding those closest vectors to another vector is a pretty hard problem.

11:24.560 --> 11:29.530
You know, if you think about that as a k nearest neighbors problem, we have to search through a lot

11:29.530 --> 11:31.540
of stuff to figure out what the closest ones are.

11:31.540 --> 11:36.670
And there are some clever ways that vector databases use to try to narrow that down and make it a little

11:36.670 --> 11:40.090
bit faster, but at the end of the day, it's not the most efficient algorithm.

11:40.270 --> 11:44.680
Now, again, this leverages embedding capabilities you might already have for machine learning systems.

11:44.800 --> 11:50.140
In this case, we're going to be taking advantage of embedding models that are already offered by Amazon

11:50.160 --> 11:54.150
Bedrock like from Titan, and often these are the same embedding models that are used at the base of

11:54.150 --> 11:56.040
these larger language models anyway.

11:56.040 --> 11:59.580
So when you want to retrieve something from a vector store, this is the process.

11:59.580 --> 12:02.730
First, you need to compute an embedding vector for the thing you're searching for.

12:02.730 --> 12:08.550
So if I'm searching for, I don't know what's what's a Star Wars ship, the Millennium Falcon right.

12:09.090 --> 12:12.420
I would need to create an embedding vector for the Millennium Falcon.

12:12.420 --> 12:16.890
And in principle, that would be somewhere up there around the other science fiction stuff that we have

12:16.890 --> 12:18.480
there in the upper right hand corner.

12:18.510 --> 12:23.700
We would then query the vector database for the top items that are close to that vector, and we get

12:23.700 --> 12:25.620
back however many things that we asked for.

12:25.620 --> 12:27.660
And again this is a lot like k nearest neighbor.

12:27.660 --> 12:27.930
Right?

12:27.930 --> 12:34.290
So let's say I want to get back the three closest vectors to the Millennium Falcon that might come back

12:34.290 --> 12:36.990
with the Starship Enterprise, Spaceballs, and The Orville Wright.

12:36.990 --> 12:39.250
So in that case, K is three and that's it.

12:39.250 --> 12:40.120
That's vector search.

12:40.120 --> 12:43.780
We're basically converting what we're searching for into a vector.

12:43.840 --> 12:48.850
Uh, comparing that to all the other stuff I've stored in my vector space and finding the closest vectors

12:48.850 --> 12:53.950
to find the things that are what we call semantically similar, because, again, these embedding vectors

12:54.130 --> 12:59.260
fundamentally have some sort of relationship to the underlying meaning of the thing you're storing.

12:59.260 --> 13:02.950
And that's just learned through training through through a deep neural network.

13:02.960 --> 13:09.500
Some examples of vector databases in AWS Amazon OpenSearch service can do this, either provisioned

13:09.500 --> 13:10.700
or serverless.

13:10.790 --> 13:13.010
Serverless is a popular choice these days.

13:13.010 --> 13:16.190
Keep in mind, though, that serverless doesn't mean that it's free if you don't use it.

13:16.190 --> 13:21.800
There are base charges there, so even an unused OpenSearch serverless database will cost you about

13:21.800 --> 13:24.860
50 bucks a month, even if it's the smallest one you can get.

13:25.160 --> 13:30.350
Also, Amazon Memorydb now has a vector search capability built into it as well.

13:30.380 --> 13:37.340
Other choices would be the PG vector extension in Amazon RDS for PostgreSQL, and there's also a PG

13:37.340 --> 13:41.840
vector extension for Amazon Aurora PostgreSQL compatible edition.

13:41.870 --> 13:44.720
Amazon Kendra also supports vector search as well.

13:44.720 --> 13:46.280
So a lot of options here.

13:46.520 --> 13:52.850
Um, I'd say OpenSearch is one of the most popular ones, and Memorydb is a new one as of this recording.

13:52.850 --> 13:55.490
But I think that's also going to be extremely popular as well.

13:56.240 --> 13:59.540
So just to make this real again, let's look at another example.

13:59.660 --> 14:03.500
Um, I realize not everybody in the world knows Star Trek The Next Generation, but it was an old science

14:03.500 --> 14:07.730
fiction TV show, and Lieutenant Commander Data was one of the characters there.

14:07.760 --> 14:13.250
Lieutenant Commander Data was an android, and in one episode, data built an android daughter called

14:13.250 --> 14:13.670
lol.

14:13.670 --> 14:19.570
So let's say I have a Rag system that has all the scripts from Star Trek The Next Generation baked into

14:19.570 --> 14:21.460
it in an external vector store.

14:21.490 --> 14:23.710
So let's say I get a prompt that says data.

14:23.710 --> 14:25.090
Tell me about your daughter lol.

14:25.090 --> 14:29.080
Well, I've built a system that pretends that it's Lieutenant Commander Data, which is kind of fun,

14:29.080 --> 14:32.650
and the first thing it will do is compute an embedding vector for that prompt.

14:32.650 --> 14:38.380
It will then search a vector database of everything data ever said in the TV series Star Trek The Next

14:38.380 --> 14:44.230
Generation, and find relevant bits of script that are relevant to data's daughter lore.

14:44.260 --> 14:44.770
Right.

14:44.770 --> 14:49.570
Once we get that set of script lines back, we can incorporate that into the prompt itself.

14:49.570 --> 14:54.820
So I might expand that prompt to say you are commander data from Star Trek, how my data respond to

14:54.820 --> 14:55.930
the question data.

14:55.930 --> 15:00.220
Tell me about your daughter lore taking into account the following related lines from data.

15:00.220 --> 15:03.010
And I mean you can word that in any number of ways, right?

15:03.010 --> 15:06.870
So there are there are many approaches to this, But that's the fundamental idea.

15:06.870 --> 15:09.000
We compute an embedding vector for the prompt.

15:09.000 --> 15:14.310
We do a vector search for relevant semantically relevant results in that vector store.

15:14.310 --> 15:17.880
And then we incorporate those top results into our prompt.

15:17.880 --> 15:20.490
And the other thing that matters too is how many prompts you choose there.

15:20.490 --> 15:23.610
So you know there's another parameter there to tune.

15:23.610 --> 15:26.490
How many similar lines in this case would I include in the prompt?

15:26.490 --> 15:28.230
You know, 1234, five.

15:28.230 --> 15:30.750
You need to experiment on that and see what actually works.

15:30.750 --> 15:32.310
But you know what we're trying to do here.

15:32.310 --> 15:34.740
We take the prompt, we do a query based on that prompt.

15:34.740 --> 15:40.020
And we include relevant data into a modified prompt that then gets handed into the underlying foundation

15:40.020 --> 15:40.560
model.

15:42.000 --> 15:43.500
How is this work in bedrock?

15:43.500 --> 15:46.470
Well, in bedrock we call this knowledge bases.

15:46.470 --> 15:52.590
So the way this works in bedrock is you can upload your own documents or structured data into S3 somewhere,

15:52.590 --> 15:57.270
and that can be raw text data, or it could be structured data that has a JSON data structure.

15:57.270 --> 16:03.270
And with a knowledge base in bedrock, you just point it to that data and it kind of sets up a Rag system

16:03.270 --> 16:05.070
for you automatically.

16:05.340 --> 16:06.720
It doesn't have to be S3, by the way.

16:06.720 --> 16:08.280
It also supports web crawlers.

16:08.280 --> 16:14.940
So you could tell bedrock go crawl these web pages and ingest their information into my Rag system.

16:14.940 --> 16:16.710
Hopefully you have the rights to do that.

16:16.710 --> 16:21.410
You can also integrate with third party systems like confluence, Salesforce and SharePoint as well

16:21.410 --> 16:27.230
to ingest data into a knowledge base for your very own bedrock based drag application.

16:27.380 --> 16:32.870
Knowledge bases do require the use of an embedding model, because again, we need to compute embedding

16:32.870 --> 16:35.630
vectors for all that stuff that we're storing in there somehow.

16:35.630 --> 16:39.740
So before you do this, you have to make sure that you've obtained model access to an embedding model

16:39.740 --> 16:45.510
from Cohere or Amazon Titan As of this recording, you do have some control over how many dimensions

16:45.510 --> 16:50.880
are in that vector, so you can make your own trade offs over how much information is in those vectors

16:50.880 --> 16:55.830
versus how much space they're consuming, and you will need a vector store of some sort as well, some

16:55.830 --> 17:00.690
place to store these vectors by default, and for development purposes, it will just create a serverless

17:00.690 --> 17:02.160
OpenSearch instance.

17:02.310 --> 17:07.340
Remember, however, you do need to explicitly shut that serverless OpenSearch instance down when you're

17:07.340 --> 17:12.110
done developing and playing around, or else it will rack up charges at a pretty steep rate.

17:12.410 --> 17:13.160
Uh memorydb.

17:13.160 --> 17:14.090
Also an option now.

17:14.090 --> 17:15.530
It also has vector search.

17:15.530 --> 17:19.040
You can also choose from Aurora MongoDB Atlas.

17:19.130 --> 17:22.520
We're getting into things outside the Amazon ecosystem here.

17:22.520 --> 17:29.120
Pinecone and Redis Enterprise Cloud are also supported for storing your vectors that are that are used

17:29.120 --> 17:30.110
in bedrock.

17:30.110 --> 17:32.320
You can also control how your data is chunked.

17:32.320 --> 17:34.690
So what do we mean by chunking?

17:34.690 --> 17:39.940
Let's say I have, you know, a huge document or a huge script in the case of data lines, right.

17:39.940 --> 17:41.080
How do I break that up?

17:41.080 --> 17:44.770
How do I store individual chunks of that in individual vectors?

17:44.770 --> 17:46.240
Well, there's many ways to do that.

17:46.240 --> 17:51.700
Ideally, you would want each vector to contain some sort of semantically coherent thought, right?

17:51.700 --> 17:55.360
Like at the end of the day, we're trying to embed the meanings of the things that we're storing in

17:55.360 --> 18:00.190
this vector store, and just some random chunk of data might not have a coherent meaning.

18:00.190 --> 18:02.170
Unfortunately, that's how this works by default.

18:02.170 --> 18:08.560
So the sort of default naive chunking method is to just take a large document, split it up into chunks

18:08.560 --> 18:15.010
of 300 characters, and consider every chunk of 300 characters its own entity in that vector database.

18:15.010 --> 18:16.540
Obviously, that's not ideal.

18:16.660 --> 18:23.130
Um, what you could do instead is sort of chunk that yourself and store individual documents with the

18:23.130 --> 18:25.590
individual things you want to store in that vector store.

18:25.590 --> 18:28.290
So that would give you a little bit more control.

18:28.380 --> 18:32.460
There's also something called semantic chunking out there that's even more intelligent about how it

18:32.460 --> 18:37.170
tries to split that up, but by default it will just chunk it up by a given number of characters.

18:37.170 --> 18:41.670
And you can specify how big those chunks are when you're doing this.

18:42.810 --> 18:44.610
How is this work at runtime?

18:44.610 --> 18:47.160
So it has what they call chat with your document.

18:47.160 --> 18:48.720
It's a way to sort of play around with it.

18:48.720 --> 18:50.610
And it's basically an automatic rack system.

18:50.610 --> 18:54.060
So once you've created a knowledge base, you can say, let's chat with it.

18:54.060 --> 18:57.780
And it works like this, uh, diagram on the right here.

18:57.780 --> 18:59.310
You start off with a user query.

18:59.310 --> 19:04.170
It performs a semantic search against some vector store, probably Amazon OpenSearch.

19:04.410 --> 19:09.960
From that, it retrieves some relevant context about your query from that data store that then gets

19:09.960 --> 19:15.300
combined with the user query into the augmented query, which then goes into your Amazon Bedrock Foundation

19:15.300 --> 19:19.710
model and generates a response that hopefully took that context into account.

19:19.740 --> 19:23.310
You can also incorporate knowledge bases into agents, which we'll talk about shortly.

19:23.310 --> 19:29.250
It can be just another tool in the tool chest for an LM agent, or you can integrate this into an application

19:29.250 --> 19:30.120
directly.

19:30.510 --> 19:35.490
We call this a genetic rag when you're just using it as one of many tools that an AI application might

19:35.490 --> 19:36.210
use.

19:36.210 --> 19:38.790
So let's make it real with an actual hands on example.

19:38.790 --> 19:41.850
Let's go ahead and make a knowledge base, also known as a vector store.

19:41.850 --> 19:46.320
And to make it real, we're going to use the text of a book that I wrote a long time ago.

19:46.440 --> 19:51.660
Um, this book here, not relevant to Amazon, but, you know, sort of a business focused book.

19:51.660 --> 19:56.220
And we'll build a system that can answer questions about this book using Amazon Bedrock and knowledge

19:56.220 --> 19:56.640
bases.

19:56.640 --> 19:57.900
So let's have at it.
