Loading...
In this video, we'll take a look at some visualizations of embeddings. When you're building a practical application, you often not be outputting the visualization as the final step, unless you're, say, taking a collection of documents and deliberately want to visualize what different texts in the documents are saying. So what documents are similar, what documents are dissimilar. But outside applications like that, I don't end up using visualizations that much, but in this video, we'll take a look at some of them to build intuition about what these embeddings are actually doing. I think we'll get some pretty parts out of this. So, let's take a look. Let's start off by authenticating myself to the Veritex AI platform, same as before. For this visualization, I want to use this collection of seven sentences. Mustang flamingo, discover that swimming pool, see all the spotters who have bought baby panda, boat ride, breakfast theme, food truck, new curry restaurants, and then two others. Python developers are wonderful people. I think that's a totally true statement. And then TypeScript, C++, and Java all are great. I have to admit, I have my own preferences, but I won't state them in this video. And then, similar to last time, let me import NumPy, and I'm going to set up my embedding model as follows, and then for my seven sentences in these input one through input seven, in one through in seven, let's run this snippet of code to loop over, you know, my seven inputs to compute embeddings for all of them. So, for input text in this list of sentences, set the embedding to, you know, call the embedding model on that, extract the values, and then stick it in this embeddings list, and. And then a little bit of data munging to turn it into a NumPy array. And so, let's run it. Returns really quickly. And so right now, this embeddings array, let me print out the shape of the embeddings array. So, the shape is 7 by 768. And there are seven rows because there were seven sentences that we embedded, and each row has 768 numbers. Now, what I'd like to do is to next visualize these seven embeddings. But I can't plot a 768-dimensional vector on this 2D computer monitor. So, I'm going to use a technique called PCA, or Principal Components Analysis. If you know what PCA is, great, if you don't, don't worry about it, is a technique for taking very high-dimensional data, say 768-dimensional data, and compressing that down to two dimensions. If you're interested to learn more about PCA, you can take an online machine learning class such as the Machine Learning Specialization. But all you really need to know for the purpose of this video is, this is a way to take this 768-dimensional data and squash it down to two dimensions. So, this code calls the Scikit-learn PCA library, It fits a PC model to compress it to two-dimensional data, and you get a set of new values. If I now print out the new values, it's now 7 by 2 instead of 7 by 768. It's compressed. Each of these embedding vectors down to two dimensions. Thus, losing a lot of information along the way, but make it easier to plot on this computer monitor. Next, here's the code we can use to plot. I'm going to use matplotlib, a common plotting function, and plot a 2D plot of the first and second values of each of these now two-dimensional embeddings on the horizontal and vertical axes of a figure. Let me do that, and here's our plot. Let's see. Here is baby panda, mussel, flamingo, sea otter. So, our animal-related sentences are pretty close to each other. Here is my food truck and the curry restaurant, pretty close to each other. Python developers are wonderful people. I mean, that's just true, and you know, all great programming languages. So, the coding-related sentences are down here. So, this illustrates how the algorithm does embed sentences that are more similar together. Strongly encourage you to pause the video, go back and plug in your own sentences. Maybe write some fun sentences about your friends and send the resulting visualization to them. It seems like there's a lot you could do to play with this. Just to be clear, if you're writing an application, I would not use a two-dimensional embedding. I would always measure distances in that original 768 dimensional, uh, feature embedding. And I'm reducing this to two dimensions just for visualization purposes. But when measuring similarity, I would not measure similarity of the 2D data. I would measure similarity in that original, much higher dimensional space, which actually gives more accurate distance metrics. Because PCA, principal components analysis, is throwing away quite a lot of information to generate the visualization. Now, let's look at another example. Input 1, N1, he couldn't desert his post at the power plant, the power plant needed him at the time. So, N1 and N2 seem pretty similar, N plus 1 and N plus 2. Cacti are able to withstand dry environments and desert plants can survive droughts. So, N3 and N4 seem pretty similar, even though sentence one and sentence four, both of the word desert and plant versus plants. But here's my list of four sentences. Then, here's my same code snippet as before to generate the embeddings. Lastly, let's plot a heat map showing the values of the embedding. Here's some code to plot a heat map. And what this is showing is, is using these different colors from shades of blue through shades of red to show for each of the four embeddings, does the first component have a high value or low value? Does the second component have high value or low value? And so on through many elements of this vector. I don't normally use heatmaps to visualize embeddings. I'll say a little bit more about this in a second. But from this heatmap, you know, hopefully you can see that the desert plant embedding and the cacti embedding in this very high dimensional space, the first two patterns in the heat map look a bit more similar to each other and the patterns on the third and fourth heat maps, you know, kind of look a little bit more similar to each other. And so, this illustrates that the embedding also maps the first two sentences close to each other than sentences three and four. And just as one fun exercise, I encourage you to pause this video and if you want to try, get the code from the earlier notebook to actually compute the similarity between these embeddings to see if the embeddings between the first two inputs are really more similar to each other than say the first and third or the first and the fourth. If you want to do that, just as a reminder, the function we used to compute similarities back in the first notebook was the cosine similarity rarity function. So, feel free to do that. If you want to see if you can compute these pairwise similarities. Before I wrap up this video, just one caveat, which is that even though I showed this visualization, this particular visualization isn't a completely mathematically legitimate thing to do. And to get a little bit technical for the next 30 seconds, so if you don't follow what I say for the next 30 seconds, don't worry about it. But it turns out that the axes used to define an embedding is relatively arbitrary and is subject to random rotation. And that's why, whereas finding pairwise similarity between embeddings is a pretty robust and sound operation, if you were to look at, say, the first component of an embedding and visualize that by looking at the number of heatmap, that first number is very difficult to ascribe a specific meaning to it which is why we can look at this as an informal visualization to build intuition about embeddings, but for practical things, I find it less useful to look at the outputs of the embeddings a single component at a time. If you don't understand what I just said, don't worry about it. The key takeaway is that this is just something of an informal visualization, and these particular visualizations are not very robust, and they could easily change if you were to use a different embedding model, say, but hopefully the intuition that certain sentences are more similar to each other in the embedding space still comes through. So that's it. Once again, please pause the video, write in your own sentences, maybe play the cosine similarity, use all this to build intuition about what the embeddings are doing. In the next video, Nikita will start to dive into how you can use these embeddings to build a variety of applications. When you're ready, please go on to the next video.