Lecture 7: Introduction to TensorFlow

Just another WordPress site

Lecture 7: Introduction to TensorFlow

[MUSIC] Stanford University >> Happened is that Nishith and Barak are going to be giving an introduction to TensorFlow So TensorFlow is Google’s deep learning framework, which I hope everyone will be excited to learn And at any rate, you have to learn it, because we’re gonna be using it in assignments two and three So this should really also help out for the second assignment And so before we get started with that, I just want to do a couple of quick announcements So the first one was on final projects So this is really the time to be thinking about final projects And if you’ve got ideas for final projects and want to do the final project, you should be working out how to talk to one of me, Kevin, Danqi, Richard, Ignacio, or Arun over the next couple of weeks And again, obviously, you’ve got to find the time, so it’s hard to fit everybody in But were are making a real effort to have project advice office hours There were also some ideas for projects that been stuck up on the projects page So encourage people to look at that Now, people have also asked us about assignment four So we’ve also stuck up a description of assignment four And so look at that if you’re considering whether to do assignment four So assignment four is gonna be doing question answering over the SQuAD dataset and you can look in more details about that So then there are two other things I wanted to mention And we’ll also put up messages on Piazza, etc, about this I mean, the first one is that for assignment three, we want people to have experience of doing things on a GPU And we’ve arranged with Microsoft Azure to use their GPUs for doing that and for people to use for the final project And so we’re trying to get that all organized at the moment There’s a limit to how many GPUs we can have So what we’re gonna be doing for assignment three and for the final project is to allow teams of up to three And really it’s in our interest and the resource limit’s interest if many people could be teamed up So we’d like to encourage people to team up for assignment three And so we’ve put up a Google form for people to enter their teams And we need people to be doing that in advance, because we need to get that set up at least a week in advance so we can get the Microsoft people to set up accounts for people so that people can use Azure So please think about groups for assignment three and then fill in the Google form for that And then the final thing is, for next week, we’re gonna make some attempts of reorganizing the office hours and get some rooms for office hours so they can hopefully run more smoothly in the countdown towards the deadline for assignment two than they did for assignment one So keep an eye out for that and expect that some of the office hour times and locations will be varying a bit compared to what they’ve been for the first three weeks And so that’s it from me, and over to Nishith >> Hi everyone Hope you had a great weekend So today we are gonna be talking about TensorFlow, which is another deep learning framework from Google So why study deep learning frameworks? First of all, much of the research in deep learning and machine learning can be attributed because of these deep learning frameworks They’ve allowed researchers to iterate extremely quickly and also have made deep learning and other algorithms in ML much more accessible to practitioners So if you see your phone a lot smarter than it was three years ago, it’s probably because one of these deep learning frameworks So the deep learning frameworks help to scale machine learning code, which is why Google and Facebook can now scale to billions of users They can compute gradients automatically Obviously, since you all must have finished your first assignment, you must know that gradient calculation isn’t trivial And so this takes care of it automatically, and we can focus on the high-level math instead It also standardizes ML across different spaces So regardless of whether I’m at Google or at Facebook, we still use some form of TensorFlow or another deep learning framework And there’s lot of cross-pollination between the frameworks as well A lot of pre-trained models are also available online, so people like us who have limited resources in terms of GPUs do not have to start from scratch every time We can stand on the shoulders of giants and on the data that they have collected and sort of take it up from there They also allow interfacing with GPUs, which is a fascinating feature, because GPUs actually speed up your code a lot faster because of the parallelization Which is why studying TensorFlow is sort of almost necessary in order to make progress in deep learning, just because it can facilitate your research and your projects We’ll be using TensorFlow for PA two, three, and also for

the final project, which also is an added incentive for studying TensorFlow today So what is TensorFlow actually? It’s just a deep learning framework, an open source software library for numerical computation using flow graphs from Google It was developed by their Brain team, which specializes in machine learning research And in their words, TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms So now I’ll allow Barak to sort of take over and give a high-level overview of how TensorFlow works and the underlying paradigms that so many researchers have spent so much time thinking about >> Thanks, Nish, for starting us off I’m gonna be introducing some of the main ideas behind TensorFlow, its programming paradigm, and some of its main features So the biggest idea of all of the big ideas about TensorFlow is that numeric computation is expressed as a computational graph If there was one lesson that you took out of this presentation today, at the back of your mind is that the backbone of any TensorFlow program is going to be a graph where the graph nodes are going to be operations, shorthand as ops in your code And they have any number of inputs and a single output And the edges between our nodes are going to be tensors that flow between them And the best way of thinking about what tensors are in practice is as n-dimensional arrays The advantage of using flow graphs as the backbone of your deep learning framework is that it allows you to build complex models in terms of small and simple operations And this is going to make your gradient calculations extremely simple when we get to that You’re going to be very, very grateful for the automatic differentiation when you’re coding large models in your final project and in the future Another way of thinking about a TensorFlow graph is that each operation is a function that can be evaluated at that point And hopefully we will see why that is the case later in the presentation So let us look at an example of a neural network with one hidden layer, and what its computational graph in TensorFlow might look like So we have some hidden layer that we are trying to compute, as the ReLU activation of some parameter matrix W times some input x plus a bias term So if you recall from last lecture, the ReLU is an activation function standing for rectified linear unit in the same way that a sigmoid is an activation function We are applying some nonlinear function over our linear input that is what gives the neural networks their expressive function And the ReLU takes the max of your input and zero On the right, we see what the graph might look like in TensorFlow We have variables for our b and W, we have a placeholder, we’ll get to that soon, with the x, and nodes for each of the operations in our graph So let’s actually dissect those node types Variables are going to be stateful nodes which output their current value In our case, it’s just b and W What we mean by saying that variables are stateful, is that they retain their current value over multiple executions, and it’s easy to restore saved values to variables So, variables have a number of other useful features They can be saved to your disk during and after training, which is what facilitates the use the niche talked about earlier, that allows people from different companies and groups to save, store, and send over their model parameters to other people And they also make gradient updates by default Will apply over all of the variables and your graph The variables are the things that you wanna tune to minimize the loss, and we will see how to do that soon It is really important to remember that variables in the graph like b and w are still operations By definition, if there can be such a thing as a definition on this, all of your nodes in the graph are operations So when you evaluate the operation that is these variables in our run time, and we will see what run time means very shortly, you will get the value of those variables The next type of nodes are placeholders So placeholders are nodes whose value is fed in at execution time If you have inputs into your network that depend on some sort of external data that you don’t want build your graph that depends on any real value So these are place folders for values that we’re going to add into our computation during training So this is going to be our input So for placeholders, we don’t give any initial values We just assign a data type, and we assign a shape of a tensor so the graph still knows what to compute even though it doesn’t have any stored values yet The third type of node are mathematical operations This is going to be your matrix multiplication, your addition, and your ReLU All of these are nodes in your TensorFlow graphs And it’s very important that we’re actually calling on

TensorFlow mathematical operations as opposed to NumPy operations Okay, so let us actually see how this works in code So we gonna do three things We’re going to create weights including initialization We’re going to create a placeholder variable for our input x, and then we’re going to build our flow graph So how does this look in code? We’re gonna import our TensorFlow package, we’re gonna build a python variable b, that is a TensorFlow variable Taking in initial zeros of size 100 A vector of 100 values Our w is going to be a TensorFlow variable taking uniformly distributed values between -1 and 1 of shapes of 184 by 100 We’re going to create a placeholder for our input data that doesn’t take in any initial values, it just takes in a data type 32 bit floats, as well as a shape Now we’re in position to actually build our flow graph We’re going to express h as the TensorFlow ReLU, of the TensorFlow matrix multiplication of x and w, and we add b So you can actually see that the form of that line, when we build our h, essentially looks exactly the same as how it would look like a NumPy, except we’re calling on our TensorFlow mathematical operations And that is absolutely essential, because up to this point, we are not actually manipulating any data, we’re only building symbols Inside our graph No data is actually moving in through our system yet You can not print off h, and actually see the value it expresses First and foremost, because x is just a place holder, it doesn’t have any real data in it yet But, even if x wasn’t, you can not print h until we run a tune We are just building a backbone for our model But, you might wonder now, where is the graph? If you look at the slide earlier, I didn’t build a separate node for this matrix multiplication node, and a different node for add, and a different node for ReLU Well, ReLU is the h We’ve only defined one line, but I claim that we have all of these nodes in our graph So if you’re actually try to analyze what’s happening in the graph, what we’re gonna do, and there are not too many reasons for you to do this when you’re actually programming a TensorFlow operation But if I’m gonna call on my default graph, and then I call get_operations on it, I see all of the nodes in my graph and there are a lot of things going on here You can see in the top three lines that we have three separate nodes just to define what is this concept of zeroes There are no values initially assigned yet to our b, but the graph is getting ready to take in those values We see that we have all of these other nodes just to define what the random uniform distribution is And on the right column we see we have another node for Variable_1 that is probably going to be our w And then at the bottom four lines we actually see the nodes as they appear in our figure, the placeholder, the matrix multiplication, the addition and the ReLU So in fact, the figure that we’re presenting on the board is simple for what TensorFlow graphs look like There are a lot of things going behind the scenes that you don’t really need to interface with as a programmer But it is extremely important to keep in mind that this is the level of abstraction that TensorFlow is working with above the Python code This is what is actually going to be computed in your graph And it is also interesting to see that if you look at the last node, ReLU It is pointing to the same object in memory as the h variable that we defined above Both of them are operations referring to the same thing So in the code before, what this h actually stands for is the last current node in the graph that we built So great We define, question? So the question was about how we’re deciding what the values are, and the types This is purely arbitrary choice, we’re just showing an example, It’s not related to, it’s just part of our example Okay Great, so we’ve defined a graph And the next question is how do we actually run it? So the way you run graphs in TensorFlow is you deploy it in something called a session A session is a binding to a particular execution context like a CPU or a GPU So we’re going to take the graph that we built, and we’re going to deploy it on to a CPU or a GPU And you might actually be interested to know that Google is developing their own integrated circuit called a tensor processing unit, just to make tensor computation extremely quickly It’s in fact orders of magnitude more quick then even a GPU, and they did use a tender alpha go match against lissdell So the session is any like hardware environment that supports the execution of all the operations in your graph So that’s how you deploy a graph, great So lets see how this is run in code We’re going to build a session object, and we’re going to call run on two arguments, fetches and feeds

Fetches are the list of graph nodes that return the outputs of the nodes These are the nodes that we’re interested in actually computing the values of The feeds is going to be a dictionary mapping from graph nodes to actual values that we want to run in our model So this is where we actually fill in the placeholders that we talked about earlier So this is the code that we have earlier, and we’re gonna add some new lines We’re first going to build a session object called tf.Session It’s gonna take some default environment Most likely a CPU, but you’re able to add in as an argument what device you want to run it on And then we’re going to call, first of all, session.run on initialize all the variables This is concept intensive flow called lazy evaluation It means that the evaluation of your graph only ever happens at run time, and run time now we can add an interpretation to run time in TensorFlow, so and so means the session Once we build the session we’re ready to actually call unlike the tensa flow run time, so it is only then that we actually stick or assign the values that we initialize our BMW on to those notes BMW never mind [LAUGHS] After those two lines, we’re finally in a position to call run on the note we’re actually interested in, the H, and we feed in our second argument of dictionary for X, it’s our placeholder With the values that we’re interested For now just some random values, question? Initialize all variables will initialize all the things that are formerly called variables in your graph like b and w in this case [BLANK AUDIO] And so the question was What is the difference between variables and place holders, and why we might, we might want to use which So, place, sorry, variables are in most cases will be the parameters that we’re interested in, you can almost think of them as the direct correspondence, X are data is not a parameter, we’re interested in tuning In the models we are working with Additionally, it’s important that our parameters have initializations in our model to begin with They have a state Our input doesn’t really have a state as part of our model If we’re gonna take our model and Export it to somebody else There’s no reason for it to actually include any real data values The data is arbitrary, it’s the model parameters that are the foundation of your model They are what makes your model interesting and computing what it computes Great, so what have we covered so far? We first built a graph using variables and placeholders We then deploy that graph onto a session which is the execution environment And next we will see how to train the model So the first question that we might ask in terms of optimization is how do we define the loss? So we’re going to use placeholder for labels as data that we feed in only at run time and then we’re going to build a loss node using our labels and prediction The first line in code here is we’re going to have this Python variable that is the prediction at the end of your neural network It’s going to be the top of some soft max over whatever it is that your neural network is outputting a probability vector could be a regression The first sign is where is the end of the feed forward stage of your neural network? It’s what your network is trying to predict We’re then going to create a variable called label that is a place holder for the ground truth that our model is trying to train against Now we are ready to create our cross entropy node which is just like in our assignment one It’s going to be the sum of the labels times the TensorFlow log of the prediction on our column So just an interesting point, so the sum and log do need to be TensorFlow functions, but TensorFlow will automatically convert addition, subtraction and element wise multiplication into TensorFlow operations Question >> [INAUDIBLE] >> Yep >> [INAUDIBLE] >> It’s going to sum the row altogether which is what we want to do since label in the label each row >> Is going to be a one hard vector So you wanna multiply that by our prediction And it’s going to multiply it at the point of the target index And when we sum that, it’s going to give us the correct result Everything else will be a zero in that row, so it’s squashing it into a column Since zero access is the rows, axis 1 is the columns So it’s gonna collapse The colons Yes? The question was are the feeds just for the placeholders? Yes, that is correct The feeds are just used as a dictionary to fill in the values of our placeholders

Great, all right, so we’ve now defined the loss and we are ready to compute the gradients So the way this is done in TensorFlow is we’re first going to create an optimizer object So there’s a general abstract class in TensorFlow called optimizer Where each of the subclasses in that class is going to be an optimizer for a particular learning algorithm So the learning algorithm that we already use in this class is the gradient descent algorithm, but there are many other choices that you might want to experiment with in your final project They have different advantages So that is just the object to create an optimization node in our graph We’re going to call on the method of it, it’s called minimize, and it’s gonna take in its argument to the node that we actually want to minimize So this adds an optimization operation to the top of our computational graph, which when we evaluate that node, when we evaluate this variable I wrote in the top line called train_step equals the line When we call session on run on trainstep, it is going to actually apply the gradients onto all of the variables in our model This is because the dot minimize function actually does two things in Tensor flow It first computes the gradient of our argument, in this case cross entropy With respect to all of the things that we defined as variables in our graph, in this case the B and W And then it’s actually going to apply the gradient updates to those variables So I’m sure the question in the back of all your minds now is how do we actually compute the gradients? So the way it works in TensorFlow is that, every graph node has an attached gradient operation, has a prebuilt gradient of the output with respect to the input And So when we want to calculate the gradient of our cross entropy with respect to all the parameters, it is extremely easy to just backpropagate through the graph using the chain rule So this is where you actually get to see the main advantage of expressing this machine-learning framework as this computational graph, because it is very easy for the application to step backwards, to traverse backwards through your graph, and at each point, multiply the error signal by the predefined gradient of our node And all of this happens automatically, and it actually happens behind the programmer’s interface Question? The question was is the gradients are competed with respect to the cross With respect to all of our variables So the argument into the minimize function is going to be the node that it’s computing the gradient of, in the numerator, with respect to automatically all of the things we defined as variables in our graph Doesn’t that you can add as another argument what variables to actually apply gradients to, but if you don’t it’s just going to automatically do it to everything defined as a variable in our graph Which also answers a question earlier about why we wouldn’t want to call x a variable because we’re not actually we don’t actually want to update that So how does this look like in code? We’re just going to add the top line in the previous slide We’re gonna create a python variable called train_step that takes in our Gradient Descen tOptimizer object with learning rate of 0.5 We’re gonna Minimize on it over the cross_entropy So you can kinda see that that line encapsulates everything, all of the important information about doing optimization It knows what gradient step algorithm to use, the gradient descent And knows what learning rate and knows what node to compute the gradients over and an oath to minimize it of course Okay So let’s actually see how to run this in code, the last thing we have to do Question Let me answer that The question was how does session know what variables to link it to? I think that this answers it The session is going to deploy all of the nodes in your graph Onto the runtime environment Everything in the graph is already on it, so when you call minimize on this particular node, it’s already there inside your session to like compute, if that answers it Okay, so the last thing we need to do now that we have the gradients, we have the gradient update It’s just to create an iterative learning schedule So we’re going to iterate over, say 1,000 iterations, the 1,000 is arbitrary We’re going to call on our favorite data sets, we’re gonna take our next batch Data is just any abstract data in this arbitrary program So we’re gonna get a batch for our inputs, a batch for our labels We’re then going to call sess.run on our training step variable So remember, when we call run on that, it already applies the gradients onto all the variables in our graph And it’s gonna take a feed dictionary for the two place holders that we’ve defined so far

The x and the label where x and label are graph nodes The keys in our dictionary are graph nodes and the items are going to be NumPy data And this is actually a good place to talk about just how well TensorFlow interfaces with NumPy because TensorFlow will automatically convert NumPy arrays when we feed it in to our graph in to tensors So we can insert in to our feed dictionary numpy arrays which are batch_x and batch_label and we are also going to get is an output from sess.run If I defined some variable like output equals sess.run, that would also be a NumPy array of what the nodes, of what the nodes evaluate to Though train_step would return you the gradient I believe Are there any other questions up to that point before we take a little bit of a turn? Yes So I actually believe there’s some ways to create queues for inputting in data and labels, that might be the answer to your question I can testify to why this might be the best method too But it certainly is a simple one, where you can just work with NumPy data, which is what Python programmers are used to And that is the insert point into our placeholder One more question, yes? Your question was, how does a cross entropy know what to compute? So the cross entropy is going to take an, I haven’t defined what prediction it is fully, I just wanted to abstract that part The prediction is going to be something at the end of your no network, where all of those are symbols inside your graph Something before that’s going to be, all these notes in your graph I think this might be a better answer to your question, when you evaluate some node in the graph, like if i were to call session.runonprediction It automatically computes all of the nodes before it in the graph that need to be computed to actually know what the value of prediction is Behind the season TensorFlow it’s going to transverse backwards in your graph and compute all of those operations and that happens behind you That happens automatically inside my session So the last important concept that I wanna talk about before we move over to the live demo is the concept of variable sharing So when you wanna build the large model, you often need to share large sets of variables and you might want to initialize them all in one place For example, I might want to instantiate my graph multiple times or even more interestingly, I want to train over like a cluster of GPUs We might not have the benefit to do that in the class because of the research limitations you wanna talk about, but especially moving on from this class it’s often the case that you wanna train your model on many different devices at one go So how does this concept work if it’s instantiating or model on each of these devices, but we wanna share the variables So one naive way you might think of doing this is creating this variable’s dictionary at the top of your code that a dictionary of some strings into the variables that they represent And in this way if I wanna build locks below it that depends on this parameters I would just use this dictionary I would call variables_dic and I would take the key as these values And that might be how I would want to share my variables But there are many reasons this is not a good idea And it’s mostly because it breaks the encapsulation So what the code that builds your graph’s intensive flow should always have all of the relevant information about the nodes and operations that you are using You want to be able to, in your code document the names of your neurons You wanna be able to document the types of your operations and the shapes of your variables And you kind of lose this information if you just have this massive variables dictionary at the top of your code So TensorFlow inspired solution for this is something called variable scope A variable scope provides a simple name spacing scheme to avoid clashes, and the other relevant function to go along with that is something called get_variable So get_variable will create a variable for you if a variable with a certain name doesn’t exist Or it will access that variable if it finds it to exist So let us see some examples about how this works Let me open a new variable scope called foo And I’m gonna called get_variable with the name v So this is the first time I’m calling get_variable on v, so it’s going to create

a new variable and you’ll find that the name of that variable is foo/v So kind of calling this variable scope on foo It’s kind of accessing a directory that we’re calling foo Let me close that variable score an reopen it with another argument called reuse to be true Now if I call get_variable with the same name v, I’m actually going to find the same variable, I’m gonna access the variable, I’m gonna access the same variable that I created before So you will see that v1 and v are pointing to the same object If I close this variable scope again, and reopen it, but I set reuse to be false, your program will absolutely crash if I try to run that line again Because you’ve set it to not reuse any variables so it tries to create this new variable but it has the same name as the variable we defined earlier The uses of variable scope will become apparent in the next assignment and over the class but it is something useful to keep in the back of your mind So in summary, what have we looked at? We learned how to build a graph in TensorFlow that has some sort of feedforward or prediction stage where you are using your model to predict some values I then showed you how to optimize those values in your neural network, how TensorFlow computes the gradients, and how to build this train_step operation that applies gradient updates to your parameters I then showed you what it means to initialize a session, which deploys your graph onto some hardware that creates like the runtime environment to run your program I then showed you how to build some sort of simple iterating schedule to continuously run and train our model Are there any questions up to this stage before we move on in this lecture? Yes? It doesn’t because in feed_dict, you can see that, in feed_dict it takes in some node We’re not really understanding feed_dict with what the names of those variables are So whenever you create a variable or a placeholder There’s always an argument that allows you to give the name of that node So when you create the name of that node, not name in my Python variable, but name as a TensorFlow symbol That’s a great question, the naming scope changes the name of the actual symbol of that operation So if I were to scroll back in the slides and look at my list of operations The names of all those operations will be appended with foo as we created earlier Maybe one more question before we move on if there’s anything? Yes? Yes, if you load a graph using the get variable, it will call the same variable across devices This is why it’s extremely important to introduce this idea of variable scope to shared devices One more question [BLANK AUDIO] >> Can we share variables- >> The question was, can we share variables across sessions? I believe the answer to that question is correct I might be wrong But I’m not entirely sure, as of this time Okay, so we just have a couple of acknowledgements When we created this presentation, we consulted with a few other people who have done TensorFlow tutorials Most of these slides are inspired by Jon Gauthier in a similar presentation he gave We also talked with Bharath and Chip Chip is teaching a class, CS20SI, TensorFlow for Deep Learning Research So we are very grateful to all the people we talked with to create these slides And now, we will move on to the research highlights, before we move on to the live demo >> Hi everyone Can you guys hear me okay? Hi my name’s Alan And let’s take a break from TensorFlow and talk about something also very interesting I’m gonna present a paper called Visual Dialog Here’s a brief introduction Basically, in recent years we are witnessing rapid development improvement in AI Especially in natural language processing and computer vision And many people believe that the next generation of intelligent systems Will be able to hold meaningful conversations with humans in natural language based on the visual content So for example it should be able to help blind people to understand their surroundings by answering their questions Or you can integrate them together with AI assistants such as Alexa to understand people’s questions better And before I move on to the paper, let’s talk about some related work

There have been a lot of efforts trying to combine natural language parsing and computer vision And the first category is image captioning Here, I’m gonna introduce two works The first one is a paper called Show, Attend, and Tell Which is an extension of another paper called Show and Tell with some attention mechanisms And the second one is an open-source code written by Andrej Karpathy In both models the models are able to give you a description of the image And for the second case, that’s a typo right there It should be Video Summary Basically, the model’s able to summarize the content of the video So imagine if you are watching a movie and you don’t wanna watch the whole movie You wanna see what’s the main content of the movie This model would be pretty useful And in this category is Visual-Semantic Alignment So instead of giving a description for each image, this model actually gives a description for each individual component in the image And as we can see on the right, the data collection process, it’s very tedious Because you actually need to draw a lot of boundary boxes, and give a description to every single one And the next one is more related to our paper, which is called visual question and answering Basically given an image and a question, the model answers the question based on the visual content And in this case as you can see the answers are either binary, yes or no, or very short So one number, or a circle or different types of shapes And this paper, Visual Dialog, actually tries to solve the issue I just mentioned And it proposes a new AI task called Visual Dialog Which requires an AI agent to hold meaningful conversation with humans based on the visual content And also develop a novel data collection protocol And in my opinion, this is the best invention ever Because you make contributions to science, make money, and socialize with people all at the same time And it also introduces the family of deep learning models for visual dialog And I’m not gonna go into too many details today, because we are gonna cover deep neural networks later in this class This model encodes the image using a convolutional neural network And encodes the question and the chat history using two recurrent neural network And then concatenates three representations together as a vector It is then followed by a fully connected layer and a decoder which generate the answer based on the representation And here’s some analysis of the dataset As you can see the dataset is much better than the previous work, because there are more unique answers And also the question and answers tend to be longer, and here are some results They actually show the model in the form of a visual chat bot Basically you can chat with a robot online, in real time And if you guys are interested, please try it, [LAUGH] and that’s it >> [APPLAUSE] >> All right, let’s get started then So we’re gonna start with linear regression I’m sure all of you if you have taken CS 221 or CS 229 then you have heard and coded up linear regression before This is just gonna be a start to get us familiarized with TensorFlow even better So we’re gonna start at, what does linear regression do again? It takes all your data and tries to find the best linear fit to it So imagine house prices with time for example or location, it’s probably a linear fit And so we generate our data set artificially using y equals 2 x plus epsilon where epsilon is sampled from a normal distribution I won’t really go much into how we obtain the data Because that, we assume, is normal Python processing and not really TensorFlow so we will move on And actually start implementing linear regression and the function run So in this first function, linear regression, we will be actually implementing the graph itself As [INAUDIBLE] said, we will be implementing and defining the flow graph So let’s get started So, first, we’re gonna create our placeholders, because we’re gonna see how we can feed in our data So, we have two placeholders here, x and y So, let’s just start with creating x first, and this is gonna be of type float,

so we are gonna make float32 And it’s gonna be of shape So we’re gonna make this likely more general and have it of shape None And what this means that you can dynamically change the number of batches that you can send to your network Or in this case, your linear model And it’s just a row vector here All right, and we’re gonna name it x We’re gonna create y which is the label and which will also be of the same type and shape as well All right, and we’re gonna name it y All right, so now that we have defined our placeholders, we’re gonna start creating other variables So we start with first by defining our scope So let’s say tf.variable_scope, and we’re gonna name it just a lreg, because linear regression And we’re gonna call it scope, all right? So now that we are here, we’re gonna create our matrix which is w So we’re gonna call it tf.Variable and since it’s just a linear regression, it’ll just be a single integer or not an integer, my bad, but just one number And we’re gonna randomly initialize it with np.random.random We’re gonna start with the normal distribution rather Let’s do that, yeah And we’re gonna call it w Now we’re gonna actually build the graph now that we have defined our variables and placeholders We’re gonna define y_pred which is just prediction and it’s gonna be given by tf.mul(w, x) So far so clear, any questions? Yes Yeah, so as I mentioned earlier, none in this case is so that you can dynamically change the number of batches that you can send to your network So imagine like if I’m doing hyper parameter tuning, I don’t want to go and change shape to be 10 or 32 or 256 later on Almost, you can imagine that you’re dynamically saying that okay, I’m gonna change the number of batches that I’m gonna send to my network Does that answer your question? Yes So as we mentioned, it’ll just go into the variable scope and then define the name as it pleases, so, yeah All right, so let’s go ahead So now that we have our prediction, the next logical thing is to actually compute the loss So this we are gonna do by just, there are two norm So let’s just first get the norm itself And that’s gonna be given by square, and we just do, (y_pred- y) And since we wanted to be of a particular shape, its gonna be over reduce some Let’s reduce_mean rather, all right Okay, so now, with this we have finished building our graph And so now we’ll return x, y, y_pred, and we’ll return the loss as well from our linear regression model Now we’re gonna actually start computing what’s in the graph And we first start by generating our dataset And I’m gonna fill in code here, which we’ll define the training procedure for it All right, so let’s get started on that part So first we get what we call the model We make an instance of it, and that’s just gonna be given by this All right, so once we have that, we are gonna create our optimizer And this is gonna be given by, as Barack mentioned earlier in his slides, DescentOptimizer And we are gonna define the learning rate to be 0.1 And we are gonna minimize over the loss that we just got from our model All right, any questions so far? We just created our optimizer Okay, now we are gonna start a session So with tf.Session, As session And we are first gonna initialize, yeah, that’s one thing I forgot We are gonna first initialize our variables as someone earlier asked Why would we do that? So this is actually a new function So it’s likely different from what Barrack mentioned And this sort of explains our tens of our base really quickly, and since the time Barrack made the slide then I need the code It’s already been updated so we’re going to change that [Albeit so this is just initializing the variables here Initializer, all right So we created a session And now, we are gonna run the init function which is just initialization of variables

Now, we are gonna create our feed_dict And so what we are gonna feed in is essentially just x_batch and y_batch which you got from our regenerate dataset function And y here would be y_batch All right, now we’re gonna actually just loop over our data set multiple times, because it’s a pretty small dataset 30s is just our arbitrary chosen here We are gonna get our loss value and optimize it I’ll explain the step in a second So now we’re gonna call run and what we want to fetch is the loss and the optimizer and we are gonna feed in our feed dict Does anyone have any questions on this line? All right, and we are just gonna print for the loss here And then since this is an array, we are just going to want the mean because we have almost 101 examples All right, so now that we’re done with that, we can actually go and train our model, but we’d also like to see how it actually ends up performing So, what we are gonna do is we are gonna actually see what it predicts, and how we get that is, again, calling the session.run on y_pred So we are gonna fetch y_pred And our feed dictionary here will be just this All right Yes So the optimizer was defined as a GradientDescentOptimizer here So you can see we are not returning anything for that, which is why I just ended up with a blank there It’s just a syntax here So over here you see I’m returning nothing over there Yeah, all right, so we can actually go and start running our code, and see how it performs, okay? All right So let’s actually go and run our [INAUDIBLE] Let’s see how that performs So you see the loss decrease, and we can actually go ahead and see how it turns out Okay, I guess it didn’t like my tmux Anyways, So you see we fed a linear line over the data All right, so far so good All right, so now we are actually gonna go and implement word2vec using Skip-gram which is slightly gonna be more complex This was just to see how we create very simple models in TensorFlow All right, let’s go ahead So now Any questions so far? Yes? All right So, in this This is refine our understanding of word2vec again If you have the following sentence A completely unbiased statement here The first cs224 homework was a lot of fun And if you were suppose to make a dataset out of this sentence here We would have Consider a window size one here And we are going have the cs221 For a 224 end [INAUDIBLE] first And so we are just basically decomposing other sentence into a data set Remember that Skipgram tries to predict each context word given this target word And since the number of context words here is just two, because our window size is one And so the task now becomes to predict D from cs224n From first, a lot And fun from off and so on And so this is our data set So just clarifying what word2vec actually was Alright So let’s go ahead and start implementing that I’ve already made the data processing functions here So we won’t have to deal with that We have our batches And this function load data already loads the pre-process training and validation set Training data is a list of batch inputs and their label pairs And we have about 30,000 of those And we are going to see a train as well here The valuation data is just a list of all validation inputs And the reverse dictionary is a Python dictionary from word index to word Right? So let’s start and go ahead and implement Skipgram first All right So we are, again, going to start by defining our placeholders

And so this is going to be batch inputs And we are going to define a placeholder here But in this case, since you just have integers We can define with int32 And the shape is going to be batch_size and nothing So we have that And we can avoid naming here Because we are not going to call multiple variable scopes That will be fine Then we go and create our batch labels Which is, again, tf.placeholder And 32 This will also be of the same shape as previous And finally, we will go and create a constant for our validation set Because that is not going to change anytime soon And that is going to be defined by a val_data which we previously loaded And we have to define what type it is And the type for that is int32 again Just like our training set All right Now that we have defined, yes? So since I’ll be applying transposes later I just wanted to make sure that it’s one It doesn’t really make that big of a difference So in this case, I’ll be calling transpose on labels Which is why I just wanted to make sure that it transposes fine You wouldn’t It’s just, I wanna make it absolutely clear that it’s a row vector Not a column vector >> [Question] >> Yeah, exactly All right So now we can go and start creating our scope for All right So, this is where we’ll define our model And first, we are going to go and create an embeddings, as we all did in our assignment And that’s going to be a huge variable And it’s going to be initialized randomly with uniform distribution And this is going to kick vocabulary size, which you have previously defined in the top And it’s going to take embedding size So this is going to be a number of words in your dictionary times the size of your embedding size And we are going to define Since it’s a randomly uniform distribution We just going to also give the parameters for that So far so good? All right, so we just created our embeddings Now, since we want to index with our batch We are going to create batch embeddings And you are going to use this function Which is actually going to be pretty handy for our current assignment And so we do an embedding lookup with the embeddings And we are going to put in the batch inputs here All right Finally, we go and create our weights And we are going to call it, here, .variable, here So we are going to use truncated normal distribution here Which is just normal distribution where it’s cut off at two standard deviations instead of going up the internet Okay All right This is also going to be of the same size as previously But this is going to be vocabulary size and embedding size And this is because I turn tracks with our input directly Since this is truncated normal, we need to define what the standard deviation is And this is going be given by one over the square root of the embedding size, itself [BLANK AUDIO] Okay Finally we go and create our biases, Which are also going to be variables And this is going to be initialized with zeros of size vocabularies All right Now we define our loss function, now that we have all our variables So in this case, we used a soft max cross entropy in our assignment are the negative log likelihood In this case, you’d be using something similar And this is where Tensorflow really shines It has a lot of loss functions built in And say we are going to use this called negative constraint, negative concentrate I forgot the exact name But it is very similar, in the sense that the words that need to come up with a higher probability are emphasized And the words which should not appear with lower probability are not emphasized And so we are going to call tf.nn

Nn is the neural network library in TensorFlow, our module And this is going to take a couple of parameters You can look up the API Yes? Embeddings? All right What vector presentation Which is what you’re trying to learn No, w is the weight matrix that is a parameter that you’re trying to also learn But it’s interacting with other presentations Effectively, you can think of these embeddings as sort of semantic representations of those words, right? Yes? Right So, imagine So, our embeddings is defined as the vocabulary size So let’s say we have 10,000 words in our dictionary And each row is now the word vector that goes with that word Index of that word And since our batch is only a subset of the vocabulary, we need to index into that EH matrix With our batch, which is why we used the embedding lookup function, okay All right, so we’re gonna go and just use, this API obviously everyone would need to look up on the TensorFlow website itself But what this would do is now take the weights and the biases and the labels as well Okay I defined them as batch_labels And they also take an input, which is batch_inputs Okay And so here’s where TensorFlow really shines again, the num_sampled So in our data set, we only have positive samples, or in the sense that we had the context words and the target word We also need context words and noisy words And this is where num_samples will come in use We have defined num_samples to be 64 earlier And what it would essentially do is look up 64 words which are not in our training set and which are noise words And this would serve as sort of negative examples so that our network learns which words are actually context words and which are not And finally, our num_classes is defined by our vocabulary size again All right With that, we have defined our loss function And now we have to take the mean of that, because loss needs to be the size of the batch itself And we get that by reduced mean This is gonna be slightly nasty So we get, the loss is given for that particular batch And yes, exactly It’s given for multiple samples And since we have multiple samples in a batch, we want to take the average of those Exactly Okay And, so great And now we have completely defined our loss function Now we can go ahead and actually, if you remember from the assignment, we take the norm of these word vectors So let’s go ahead and do that first So that will be reduce_mean, this is just API calling, which is very valuable and detailed on the TensorFlow website itself All right Keep_dims=True So this is where, in this, I have added an argument called keep dimensions And this is where, if you sum over a dimension, you don’t it to disappear, but just leave it as 1 Okay And now we divide the embeddings with the norm, to get the normalized_embeddings embeddings/norm Great And now we return from, we get batch inputs, we return batch labels because this will be our feed We have normalized embeddings And we have loss All right With this done, we can come back to this function later There’s a slide part missing, which we’ll get back to Yes? Thank you All right So now we go and define our run function How are we doing on time? Okay, we have 20 minutes, great We actually make a object of our model

And that’s just by calling Embeddings And loss from our function, which was just called word2, or skipgram, rather Okay, and now we initialize the session And over here, again, I forgot to initialize our variables We can call We just initialized all of our variables for the default values, as Barak mentioned again Now we are gonna go and actually loop over our data to see if we can actually go ahead and train our model And so let’s actually do that first step And batch_data Train_data So for each iteration in this for loop, we are gonna obtain a batch, which has its input data as well as the labels Okay, so we have inputs and labels from our batch Great And we can now define our feed_dictionary accordingly where the batch_inputs So this is a dictionary And our batch_labels would just be labels Any questions so far? Okay We go ahead and call our loss function again And we do this by calling session.run, where we fetch the optimizer again and the loss And we pass in our feed dictionary, which we already defined above Okay We’ll get the loss And since we are trying to get the average, we’re gonna add it first and then divide by the number of examples that we just saw All right So we’re just gonna put a couple of print statements now just to make sure, to see if our model actually goes and trains And see Print loss Step [INAUDIBLE] and then average loss Since the loss will be zero in the first step, we can just [INAUDIBLE] All right, so now we have our average loss And we reset our average loss again, just so that we don’t for every iteration loop Okay So we have almost finished our implementation here However, one thing that’s, yes? I forgot to define that Good call So we can define that as the beginning of a run step Gradient Optimizer And we’ll take a learning rate of zero, and we’re gonna minimize the loss All right, thanks for that, okay One thing that we’re missing here is we haven’t really dealt with our value addition set So, although we are training in our training set, we would wanna make sure that it actually generalizes to the value addition set And that’s the last part that’s missing And we just gonna do that now But before we do that, there’s only one step missing Where we, once we have the validation set, we still need to see how similar our word vectors are with that And we do that in our flow graph itself So, let’s go back to our skip gram function Anyway here we can implement that, okay So, we have our val_embeddings against index into the embeddings matrix to get the embeddings that correspond to the validation words

And we use the embedding look up function here, embedding_lookup, embedding and we call in train data set or val data set We’ll actually use the normalized_embedding because we are very concerned about the cosine similarity and not necessarily about the magnitude of the similarity Okay, and we use val_dataset here Okay, and the similarity is essentially just a cosine similarity So, how this works is we matrix multiply the val_embeddings which we just obtained and the normalized_embeddings And since they won’t work, you can just may just multiply both of them because of dimensional incompatibility, we’ll have to transpose_b And this is just another flag All right, since we also returned this from our function, again this is just a part of the graph And we need to actually execute the graph, in order to obtain values from it And here we have similarity Okay, and let’s do, since this is a pretty expensive process, computationally expensive, let’s do this only every 5,000 iterations All right So, the way you’re gonna do this is by calling eval on the similarity matrix, what this does is, since we had this noted it actually goes and evaluates This is equal on to calling session.run on similarity and fetching that, okay So, we go on call, and we get similarity, and for every word in our validation set, You gonna find the top_k words that are closest to it And we can define k to be 8 here And we will now get the nearest words So let’s do that And we’ll sort it according to their magnitude And since the first word that will be closest will be the word itself, we’ll want the other eight words and not the first eight words and this will be given by top_k+1, any questions so far? Yes? Right, so your embedding is on number of words you have in your vocabulary, times the size of your word embedding for each word So, it’s a huge matrix, and since your batch that you’re currently working with is only subset of that vocabulary, this function, embedding to lookup, actually indexes into that matrix for you and obtains the word This is the equivalent to some complicated Python splicing that you do with matrices, but it’s just good syntax should go over it Okay, all right, almost there, okay So, we have our nearest key words, we’ll just go and, I have this function in my utils, you can check this on the GitHub that we’ll post after the class is over And you can play with it as you wish, and In the past a nearest, and a reverse_dictrionary gesture actually see the words and not just numbers, all right Finally, be open our final_embeddings which will be a normalized_embedding at the end of the train and we’re just going to call eval on that again which is equal to calling session.run and passing and fetching this All right, we are done with the coding and we can actually see and visualize how this performs Okay And python word2vec Oops, I must have missed something I missed a bracket again So, we’ll first load up our data set, and

then it will iterate over it and we will use our scripting model Oops, let’s see that All right, where did this group? You know why, please tell me why have to be there, okay Perfect So, as you can see here we have 30,000 batches each with a bad set of 128 Ahh, man, [LAUGH] Let’s see All right, so as we see, the loss started off as 259, all right, ends up at 145 and then decreases, I think it goes somewhere to around 6 Here we can also see as a printing, the nearest word for this is e leaders orbit, this gets better with time and with more training data We only use around 30,000 examples, so there’s a lot of potential to actually get better And in the interest of time, I’m only limited to around 30 epochs, yes So TensorFlow comes with TensorBoard, which I didn’t show in the interest of time Essentially, you can go up to your local host, and then see the entire graph, and how it’s organized And so, that’ll actually be a huge debugging help, and you can use that for your final project Enter bold, yeah All right, well, thank you for your time >> [APPLAUSE]