Hello everyone, welcome to the course of machine learning with Python. In this video we shall discuss about multiple linear regression. So what is the problem statement suppose there are key many features or predictor variables such as x one x two x three up to x k and our target variable y. Now, we want to explain why as a linear combination of all those predictor variables, we have a vector of features or creditors that means x vector is equal so, the x vector is equals to x one x two x three up to x k transpose. So, why transpose because we know that this should be a column vector. Now, the model representation here we can express our multiple linear regression problem by means of following probabilistic model y equals to theta zero plus theta one x one plus beta two x two plus beta three x three plus up to theta k x k plus epsilon.

So, epsilon is here Random residual error. So, here we have a vector of the model parameters, which is theta zero theta one theta two up to theta K and the other total key plus one model parameters. So, the model parameter vector is of dimension k plus one and as I said epsilon is the randomness to it, like in the case of simple linear regression, here also we shall use our sample data to estimate the model parameter vector theta that is our estimated model parameter vector will be theta hat note that this is a capital theta to denote the victim. Now the multiple linear regression as options so the predictor should be uncorrelated and linearly independent of one another. That means if one predictor depends on another variable, we simply discard that predictor the models error term should have constant variance the error term is preferably normally distributed with zero mean now the hypothesis function, the prediction value of the target variable is obtained from the estimated model parameters as y hat is equal to beta zero hat plus beta one hat x one plus beta two hat x two plus up to theta k hat x k that means, it is nothing but theta zero hat plus some over T does he have X g, where g runs from one to K now, the cost function here we shall define cost function in the following way.

So, g of the model parameter vector So, this is a function of the model parameter vector will be equals to one by twice in sum over the actual value of y minus the predicted value of y whole square items from one to n. So, there are total m number of training samples if I replace the value of predicted value of y by y hat is equal to theta zero plus theta t x ci g runs from one two key we shall have this equation of the cost function Note that here x Victor superscript i and Y superscript i is the IEA data sample and Y hat superscript i is the corresponding output of the model note that we have we are using superscript to denote the instance of the data or that particular data sample. Now, our objective is to estimate the model parameters theta cap from the given sample of data so, that the cost function g is minimized now, gradient descent algorithm for cost function optimization gradient descent algorithm is an optimization technique for minimizing the cost function J gradient descent is an iterative algorithm and in a nutshell it does the following.

So, we start with some random guesses or the some random initial values of theta which is a model parameter. It keeps on changing the values of theta that is to zero theta one head to head and up to Cheetah head to reduce the G We end up at the minima. Okay, so how do we do these algorithms okay. So, this is the algorithm of gradient descent. So, first the input is the feature set that means the values of x and the corresponding values of the target variable, that is the values of y the output will be the model parameters that we want to estimate for which the cost function is minimum now the initializations So, first we initialize the model parameters with some random values and we initialize iteration number t to be equals to one now, we should repeat until convergence the following steps. So, first we'll compute the cost function for the given value of the theta cap.

And for a particular theta j at t plus one iteration will be equals to the value of the theta j at TF iteration minus alpha times the gradient of the cost function with respect to theta j and will ceremoniously operate the same for every g equals to 012 up two key and then we increase the iteration number by one. And we repeat this tape until we get convergence. Okay, so we'll discuss about convergence a little later. So, here alpha is called a small positive insert different parameter and it is called the learning rate. Now, what is the intuition of the gradient descent algorithm the plot of the cost function with any of the parameters considering the other parameters fix is one of the following form we have already seen this now, as for the gradient descent rule, the update is So, this is what is called the update rule of the gradient descent.

So, theta t at t plus monetization will be equals to the value of theta j at TF iteration minus alpha times the gradient of the cost function with respect to theta J. Now, if the gradient is positive that means, this particular value is positive and we have already specified that alpha is a small positive number then this entire thing is positive then the upgrade in the value of theta z. So, what is The particular update so, the update is minus alpha Dale then theta j of capital T right. So, if this particular gradient is positive then the update in the value of theta j will be negative indicating that in the next update that theta z will be smaller than its previous family okay. So, we can pictorially represent it like this. So, let's say we first had to determine the value of theta j is here then So, this is the gradient we see that the gradient is positive okay.

So, in the next tape or in the next iteration, the value of the theta j will be deduced okay it is denoted by these red dots and again the gradient is positive, so, it will keep on decreasing. On the other hand, if the gradient is negative, then the update in the value of theta j is positive indicated that the next update theta z would be greater than it will be as well. So, we can see these big typically like this. So, here the gradient of the value of theta j is the If so, in the next iteration that it is available will be more than its previous value. So, as we can see, wherever we are in this particular graph will be always moving towards the minimum of the function okay. And as we reach to the minimum of the function the gradient will become zero and there will not be any update in the value of theta z okay.

So, this is the equation of the gradient descent algorithm. Now, the cost function of the linear regression with multiple parameter looks like this we have already specified we can calculate the derivative of the cost function with respect to theta zero and theta d for mpg runs from 123 up to k. So, how do we do that? So, this is basically the gradient calculation of cost function g with respect to t does zero head and this is the calculation of the cost function with respect to theta g for j equals 2123 up to K so, I will recommend to pause this video and obtain these result by yourself okay hence the upgrade rule in the gradient descent algorithm for the iteration will look like this fine because as you can see this is basically we are optimizing the gradient if we multiply these with the alpha and this will be the update.

So, this is basically the uptick rule of the gradient descent in the context of multiple linear regression or the naughties if we consider another predictor x zero whose value is always one, then we can use the last equation as the generalized gradient descent update rule. That means, if we add a particular column of one all ones at the left of the data set, then we can say that x cheedo is equals to one for all eyes and then we can use this last equation as a generalized rule for the gradient descent update in case of linear regression. Now, the stopping criteria gradient descent would be we can consider the following two methods as a stopping criteria for the descent elbow random one being the convergence of the model parameters. So, in two successive iterations, the change in the model parameter is not significant, then we can say that the algorithm has converged as the model parameters are vectors to compute the values of the model parameters in two successive iterations, we can either take norm or absolute differences.

Alternatively, if at least one of the parameters is not conversed, then we keep on running the algorithm until all the parameters are converged. And the second method would be predefined number of iterations where we run the algorithm for a predefined number of iteration So, let's the thousand iterations or 2000 traditions and whatever we get, we'll be happy with that okay. Now, how to check whether gradient descent is working or not, we can plot cost function versus the number of iterations if it is continuously decreasing, then we can say that the gradient descent is working however, we may have to fine tune the value of the learning rate that is alpha depending upon how fast flow the cost function is converging. So, let us consider the example here. So, here the cost function is decreasing smoothly and gradient descent is working properly. However, if we obtain a cost function was a traditional graph like this, which is oscillatory we may have to decrease the alpha and similarly, if it is too sluggish that means, the change in the cost function with respect to iteration is not that much then in order to speed up the gradient descent we may increase the alpha.

So, in both cases the gradient descent is not working properly right. Now, the implementation note choosing the value of learning rate alpha So, as we can see, there is no general apply to all value of the learning rate. However, we can choose the value of alpha as 0.001 0.003 so many values again this list is not exhaustive one may choose different values like 00 5.005 as a learning rate of openness And again another implementation note that is the feature scaling. So, linear regression and most of the other machine learning problems performs well when the range of the predictor variables are roughly in the same range. This is because if the range of the one predictor variable is significantly higher than the another predictor variable, then the first one starts to dominate over the second one making gradient descent algorithm sluggish hence, feature scaling is necessary for faster convergence of the gradient descent algorithm.

So, this is done either by the following two methods. So, one is called the standardization. So, for a particular predictor variable exchange, whose mean is exchange par, and the standard deviation is sigma x shaped therefore, features can replace x j with XT minus Xj bar all divided by sigma of exchange. So this ensures that the after scaling exchange will have zero mean and English Standard deviation or you can also do min max killing where we replace x with x j minus x j part divided by maximum value of Xj minus minimum value of exactly this is a range of the values of x j. These issues that after scaling Xj will lie between minus one to plus one. So in the next video, we'll see how to implement gradient descent in Python.

So see you in the next lecture. Thank you.