Hello everyone welcome to the course of machine learning with Python. In this video, we shall learn about some advanced revision of our first topic is normal equation. Now, the training data set can be written as follows, there are total key many features and in many training samples notice that we have added one extra feature column extra row with all values one to the left, the training samples can now be written as x superscript r comma y superscript is from one to n x superscript i is nothing but victor of the values extra x one x two up to escape, notice that these external superscript will be always one for all items from one to n. Now the equation y equals two theta zero plus theta one x one plus beta two x two up to theta k x k can be written in the vector form as y equals to theta transpose x, where capital theta is nothing but the victor of the model parameters suitable theta is theta zero theta one theta two theta K, notice that we have used the transpose in order to denote that this is a column vector.

Now, in linear regression model, we are trying to estimate the model parameter vector from the given set of data, let the estimated parameter vector v theta hat and the corresponding predicted values the Y hat then in vector matrix notation y hat will be equals to the victim of the predicted values y hat one y hat two up to y hat aim, which is nothing but this matrix of x multiplied with the theta. So, y hat Victor is equals to x multiplied with that theta hat. So, what is this x matrix this is the x matrix we have already seen this now, the mean square error cost function in matrix notation can be written as forming. So, what is this x theta hat It is nothing but y hat I have already pointed it out. So, it is Y hat minus y whole transpose multiplied with y hat minus one.

So, it is a scalar okay. So, ultimately what we are actually trying to do is Y hat minus y whole square. So I am taking the norm of the vector y hat minus y and then squaring it up. So that will give me the mean squared error cost function. Note that y hat equals to x theta hat is the victor of the predicted values. And why is the victor of the actual values now, we are trying to simplify the cost function.

So, our first tip would be within these parentheses we'll move the transpose operation. So, now, it will become within these parentheses x theta hat transpose minus y transpose. Now, we shall multiply these first parentheses as a second penalty that means, the first term and the second term So, this is the expanded form Okay. Now, note that Eb pole transpose is nothing but V transpose multiplied with a transpose. So, x theta hat full transpose will be equals to theta transpose x transpose again note that y can Suppose multiplied with x theta hat and x theta hat whole transpose multiplied with y vector are scalars. And if you produce the transpose of these particular terms is nothing but this term and we know that the transpose of a particular scalar will be the scalar itself.

So, we can say that y transpose multiplied with x theta hat is equal to x theta hat whole transpose multiplied with y again using the same principle, we can expand it as theta hat transpose X transpose Y. So, that is why I am getting an Lu over here. Now, if we differentiate this cost function with respect to theta hat and save the derivative to zero what we shall get so, we shall get x transpose x theta hat is equal to x transpose one or in other words theta hat is equal to X transpose X inverse multiplied with X transpose Y as you bring that extra Suppose x is invertible. So, this is our normal equation from where we can actually find the model parameters directly if we know the feature matrix X and the corresponding actual values y. Now, between gradient descent a normal equation, which one is preferable, though the normal equation directly gives the solution without iteration like gradient descent, it has many drawbacks for example, for large data set computing X transpose X inverse is a costly operation.

Moreover, X transpose X is non invertible we cannot use normal equation directly as the water in the case when X transpose X is non invertible is to use pseudo inverse pseudo inverse is a numerical method based on some matrix operations and linear algebra, which will compute the approximate inverse of all non invertible matrix gradient descent is more popular and good choice for solving linear regression problem though gradient descent has its own demerits, like it is susceptible to local minima, and we have to choose certain parameters like learning rate, and it is not trivial to choose learning rate when we have different kind of cost versus iteration costs. So, but overall gradient descent is more popular, and we'll be using that for solving the linear regression problem. Okay, so our next topic is polynomial regression. Consider the following example, where I have shown the scatter plot of a biomedical data, we can fit a straight line through the data points of the form y equals to theta zero plus theta one x, but we can do better, right if we feed a second order polynomial of the form y equals to theta zero plus theta one expected a two x squared or if we feed a third polynomial of the form y equals to theta zero plus theta one expositor two x squared plus two to three x three.

In general, we can feed in it order polynomial To the data points, so the model for the electoral polynomial is y equals two theta zero plus theta one x plus theta two x squared plus beta three x cubed plus theta n x to the power n. Now smaller the value of n, the complexity of the model is less, but the model may not fit the data set appropriately. So we have to choose in accordingly, such that we get reasonably good feet within this complexity, we can convert the polynomial intuition problem into multiple linear decision problem just by assigning x one equals two small x x two equals two x square x equals two x cube and x in equal straights to the bar in and then construct the multiple linear regression model as y equals to theta zero plus sum over theta i x i have items from one to n. Note that x i is nothing but x to the power.

For more than one predictor variables, the polynomial regression becomes more complicated for two predictor variables x one and x two the generalized form of signal polynomial becomes y equals two theta zero plus theta one x one plus beta two x two plus three x two x one x two plus theta four x one square plus feet of high x two square. So for two variables two predictor variables x one and x two, for a second order polynomial, it become six factor model, which is quite complex. Let's move ahead and discuss about something called coefficient of determination to determine the goodness of the feet in a linear regression model, we use a quantitative measure and that is called coefficient of determination or r square score, it is defined as follows. So let their smallest number of data points and y equals two y one y two y three up to y n is the vector of actual values of the target variable and Y hat equals two y one hat y two y three hat up to y m is a vector of predicted values of the target variable.

Note that I have used bold faces of y and y hat in order to evoke that These elevators and these without bold faces these denote the scales Okay. Now let y bar is the mean of the target variable then the total sum of squares is defined as follows the total sum of squares is equals to y minus y bar whole square and we sum it from items from one to M, the total sum of squares is proportional to the variance of the target variable. Now, we already know what is called as it was our square. So how do we compute residual sum of squares. So, residual sum of squares is equals to actual minus predicted whole square and we take this summation for all data samples. So in this case, it is y minus y i had whole square items from one to m. Another faction of unexplained variance is defined as if u equals to RSS by TSS so residual sum of squares divided by two to solve square the coefficient of determination or r square also called the fraction of explained variance.

Is defined as r squared equals to one minus fraction of unexpressed videos that is if you V which is equals to R SS by cases so r square is equal to one minus R is by TSS coefficient of determination r square lies between zero to one closer to the values of r squared to one regression model fits better to our data set and can better explain the observed variability of the target variable. smaller value of r squared implies that that equation model is not that good. It can be shown that for my video data set r square is equals to square of the correlation coefficient between the predictor and the target variable. So, so far this one in the next video, we shall learn about how to code these normal equation in Python. So see you in the next lecture. Thank you.