Hello everyone, welcome to the course of machine learning with Python. In this video, we shall learn about principal component analysis, also known as PCA. Let's look at first variance and covariance, variance and covariance are the measures of spread of a set of points around their center of mass or me variance measures the spread of feature around its center of mass for a feature x, which takes the values x one x two X, two x in the variance is calculated as variance of x is equals to one upon in sum over x i minus mu squared, that is from one to n, where mu is nothing but the mean of all these values excite that means movies nothing but one upon pain, sum over excite ideas from one to N, no covariance between two features how Watch each of the feature vary from the mean with respect to each other.

For example, consider the following table which records the values of two features x one and x two, okay x one can take the value x one one x one two x one three up to x one n and x two can take the values x two one x two two x two three and up to x two n. The covariance between x one and x two is nothing but one upon n, some of her items from one to n x one IE minus nu one multiplied with x two IE minus two where mew one or new two is equals to one upon n sum over items from one to N exchange I where g is either one or two. Okay, G equals to one we'll get new one j goes to visit. Often observation overviews of one feature with itself is nothing but It's Friday and that means covariance between X comma x is nothing but variance of x.

Now covariance matrix if we have three dimensional data set such as x one x two x three, then we can measure the covariance between x one and x two, x two and x three and x one and x three measuring the covariance between x one and x one x one and x two and x one x three and x three would give us the variances of x one x two and x three respectively. We can consider the following metrics of dimension three cross three which records the values of these variances and covariances This is known as covariance matrix. So, as you can see, the diagonal entries of this covariance matrix are nothing but the variances. So, covariance x one comma x one is nothing but variance of x one covariance of x two comma x two is nothing but variance of x two, covariance of x three comma x three is nothing but areas of extreme and the off diagonal elements and Nothing but the covariance between the different features or variables.

Similarly, for D dimensional data we shall have a D plus D covariance matrix. Now properties of covariance matrix covariance matrix is a symmetric matrix, because covariance between Xi comma x ship is nothing but covariance between exit comma exit okay and covariance matrix is positive semi definite that is all is a key values are non negative. This can be proved by linear algebra, which we are skipping over here. Now, principal component analysis let's talk about the intuition of principal component analysis. First, consider the data points as shown in the figure. This is basically split inside a two dimensional plane.

Now, let's look at the data set from a different perspective. Now, all the data points now shall be projected into the tilted orthogonal axes noted by u one and what is the uniqueness about a one ecosystem of coordinates as compared to the other You know x one comma x two system of coordinates, the variance of the data points is maximum along u one and the X is u one contains the maximum information about the data set, this is called the first principal component of the data set. Similarly, you two, which contains relatively less information is known as second principal component. Hence, the data set can be represented by only one feature that is human without much lots of information. This is how PCA acts as a dimension reduction technique. Note that the principal components are linear combinations of original bases of the data set.

His PCA is a linear feature extraction technique, while standardization or normalization is required for PCA. That means, before performing PCA we have to normalize the data. Now, why is this required? Usually the data set contains variables or features which are in different units of measurements. For example, we of a person measured in kg and the sugar level in milligram per deciliter. Now, if the unit of weight is changed to ground, then the weight column would now content values which is simply thousand times the weights in kg.

Now, we know that variance of some scalar multiply to the random variable x is nothing but that scalar squared times the variance of X Okay, here he is the scalar So, variance of E x is nothing but a square variance of x. Hence, changing the scale of measurement will change the variance significantly. However, the interrelationship between the variables will not change does few variables whose values are high in range will now dominate principal component and give the misleading directions of principal components. Hence, it is always advisable to perform normalization on the data set before applying PCA normalization of standardization we make sure showed that all the data falls in the same scale. Now steps for standardizing the data. So, consider this is our D dimensional data click New Victor is equals to new one comma move up to new D denotes the mean of the columns.

That means, new j is nothing but expectation of exchange data from one to D. Similarly, sigma kilos the standard deviation of the columns sigma filters are nothing but sigma one sigma two sigma d, here also sigma is nothing but square root of the headings of the column C. Now, we can make the data sets zero centered that means, we can replace the column by subtracting all the elements in the column with their mean okay, make each column unit variance This is done after the first tape that means the Judo center instead. That way in this case, we will divide the entire data set by its standard deviation, okay. Now, the variance of each column of the transformed data is one. So, after normalization the covariance matrix of the normalization data or the normalized data is simply nothing but the correlation matrix variance probe, let us assume that we have a zero set of data that means, expectation of X vector is nothing but zero let u is the unit vector along the direction of principal component and as he is a unit vector that means, its norm should be close to one or norm squared should also be close to one and Oscar is nothing but the dot product between you with you that is used transpose you should be close to one the projection of the exit on you return is a it is obtained as he is nothing but x transpose u transpose x.

Now let's see few statistical properties of a expected value. Have is nothing but expected value of u transpose x which is nothing but u transpose expected value of x because u is fixed in space with respect to x and as expected value of X equals to zero it is nothing but zero. Now, variance of A is expected value of a squared minus expected value of a whole square. Now, as expected value of E is equal to zero, so, it is simply expected value of a square. So, here is of E is nothing but the expected value of a dot A now, the first e I am writing as you transpose x while the second am writing as x transpose u okay. So, again you transpose and you are basically constant over space as compared to x.

So, we can take out of these are two the expectation equation and what remains is that the variance of Is you transpose expectation of x x transpose u okay. So, now expectation of x x transpose is nothing but sigma which is nothing but our covariance matrix that varies is a function of u okay let's call it shine you shine U is nothing but you transpose sigma you okay. This is known as variance scope or PCA as optimization problem. Our objective is to find a unit fitted you along which the variance of the data set is maximum as the various flow shy you estimates the variance of the data set along the direction of the EU. We can reformulate the PC as the following optimization problem, find the vector u, which will maximize these various probe shy you subjected to the constraint that u is a unit vector.

So, we have to maximize u transpose sigma u subject to the constraint that you transpose u is equal to one, this is a constant optimization problem and we shall use languages multiplied with thought to solve this. So, we shall make a lateral engine like this and we have to maximize the modified objective function for this we shall differentiate the left engine with respect to you has set it to close to zero. Now, as we have set it equal to zero, we can differentiate this whole expression and from some help of the standard formula of matrix calculus, we arrive at that the final expression is sigma u equals two lambda u does principal component analysis simply reduced to an eigenvalue problem? Here we have to find the eigenvalues and the corresponding Eigen vectors of covariance matrix of the normalized data set. So, we can find the median values or corresponding Eigen vectors as the dimension of the data set is D We dad is a convolution descending orders.

Let the inflows in descending order, sir lambda one, lambda two lambda two lambda d and the corresponding Eigen vectors are u one u two u three up to UT okay as part of auditing, we can see that the vector u one is known as the first principal component of the data set. Similarly, u two points was the second principal component of the data set. So, lambda j is the various along the Jade principal component and one interesting property of the eigenvectors are the principal component is that one unit or the principal component is orthogonal to another okay. So, the proof is written over here you can go through this proof and see that you i transpose u j is equal to zero that means the dot product between two principal components is close to zero. That means principal components are orthogonal to each other. Now, scree plot scree plot is a line plot of a game Palouse of correlation matrix in descending order.

So, this is a script Have some data set usually around the first principal component the datasheet shows maximum variance scree plot is a monotonically decreasing plot and helps us to return mine how many principal components are enough to faithfully represent the data without much loss of information. So these are the properties of script doc. So his usual scree plot, one can identify the best read KD presentation of the data set. Okay, so in the next video, we will discuss about how to implement principal component analysis in Python using this cannon library. So see you in the next lecture. Thank you.