Hello everyone, welcome to the course of machine learning with Python. In this video, we shall learn about dimensionality reduction. So, problem with very high dimensional data. This is also known as curse of dimensionality data at very high dimensional space becomes pass for highly sparse data machine learning algorithm does not give satisfactory results always this is simply because the available training data set might not have every possible combinations of the features. Hence, machine learning algorithm struggles to find any meaningful relationship among the features and the target variables. For example, consider a data set which has the mini categorical features with key many possible values, then, the number of possible combinations of the data point is k to the power d, which grows exponentially as the dimension increases.

Our training data set may not contain all sorts of possible combinations. So, what is the workaround of this problem? dimensionality reduction to find a suitable representation of the data in one dimensional space, which will eventually help us for machine learning, such as classification, clustering, regression and data visualization. There are several types of dimensionality reduction techniques. However, all those techniques can be broadly classified into two categories feature selection technique and feature extraction techniques. So, feature selection techniques, we select few features based on certain criteria, and thus get rid of redundant or less important features to make our data set less bulky.

Note that feature selection depends on the data set as well as on the task we want to perform with the data set feature extraction techniques we project or original data set which belongs to a high dimensional space to a low dimensional space such that the inherent information inside the data is not lost. This projection could be either linear or nonlinear. Based on which we have linear or nonlinear feature extraction techniques a very simple example of feature selection let's say a school student in a particular school can have following features x one name of the student x two contact details of the student x three class export roll number x five marks in mathematics, x six marks in language group x seven marks in social science group x eight marks in natural science group x nine height extent bodyweight excellent blood group etc. Each student can be represented by a vector such as x equals two x one x two x three whole transpose y transpose because we are considering column vectors.

Now, suppose our job is to find the suitable set of students for Science Olympiad which features play the important role in selecting the students for Science Olympiad. So, obviously not all of these features are important in this case, the most important features are marks in mathematics and marks in natural science cool. Now, let us want to select the students for sports to numb it. In this case, we have to select the students based on the height weight fitness and skill in that particular sports. So, based on the problems in hand, we can discuss some of the features and consider others. So, what is the objective of feature selection to opt in a set of features key out of in many features in the data set key usual is done in such that the selected features are based suitable for a required task D. Now, the actual value of k depends on many things such as the task itself.

That is if it is for visualization, then maybe we'll go for two or three features, the amount of the data that machine can handle also determines the value of k number of available training data points so that the data set becomes less parts in the reduced feature space. Excellent. QRA determines how much should be the value of key. That means in the modified he just pays how many features should present. Now how to select the features, we can use domain knowledge and experience to see a set of basic features suitable for a particular task exhaustive sets select the best possible set of K mini features out of all possible combinations of the features, these methods might work well if the number of features in the data set is not very high. Otherwise this method does not scale well.

If the number of features in the data set is too high heuristic search in this case, we keep on adding features which will provide based value of some criterion function. We stopped the algorithm till the required number of features are selected. Other such strategies such as randomized search are also used for feature selection. Now feature extraction let the original data point x belong to a higher dimension. Space capital D, we want to reduce the dimensionality of the data set into small d small d is less than capital D. in feature extraction, we shall project the data set from the higher dimension capital D to the lower dimension small d using some function which maps the higher dimensional space to learn what dimensional space such that the information contained inside the data set is preserved significantly. Here we are not selecting or discarding any particular feature.

The whole idea is to project the data set into some smaller dimensional subspace such that the interrelationship of the data points are preserved late capital X is our original data set in dimension capital D and small x vector is a data point late in the transform space, the dimension is small d and x prime is the corresponding data point that means, f of x vector is nothing but Vector prime if x one vector and x two vector A to neighboring points in the origin data set x then in the transform Space X one prime vector and x two prime vector should also be neighbors. There are various feature extraction techniques, few techniques employ linear transformation or projection of the data set for example, principal component analysis and linear discriminant analysis some techniques involve nonlinear transformation of the data set, for example, locally linear embedding or kernel PCA.

Now, let's look at some mathematical prerequisite Egon values and eigenvectors of a square matrix matrix vector multiplication consider the vector x one x one is denoted by two comma three in these two dimensional space. Now, the matrix E which is written over here if it is post quantified with x one then we are Another vector x one prime. So x one prime is equal to E x one, we can see that x one prime after computing this multiplication, it is nothing but five, seven. Okay. So in general, we can say that multiplying a vector with a matrix changes the direction and the magnitude of the vector. However, for a given square matrix, there are some factors whose direction remain unchanged.

After multiplying with the matrix, consider the fixed term x two which is nothing but one one. So this is our x two vector. If we multiply a with x two, then the transform factor x two prime is nothing but three three, which is nothing but three times one one, okay. So we can see that the magnitude of the victor has been changed after multiplying the vector with the matrix, but the direction remains unchanged. That is the transformed vector x two prime is nothing but a scalar multiplication of the original vector x to know for a given square matrix E, if we find a vector v, such that the mighty plane a with vector v does not change the direction of the vector v, but scales it, then the vector v is known as the eigenvector of a and the scalar value by which it will be scaled after being multiplied with a is known as the corresponding Eigen value.

Mathematically, if e times V is equals to lambda times p, then lambda is called the eigenvalue and v is the Eigen vector of matrix A note that lambda here is a scalar. Now, e times v is equal to lambda times V, we can write this as e minus lambda times I hold times P is equals to zero. Here AI is nothing but the identity matrix of the same Order of matrix A for non trivial solution of the eigenvectors determining value of A minus lambda I should be equals to zero, the above equation is a polynomial of lambda of degree same as the order of the matrix A. This equation is also known as the character characteristic equation of matrix A, we can calculate the eigenvalues by solving the characteristic equation Putting the value of the Eigen value one can calculate the corresponding Eigen vectors. In the next video, we shall learn about principal component analysis.

So, see you in the next lecture. Thank you