In this article, we discuss how Principal Component Analysis (PCA) works, and how it can be used as a dimensionality reduction technique for classification problems. At the end of this article, Matlab source code is provided for demonstration purposes.

In an earlier article, we discussed the so called Curse of Dimensionality and showed that classifiers tend to overfit the training data in high dimensional spaces. The question then rises which features should be preferred and which ones should be removed from a high dimensional feature vector.

If all features in this feature vector were statistically independent, one could simply eliminate the least discriminative features from this vector. The least discriminative features can be found by various greedy feature selection approaches. However, in practice, many features depend on each other or on an underlying unknown variable. A single feature could therefore represent a combination of multiple types of information by a single value. Removing such a feature would remove more information than needed. In the next paragraphs, we introduce PCA as a feature extraction solution to this problem, and introduce its inner workings from two different perspectives.

PCA as a decorrelation method

More often than not, features are correlated. As an example, consider the case where we want to use the red, green and blue components of each pixel in an image to classify the image (e.g. detect dogs versus cats). Image sensors that are most sensitive to red light also capture some blue and green light. Similarly, sensors that are most sensitive to blue and green light also exhibit a certain degree of sensitivity to red light. As a result, the R, G, B components of a pixel are statistically correlated. Therefore, simply eliminating the R component from the feature vector, also implicitly removes information about the G and B channels. In other words, before eliminating features, we would like to transform the complete feature space such that the underlying uncorrelated components are obtained.

Consider the following example of a 2D feature space:

Figure 1 2D Correlated data with eigenvectors shown in color.

The features and , illustrated by figure 1, are clearly correlated. In fact, their covariance matrix is:

In an earlier article we discussed the geometric interpretation of the covariance matrix. We saw that the covariance matrix can be decomposed as a sequence of rotation and scaling operations on white, uncorrelated data, where the rotation matrix is defined by the eigenvectors of this covariance matrix. Therefore, intuitively, it is easy to see that the data shown in figure 1 can be decorrelated by rotating each data point such that the eigenvectors become the new reference axes:

(1)  

Figure 2.2D Uncorrelated data with eigenvectors shown in color.

The covariance matrix of the resulting data is now diagonal, meaning that the new axes are uncorrelated:

In fact, the original data used in this example and shown by figure 1 was generated by linearly combining two 1D Gaussian feature vectors and as follows:

Since the features and are linear combinations of some unknown underlying components and , directly eliminating either or as a feature would have removed some information from both and . Instead, rotating the data by the eigenvectors of its covariance matrix, allowed us to directly recover the independent components and (up to a scaling factor). This can be seen as follows: The eigenvectors of the covariance matrix of the original data are (each column represents an eigenvector):

The first thing to notice is that in this case is a rotation matrix, corresponding to a rotation of 45 degrees (cos(45)=0.7071), which indeed is evident from figure 1. Secondly, treating as a linear transformation matrix results in a new coordinate system, such that each new feature and is expressed as a linear combination of the original features and :

(2)  

and

(3)  

In other words, decorrelation of the feature space corresponds to the recovery of the unknown, uncorrelated components and of the data (up to an unknown scaling factor if the transformation matrix was not orthogonal). Once these components have been recovered, it is easy to reduce the dimensionality of the feature space by simply eliminating either or .

In the above example we started with a two-dimensional problem. If we would like to reduce the dimensionality, the question remains whether to eliminate (and thus ) or (and thus ). Although this choice could depend on many factors such as the separability of the data in case of classification problems, PCA simply assumes that the most interesting feature is the one with the largest variance or spread. This assumption is based on an information theoretic point of view, since the dimension with the largest variance corresponds to the dimension with the largest entropy and thus encodes the most information. The smallest eigenvectors will often simply represent noise components, whereas the largest eigenvectors often correspond to the principal components that define the data.

Dimensionality reduction by means of PCA is then accomplished simply by projecting the data onto the largest eigenvectors of its covariance matrix. For the above example, the resulting 1D feature space is illustrated by figure 3:

Figure 3. PCA: 2D data projected onto its largest eigenvector.

Obivously, the above example easily generalizes to higher dimensional feature spaces. For instance, in the three-dimensional case, we can either project the data onto the plane defined by the two largest eigenvectors to obtain a 2D feature space, or we can project it onto the largest eigenvector to obtain a 1D feature space. This is illustrated by figure 4:

Figure 4. 3D data projected onto a 2D or 1D linear subspace by means of Principal Component Analysis.

In general, PCA allows us to obtain a linear M-dimensional subspace of the original N-dimensional data, where . Furthermore, if the unknown, uncorrelated components are Gaussian distributed, then PCA actually acts as an independent component analysis since uncorrelated Gaussian variables are statistically independent. However, if the underlying components are not normally distributed, PCA merely generates decorrelated variables which are not necessarily statistically independent. In this case, non-linear dimensionality reduction algorithms might be a better choice.

PCA as an orthogonal regression method

In the above discussion, we started with the goal of obtaining independent components (or at least uncorrelated components if the data is not normally distributed) to reduce the dimensionality of the feature space. We found that these so called ‘principal components’ are obtained by the eigendecomposition of the covariance matrix of our data. The dimensionality is then reduced by projecting the data onto the largest eigenvectors.

Now let’s forget about our wish to find uncorrelated components for a while. Instead, we will now try to reduce the dimensionality by finding a linear subspace of the original feature space onto which we can project our data such that the projection error is minimized. In the 2D case, this means that we try to find a vector such that projecting the data onto this vector corresponds to a projection error that is lower than the projection error that would be obtained when projecting the data onto any other possible vector. The question is then how to find this optimal vector.

Consider the example shown by figure 5. Three different projection vectors are shown, together with the resulting 1D data. In the next paragraphs, we will discuss how to determine which projection vector minimizes the projection error. Before searching for a vector that minimizes the projection error, we have to define this error function.

Figure 5 Dimensionality reduction by projection onto a linear subspace

A well known method to fit a line to 2D data is least squares regression. Given the independent variable and the dependent variable , the least squares regressor corresponds to the line , such that the sum of the squared residual errors is minimized. In other words, if is treated as the independent variable, then the obtained regressor is a linear function that can predict the dependent variable such that the squared error is minimal. The resulting model is illustrated by the blue line in figure 5, and the error that is minimized is illustrated in figure 6.

Figure 6. Linear regression where x is the independent variable and y is the dependent variable, corresponds to minimizing the vertical projection error.

However, in the context of feature extraction, one might wonder why we would define feature as the independent variable and feature as the dependent variable. In fact, we could easily define as the independent variable and find a linear function that predicts the dependent variable , such that is minimized. This corresponds to minimization of the horizontal projection error and results in a different linear model as shown by figure 7:

Figure 7. Linear regression where y is the independent variable and x is the dependent variable, corresponds to minimizing the horizontal projection error.

Clearly, the choice of independent and dependent variables changes the resulting model, making ordinary least squares regression an asymmetric regressor. The reason for this is that least squares regression assumes the independent variable to be noise-free, whereas the dependent variable is assumed to be noisy. However, in the case of classification, all features are usually noisy observations such that neither or should be treated as independent. In fact, we would like to obtain a model that minimizes both the horizontal and the vertical projection error simultaneously. This corresponds to finding a model such that the orthogonal projection error is minimized as shown by figure 8.

Figure 8. Linear regression where both variables are independent corresponds to minimizing the orthogonal projection error.

The resulting regression is called Total Least Squares regression or orthogonal regression, and assumes that both variables are imperfect observations. An interesting observation is now that the obtained vector, representing the projection direction that minimizes the orthogonal projection error, corresponds the the largest principal component of the data:

Figure 9. The vector which the data can be projected unto with minimal orthogonal error corresponds to the largest eigenvector of the covariance matrix of the data.

In other words, if we want to reduce the dimensionality by projecting the original data onto a vector such that the squared projection error is minimized in all directions, we can simply project the data onto the largest eigenvectors. This is exactly what we called Principal Component Analysis in the previous section, where we showed that such projection also decorrelates the feature space.

A practical PCA application: Eigenfaces

Although the above examples are limited to two or three dimensions for visualization purposes, dimensionality reduction usually becomes important when the number of features is not negligible compared to the number of training samples. As an example, suppose we would like to perform face recognition, i.e. determine the identity of the person depicted in an image, based on a training dataset of labeled face images. One approach might be to treat the brightness of each pixel of the image as a feature. If the input images are of size 32×32 pixels, this means that the feature vector contains 1024 feature values. Classifying a new face image can then be done by calculating the Euclidean distance between this 1024-dimensional vector, and the feature vectors of the people in our training dataset. The smallest distance then tells us which person we are looking at.

However, operating in a 1024-dimensional space becomes problematic if we only have a few hundred training samples. Furthermore, Euclidean distances behave strangely in high dimensional spaces as discussed in an earlier article. Therefore, we could use PCA to reduce the dimensionality of the feature space by calculating the eigenvectors of the covariance matrix of the set of 1024-dimensional feature vectors, and then projecting each feature vector onto the largest eigenvectors.

Since the eigenvector of 2D data is 2-dimensional, and an eigenvector of 3D data is 3-dimensional, the eigenvectors of 1024-dimensional data is 1024-dimensional. In other words, we could reshape each of the 1024-dimensional eigenvectors to a 32×32 image for visualization purposes. Figure 10 shows the first four eigenvectors obtained by eigendecomposition of the Cambridge face dataset:

Figure 10. The four largest eigenvectors, reshaped to images, resulting in so called EigenFaces. (source: https://nl.wikipedia.org/wiki/Eigenface)

Each 1024-dimensional feature vector (and thus each face) can now be projected onto the N largest eigenvectors, and can be represented as a linear combination of these eigenfaces. The weights of these linear combinations determine the identity of the person. Since the largest eigenvectors represent the largest variance in the data, these eigenfaces describe the most informative image regions (eyes, noise, mouth, etc.). By only considering the first N (e.g. N=70) eigenvectors, the dimensionality of the feature space is greatly reduced.

The remaining question is now how many eigenfaces should be used, or in the general case; how many eigenvectors should be kept. Removing too many eigenvectors might remove important information from the feature space, whereas eliminating too few eigenvectors leaves us with the curse of dimensionality. Regrettably there is no straight answer to this problem. Although cross-validation techniques can be used to obtain an estimate of this hyperparameter, choosing the optimal number of dimensions remains a problem that is mostly solved in an empirical (an academic term that means not much more than ‘trial-and-error’) manner. Note that it is often useful to check how much (as a percentage) of the variance of the original data is kept while eliminating eigenvectors. This is done by dividing the sum of the kept eigenvalues by the sum of all eigenvalues.

The PCA recipe

Based on the previous sections, we can now list the simple recipe used to apply PCA for feature extraction:

1) Center the data

In an earlier article, we showed that the covariance matrix can be written as a sequence of linear operations (scaling and rotations). The eigendecomposition extracts these transformation matrices: the eigenvectors represent the rotation matrix, while the eigenvalues represent the scaling factors. However, the covariance matrix does not contain any information related to the translation of the data. Indeed, to represent translation, an affine transformation would be needed instead of a linear transformation.

Therefore, before applying PCA to rotate the data in order to obtain uncorrelated axes, any existing shift needs to be countered by subtracting the mean of the data from each data point. This simply corresponds to centering the data such that its average becomes zero.

2) Normalize the data

The eigenvectors of the covariance matrix point in the direction of the largest variance of the data. However, variance is an absolute number, not a relative one. This means that the variance of data, measured in centimeters (or inches) will be much larger than the variance of the same data when measured in meters (or feet). Consider the example where one feature represents the length of an object in meters, while the second feature represents the width of the object in centimeters. The largest variance, and thus the largest eigenvector, will implicitly be defined by the first feature if the data is not normalized.

To avoid this scale-dependent nature of PCA, it is useful to normalize the data by dividing each feature by its standard deviation. This is especially important if different features correspond to different metrics.

3) Calculate the eigendecomposition

Since the data will be projected onto the largest eigenvectors to reduce the dimensionality, the eigendecomposition needs to be obtained. One of the most widely used methods to efficiently calculate the eigendecomposition is Singular Value Decomposition (SVD).

4) Project the data

To reduce the dimensionality, the data is simply projected onto the largest eigenvectors. Let be the matrix whose columns contain the largest eigenvectors and let be the original data whose columns contain the different observations. Then the projected data is obtained as . We can either choose the number of remaining dimensions, i.e. the columns of , directly, or we can define the amount of variance of the original data that needs to kept while eliminating eigenvectors. If only eigenvectors are kept, and represent the corresponding eigenvalues, then the amount of variance that remains after projecting the original -dimensional data can be calculated as:

(4)  

PCA pitfalls

In the above discussion, several assumptions have been made. In the first section, we discussed how PCA decorrelates the data. In fact, we started the discussion by expressing our desire to recover the unknown, underlying independent components of the observed features. We then assumed that our data was normally distributed, such that statistical independence simply corresponds to the lack of a linear correlation. Indeed, PCA allows us to decorrelate the data, thereby recovering the independent components in case of Gaussianity. However, it is important to note that decorrelation only corresponds to statistical independency in the Gaussian case. Consider the data obtained by sampling half a period of :

Figure 11 Uncorrelated data is only statistically independent if normally distributed. In this example a clear non-linear dependency still exists: y=sin(x).

Although the above data is clearly uncorrelated (on average, the y-value increases as much as it decreases when the x-value goes up) and therefore corresponds to a diagonal covariance matrix, there still is a clear non-linear dependency between both variables.

In general, PCA only uncorrelates the data but does not remove statistical dependencies. If the underlying components are known to be non-Gaussian, techniques such as ICA could be more interesting. On the other hand, if non-linearities clearly exist, dimensionality reduction techniques such as non-linear PCA can be used. However, keep in mind that these methods are prone to overfitting themselves, since more parameters are to be estimated based on the same amount of training data.

A second assumption that was made in this article, is that the most discriminative information is captured by the largest variance in the feature space. Since the direction of the largest variance encodes the most information this is likely to be true. However, there are cases where the discriminative information actually resides in the directions of the smallest variance, such that PCA could greatly hurt classification performance. As an example, consider the two cases of figure 12, where we reduce the 2D feature space to a 1D representation:

Figure 12. In the first case, PCA would hurt classification performance because the data becomes linearly unseparable. This happens when the most discriminative information resides in the smaller eigenvectors.

If the most discriminative information is contained in the smaller eigenvectors, applying PCA might actually worsen the Curse of Dimensionality because now a more complicated classification model (e.g. non-linear classifier) is needed to classify the lower dimensional problem. In this case, other dimensionality reduction methods might be of interest, such as Linear Discriminant Analysis (LDA) which tries to find the projection vector that optimally separates the two classes.

Source Code

The following code snippet shows how to perform principal component analysis for dimensionality reduction in Matlab:
Matlab source code

Conclusion

In this article, we discussed the advantages of PCA for feature extraction and dimensionality reduction from two different points of view. The first point of view explained how PCA allows us to decorrelate the feature space, whereas the second point of view showed that PCA actually corresponds to orthogonal regression.

Furthermore, we briefly introduced Eigenfaces as a well known example of PCA based feature extraction, and we covered some of the most important disadvantages of Principal Component Analysis.

If you’re new to this blog, don’t forget to subscribe, or follow me on twitter!

The post Feature extraction using PCA appeared first on Computer vision for dummies.

In this article, we will discuss the so called ‘Curse of Dimensionality’, and explain why it is important when designing a classifier. In the following sections I will provide an intuitive explanation of this concept, illustrated by a clear example of overfitting due to the curse of dimensionality.

Consider an example in which we have a set of images, each of which depicts either a cat or a dog. We would like to create a classifier that is able to distinguish dogs from cats automatically. To do so, we first need to think about a descriptor for each object class that can be expressed by numbers, such that a mathematical algorithm, i.e. a classifier, can use these numbers to recognize the object. We could for instance argue that cats and dogs generally differ in color. A possible descriptor that discriminates these two classes could then consist of three number; the average red color, the average green color and the average blue color of the image under consideration. A simple linear classifier for instance, could combine these features linearly to decide on the class label:

If 0.5*red + 0.3*green + 0.2*blue > 0.6 : return cat;
else return dog;

However, these three color-describing numbers, called features, will obviously not suffice to obtain a perfect classification. Therefore, we could decide to add some features that describe the texture of the image, for instance by calculating the average edge or gradient intensity in both the X and Y direction. We now have 5 features that, in combination, could possibly be used by a classification algorithm to distinguish cats from dogs.

To obtain an even more accurate classification, we could add more features, based on color or texture histograms, statistical moments, etc. Maybe we can obtain a perfect classification by carefully defining a few hundred of these features? The answer to this question might sound a bit counter-intuitive: no we can not!. In fact, after a certain point, increasing the dimensionality of the problem by adding new features would actually degrade the performance of our classifier. This is illustrated by figure 1, and is often referred to as ‘The Curse of Dimensionality’.

Figure 1. As the dimensionality increases, the classifier’s performance increases until the optimal number of features is reached. Further increasing the dimensionality without increasing the number of training samples results in a decrease in classifier performance.

In the next sections we will review why the above is true, and how the curse of dimensionality can be avoided.

The curse of dimensionality and overfitting

In the earlier introduced example of cats and dogs, let’s assume there are an infinite number of cats and dogs living on our planet. However, due to our limited time and processing power, we were only able to obtain 10 pictures of cats and dogs. The end-goal in classification is then to train a classifier based on these 10 training instances, that is able to correctly classify the infinite number of dog and cat instances which we do not have any information about.

Now let’s use a simple linear classifier and try to obtain a perfect classification. We can start by a single feature, e.g. the average ‘red’ color in the image:

Figure 2. A single feature does not result in a perfect separation of our training data.

Figure 2 shows that we do not obtain a perfect classification result if only a single feature is used. Therefore, we might decide to add another feature, e.g. the average ‘green’ color in the image:

Figure 3.Adding a second feature still does not result in a linearly separable classification problem: No single line can separate all cats from all dogs in this example.

Finally we decide to add a third feature, e.g. the average ‘blue’ color in the image, yielding a three-dimensional feature space:

Figure 4. Adding a third feature results in a linearly separable classification problem in our example. A plane exists that perfectly separates dogs from cats.

In the three-dimensional feature space, we can now find a plane that perfectly separates dogs from cats. This means that a linear combination of the three features can be used to obtain perfect classification results on our training data of 10 images:

Figure 5. The more features we use, the higher the likelihood that we can successfully separate the classes perfectly.

The above illustrations might seem to suggest that increasing the number of features until perfect classification results are obtained is the best way to train a classifier, whereas in the introduction, illustrated by figure 1, we argued that this is not the case. However, note how the density of the training samples decreased exponentially when we increased the dimensionality of the problem.

In the 1D case (figure 2), 10 training instances covered the complete 1D feature space, the width of which was 5 unit intervals. Therefore, in the 1D case, the sample density was 10/5=2 samples/interval. In the 2D case however (figure 3), we still had 10 training instances at our disposal, which now cover a 2D feature space with an area of 5×5=25 unit squares. Therefore, in the 2D case, the sample density was 10/25 = 0.4 samples/interval. Finally, in the 3D case, the 10 samples had to cover a feature space volume of 5x5x5=125 unit cubes. Therefore, in the 3D case, the sample density was 10/125 = 0.08 samples/interval.

If we would keep adding features, the dimensionality of the feature space grows, and becomes sparser and sparser. Due to this sparsity, it becomes much more easy to find a separable hyperplane because the likelihood that a training sample lies on the wrong side of the best hyperplane becomes infinitely small when the number of features becomes infinitely large. However, if we project the highly dimensional classification result back to a lower dimensional space, a serious problem associated with this approach becomes evident:

Figure 6. Using too many features results in overfitting. The classifier starts learning exceptions that are specific to the training data and do not generalize well when new data is encountered.

Figure 6 shows the 3D classification results, projected onto a 2D feature space. Whereas the data was linearly separable in the 3D space, this is not the case in a lower dimensional feature space. In fact, adding the third dimension to obtain perfect classification results, simply corresponds to using a complicated non-linear classifier in the lower dimensional feature space. As a result, the classifier learns the appearance of specific instances and exceptions of our training dataset. Because of this, the resulting classifier would fail on real-world data, consisting of an infinite amount of unseen cats and dogs that often do not adhere to these exceptions.

This concept is called overfitting and is a direct result of the curse of dimensionality. Figure 7 shows the result of a linear classifier that has been trained using only 2 features instead of 3:

Figure 7. Although the training data is not classified perfectly, this classifier achieves better results on unseen data than the one from figure 5.

Although the simple linear classifier with decision boundaries shown by figure 7 seems to perform worse than the non-linear classifier in figure 5, this simple classifier generalizes much better to unseen data because it did not learn specific exceptions that were only in our training data by coincidence. In other words, by using less features, the curse of dimensionality was avoided such that the classifier did not overfit the training data.

Figure 8 illustrates the above in a different manner. Let’s say we want to train a classifier using only a single feature whose value ranges from 0 to 1. Let’s assume that this feature is unique for each cat and dog. If we want our training data to cover 20% of this range, then the amount of training data needed is 20% of the complete population of cats and dogs. Now, if we add another feature, resulting in a 2D feature space, things change; To cover 20% of the 2D feature range, we now need to obtain 45% of the complete population of cats and dogs in each dimension (0.45^2 = 0.2). In the 3D case this gets even worse: to cover 20% of the 3D feature range, we need to obtain 58% of the population in each dimension (0.58^3 = 0.2).

Figure 8. The amount of training data needed to cover 20% of the feature range grows exponentially with the number of dimensions.

In other words, if the amount of available training data is fixed, then overfitting occurs if we keep adding dimensions. On the other hand, if we keep adding dimensions, the amount of training data needs to grow exponentially fast to maintain the same coverage and to avoid overfitting.

In the above example, we showed that the curse of dimensionality introduces sparseness of the training data. The more features we use, the more sparse the data becomes such that accurate estimation of the classifier’s parameters (i.e. its decision boundaries) becomes more difficult. Another effect of the curse of dimensionality, is that this sparseness is not uniformly distributed over the search space. In fact, data around the origin (at the center of the hypercube) is much more sparse than data in the corners of the search space. This can be understood as follows:

Imagine a unit square that represents the 2D feature space. The average of the feature space is the center of this unit square, and all points within unit distance from this center, are inside a unit circle that inscribes the unit square. The training samples that do not fall within this unit circle are closer to the corners of the search space than to its center. These samples are difficult to classify because their feature values greatly differs (e.g. samples in opposite corners of the unit square). Therefore, classification is easier if most samples fall inside the inscribed unit circle, illustrated by figure 9:

Figure 9.Training samples that fall outside the unit circle are in the corners of the feature space and are more difficult to classify than samples near the center of the feature space.

An interesting question is now how the volume of the circle (hypersphere) changes relative to the volume of the square (hypercube) when we increase the dimensionality of the feature space. The volume of a unit hypercube of dimension d is always 1^d = 1. The volume of the inscribing hypersphere of dimension d and with radius 0.5 can be calculated as:

(1)  

Figure 10 shows how the volume of this hypersphere changes when the dimensionality increases:

Figure 10. The volume of the hypersphere tends to zero as the dimensionality increases.

This shows that the volume of the hypersphere tends to zero as the dimensionality tends to infinity, whereas the volume of the surrounding hypercube remains constant. This surprising and rather counter-intuitive observation partially explains the problems associated with the curse of dimensionality in classification: In high dimensional spaces, most of the training data resides in the corners of the hypercube defining the feature space. As mentioned before, instances in the corners of the feature space are much more difficult to classify than instances around the centroid of the hypersphere. This is illustrated by figure 11, which shows a 2D unit square, a 3D unit cube, and a creative visualization of an 8D hypercube which has 2^8 = 256 corners:

Figure 11. As the dimensionality increases, a larger percentage of the training data resides in the corners of the feature space.

For an 8-dimensional hypercube, about 98% of the data is concentrated in its 256 corners. As a result, when the dimensionality of the feature space goes to infinity, the ratio of the difference in minimum and maximum Euclidean distance from sample point to the centroid, and the minimum distance itself, tends to zero:

(2)  

Therefore, distance measures start losing their effectiveness to measure dissimilarity in highly dimensional spaces. Since classifiers depend on these distance measures (e.g. Euclidean distance, Mahalanobis distance, Manhattan distance), classification is often easier in lower-dimensional spaces where less features are used to describe the object of interest. Similarly, Gaussian likelihoods become flat and heavy tailed distributions in high dimensional spaces, such that the ratio of the difference between the minimum and maximum likelihood and the minimum likelihood itself tends to zero.

How to avoid the curse of dimensionality?

Figure 1 showed that the performance of a classifier decreases when the dimensionality of the problem becomes too large. The question then is what ‘too large’ means, and how overfitting can be avoided. Regrettably there is no fixed rule that defines how many feature should be used in a classification problem. In fact, this depends on the amount of training data available, the complexity of the decision boundaries, and the type of classifier used.

If the theoretical infinite number of training samples would be available, the curse of dimensionality does not apply and we could simply use an infinite number of features to obtain perfect classification. The smaller the size of the training data, the less features should be used. If N training samples suffice to cover a 1D feature space of unit interval size, then N^2 samples are needed to cover a 2D feature space with the same density, and N^3 samples are needed in a 3D feature space. In other words, the number of training instances needed grows exponentially with the number of dimensions used.

Furthermore, classifiers that tend to model non-linear decision boundaries very accurately (e.g. neural networks, KNN classifiers, decision trees) do not generalize well and are prone to overfitting. Therefore, the dimensionality should be kept relatively low when these classifiers are used. If a classifier is used that generalizes easily (e.g. naive Bayesian, linear classifier), then the number of used features can be higher since the classifier itself is less expressive. Figure 6 showed that using a simple classifier model in a high dimensional space corresponds to using a complex classifier model in a lower dimensional space.

Therefore, overfitting occurs both when estimating relatively few parameters in a highly dimensional space, and when estimating a lot of parameters in a lower dimensional space. As an example, consider a Gaussian density function, parameterized by its mean and covariance matrix. Let’s say we operate in a 3D space, such that the covariance matrix is a 3×3 symmetric matrix consisting of 6 unique elements (3 variances on the diagonal and 3 covariances off-diagonal). Together with the 3D mean of the distribution this means that we need to estimate 9 parameters based on our training data, to obtain the Gaussian density that represent the likelihood of our data. In the 1D case, only 2 parameters need to be estimated (mean and variance), whereas in the 2D case 5 parameters are needed (2D mean, two variances and a covariance). Again we can see that the number of parameters to be estimated grows quadratic with the number of dimensions.

In an earlier article we showed that the variance of a parameter estimate increases if the number of parameters to be estimated increases (and if the bias of the estimate and the amount of training data are kept constant). This means that the quality of our parameter estimates decreases if the dimensionality goes up, due to the increase of variance. An increase of classifier variance corresponds to overfitting.

Another interesting question is which features should be used. Given a set of N features; how do we select an optimal subset of M features such that M<N? One approach would be to search for the optimum in the curve shown by figure 1. Since it is often intractable to train and test classifiers for all possible combinations of all features, several methods exist that try to find this optimum in different manners. These methods are called feature selection algorithms and often employ heuristics (greedy methods, best-first methods, etc.) to locate the optimal number and combination of features.

Another approach would be to replace the set of N features by a set of M features, each of which is a combination of the original feature values. Algorithms that try to find the optimal linear or non-linear combination of original features to reduce the dimensionality of the final problem are called Feature Extraction methods. A well known dimensionality reduction technique that yields uncorrelated, linear combinations of the original N features is Principal Component Analysis (PCA). PCA tries to find a linear subspace of lower dimensionality, such that the largest variance of the original data is kept. However, note that the largest variance of the data not necessarily represents the most discriminative information.

Finally, an invaluable technique used to detect and avoid overfitting during classifier training is cross-validation. Cross validation approaches split the original training data into one or more training subsets. During classifier training, one subset is used to test the accuracy and precision of the resulting classifier, while the others are used for parameter estimation. If the classification results on the subsets used for training greatly differ from the results on the subset used for testing, overfitting is in play. Several types of cross-validation such as k-fold cross-validation and leave-one-out cross-validation can be used if only a limited amount of training data is available.

Conclusion

In this article we discussed the importance of feature selection, feature extraction, and cross-validation, in order to avoid overfitting due to the curse of dimensionality. Using a simple example, we reviewed an important effect of the curse of dimensionality in classifier training, namely overfitting.

If you’re new to this blog, don’t forget to subscribe, or follow me on twitter!

The post The Curse of Dimensionality in classification appeared first on Computer vision for dummies.