What’s Principal Component in PCA?

4 min readApr 3, 2021

This blog will try to help you understand PCA even if you are a beginner or an expert.

The depth of the content will keep increasing as you go on reading. You can continue according to your requirement of understanding….

∘ Dimensionality Reduction! Curse of Dimensionality.
∘ How PCA works?
∘ PCA working on high dimensions.(Matrix intuition)
∘ Numerical explanation.

Dimensionality Reduction! Curse of Dimensionality.

In Data science, you deal with huge amount of data. The data consists of many rows and columns. You have to use the data and train a model or try to extract some insights or detect a pattern in the given data. Here comes the curse of dimensionality.

Lets take a real world example and try to accumulate all the possible points on which value of a stock depends.

Latest news regarding the stock
Nifty and Sensex directional emotion.
News about our stock sector.
Foreign Direct Investment.
Government policies.
Monthly expiry of options.
International indices and so on…

This example shows that there are N number of variables on which our stock price is dependent on. But the important point is not all the above listed features impact the stock prices directly. There are many features which does not create a large change in dependent variable. So, that variable will create unnecessary burden while analysis. Thus it is important to get rid of such features and this is where dimensionality reduction techniques are used.

How PCA works?

Basic idea behind PCA is to remove features with low variance because they does not have much information about the target variable. Instead we preserve high variance features that is useful in analysis.

Now lets visualize how PCA works.

Suppose you are trying to analyze and order uniform for school students. You have their heights and age as shown in the below image.

Scatter plot of Age vs Height of students.

You can observe that there is very less variation in age as compared to height. Age varies by 3 values(9,10,11) where as height varies 5 values(116 to 124). So for analysis we can neglect the age variable as it does not have much information. To do this, we can project the points on the height axis and reduce the data to 1 dimension.

In reality data will not be parallel to X or Y axis. So we have to rotate our axis in the direction of the data points.

PCA working on high dimensions.(Matrix intuition)

You want to visualize data with 1,000 columns. But humans are only able to visualize 2d and 3d data. So we can use PCA to reduce 1000D data to 2D data. Data=>20,000 rows,1,000 columns/features/dimensions.

The steps and matrix intuition are as follows:

Compute covariance matrix of D:

Covariance matrix = D’(1000, 20000) X D(20000, 1000)

∴ Covariance matrix = (1000,1000)

2. Compute all Eigen vectors. We will get 1000 Eigen vectors. Then Sort them in ascending order and select N(no of reduced dimension) largest components i.e. Principal component1(999),Principal component2(1000).

These two are the PRINCIPAL COMPONENT IN PCA.

∴ Eigen vector=>(1000,2)

3. Multiply transpose of Eigen vector with transpose of data.(This step is used for projecting the data points on another axis).

PCA’ = Eigen vector’(2,1000) X D’(1000,20000)

∴ PCA’ => (2,20000)

4. Transpose of the obtained matrix gives us the matrix with dimension reduced from 1000D to 2D.

∴PCA => 20,000 rows , 2 columns/dimensions