# Solved – Eigenvalues of correlation matrices exhibit exponential decay

I have a data-set of \$P\$ samples of size \$N\$, and noticed that the eigenvalues of the correlation matrices \$A^TA\$, when presented in descending order, can in many cases be described as an exponential decaying function. That is, there is a good fit linear of from \$i=1..N\$ to \$log|lambda_i|\$. Moreover, for several data sets I found that the exponent of the decay is fairly constant.

Is this a well-known fact or just one's tendency for finding patterns?

Obviously from PCA / SVD it tells me something about the ability to approximate the data using low-dimensional matrix.

Are there any solid mathematical results on the size of the exponent of this decay?

Contents

Everything has already been pretty much figured out in the comments, thanks to @AndyW, @whuber, and @UriCohen, but I would still like to write it up as a coherent answer.

First, let me illustrate the original question. Here is the eigenspectrum of some actual real data (neural recordings) that I happen to work with right now. First few (~20-30) PCs obviously carry some signal, but after that the eigenvalues start slowly decreasing in what does seem like an exponential fashion: note that the middle part of the spectrum is almost a straight line on this log-plot. I am not showing the last part of the spectrum, because there the eigenvalues decrease pretty much to 0, due to some temporal smoothing that I used before PCA. Question is: why exponential decay?

The answer is, I believe, that any high-dimensional real data are highly contaminated by noise, so the bulk of the eigenspectrum shows the spectral behaviour of pure noise. What is the spectrum of a random covariance matrix? Turns out, there is a nice asymptotic result given by Marchenko–Pastur distribution, see the pdf of the original 1967 paper in Russian if you like.

Marchenko and Pastur tell us to consider a random data matrix of \$Ntimes D\$ size filled with independent Gaussian random values from \$mathcal{N}(0,sigma^2)\$. If \$sigma^2=1\$ and \$N=D\$, then in the limit \$N to infty\$ the distribution of eigenvalues of its covariance matrix is given by \$\$mu(x)=frac{sqrt{4x-x^2}}{2pi x}.\$\$

Let us verify. I generated a random matrix of the \$1000 times 1000\$ size, computed its covariance matrix and then calculated the eigenspectrum. The first subplot below shows the covariance matrix. The second shows the distribution (histogram) of the eigenvalues and the Marchenko-Pastur function given above. It does fit perfectly. But we are interested not so much in the distribution of eigenvalues, but in the eigenspectrum itself. If we draw 1000 values from the Marchenko-Pastur distribution (forming the spectrum) and sort them in decreasing order, then the resulting function will be given by \$S(x)=(1-M(x))^{-1}\$ rescaled to \$[1, 1000]\$, where \$M(x)\$ is the Marchenko-Pastur cumulative distribution function, i.e. \$M(x) = int_0^x mu(t) dt\$. The third subplot on the figure above shows the empirical spectrum vs Marchenko-Pastur fit.

It is quite a mess to compute \$M(x)\$, here is Wolfram Alpha's attempt. But we can note that \$mu(x)\$ in the middle of its domain (around \$xapprox 2\$) is very well approximated by a straight line. This means that \$M(x)\$ will be approximately quadratic, and so its inverse \$S(x) sim mathrm{const}-sqrt{x}\$.

In other words, the decay is not exponential at all, it is a square-root decay! However, funny enough, it is close enough to the exponential shape so that on the log-plot (see the fourth subplot above) the middle part of the spectrum looks pretty straight.

Rate this post

# Solved – Eigenvalues of correlation matrices exhibit exponential decay

I have a data-set of \$P\$ samples of size \$N\$, and noticed that the eigenvalues of the correlation matrices \$A^TA\$, when presented in descending order, can in many cases be described as an exponential decaying function. That is, there is a good fit linear of from \$i=1..N\$ to \$log|lambda_i|\$. Moreover, for several data sets I found that the exponent of the decay is fairly constant.

Is this a well-known fact or just one's tendency for finding patterns?

Obviously from PCA / SVD it tells me something about the ability to approximate the data using low-dimensional matrix.

Are there any solid mathematical results on the size of the exponent of this decay?

Everything has already been pretty much figured out in the comments, thanks to @AndyW, @whuber, and @UriCohen, but I would still like to write it up as a coherent answer.

First, let me illustrate the original question. Here is the eigenspectrum of some actual real data (neural recordings) that I happen to work with right now. First few (~20-30) PCs obviously carry some signal, but after that the eigenvalues start slowly decreasing in what does seem like an exponential fashion: note that the middle part of the spectrum is almost a straight line on this log-plot. I am not showing the last part of the spectrum, because there the eigenvalues decrease pretty much to 0, due to some temporal smoothing that I used before PCA. Question is: why exponential decay?

The answer is, I believe, that any high-dimensional real data are highly contaminated by noise, so the bulk of the eigenspectrum shows the spectral behaviour of pure noise. What is the spectrum of a random covariance matrix? Turns out, there is a nice asymptotic result given by Marchenko–Pastur distribution, see the pdf of the original 1967 paper in Russian if you like.

Marchenko and Pastur tell us to consider a random data matrix of \$Ntimes D\$ size filled with independent Gaussian random values from \$mathcal{N}(0,sigma^2)\$. If \$sigma^2=1\$ and \$N=D\$, then in the limit \$N to infty\$ the distribution of eigenvalues of its covariance matrix is given by \$\$mu(x)=frac{sqrt{4x-x^2}}{2pi x}.\$\$

Let us verify. I generated a random matrix of the \$1000 times 1000\$ size, computed its covariance matrix and then calculated the eigenspectrum. The first subplot below shows the covariance matrix. The second shows the distribution (histogram) of the eigenvalues and the Marchenko-Pastur function given above. It does fit perfectly. But we are interested not so much in the distribution of eigenvalues, but in the eigenspectrum itself. If we draw 1000 values from the Marchenko-Pastur distribution (forming the spectrum) and sort them in decreasing order, then the resulting function will be given by \$S(x)=(1-M(x))^{-1}\$ rescaled to \$[1, 1000]\$, where \$M(x)\$ is the Marchenko-Pastur cumulative distribution function, i.e. \$M(x) = int_0^x mu(t) dt\$. The third subplot on the figure above shows the empirical spectrum vs Marchenko-Pastur fit.

It is quite a mess to compute \$M(x)\$, here is Wolfram Alpha's attempt. But we can note that \$mu(x)\$ in the middle of its domain (around \$xapprox 2\$) is very well approximated by a straight line. This means that \$M(x)\$ will be approximately quadratic, and so its inverse \$S(x) sim mathrm{const}-sqrt{x}\$.

In other words, the decay is not exponential at all, it is a square-root decay! However, funny enough, it is close enough to the exponential shape so that on the log-plot (see the fourth subplot above) the middle part of the spectrum looks pretty straight.

Rate this post