MATLAB: Fast correlation computation for all indices in 2 vectors - matlab

Question

I have 2 vectors A and B, each of length 10,000. For each of ind=1:10000, I want to compute the Pearson's correlation of A(1:ind) and B(1:ind). When I do this in a for loop, it takes too much time. parfor does not work with more than 2 workers in my machine. Is there a way to do this operation fast and save results in a vector C (apparently of length 10,000 where the first element is NaN)? I found the question Fast rolling correlation in Matlab, but this is a little different than what I need.

Answer 1

You can use this method to compute cumulative correlation coefficient:
function result = cumcor(x,y)
n = reshape(1:numel(x),size(x));
sumx = cumsum(x);
sumy = cumsum(y);
sumx2 = cumsum(x.^2);
sumy2 = cumsum(y.^2);
sumxy = cumsum(x.*y);
result = (n.*sumxy-sumx.*sumy)./(sqrt((sumx.^2-n.*sumx2).*(sumy.^2-n.*sumy2)));
end

Answer 2

Solution
I suggest the following approach:
Pearson correlation can be calculated by using the following formula:
calculating the accumulative mean of each of the random variabiles above efficiently is realtively easy
(X, Y, XY, X^2, Y^2).
given the accumulative mean calculated in 2, we can calculate the accumulative std of X and Y.
given the accumulative std of X,Y and accumulative mean above, we can calculate the accumulative pearson coefficient.
Code
%defines inputs
N = 10000;
X = rand(N,1);
Y = rand(N,1);
%calculates accumolative mean for X, Y, X^2, Y^2, XY
EX = accumMean(X);
EY = accumMean(Y);
EX2 = accumMean(X.^2);
EY2 = accumMean(Y.^2);
EXY = accumMean(X.*Y);
%calculates accumolative pearson correlation
accumPearson = zeros(N,1);
for ii=2:N
stdX = (EX2(ii)-EX(ii)^2).^0.5;
stdY = (EY2(ii)-EY(ii)^2).^0.5;
accumPearson(ii) = (EXY(ii)-EX(ii)*EY(ii))/(stdX*stdY);
end
%accumulative mean function, to be defined in an additional m file.
function [ accumMean ] = accumMean( vec )
accumMean = zeros(size(vec));
accumMean(1) = vec(1);
for ii=2:length(vec)
accumMean(ii) = (accumMean(ii-1)*(ii-1) +vec(ii))/ii;
end
end
Runtime
for N=10000:
Elapsed time is 0.002096 seconds.
for N=1000000:
Elapsed time is 0.240669 seconds.
Correctness
Testing the correctness of the code above could be done by calculative the accumulative pearson coefficient by corr function, and comparing it to the result given from the code above:
%ground truth for correctness comparison
gt = zeros(N,1)
for z=1:N
gt(z) = corr(X(1:z),Y(1:z));
end
Unfortunately, I dont have the Statistics and Machine Learning Toolbox, so I cant make this check.
I do think that it is a good start though, and you can continue from here :)