Correlation matrix ignoring NaN - matlab

Question

I am using matlab and I have a (60x882) matrix and I need to compute pairwise correlations between columns. However I want to ignore all the columns which have a NaN or more (i.e. the result for any pair of columns in which at least one entry is NaN should be NaN).
Here's my code so far:
for i=1:size(auxret,2)
for j=1:size(auxret,2)
rho(i,j)=corr(auxret(:,i),auxret(:,j));
end
end
end
But this is extremely innefficient. I considered using the function:
corr(auxret, 'rows','pairwise');
But it didn't produce the same result (it ignores NaNs but still computes the correlation - so unless all entries of a column except one are NaN it will still give an output).
Any suggestions on how to improve efficiency?

Answer 1

To get the same output as your code using corr(auxret, 'rows','pairwise'), the following does the job
auxret(:,any(isnan(auxret))) = NaN;
r = corr(auxret, 'rows','pairwise');

Answer 2

What you describe is the default behavior of corr without any special options. For example,
auxret = [8 2 3
3 5 NaN
7 10 3
7 4 6
2 6 7];
rho = corr(auxret)
results in
rho =
1.0000 -0.1497 NaN
-0.1497 1.0000 NaN
NaN NaN NaN

Answer 3

This would be an efficient approach, specially when dealing with input data involving NaNs -
%// Get mask of invalid columns and thus extract columns without any NaN
mask = any(isnan(auxret),1);
A = auxret(:,~mask);
%// Use correlation formula to get correlation outputs for valid columns
n = size(A,1);
sum_cols = sum(A,1);
sumsq_sqcolsum = n*sum(A.^2,1) - sum_cols.^2;
val1 = n.*(A.'*A) - bsxfun(#times,sum_cols.',sum_cols); %//'
val2 = sqrt(bsxfun(#times,sumsq_sqcolsum.',sumsq_sqcolsum)); %//'
valid_outvals = val1./val2;
%// Setup output array and store the valid outputs in it
ncols = size(auxret,2);
valid_idx = find(~mask);
out = nan(ncols);
out(valid_idx,valid_idx) = valid_outvals;
Basically, as the pre-processing step, it altogether removes all columns having one or more NaNs and calculates the correlation outputs. Then we initialize an output array of NaNs with appropriate size and puts back the valid outputs into it at appropriate places.
Benchmarking
It seems the results are valid whether you stay with your loopy approach or go with the optional corr(auxret, 'rows','pairwise'). But, there is a big catch here: Even a single NaN
in any of the columns slows down the performance quite a bit and this performance drop is huge with the original loopy approach and still big with the rows + pairwise option as we will
find out in the benchmarking results next up.
Benchmarking code
nrows = 60;
ncols = 882;
percent_nans = 1; %// decides the percentage of NaNs in input
auxret = rand(nrows,ncols);
auxret(randperm(numel(auxret),round((percent_nans/100)*numel(auxret))))=nan;
disp('------------------------------- With Proposed Approach')
tic
%// Solution code from earlier
toc
disp('------------------------------- With ROWS + PAIRWISE Approach')
tic
auxret(:,any(isnan(auxret))) = NaN;
out1 = corr(auxret, 'rows','pairwise');
toc
disp('------------------------------- With Original Loopy Approach')
tic
out2 = zeros(size(auxret,2));
for i=1:size(auxret,2)
for j=1:size(auxret,2)
out2(i,j)=corr(auxret(:,i),auxret(:,j));
end
end
toc
So, there are few cases possible based on the input datasizes and percentage of NaNs and correspondingly we have the runtime results -
Case 1: Input is 6 x 88 and Percentage of NaNs is 10
------------------------------- With Proposed Approach
Elapsed time is 0.006371 seconds.
------------------------------- With ROWS + PAIRWISE Approach
Elapsed time is 0.052563 seconds.
------------------------------- With Original Loopy Approach
Elapsed time is 0.875620 seconds.
Case 2: Input is 6 x 88 and Percentage of NaNs is 1
------------------------------- With Proposed Approach
Elapsed time is 0.006303 seconds.
------------------------------- With ROWS + PAIRWISE Approach
Elapsed time is 0.049194 seconds.
------------------------------- With Original Loopy Approach
Elapsed time is 0.871369 seconds.
Case 3: Input is 6 x 88 and Percentage of NaNs is 0.001
------------------------------- With Proposed Approach
Elapsed time is 0.006738 seconds.
------------------------------- With ROWS + PAIRWISE Approach
Elapsed time is 0.025754 seconds.
------------------------------- With Original Loopy Approach
Elapsed time is 0.867647 seconds.
Case 4: Input is 60 x 882 and Percentage of NaNs is 10
------------------------------- With Proposed Approach
Elapsed time is 0.007766 seconds.
------------------------------- With ROWS + PAIRWISE Approach
Elapsed time is 2.479645 seconds.
------------------------------- With Original Loopy Approach
...... Taken Too long ...
Case 5: Input is 60 x 882 and Percentage of NaNs is 1
------------------------------- With Proposed Approach
Elapsed time is 0.014144 seconds.
------------------------------- With ROWS + PAIRWISE Approach
Elapsed time is 2.324878 seconds.
------------------------------- With Original Loopy Approach
...... Taken Too long ...
Case 6: Input is 60 x 882 and Percentage of NaNs is 0.001
------------------------------- With Proposed Approach
Elapsed time is 0.020410 seconds.
------------------------------- With ROWS + PAIRWISE Approach
Elapsed time is 1.830632 seconds.
------------------------------- With Original Loopy Approach
...... Taken Too long ...