Calculate pairwise distances of xyz values within a matrix using matlab - matlab

I am fairly new to Matlab and have been trying to solve this issue for some weeks.
I have several large matrices, which vary in their amount of rows, although all have 69 columns. Within these matrices are 23 xyz values, e.g. column 1,2,3 are the x,y,z values for point 1, column 4,5,6 the x,y,z values for point 2 and so on, up to column 69.
There are over 1000 rows for each matrix, with each row representing a point in time.
I am trying to use a for loop to calculate the difference between row 1 and 2, then row 2 and 3, then 3 and 4 etc, using pythagoras theorem.
Essentially, the end output should be a N*23 matrix of the distance values but I'm getting stuck on generating the for loop.
I've been trying to solve this using a smaller 4*3 array for the meantime, (n is the matrix) with no luck.
for i = 1:row
for j = 1:col
pythag(i,j) = sqrt((n(i,1)-n(j,1))^2 +((n(i,2)-n(j,2))^2 +((n(i,3)-n(j,3))^2)));
end
end
Any help would be appreciated.

You can actually solve the problem in a single line of code.
The end output should be (N-1)*23 and not N*23 matrix.
In your loop, the column index in n should be advanced by 3 for each advancement of j.
You also need to subtract row i+1 from row i.
I renamed your matrix n with A.
Here is a sample solution using a for loop (please read the comments):
%I have a large number of N*69 matrices
%Within these matrices are 23 xyz values, e.g. column 1,2,3 are the x,y,z values for point 1, column 4,5,6 the x,y,z values for point 2 and so on, up to column 69.
%One row:
%A = [x1, y1, z1, x2, y2, z2, x3, y3, z3, x4, y4, z4]; %For example take 4 xyz values (instead of 23)
%There are over 1000 rows for each matrix,
%Example include 4 rows:
% A = [x11, y11, z11, x12, y12, z12, x13, y13, z13, x14, y14, z14;...
% x21, y21, z21, x22, y22, z22, x23, y23, z23, x24, y24, z24;...
% x31, y31, z31, x32, y32, z32, x33, y33, z33, x34, y34, z34;...
% x41, y41, z41, x42, y42, z42, x43, y43, z43, x44, y44, z44];
%Initialize A with arbitrary values (I decided to name the matrix A instead of n).
A = [magic(4) magic(4) magic(4)];
% A = [16 2 3 13 16 2 3 13 16 2 3 13
% 5 11 10 8 5 11 10 8 5 11 10 8
% 9 7 6 12 9 7 6 12 9 7 6 12
% 4 14 15 1 4 14 15 1 4 14 15 1];
%Calculate the difference between row 1 and 2, then row 2 and 3, then 3 and 4
%Essentially, the end output should be a N*23 matrix. Wrong!!! it should be (N-1)*23 matrix
%Assume N is the number of rows (over 1000 rows).
N = size(A, 1);
%Size of pythag matrix is (N-1)x(69/3) and in the example size is 3x4
row = N-1;
col = size(A,2)/3;
%Initialize output matrix with zeros.
pythag = zeros(row, col);
%Solving using a nested for loops:
for i = 1:row
%k, k+1, k+2 are the x,y,z, columns indeces in matrix A
k = 1;
for j = 1:col
%pythag(i,j) = sqrt((n(i,1)-n(j,1))^2 +((n(i,2)-n(j,2))^2 +((n(i,3)-n(j,3))^2)));
%We want the to distance from row i+1 to row i (in matrix A).
pythag(i, j) = sqrt((A(i+1, k)-A(i, k))^2 +((A(i+1, k+1)-A(i, k+1))^2 +((A(i+1, k+2)-A(i, k+2))^2)));
% | | |
% x y z
%The column index in A must skip by 3 for each increment of j
k = k + 3;
end
end
More elegant: replacing the sqrt(x^2 + y^2 + z^2) with norm([x,y,z]):
for i = 1:row
%k, k+1, k+2 are the x,y,z, columns indices in matrix A
k = 1;
for j = 1:col
%We want the to distance from row i+1 to row i (in matrix A).
pythag(i, j) = norm([A(i+1, k) - A(i, k), A(i+1, k+1) - A(i, k+1), A(i+1, k+2) - A(i, k+2)]);
%The column index in A must skip by 3 for each increment of j
k = k + 3;
end
end
Vectorizing the inner loop (use vecnorm instead of norm):
for i = 1:row
pythag(i, 1:col) = vecnorm([A(i+1, 1:3:end) - A(i, 1:3:end); A(i+1, 2:3:end) - A(i, 2:3:end); A(i+1, 3:3:end) - A(i, 3:3:end)]);
end
Vectorizing the outer loop (too confusing with vecnorm - back to Pythagoras):
pythag = sqrt((A(2:end, 1:3:end)-A(1:end-1, 1:3:end)).^2 +((A(2:end, 2:3:end)-A(1:end-1, 2:3:end)).^2 +((A(2:end, 3:3:end)-A(1:end-1, 3:3:end)).^2)));

Make use of the full power of Matlab using vector-wise calculations: you can omit the inner loop and calculate the distance immediately for a vector information (note that you need to provide the direction of the sum()).
Matlab stores all data in the RAM consistency, i.e. it will sometimes have to copy the vector to a different location in the RAM as it grows. Therefore, allocate the full space right at the start e.g. with NaN().
% allocate memory
pythag = NaN(row,col);
for i = 1:row
pythag(i,:) = sqrt(sum( (n(i,1:3)-n(:,1:3)).^2,2 ));
end
If you provide a minimal working example, we can check your full code and walk you through the problem.

Related

Add a diagonal of zeros to a matrix in MATLAB

Suppose I have a matrix A of dimension Nx(N-1) in MATLAB, e.g.
N=5;
A=[1 2 3 4;
5 6 7 8;
9 10 11 12;
13 14 15 16;
17 18 19 20 ];
I want to transform A into an NxN matrix B, just by adding a zero diagonal, i.e.,
B=[ 0 1 2 3 4;
5 0 6 7 8;
9 10 0 11 12;
13 14 15 0 16;
17 18 19 20 0];
This code does what I want:
B_temp = zeros(N,N);
B_temp(1,:) = [0 A(1,:)];
B_temp(N,:) = [A(N,:) 0];
for j=2:N-1
B_temp(j,:)= [A(j,1:j-1) 0 A(j,j:end)];
end
B = B_temp;
Could you suggest an efficient way to vectorise it?
You can do this with upper and lower triangular parts of the matrix (triu and tril).
Then it's a 1 line solution:
B = [tril(A,-1) zeros(N, 1)] + [zeros(N,1) triu(A)];
Edit: benchmark
This is a comparison of the loop method, the 2 methods in Sardar's answer, and my method above.
Benchmark code, using timeit for timing and directly lifting code from question and answers:
function benchie()
N = 1e4; A = rand(N,N-1); % Initialise large matrix
% Set up anonymous functions for input to timeit
s1 = #() sardar1(A,N); s2 = #() sardar2(A,N);
w = #() wolfie(A,N); u = #() user3285148(A,N);
% timings
timeit(s1), timeit(s2), timeit(w), timeit(u)
end
function sardar1(A, N) % using eye as an indexing matrix
B=double(~eye(N)); B(find(B))=A.'; B=B.';
end
function sardar2(A,N) % similar to sardar1, but avoiding slow operations
B=1-eye(N); B(logical(B))=A.'; B=B.';
end
function wolfie(A,N) % using triangular parts of the matrix
B = [tril(A,-1) zeros(N, 1)] + [zeros(N,1) triu(A)];
end
function user3285148(A, N) % original looping method
B = zeros(N,N); B(1,:) = [0 A(1,:)]; B(N,:) = [A(N,:) 0];
for j=2:N-1; B(j,:)= [A(j,1:j-1) 0 A(j,j:end)]; end
end
Results:
Sardar method 1: 2.83 secs
Sardar method 2: 1.82 secs
My method: 1.45 secs
Looping method: 3.80 secs (!)
Conclusions:
Your desire to vectorise this was well founded, looping is way slower than other methods.
Avoiding data conversions and find for large matrices is important, saving ~35% processing time between Sardar's methods.
By avoiding indexing all together you can save a further 20% processing time.
Generate a matrix with zeros at diagonal and ones at non-diagonal indices. Replace the non-diagonal elements with the transpose of A (since MATLAB is column major). Transpose again to get the correct order.
B = double(~eye(N)); %Converting to double since we want to replace with double entries
B(find(B)) = A.'; %Replacing the entries
B = B.'; %Transposing again to get the matrix in the correct order
Edit:
As suggested by Wolfie for the same algorithm, you can get rid of conversion to double and the use of find with:
B = 1-eye(N);
B(logical(B)) = A.';
B = B.';
If you want to insert any vector on a diagonal of a matrix, one can use plain indexing. The following snippet gives you the indices of the desired diagonal, given the size of the square matrix n (matrix is n by n), and the number of the diagonal k, where k=0 corresponds to the main diagonal, positive numbers of k to upper diagonals and negative numbers of k to lower diagonals. ixd finally gives you the 2D indices.
function [idx] = diagidx(n,k)
% n size of square matrix
% k number of diagonal
if k==0 % identity
idx = [(1:n).' (1:n).']; % [row col]
elseif k>0 % Upper diagonal
idx = [(1:n-k).' (1+k:n).'];
elseif k<0 % lower diagonal
idx = [(1+abs(k):n).' (1:n-abs(k)).'];
end
end
Usage:
n=10;
k=3;
A = rand(n);
idx = diagidx(n,k);
A(idx) = 1:(n-k);

Matlab: How to read data into a matrix

I have a data file matrix.txt, it has three columns. The first column stores the row index, the second column stores the column index, the third column stores the value. How do I read these into a matrix called mat. To be explicit, suppose our mat is a n*n square matrix, let n=2 for instance. In the text file, it has:
0 0 10
1 1 -10
The element in mat not specified is 0. Thus mat is supposed to be:
mat = 10 0
0 -10
How do I achieve this?
This should work for the generic 2-D case.
% Read in matrix specification
fID = fopen('matrix.txt');
tmp = fscanf(fID, '%u%u%f', [3 inf])';
fclose(fID);
% Use the maximum row and column subscripts to obtain the matrix size
tmp(:, 1:2) = tmp(:, 1:2) + 1; % MATLAB doesn't use 0-based indexing
matsize = [max(tmp(:,1)), max(tmp(:,2))];
% Convert subscripts to linear indices
lidx = sub2ind(matsize, tmp(:,1), tmp(:,2));
mat = zeros(matsize); % Initialize matrix
mat(lidx) = tmp(:,3); % Assign data
Using a sample matrix.txt:
0 0 10
1 1 -10
1 2 20
We receive:
>> mat
mat =
10 0 0
0 -10 20
Since in MATLAB, indices begin with 1 (not zero), we should add 1 to our indices in code.
r and c stand for row and column.
Alsom and n is for m by n zero matrix
A = importdata('matrix.txt');
r = A(:, 1)';
c = A(:, 2)';
m = max(r);
n = max(c);
B = zeros(m + 1, n + 1);
for k = 1:size(A,1);
B(r(k) + 1, c(k) + 1) = A(k, 3);
end
Result:
B =
10 0
0 -10
I see I am too slow, but I decided post my answer anyway...
I initialized matrix A as a vector, and used reshape:
%Load all file to matrix at once
%You may consider using fopen and fscanf, in case Matrix.txt is not ordered perfectly.
row_column_val = load('Matrix.txt', '-ascii');
R = row_column_val(:, 1) + 1; %Get vector of row indexes (add 1 - convert to Matalb indeces).
C = row_column_val(:, 2) + 1; %Get vector of column indexes (add 1 - convert to Matalb indeces).
V = row_column_val(:, 3); %Get vector of values.
nrows = max(R); %Number of rows in matrix.
ncols = max(C); %Number of columns in matrix.
A = zeros(nrows*ncols, 1); %Initialize A as a vector instead of a matrix (length of A is nrows*ncols).
%Put value v in place c*ncols + r for all elements of V, C and R.
%The formula is used for column major matrix (Matlab stored matrices in column major format).
A((C-1)*nrows + R) = V;
A = reshape(A, [nrows, ncols]);

Matrix with sliding window elements

I have time series, and I applying some user defined function to every W elements in time series.
Right now I am just using for loop, slide window of size W an apply my function to elements in a window at every iteration.
I am using Matlab and it is very inefficient with a "for loops" so I would love to vectorize this operation.
As a solution I see transforming signal with length N to a matrix with size (N - 1, W) where each row is time series in different windows and applying function to this matrix.
So, my questions are:
How to transform my initial time series to such a matrix?
Let's say I am sliding window with step X. So that not (N - 1, W) matrix will appear, but ((N - 1) / X, W). (Every Xth row of the matrix in [1])
Example:
Let's say my time series is:
T = [1, 5, 6, 8, 10, 14, 22]
W = 3
X = 1
=> I would love to get
[[1, 5, 6],
[5, 6, 8],
[6, 8, 10],
[8, 10, 14],
[10, 14, 22]]
If
W = 3
X = 2
=> I would love to get
[[1, 5, 6],
[6, 8, 10],
[10, 14, 22]]
Creating the right indices with bsxfun should most certainly help:
ind = bsxfun(#plus, 1:W, (0:X:numel(T)-W).');
out = T(ind);
Creating the right indices is the first step, delineated by the first line of code. What this code does is that it creates a 2D matrix where each row are the elements to access per window of interest. If you want to gain intuition on how the code generates the indices, look specifically at the first case where X = 1; and W = 3;.
We can see that the first row consists of accessing elements 1, 2, 3. The second row consists of accessing elements 2, 3, 4... up until the last row, which is 5, 6, 7. We can see that we have to access neighbouring elements in a window, and so the base indices need to go from 1, 2, 3, or in general from 1 to W. We now need to offset these indices so that they are centred at the right elements in T per window. The offset for the first window is simply 0, the next offset for the second window is simply 1 up until the last row which is 3. We see that for each row, we add 1 more to the base indices as the rows increase. Therefore, we add 1 to each base index for the second row, then 2 for each base index in the third row and so on. If you add the base indices with the offset indices, you thus finally get the correct indices to access the right elements in T.
Similarly if X = 2; and W = 3;, we see that we still have base indices of 1, 2, 3. However, the right elements to access now are 1, 2, 3 for the first row, then 3, 4, 5 for the second row then 5, 6, 7 for the third row. For each row, we now offset the base indices by 2 instead of 1 now. Therefore the second row we add 2 to each base index, then we add 4 to each base index for the third row and so on.
In general, the base indices are created using a vector 1:W and the offset indices are created using a vector 0:X:numel(T)-W. The subtraction of W is required so that we don't go out of bounds when sampling the signal as per the requirement. To create these indices that we just talked about, bsxfun handles this for us.
We create a row vector of 1:W which corresponds to the base indices and a column vector of (0:X:numel(T)-W).' which corresponds to the offsets per window. Note that the first offset starts at 0, then we increment by X amount to ensure that the correct centre is calculated to place our base indices at. We stop until we hit numel(T)-W elements, which is the condition you have stated. By using bsxfun, two temporary 2D matrices are created where the row vector is duplicated for as many rows as there are rows as there are in the column vector and the column vector is duplicated for as many columns as there are in the row vector. Once you add these two matrices together, you get the resulting index matrix.
Running the code with W = 3; and X = 1; gives:
>> T = [1, 5, 6, 8, 10, 14, 22];
>> X = 1;
>> W = 3;
>> ind = bsxfun(#plus, 1:W, (0:X:numel(T)-W).')
ind =
1 2 3
2 3 4
3 4 5
4 5 6
5 6 7
Similarly if W = 3; and X = 2; we also get:
>> T = [1, 5, 6, 8, 10, 14, 22];
>> X = 2;
>> W = 3;
>> ind = bsxfun(#plus, 1:W, (0:X:numel(T)-W).')
ind =
1 2 3
3 4 5
5 6 7
You can verify for yourself that these indices correspond to the correct elements in T to create your desired matrix in this case.
We finally use this to index into our matrix to grab the right elements:
out = T(ind);
Doing this for X = 1; and W = 3; gives:
>> out = T(ind)
out =
1 5 6
5 6 8
6 8 10
8 10 14
10 14 22
Similarly for X = 2; and W = 3; gives:
>> out = T(ind)
out =
1 5 6
6 8 10
10 14 22
Based on rayryeng's answer I wrote a function that does exactly this, plus some additional functionality. It's designed for generating indices for autoregression on a univariate time series. It can easily be used for the multivariate case by simply using the same indices and concatenating the referenced data.
It returns the indices to the predictor variable X (as per your request) and for the regressors y as well. In addition you have the option of applying a 'mask' to the predictor variables X while sliding the window. For example, with a window of 21 steps, you can select [T-2 T-3 T-5 T-8 T-13 T-21] for X and T for y
You can also change the prediction horizon - how many steps into the future the indices for y are. For example X = [T-1 T-2] and y = T+2
Hopefully someone else will find this useful.
% get_Sliding_Indexes:
% Useful for autoregression on a univariate time series.
% Returns the indexes for the predictor and response variables
% according to a sliding window.
%
% Copyright (C) 20016 Florin Schimbinschi
%
% Parameters:
% numRecords - the number of records in the dataset
%
% windowLag - number of past samples to take - it will be equal to
% the size of the predictor vector X. Default 10
%
% predHorizon - the prediction horizon is the number of steps into
% the future that predictions are to be made. Default 1
%
% windowPattern - by default the window will take all consecutive
% values in the past over the window lag size, however it is
% possible to sample using a custom pattern.
% For example taking every second value can be done by setting
% this parameter to 1:2:5. Default 1:windowLag
%
% stepSize - number of steps taken when window is moved. Default 1
%
% Returns:
% indX - predictor variable indexes
% indY - response variable indexes
%
%
% windowPattern = 1:2:9 __ structure between [] is moved to
% / \ / the right by stepSize units
% >------[(9-7-5-3-1)---(y)]--------------->
% \_______/ \_/
% X = [13579] predHorizon = 3
%
%
% Example on a multivariate time series (two) with 6 records:
%
% data2d = [ .1 .2 .3 .4 .5 .6
% .11 .22 .33 .44 .55 .66]';
%
% [X, y] = getSlidingIndexes(size(data2d,1), 4)
% X =
% 1 2 3 4
% 2 3 4 5
% y =
% 5
% 6
%
% Assuming we are interested in the second series (column):
%
% series2 = data2d(:,2);
%
% series2(X)
% ans =
% 0.1100 0.2200 0.3300 0.4400
% 0.2100 0.3300 0.4400 0.5500
%
% series2(y)
% ans =
% 0.5500
% 0.6600
%
function [indX, indY] = get_Sliding_Indexes(numRecords, ...
windowLag, predHorizon, windowPattern, stepSize)
if ~exist('numRecords','var') || isempty(numRecords)
error('The number of records in the dataset is not specified');
end
if ~exist('stepSize','var') || isempty(stepSize)
stepSize = 1; % steps taken when moving the window
end
if ~exist('predHorizon','var') || isempty(predHorizon)
predHorizon = 1; % aiming to predict this many steps in the future
end
if ~exist('windowLag','var') || isempty(windowLag)
windowLag = 10; % number of time steps to look back
end
if exist('windowLag','var') && (windowLag > numRecords)
error('The size of the window is larger than the number of observations');
end
if ~exist('windowPattern','var') || isempty(windowPattern)
windowPattern = 1:windowLag; % pattern of sampling data
end
if exist('windowPattern','var') && windowPattern(end) > windowLag
error('The window pattern must stop at the window lag specified');
end
% the number of samples in the window
maxSample = max(windowPattern);
indX = bsxfun(#plus, windowPattern, ...
(0:stepSize:(numRecords - maxSample - predHorizon))');
indY = bsxfun(#plus, max(windowPattern) + predHorizon, ...
(0:stepSize:(numRecords - maxSample - predHorizon))');
end
You can also find the code here: https://au.mathworks.com/matlabcentral/fileexchange/58730-get-sliding-indexes-numrecords--windowlag--predhorizon--windowpattern--stepsize-

Getting the N-dimensional product of vectors

I am trying to write code to get the 'N-dimensional product' of vectors. So for example, if I have 2 vectors of length L, x & y, then the '2-dimensional product' is simply the regular vector product, R=x*y', so that each entry of R, R(i,j) is the product of the i'th element of x and the j'th element of y, aka R(i,j)=x(i)*y(j).
The problem is how to elegantly generalize this in matlab for arbitrary dimensions. This is I had 3 vectors, x,y,z, I want the 3 dimensional array, R, such that R(i,j,k)=x(i)*y(j)*z(k).
Same thing for 4 vectors, x1,x2,x3,x4: R(i1,i2,i3,i4)=x1(i1)*x2(i2)*x3(i3)*x4(i4), etc...
Also, I do NOT know the number of dimensions beforehand. The code must be able to handle an arbitrary number of input vectors, and the number of input vectors corresponds to the dimensionality of the final answer.
Is there any easy matlab trick to do this and avoid going through each element of R specifically?
Thanks!
I think by "regular vector product" you mean outer product.
In any case, you can use the ndgrid function. I like this more than using bsxfun as it's a little more straightforward.
% make some vectors
w = 1:10;
x = w+1;
y = x+1;
z = y+1;
vecs = {w,x,y,z};
nvecs = length(vecs);
[grids{1:nvecs}] = ndgrid(vecs{:});
R = grids{1};
for i=2:nvecs
R = R .* grids{i};
end;
% Check results
for i=1:10
for j=1:10
for k=1:10
for l=1:10
V(i,j,k,l) = R(i,j,k,l) == w(i)*x(j)*y(k)*z(l);
end;
end;
end;
end;
all(V(:))
ans = 1
The built-in function bsxfun is a fast utility that should be able to help. It is designed to perform 2 input functions on a per-element basis for two inputs with mismatching dimensions. Singletons dimensions are expanded, and non-singleton dimensions need to match. (It sounds confusing, but once grok'd it useful in many ways.)
As I understand your problem, you can adjust the dimension shape of each vector to define the dimension that it should be defined across. Then use nested bsxfun calls to perform the multiplication.
Example code follows:
%Some inputs, N-by-1 vectors
x = [1; 3; 9];
y = [1; 2; 4];
z = [1; 5];
%The computation you describe, using nested BSXFUN calls
bsxfun(#times, bsxfun(#times, ... %Nested BSX fun calls, 1 per dimension
x, ... % First argument, in dimension 1
permute(y,2:-1:1) ) , ... % Second argument, permuited to dimension 2
permute(z,3:-1:1) ) % Third argument, permuted to dimension 3
%Result
% ans(:,:,1) =
% 1 2 4
% 3 6 12
% 9 18 36
% ans(:,:,2) =
% 5 10 20
% 15 30 60
% 45 90 180
To handle an arbitrary number of dimensions, this can be expanded using a recursive or loop construct. The loop would look something like this:
allInputs = {[1; 3; 9], [1; 2; 4], [1; 5]};
accumulatedResult = allInputs {1};
for ix = 2:length(allInputs)
accumulatedResult = bsxfun(#times, ...
accumulatedResult, ...
permute(allInputs{ix},ix:-1:1));
end

How do i add a random number to each matrix element in matlab?

I have the following matrix X and Column vector H:
X=[ 1 2 3; 2 3 4; 3 4 5];
H=[1; 2 ;3];
m=[X(:,1) H(:) X(:,2) H(:) X(:,3) H(:)];
How do i add a small random number to all elements of m while keeping every column H in the matrix m the same after adding the random number?
You should add the random numbers separately, and then build your combined matrix.
Xnew = X + rand(size(X)); % replace 'rand' with the random numbers you want
Hnew = H + rand(size(H));
mnew = [Xnew(:,1) Hnew(:) Xnew(:,2) Hnew(:) Xnew(:,3) Hnew(:)];
you can add the random to X instead of m
rX = X + rand(size(X)); % add uniformly distributed random numbers in [0,1] interval
m=[rX(:,1) H(:) rX(:,2) H(:) rX(:,3) H(:)];