I have 2 data vectors with corresponding time vectors. This data is sampled nearly simultaneously but they have slightly different timestamps (from machine precision transmission delays etc.). One or both of the data vectors experience occasional data losses & occasional double samples due to telemetry issues.
I want to match up the data arrays to where their times match to perform some math operations between them. Essentially remove points from y1 & y2 where they do not have corresponding times x1 & x2 (within about 1/2 of the sample rate to be considered a match).
Note I do not want to interpolate y1 & y2
%Sample time stamps: Real ones are much faster and not as neat.
x1 = [1 2 3 4 5 5.1 6 7 8 10 ]; %note double sample at ~5.
x2 = [.9 4.9 5.9 6.9 8.1 9.1 10.1]; %Slightly different times.
%Sample data: y is basically y1+1 if no data was missing
y1 = [1 2 3 4 5 5 6 7 8 10];
y2 = [2 6 7 8 9 10 11];
So the result should look like:
y1_m = [1 5 6 7 8 10];
y2_m = [2 6 7 8 9 11];
What I have so far: I used interp1 to find the closest time points between the 2 time arrays. Then got the time delta between them like this:
>> idx = interp1(x2,1:numel(x2),x1,'nearest','extrap')
idx =
1 1 2 2 2 2 3 4 5 7
>> xDelta = abs(x2(idx) - x1)
xDelta =
0.1000 1.1000 1.9000 0.9000 0.1000 0.2000 0.1000 0.1000 0.1000 0.1000
Now what I think I need to do is for each unique idx find the min xDelta and that should get me all the matching points. However, I haven't come up with a clever way of doing that... It seems like accumarray should be useful here but so far I failed at using it.
Here's a rough idea which you could likely improve upon, using unique and ismembertol:
function [y1_m, y2_m] = q48723002
%% Stage 0 - Setup:
%Sample time stamps: Real ones are much faster and not as neat.
x1 = [1 2 3 4 5 5.1 6 7 8 10 ]; %note double sample at ~5.
x2 = [.9 4.9 5.9 6.9 8.1 9.1 10.1]; %Slightly different times.
%Sample data: y is basically y1+1 if no data was missing
y1 = [1 2 3 4 5 5 6 7 8 10];
y2 = [2 6 7 8 9 10 11];
%% Stage 1 - Remove repeating samples:
SR = 0.5; % Sampling rate, for rounding.
[~,Loc1] = ismembertol(x1,round(x1/SR)*SR,SR/2,'DataScale',1);
[~,Loc2] = ismembertol(x2,round(x2/SR)*SR,SR/2,'DataScale',1);
u1 = unique(Loc1);
u2 = unique(Loc2);
x1u = x1(u1);
y1u = y1(u1);
x2u = x2(u2);
y2u = y2(u2);
clear Loc1 Loc2
%% Stage 2 - Get a vector of reference time steps:
ut = union(u1,u2);
%% Stage 3 - Only keep times found in both
[In1,Loc1] = ismembertol(ut,x1u,SR/2,'DataScale',1);
[In2,Loc2] = ismembertol(ut,x2u,SR/2,'DataScale',1);
valid = In1 & In2;
%% Stage 4 - Output:
y1_m = ut(Loc1(valid)); % equivalently: y1_m = ut(valid)
y2_m = y1_m + 1;
ans =
1 5 6 7 8 9
See also: uniquetol.
Here is a solution based on #Cris Luengo's comment on the original question.
It uses a sortrows & unique to get the lowest time error for each pairing of data points.
%Sample time stamps: Real ones are much faster and not as neat.
x1 = [1 2 3 4 5 5.1 6 7 8 10 ]; %note double sample at ~5.
x2 = [.9 4.9 5.9 6.9 8.1 9.1 10.1]; %Slightly different times.
%Sample data: y is basically y1+1 if no data was missing
y1 = [1 2 3 4 5 5 6 7 8 10];
y2 = [2 6 7 8 9 10 11];
%Find the nearest match
idx = interp1(x2,1:numel(x2),x1,'nearest','extrap');
xDiff = abs(x2(idx) - x1);
% Combine the matched indices & the deltas together & sort by rows.
%So lowest delta for a given index is first.
[A, idx1] = sortrows([idx(:) xDiff(:)]);
[idx2, uidx] = unique(A(:,1),'first');
idx1 = idx1(uidx); %resort idx1
%output
y1_m = y1(idx1)
y2_m = y2(idx2)
y1_m =
1 5 6 7 8 10
y2_m =
2 6 7 8 9 11
Related
I would like to align and count vectors with different time stamps to count the corresponding bins.
Let's assume I have 3 matrix from [N,edges] = histcounts in the following structure. The first row represents the edges, so the bins. The second row represents the values. I would like to sum all values with the same bin.
A = [0 1 2 3 4 5;
5 5 6 7 8 5]
B = [1 2 3 4 5 6;
2 5 7 8 5 4]
C = [2 3 4 5 6 7 8;
1 2 6 7 4 3 2]
Now I want to sum all the same bins. My final result should be:
result = [0 1 2 3 4 5 6 7 8;
5 7 12 16 ...]
I could loop over all numbers, but I would like to have it fast.
You can use accumarray:
H = [A B C].'; %//' Concatenate the histograms and make them column vectors
V = [unique(H(:,1)) accumarray(H(:,1)+1, H(:,2))].'; %//' Find unique values and accumulate
V =
0 1 2 3 4 5 6 7 8
5 7 12 16 22 17 8 3 2
Note: The H(:,1)+1 is to force the bin values to be positive, otherwise MATLAB will complain. We still use the actual bins in the output V. To avoid this, as #Daniel says in the comments, use the third output of unique (See: https://stackoverflow.com/a/27783568/2732801):
H = [A B C].'; %//' stupid syntax highlighting :/
[U, ~, IU] = unique(H(:,1));
V = [U accumarray(IU, H(:,2))].';
If you're only doing it with 3 variables as you've shown then there likely aren't going to be any performance hits with looping it.
But if you are really averse to the looping idea, then you can do it using arrayfun.
rng = 0:8;
output = arrayfun(#(x)sum([A(2,A(1,:) == x), B(2,B(1,:) == x), C(2,C(1,:) == x)]), rng);
output = cat(1, rng, output);
output =
0 1 2 3 4 5 6 7 8
5 7 12 16 22 17 8 3 2
This can be beneficial for particularly large A, B, and C variables as there is no copying of data.
I am trying to replicate the Kron product using only repmat and reshape and I believe I am pretty close but I can't manage to do the last correct reshape.
Particularly I have problem in reshaping A
To make things simple let suppose we have
A=[1 3; 2 4]
B=[5 10; 10 5]
so my kron(A,B) is going to be a 4x4 matrix.
kron=[5 10 15 30
10 5 30 15
10 20 20 40
20 10 40 20]
I am proceeding this way:
Y=repmat(B,2,2)
X=A(:);
X=repmat(X,1,2)';
X=X(:);
X=repmat(X,1,2);
which gives me the following 8x2 Matrix:
X= [1 1
1 1
2 2
2 2
3 3
3 3
4 4
4 4]
I can't just figure out how to do the correct reshape to obtain my 4x4 matrix:
X=[1 1 3 3
1 1 3 3
2 2 4 4
2 2 4 4]
Then I will be able to compute: X.*Y=kron(A,B)
Here's one approach using the powerful trio of bsxfun, permute and reshape -
M = bsxfun(#times,B,permute(A,[3 4 1 2]));
out = reshape(permute(M,[1 3 2 4]),size(A,1)*size(B,1),[]);
If you are hell bent on using repmat, perform the calculation of M with it, like so -
M = repmat(B,[1 1 size(A)]).*permute(repmat(A,[1 1 size(B)]),[3 4 1 2])
Verify output by comparing against kron for generic matrix sizes -
>> A = rand(4,5);
>> B = rand(6,7);
>> M = bsxfun(#times,B,permute(A,[3 4 1 2]));
>> out = reshape(permute(M,[1 3 2 4]),size(A,1)*size(B,1),[]);
>> out_kron = kron(A,B);
>> max(abs(out(:) - out_kron(:)))
ans =
0
Here's one using matrix-multiplication and as such must be pretty efficient -
[mA,nA] = size(A);
[mB,nB] = size(B);
out = reshape(permute(reshape(B(:)*A(:).',mB,nB,mA,nA),[1 3 2 4]),mA*mB,[])
If you don't want to use any loops or bsxfun/arrayfun-solutions, you can do as follows:
[ma,na] = size(A);
[mb,nb] = size(B);
Y = repmat(B,ma,mb);
X = reshape(repmat(reshape(repmat(A(:),1,mb)',ma*mb,na),nb,1),ma*mb,na*nb);
X.*Y
Referring to Reshape row wise w/ different starting/ending elements number #Divakar came with a nice solution but, what if the number of columns is not always the same?
Sample run -
>> A'
ans =
4 9 8 9 6 1 8 9 7 7 7 4 6 2 7 1
>> out
out =
4 9 8 9 0 0
6 1 8 9 7 7
7 4 6 2 7 1
I took only the first 4 terms of A and put them in out, then fill the rest 2 empty cell with 0's. So the ncols = [4 6 6]. Unfortunately vet2mat doesn't allow vector as columns number.
Any suggestions?
You can employ bsxfun's masking capability here -
%// Random inputs
A = randi(9,1,15)
ncols = [4 6 5]
%// Initialize output arary of transposed size as compared to the desired
%// output arary size, as we need to insert values into it row-wise and MATLAB
%// follows column-major indexing
out = zeros(max(ncols),numel(ncols));
mask = bsxfun(#le,[1:max(ncols)]',ncols); %//'# valid positions mask for output
out(mask) = A; %// insert input array elements
out = out.' %//'# transpose output back to the desired output array size
Code run -
A =
5 3 7 2 7 2 4 6 8 1 9 7 5 4 5
ncols =
4 6 5
out =
5 3 7 2 0 0
7 2 4 6 8 1
9 7 5 4 5 0
You could use accumarray for that:
A = [4 9 8 9 6 1 8 9 7 7 7 4 6 2 7 1].'; %'// data
ncols = [4 6 6]; %// columns
n = max(ncols);
cs = cumsum(ncols);
ind = 1;
ind(cs+1) = 1;
ind = cumsum(ind(1:end-1)); %// `ind` tells the row for each element of A
result = accumarray(ind(:), A(:), [], #(x) {[x; zeros(n-numel(x),1)]}); %// split `A` as
%// dictated by `ind`, and fill with zeros. Each group is put into a cell.
result = [result{:}].'; %'// concatenate all cells
This question pops up quite often in one form or another (see for example here or here). So I thought I'd present it in a general form, and provide an answer which might serve for future reference.
Given an arbitrary number n of vectors of possibly different sizes, generate an n-column matrix whose rows describe all combinations of elements taken from those vectors (Cartesian product) .
For example,
vectors = { [1 2], [3 6 9], [10 20] }
should give
combs = [ 1 3 10
1 3 20
1 6 10
1 6 20
1 9 10
1 9 20
2 3 10
2 3 20
2 6 10
2 6 20
2 9 10
2 9 20 ]
The ndgrid function almost gives the answer, but has one caveat: n output variables must be explicitly defined to call it. Since n is arbitrary, the best way is to use a comma-separated list (generated from a cell array with ncells) to serve as output. The resulting n matrices are then concatenated into the desired n-column matrix:
vectors = { [1 2], [3 6 9], [10 20] }; %// input data: cell array of vectors
n = numel(vectors); %// number of vectors
combs = cell(1,n); %// pre-define to generate comma-separated list
[combs{end:-1:1}] = ndgrid(vectors{end:-1:1}); %// the reverse order in these two
%// comma-separated lists is needed to produce the rows of the result matrix in
%// lexicographical order
combs = cat(n+1, combs{:}); %// concat the n n-dim arrays along dimension n+1
combs = reshape(combs,[],n); %// reshape to obtain desired matrix
A little bit simpler ... if you have the Neural Network toolbox you can simply use combvec:
vectors = {[1 2], [3 6 9], [10 20]};
combs = combvec(vectors{:}).' % Use cells as arguments
which returns a matrix in a slightly different order:
combs =
1 3 10
2 3 10
1 6 10
2 6 10
1 9 10
2 9 10
1 3 20
2 3 20
1 6 20
2 6 20
1 9 20
2 9 20
If you want the matrix that is in the question, you can use sortrows:
combs = sortrows(combvec(vectors{:}).')
% Or equivalently as per #LuisMendo in the comments:
% combs = fliplr(combvec(vectors{end:-1:1}).')
which gives
combs =
1 3 10
1 3 20
1 6 10
1 6 20
1 9 10
1 9 20
2 3 10
2 3 20
2 6 10
2 6 20
2 9 10
2 9 20
If you look at the internals of combvec (type edit combvec in the command window), you'll see that it uses different code than #LuisMendo's answer. I can't say which is more efficient overall.
If you happen to have a matrix whose rows are akin to the earlier cell array you can use:
vectors = [1 2;3 6;10 20];
vectors = num2cell(vectors,2);
combs = sortrows(combvec(vectors{:}).')
I've done some benchmarking on the two proposed solutions. The benchmarking code is based on the timeit function, and is included at the end of this post.
I consider two cases: three vectors of size n, and three vectors of sizes n/10, n and n*10 respectively (both cases give the same number of combinations). n is varied up to a maximum of 240 (I choose this value to avoid the use of virtual memory in my laptop computer).
The results are given in the following figure. The ndgrid-based solution is seen to consistently take less time than combvec. It's also interesting to note that the time taken by combvec varies a little less regularly in the different-size case.
Benchmarking code
Function for ndgrid-based solution:
function combs = f1(vectors)
n = numel(vectors); %// number of vectors
combs = cell(1,n); %// pre-define to generate comma-separated list
[combs{end:-1:1}] = ndgrid(vectors{end:-1:1}); %// the reverse order in these two
%// comma-separated lists is needed to produce the rows of the result matrix in
%// lexicographical order
combs = cat(n+1, combs{:}); %// concat the n n-dim arrays along dimension n+1
combs = reshape(combs,[],n);
Function for combvec solution:
function combs = f2(vectors)
combs = combvec(vectors{:}).';
Script to measure time by calling timeit on these functions:
nn = 20:20:240;
t1 = [];
t2 = [];
for n = nn;
%//vectors = {1:n, 1:n, 1:n};
vectors = {1:n/10, 1:n, 1:n*10};
t = timeit(#() f1(vectors));
t1 = [t1; t];
t = timeit(#() f2(vectors));
t2 = [t2; t];
end
Here's a do-it-yourself method that made me giggle with delight, using nchoosek, although it's not better than #Luis Mendo's accepted solution.
For the example given, after 1,000 runs this solution took my machine on average 0.00065935 s, versus the accepted solution 0.00012877 s. For larger vectors, following #Luis Mendo's benchmarking post, this solution is consistently slower than the accepted answer. Nevertheless, I decided to post it in hopes that maybe you'll find something useful about it:
Code:
tic;
v = {[1 2], [3 6 9], [10 20]};
L = [0 cumsum(cellfun(#length,v))];
V = cell2mat(v);
J = nchoosek(1:L(end),length(v));
J(any(J>repmat(L(2:end),[size(J,1) 1]),2) | ...
any(J<=repmat(L(1:end-1),[size(J,1) 1]),2),:) = [];
V(J)
toc
gives
ans =
1 3 10
1 3 20
1 6 10
1 6 20
1 9 10
1 9 20
2 3 10
2 3 20
2 6 10
2 6 20
2 9 10
2 9 20
Elapsed time is 0.018434 seconds.
Explanation:
L gets the lengths of each vector using cellfun. Although cellfun is basically a loop, it's efficient here considering your number of vectors will have to be relatively low for this problem to even be practical.
V concatenates all the vectors for easy access later (this assumes you entered all your vectors as rows. v' would work for column vectors.)
nchoosek gets all the ways to pick n=length(v) elements from the total number of elements L(end). There will be more combinations here than what we need.
J =
1 2 3
1 2 4
1 2 5
1 2 6
1 2 7
1 3 4
1 3 5
1 3 6
1 3 7
1 4 5
1 4 6
1 4 7
1 5 6
1 5 7
1 6 7
2 3 4
2 3 5
2 3 6
2 3 7
2 4 5
2 4 6
2 4 7
2 5 6
2 5 7
2 6 7
3 4 5
3 4 6
3 4 7
3 5 6
3 5 7
3 6 7
4 5 6
4 5 7
4 6 7
5 6 7
Since there are only two elements in v(1), we need to throw out any rows where J(:,1)>2. Similarly, where J(:,2)<3, J(:,2)>5, etc... Using L and repmat we can determine whether each element of J is in its appropriate range, and then use any to discard rows that have any bad element.
Finally, these aren't the actual values from v, just indices. V(J) will return the desired matrix.
This is the input matrix
7 9 6
8 7 9
7 6 7
Based on the frequency their appearance in the matrix (Note. these values are for explanation purpose. I didn't pre-calculate them in advance. That why I ask this question)
number frequency
6 2
7 4
8 1
9 2
and the output I expect is
4 2 2
1 4 2
4 2 4
Is there a simple way to do this?
Here's a three-line solution. First prepare the input:
X = [7 9 6;8 7 9;7 6 7];
Now do:
[a m n] = unique(X);
b = hist(X(:),a);
c = reshape(b(n),size(X));
Which gives this value for c:
4 2 2
1 4 2
4 2 4
If you also wanted the frequency matrix, you can get it with this code:
[a b']
Here is a code with for-loop (a is input matrix, freq - frequency matrix with 2 columns):
weight = zeros(size(a));
for k = 1:size(freq,1)
weight(a==freq(k,1)) = freq(k,2);
end
Maybe it can be solved without loops, but my code looks like:
M = [7 9 6 ;
8 7 9 ;
7 6 7 ;];
number = unique(M(:));
frequency = hist(M(:), number)';
map = containers.Map(number, frequency);
[height width] = size(M);
result = zeros(height, width); %allocate place
for i=1:height
for j=1:width
result(i,j) = map(M(i,j));
end
end