Suppose I have a date vector shown here by tt and a corresponding data series corresponding to aa. For example:
dd = datestr(datenum('2007-01-01 00:00','yyyy-mm-dd HH:MM'):1/24:...
datenum('2011-12-31 23:00','yyyy-mm-dd HH:MM'),...
'yyyy-mm-dd HH:MM');
tt = datevec(datenum(dd,'yyyy-mm-dd HH:MM'));
tt(1002,:) = [];
aa = rand(length(tt),1)
How is it possible to ensure that the hours and days are consistent among the years?
For example, I only want to keep times that are the same among years e.g.
2009-01-01 01:00
would be the same as
2010-01-01 01:00
ad so on.
If one year has a measurements at
2009-01-01 02:00
but yyyy-01-01 02:00
is not present in the other years, this time should re removed.
I would like the to return tt and aa where only those times that are consistent among the years are kept. how can this be done?
I was considering finding the indices for the unique years first as:
[~,~,iyears] = unique(tt(:,1),'rows');
and then find the indices for the unique month, day, and hour as:
[~,~,iid] = unique(tt(:,2:4),'rows');
but I am not sure how to combine these to give the desired output?
The solution below uses a loop to store data in an unitialized array, which can be inefficient, but unless your dataset is huge (with many, many years) it should do the job. The general idea is to break the dataset up into years. I am storing the resulting time-vectors in a cell array, because they probably won't have the same length. I then do a set-intersection of all the time vectors, to get a vector of common times. From there it's straight forward.
years = unique(tt(:,1), 'rows');
% Put the "sub-times" of each year into cell array
for ii = 1:length(years)
times_each_year{ii} = tt(tt(:,1)==years(ii),2:end);
end
% Do intersection of all "sub-times" sets
common_times = times_each_year{1};
for ii = 2:length(years)
common_times = intersect(common_times, times_each_year{ii},'rows');
end
% Find and delete the points that are not member of the "sub-times":
idx = ~ismember(tt(:,2:end),common_times,'rows');
deleted_points = datestr(tt(idx,:)); % for later review
tt(idx,:) = [];
However, note that the deleted_points vector contains more points than one might expect. That's because 2008 was a leap year, and all the points corresponding to Febr. 29th were deleted.
Another such oddity might await you if your data is "contaminated" by daylight savings time.
Code
a1 = str2num(datestr(tt,'mmddHHMM')); %// If in your data minutes are always 00, you can use 'mmddHH' instead and save some runtime
k1 = unique(a1);
gt1 = histc(a1,k1);
valid_rows = ismember(a1,k1(gt1==max(gt1)));
new_tt = tt(valid_rows,:); %// Desired tt output
new_aa = aa(valid_rows,:); %// Desired aa output
Explanation
To understand how it works, let's test out the code at a micro-level. Let's assume some small data that corresponds to tt -
data1 = [4 5 1 4 5 1 4 5 6]
data1 is the data collected over few sets and resembles tt that has data over few years with month, date, hour and minutes when these four parameters are conglomerated into a single parameter.
One can notice it would represent data from three sets/years with data as {4,5}, {1,4,5} and {1,4,5,6}. Our job is to found out all those values in data1 that is repeated across all the three years/sets of data. Thus, the final output must be {4,5}.
Let's see how this can be coded up.
Step 1: Get the unique values
unique_val = unique(data1)
We would have - [1 4 5 6]
Step 2: Get the count of unique values in the data
count_unique_val = histc(data1,unique_val)
Output is - [2 3 3 1]
Step 3: Get the indices from the unique values array where their counts are equal to the maximum of counts, indicating those are the unique values that are repeated across all the sets.
index1 = count_unique_val==max(count_unique_val)
Output comes out as - [0 1 1 0]
Step 4: Get those "consistent" unique values
consistent_val = unique_val(index1)
Gives us - [4 5], which is what we were looking for.
Step 5: Finally get the indices where the consistent data is present,
which can be used later on to select the rows with "consistent" data.
index_consistent_val = ismember(data1,consistent_val)
Output is - [1 1 0 1 1 0 1 1 0], which makes sense too.
Please note that in the original code a1 = str2num(datestr(tt,'mmddHHMM')); gets us the single parameter from the four parameters of month, date, hour and minutes as discussed in the comments earlier too.
Related
I have a very long text file which contains the data from 4 different stations with different time steps:
1:00
station 1
a number 1 (e.g.0.6E-06)
matrix1 (41x36)
station 2
number 2 (e.g.0.1E-06)
matrix2 (41x36)
station 3
number 3 (e.g.0.2E-06)
matrix3 (41x36)
station 4
number 4 (e.g.0.4E-06)
matrix4 (41x36)
2:00
station 1
a number (e.g.0.24E-06)
matrix5 (41x36)
station 2
a number (e.g.0.3E-06)
matrix6 (41x36)
station 3
number (e.g.0.12E-06)
matrix7 (41x36)
station 4
number (e.g.0.14E-06)
matrix8 (41x36)
.....
and so on
I need to read this data by each station and each step, and noted that each matrix should be scaled by multiplying with a number above it. An example is here: https://files.fm/u/sn447ttc#/view/example.txt
Could you please help?
Thank you a lot.
My idea here would be to read the textfile using fopen and textscan. Afterwards you can search for appearances of the Keyword FACTOR to subdivide the output. Here's the code:
fid=fopen('example.txt'); % open the document
dataRaw=textscan(fid,'%s','Delimiter',''); % read the file with no delimiter to achieve a cell array with 1 cell per line of the text file
fclose(fid); % close the document
rows=cellfun(#(x) strfind(x,'FACTOR'),dataRaw,'uni',0); % search for appearances of 'FACTOR'
hasFactor=find(~cellfun(#isempty,rows{1})); % get rownumbers of the lines that contain the word FACTOR
dataRaw=dataRaw{1}; % convert array for easier indexing
for ii=1:(numel(hasFactor)-1) % loop over appearances of the word FACTOR
array=cellfun(#str2num,dataRaw(hasFactor(ii)+2:hasFactor(ii+1)-1),'uni',0); % extract numerical data
output{ii}=str2num(dataRaw{hasFactor(ii)+1})*cat(1,array{:}); % create output scaled by the factor
end
array=cellfun(#str2num,dataRaw(hasFactor(end)+2:end),'uni',0);
output{end+1}=str2num(dataRaw{hasFactor(end)+1})*cat(1,array{:}); % These last 2 lines add the last array to the ouput
outputMat=cat(3,output{:}); % convert to a 3-dimensional matrix
outputStations=[{output(1:4:end)} {output(2:4:end)} {output(3:4:end)} {output(4:4:end)}]; % Sort the output to have 1 cell for each station
outputColumnSums=cellfun(#(x) cellfun(#sum,x,'uni',0),outputStations,'uni',0); % To sum up all the columns of each matrix
outputRowSums=cellfun(#(x) cellfun(#(y) sum(y,2),x,'uni',0),outputStations,'uni',0);
This approach is pretty slow and probably can be vectorized, but if you don't need it to be fast it should do the job. I created a cell-output with 1 cell per array and a 3 dimensional array as optional output. Hope that's fine with you
I have looked into your situation and it seems that the problem not trivial as anticipated. Keep in mind that if I have made mistakes on the assumption of the location of the data, you can let me know so I can edit it, or you can just change the numbers to that which suits your case. In this case, I initially loaded the delimited file into an Excel spreadsheet, just to visualize it.
After reading up on dlmread, I found that one can specify the exact rows and columns to pull from example.txt, as shown here:
data = dlmread('example.txt', ' ', [4 1 45 37]); % [r1 c1 r2 c2]
data2 = dlmread('example.txt', ' ', [47 1 88 37]);
The result of which is two matrices that are 41-by-37, containing only numbers. I started data at row 4 to bypass the header information/strings. Noticing the pattern, I set it up as a loop:
No_of_matrices_expected = 4;
dataCell = cell(No_of_matrices_expected, 1);
iterations = length(dataCell)
% Initial Conditions
rowBeginning = 4;
col1 = 1; % Constant
rowEnd = rowBeginning + 40; % == 44, right before next header information
col2 = 36; % Constant
for n = 1 : iterations
dataCell{n} = dlmread('example.txt', ' ', [rowBeginning, col1, rowEnd, col2]);
rowBeginning = rowBeginning + 41 + 2; % skip previous matrix and skip header info
rowEnd = rowBeginning + 40;
end
However, I stumbled across what you stated earlier which was that there are four different stations, each with their own time stamps. So running this loop more than 4 times led to unexpected results and MATLAB crashed. The reason is that the new timestamp creates an extra row for the date. Now, you could change the loop above to compensate for this extra row, or you can make multiple for loops for each station. This will be your decision to make.
Now if you wanted to save the header information, I would recommend taking a look into textscan. You can simply use this function to pull the first column of all the data into a cell array of strings. Then you can pull out the header information that you want. Keep in mind, use fopen if you want to use textscan.
I'll let you use what I have found thus far, but let me know if you need more help.
Numbers
I have a column of years from 1981 to 2000 that corresponds to another column of prices for a good. I am trying to make a loop that iterates through only the years from 1990 to 2000 and prints the prices in order that correlates with their year. I have this code so far but I'm not sure why it won't run, any help would be awesome.
for x=1:year == 1990:2000
v = find(isfinite(price));
v
end
If your input data is something like this where the first column is year and the second column is price
data = [1990, 2.50;
1991, 3.00;
...
2000, 4.00];
You can loop through the years in your for loop (Note the syntax and how this compares to the one in your post) and then find the second column where the price corresponds to that year using logical indexing.
for year = 1990:2000
% Grabs column 2 where column 1 is equal to the year
price = data(data(:,1) == year, 2);
end
Even if your data lives in two different data structures you can do something similar (as long as they are the same size).
years = [1990, 1991, 1992, ... 2000];
prices = [2.50, 3.00, 3.50, ... 4.00];
for year = 1990:2000
price = prices(years == year);
end
Edit
If you are for-loop averse, you can definitly do the same thing without a for loop. The most robust solution is to use arrayfun.
annualPrices = arrayfun(#(x)prices(years == x), years, 'uniform', 0);
This will return a cell array where each element is all prices for a given year.
If you're guaranteed to only have one price per year, however, you can omit the uniform input and you'll get an array of prices.
annualPrices = arrayfun(#(x)prices(years == x), years);
One of the benefits is that neither of these approaches requires extra operations (such as sorting) on your data.
Example 1:
Let's make a matrix holding your data:
M = ones(100,2); % 1st column for the year and the second column for the prices
M(:,1) = (1951:2050).';
M(:,2) = rand(100,1);
A one liner to your question can be as follows:
M((M(:,1)<= 2000 & M(:,1) >= 1990),2)
Example 2:
If you have prices and years in two vectors, first make sure your years are sorted:
[sortedYears,Idx] = sort(years); % sort the years vector
sortedPrices = prices(Idx); % use the index to sort the prices in the same order
Now use the following one liner:
sortedPrices((sortedYears<= 2000 & sortedYears >= 1990));
I need help with taking the following data which is organized in a large matrix and averaging all of the values that have a matching ID (index) and outputting another matrix with just the ID and the averaged value that trail it.
File with data format:
(This is the StarData variable)
ID>>>>Values
002141865 3.867144e-03 742.000000 0.001121 16.155089 6.297494 0.001677
002141865 5.429278e-03 1940.000000 0.000477 16.583748 11.945627 0.001622
002141865 4.360715e-03 1897.000000 0.000667 16.863406 13.438383 0.001460
002141865 3.972467e-03 2127.000000 0.000459 16.103060 21.966853 0.001196
002141865 8.542932e-03 2094.000000 0.000421 17.452007 18.067214 0.002490
Do not be mislead by the examples I posted, that first number is repeated for about 15 lines then the ID changes and that goes for an entire set of different ID's, then they are repeated as a whole group again, think first block of code = [1 2 3; 1 5 9; 2 5 7; 2 4 6] then the code repeats with different values for the columns except for the index. The main difference is the values trailing the ID which I need to average out in matlab and output a clean matrix with only one of each ID fully averaged for all occurrences of that ID.
Thanks for any help given.
A modification of this answer does the job, as follows:
[value_sort ind_sort] = sort(StarData(:,1));
[~, ii, jj] = unique(value_sort);
n = diff([0; ii]);
averages = NaN(length(n),size(StarData,2)); % preallocate
averages(:,1) = StarData(ii,1);
for col = 2:size(StarData,2)
averages(:,col) = accumarray(jj,StarData(ind_sort,col))./n;
end
The result is in variable averages. Its first column contains the values used as indices, and each subsequent column contains the average for that column according to the index value.
Compatibility issues for Matlab 2013a onwards:
The function unique has changed in Matlab 2013a. For that version onwards, add 'legacy' flag to unique, i.e. replace second line by
[~, ii, jj] = unique(value_sort,'legacy')
I have observed daily data that I need to compare to generated Monthly data so I need to get a mean of each month over the thirty year period.
My observed data set is currently in 365x31 with rows being each day (no leap years!) and the extra column being the month number (1-12).
the problem I am having is that I can only seem to get a script to get the mean of all years. ie. I cannot figure how to get the script to do it for each column separately. Example of the data is below:
1 12 14
1 -15 10
2 13 3
2 2 37
...all the way to 12 for 365 rows
SO: to recap, I need to get the mean of [12; -15; 13; 2] then [14; 10; 3; 37] and so on.
I have been trying to use the unique() function to loop through which works for getting the number rows to average but incorrect means. Now I need it to do each month(28-31 rows) and column individually. Result should be a 12x30 matrix. I feel like I am missing something SIMPLE. Code:
u = unique(m); %get unique values of m (months) ie) 1-12
for i=1:length(u)
month(i) = mean(obatm(u(i), (2:31)); % the average for each month of each year
end
Appreciate any ideas! Thanks!
You can simply filter the rows for each month and then apply mean, like so:
month = zeros(12, size(obatm, 2));
for k = 1:12
month(k, :) = mean(obatm(obatm(:, 1) == k, :));
end
EDIT:
If you want something fancy, you can also do this:
cols = size(obatm, 2) - 1;
subs = bsxfun(#plus, obatm(:, 1), (0:12:12 * (cols - 1)));
vals = obatm(:, 2:end);
month = reshape(accumarray(subs(:), vals(:), [12 * cols, 1], #mean), 12, cols)
Look, Ma, no loops!
I have 19 cells (19x1) with temperature data for an entire year where the first 18 cells represent 20 days (each) and the last cell represents 5 days, hence (18*20)+5 = 365days.
In each cell there should be 7200 measurements (apart from cell 19) where each measurement is taken every 4 minutes thus 360 measurements per day (360*20 = 7200).
The time vector for the measurements is only expressed as day number i.e. 1,2,3...and so on (thus no decimal day),
which is therefore displayed as 360 x 1's... and so on.
As the sensor failed during some days, some of the cells contain less than 7200 measurements, where one in
particular only contains 858 rows, which looks similar to the following example:
a=rand(858,3);
a(1:281,1)=1;
a(281:327,1)=2;
a(327:328,1)=5;
a(329:330,1)=9;
a(331:498,1)=19;
a(499:858,1)=20;
Where column 1 = day, column 2 and 3 are the data.
By knowing that each day number should be repeated 360 times is there a method for including an additional
amount of every value from 1:20 in order to make up the 360. For example, the first column requires
79 x 1's, 46 x 2's, 360 x 3's... and so on; where the final array should therefore have 7200 values in
order from 1 to 20.
If this is possible, in the rows where these values have been added, the second and third column should
changed to nan.
I realise that this is an unusual question, and that it is difficult to understand what is asked, but I hope I have been clear in expressing what i'm attempting to
acheive. Any advice would be much appreciated.
Here's one way to do it for a given element of the cell matrix:
full=zeros(7200,3)+NaN;
for i = 1:20 % for each day
starti = (i-1)*360; % find corresponding 360 indices into full array
full( starti + (1:360), 1 ) = i; % assign the day
idx = find(a(:,1)==i); % find any matching data in a for that day
full( starti + (1:length(idx)), 2:3 ) = a(idx,2:3); % copy matching data over
end
You could probably use arrayfun to make this slicker, and maybe (??) faster.
You could make this into a function and use cellfun to apply it to your cell.
PS - if you ask your question at the Matlab help forums you'll most definitely get a slicker & more efficient answer than this. Probably involving bsxfun or arrayfun or accumarray or something like that.
Update - to do this for each element in the cell array the only change is that instead of searching for i as the day number you calculate it based on how far allong the cell array you are. You'd do something like (untested):
for k = 1:length(cellarray)
for i = 1:length(cellarray{k})
starti = (i-1)*360; % ... as before
day = (k-1)*20 + i; % first cell is days 1-20, second is 21-40,...
full( starti + (1:360),1 ) = day; % <-- replace i with day
idx = find(a(:,1)==day); % <-- replace i with day
full( starti + (1:length(idx)), 2:3 ) = a(idx,2:3); % same as before
end
end
I am not sure I understood correctly what you want to do but this below works out how many measurements you are missing for each day and add at the bottom of your 'a' matrix additional lines so you do get the full 7200x3 matrix.
nbMissing = 7200-size(a,1);
a1 = nan(nbmissing,3)
l=0
for i = 1:20
nbMissing_i = 360-sum(a(:,1)=i);
a1(l+1:l+nbMissing_i,1)=i;
l = l+nb_Missing_i;
end
a_filled = [a;a1];