I have the following script for importing text files into matlab which include hourly data, where I am then trying to convert them into daily averages:
clear all
pathName = ...
TopFolder = pathName;
dirListing = dir(fullfile(TopFolder,'*.txt'));%Lists the folders in the directory specified by pathName.
for i = 1:length(dirListing);
SubFolder{i} = dirListing(i,1).name;%obtain the name of each folder in
%the specified path.
end
%import data
for i=1:length(SubFolder);
rawData1{i} = importdata(fullfile(pathName,SubFolder{i}));
end
%convert into daily averages
rawData2=cell2mat(rawData1);
%create one matrix for entire data set
altered=reshape(rawData2,24,(size(rawData2,2)*365));
%convert into daily values
altered=mean(altered)';
%take the average for each day
altered=reshape(altered,365,size(rawData2,2));
%convert back into original format
My problem lies in trying to convert the data back into the same format as 'rawData1' which was a cell for each variable (where each variable is denoted by 'SubFolder'. The reason for doing this is that all but one of the variables are vectors, where the remaining variable is a matrix (8760*11).
So, an example of this would be:
clear all
cell_1 = rand(8760,1);
cell_2 = rand(8760,1);
cell_3 = rand(8760,1);
cell_4 = rand(8760,1);
cell_5 = rand(8760,1);
cell_6 = rand(8760,11);
cell_7 = rand(8760,1);
cell_8 = rand(8760,1);
cell_9 = rand(8760,1);
data = {cell_1,cell_2,cell_3,cell_4,cell_5,cell_6,cell_7,cell_8,cell_9};
Where I need to convert each cell in 'data' from hourly values into daily averages (i.e. 365 rows).
Any advice would be much appreciated.
I think this does what you want.
data = cellfun(#(x) reshape(mean(reshape(x,24,[]))',365,[]),data,'uniformoutput',false);
However that is kind of confusing so I will explain a little.
This part mean(reshape(x,24,[]))' inside of the cellfun will reshape each cell in data into a 24 by 365, compute the mean, then turn it back into a single column. This works fine when the original data only has 1 column ... but for cell_6 with 11 columns it puts all the data end to end. So I added an addition reshape(...) wrapper around the mean(...) part to put it back into the original 11 columns ... or more precises N columns that are 365 rows in length.
Note: This is going to give you errors if you ever have data sets dimensions are not 8760 by X.
Related
I got a couple 100 sensor measurement files all containing the date and time of measurement. All the files have names that include date and time. Example:
07-06-2016_17-58-32.wf
07-06-2016_18-02-32.wf
...
...
08-06-2016_17:48-26.wf
I have a function (importfile) and a loop that imports my data. The loop looks like this:
Files = dir('C:\Osci\User\*.waveform');
numFiles = length(Files);
Data = cell(1, numFiles);
for fileNum = 1:numFiles
Data{fileNum} = importfile(Files(fileNum).name);
end
Not all of these waveform files are useful. The measurement files are only useful if they were generated in a certain time period. I got a table that shows my allowed time periods:
07-Jun-2016 18:00:01
07-Jun-2016 18:01:31
07-Jun-2016 18:02:01
...
I want to modify my loop, so that the files (.waveform files) are only imported if the numbers for day (first number), hour (4th number) and minute (5th number) from the files match the numbers of the table containing the allowed time periods.
EDIT: Rather than a scalar hour, minute, and second, there is a vector of each. In my case, MyDay, MyHour and MyMinute are 1100x1 matrices while fileTimes only consists of 361 rows.
So, using the provided example the loop should only import file
07-06-2016_18-02-32.wf
since it is the only one where the numbers match (in this case 7, 18, 02).
EDIT2: Using #erfan's answer (and changing some directories and variable names) I have the following working code:
fmtstr = 'O:\\Basic_Research_All\\Lange\\Skripe ISAT\\Rohdaten\\*_%02i-*-*_%02i-%02i-*.wf';
Files = struct([]);
n = size(MyDayMyHourMyMinute);
for N = 1:n;
Files = [Files; dir(sprintf(fmtstr, MyDayMyHourMyMinute(N,:)))];
end
numFiles = length(Files);
WaveformData = cell(1, numFiles);
for fileNum = 1:numFiles
WaveformData{fileNum} = importfile(Files(fileNum).name);
end
Since your filenames are pretty well defined as dates and times, you can prefilter your list by turning them into actual dates and times:
% Get the file list
Files = dir('C:\Osci\User\*.waveform');
% You only need the names
Files = {Files.name};
% Get just the filename w/o the extension
[~, baseFileNames] = cellfun(#(x) fileparts(x), Files, 'UniformOutput', false);
% Your filename is just a date, so parse it as such
fileTimes = datevec(baseFileNames, 'mm-dd-yyyy_HH-MM-SS');
% Now pick out the files you want
% goodFiles = fileTimes(:, 4) == myHour & fileTimes(:, 5) == myMinute & fileTimes(:, 6) == mySecond;
goodFiles = ismember(fileTimes(:, 4:6), [myHour(:), myMinute(:), mySecond(:)], 'rows');
% Pare down your list of filenames
Files = Files(goodFiles);
% Preallocate your data cell
Data = cell(1, numel(Files));
% Now do your loop
for idx = 1:numel(Data)
Data{idx} = importfile(Files{idx});
end
You will, of course, need to define myHour, myMinute and mySecond. Of course, using the logical indexing in goodFiles, you could impose any sort of time criteria, like time or date range. If you find that your filenames aren't so well defined, you could parse out the filename using textscan or strfind to get the bits you want. The important thing is that cell arrays can be indexed into in much the same way as numerical or string arrays and it's often better to vectorize your filter criteria and then only do the loop on the parts you have to.
The OP indicated in a comment below that rather than a scalar hour, minute, and second, there is a vector of each. In that case, use ismember to match the two time vectors and return a logical index vector. With 2015a, MathWorks introduced the function ismembertol, which allows one to check membership within a certain tolerance.
You can apply your selection from the beginning. Imagine the acceptable values for day, hour and minute are saved in acc as an n*3 matrix. If you replace the first line of your code with:
fmtstr = 'C:\Osci\User\%02i-*-*_%02i-%02i-*.wf';
Files = struct([]);
for ii = 1:n
Files = [Files; dir(sprintf(fmtstr, acc(ii,:)))];
end
Then you have already applied your criteria to Files. The rest is the same.
I'm working to get a 1st level analysis completed on some fMRI data, and as it's my first time using SPM in this way, it seems as though there is no end to my frustrations. SPM includes these specific instructions:
"This *.mat file must include the following cell arrays (each 1 x n): names, onsets, and durations. eg. names=cell(1,5), onsets=cell(1,5), durations=cell(1,5), then names{2}="Second condition", onsets{2}=[3,5,19,22],durations{2}=[0,0,0,0], contain the required details of the second condition."
The code I'm using grabs the data I need from the various excel files the behavioral data is stored in, and adds them to these cell arrays.
sessionFiles = dir('*.xlsx');
allNames = {sessionFiles.name}';
conditions = 36;
% go through excel files to grab relevant column information for SPM
for i=1:length(sessionFiles)
[~,fileName,~] = fileparts(allNames{i});
% initialize cells SPM needs
names = cell(1,conditions);
onsets = cell(1,conditions);
durations = {1.75};
durations = repmat(durations,1,conditions);
% read in excel file
[num,~,~] = xlsread(sessionFiles(i).name);
trialType = num(:,6);
% grab condition information from columns: seconds=9, name=6
for j=1:conditions
index = find(trialType==j);
trialOnsets = cell(1,length(index));
names{1,j} = j;
for k=1:length(index)
trialOnsets{1,k}=double(num(index(k),9));
end
onsets{1,j} = trialOnsets;
end
% save new data for SPM
save(fileName,'names','onsets','durations');
clear names onsets durations fileName num raw text
end
I found an example which shows each cell should look like this:
I just can't figure out how to grab the numbers automatically and put them in cells like that.
I know this isn't the SPM forums, but I've seen a few questions posted and I thought I would try my luck.
With the line trialOnsets = cell(1,length(index));, trialOnsets is specified as a cell array of size 1xlength(index). Then, trialOnsets is assigned to onsets{1,j}. With this workflow, each cell of onsets will be of size 1xlength(index).
Instead, each cell of onsets should be of size 1x1 and each 1x1 cell in onsets should have a matrix of size 1xlength(index). To do this, do the following.
Specify trialOnsets as a matrix, instead of as a cell array. To do this, replace trialOnsets = cell(1,length(index)); with trialOnsets = zeros(1,length(index));.
Assign the values from num to trialOnsets, which is now a matrix (previously was a cell array). To do this, replace trialOnsets{1,k}=double(num(index(k),9)); with trialOnsets(1,k)=double(num(index(k),9));.
The edited code should be as follows:
sessionFiles = dir('*.xlsx');
allNames = {sessionFiles.name}';
conditions = 36;
% go through excel files to grab relevant column information for SPM
for i=1:length(sessionFiles)
[~,fileName,~] = fileparts(allNames{i});
% initialize cells SPM needs
names = cell(1,conditions);
onsets = cell(1,conditions);
durations = {1.75};
durations = repmat(durations,1,conditions);
% read in excel file
[num,~,~] = xlsread(sessionFiles(i).name);
trialType = num(:,6);
% grab condition information from columns: seconds=9, name=6
for j=1:conditions
index = find(trialType==j);
trialOnsets = zeros(1,length(index));
names{1,j} = j;
for k=1:length(index)
trialOnsets(1,k)=double(num(index(k),9));
end
onsets{1,j} = trialOnsets;
end
% save new data for SPM
save(fileName,'names','onsets','durations');
clear names onsets durations fileName num raw text
end
I could not test this code, since there was no sample data. Let me know if this works for you.
Can someone help me to understand how I can save in matlab a group of .csv files, select only the columns in which I am interested and get as output a final file in which I have the average value of the y columns and standard deviation of y axes? I am not so good in matlab and so I kindly ask if someone to help me to solve this question.
Here what I tried to do till now:
clear all;
clc;
which_column = 5;
dirstats = dir('*.csv');
col3Complete=0;
col4Complete=0;
for K = 1:length(dirstats)
[num,txt,raw] = xlsread(dirstats(K).name);
col3=num(:,3);
col4=num(:,4);
col3Complete=[col3Complete;col3];
col4Complete=[col4Complete;col4];
avgVal(K)=mean(col4(:));
end
col3Complete(1)=[];
col4Complete(1)=[];
%columnavg = mean(col4Complete);
%columnstd = std(col4Complete);
% xvals = 1 : size(columnavg,1);
% plot(xvals, columnavg, 'b-', xvals, columnavg-columnstd, 'r--', xvals, columnavg+columstd, 'r--');
B = reshape(col4Complete,[5000,K]);
m=mean(B,2);
C = reshape (col4Complete,[5000,K]);
S=std(C,0,2);
Now I know that I should compute mean and stdeviation inside for loop, using mean()function, but I am not sure how I can use it.
which_column = 5;
dirstats = dir('*.csv');
col3Complete=[]; % Initialise as empty matrix
col4Complete=[];
avgVal = zeros(length(dirstats),2); % initialise as columnvector
for K = 1:length(dirstats)
[num,txt,raw] = xlsread(dirstats(K).name);
col3=num(:,3);
col4=num(:,4);
col3Complete=[col3Complete;col3];
col4Complete=[col4Complete;col4];
avgVal(K,1)=mean(col4(:)); % 1st column contains mean
avgVal(K,2)=std(col4(:)); % 2nd column contains standard deviation
end
%columnavg = mean(col4Complete);
%columnstd = std(col4Complete);
% xvals = 1 : size(columnavg,1);
% plot(xvals, columnavg, 'b-', xvals, columnavg-columnstd, 'r--', xvals, columnavg+columstd, 'r--');
B = reshape(col4Complete,[5000,K]);
meanVals=mean(B,2);
I didn't change much, just initialised your arrays as empty arrays so you do not have to delete the first entry later on and made avgVal a column vector with the mean in column 1 and the standard deviation in column 1. You can of course add two columns if you want to collect those statistics for your 3rd column in the csv as well.
As a side note: xlsread is rather heavy for reading files, since Excel is horribly inefficient. If you want to read a structured file such as a csv, it's faster to use importdata.
Create some random matrix to store in a file with header:
A = rand(1e3,5);
out = fopen('output.csv','w');
fprintf(out,['ColumnA', '\t', 'ColumnB', '\t', 'ColumnC', '\t', 'ColumnD', '\t', 'ColumnE','\n']);
fclose(out);
dlmwrite('output.csv', A, 'delimiter','\t','-append');
Load it using csvread:
data = csvread('output.csv',1);
data now contains your five columns, without any headers.
I am trying to copy the results to a matrix and want the output in a 32768*8 array. This is the code I am using, but it stops working after the last line.
As you can see for the first file ( i=1), the decimal data,T(32768*1) is converted to M(32768*8). Now I want this M to be stored for each iteration of i, without overwriting anything.
Files_list = getAllFiles('C:\Stellaris Measurements\Stellaris-LM4F120_all');
for i = 1:15000
B=num2str(cell2mat(Files_list(i)));
fid = fopen(B,'rb');
T= fread(fid,inf,'uint8','ieee-be');
total = numel(T);
%M=textread('C:\Users\admin\Workspace\STELLARIS-LM4F120_00_210214_104000_0001_temp_025.bin','%2c');
%M=dec2bin(M);
M= de2bi(T,8,'left-msb');
M = measure(i);
end
So, basically I want to create a martix for each of the measurement, which will store the converted binary results in a 32768*8 array.
Thanks!
BR,
\Kashif
I want to import a sequence of excel files with a large amount of data in them. The problem that I have is I want to process the data in each file at a time and store the output from this into a variable, but each time I try to process a different file the variable gets overwritten in the variable workspace. Is there anyway I could store these files and process each file at a time?
numFiles = 1;
range = 'A2:Q21';
sheet = 1;
myData = cell(1,numFiles); % Importing data from Excel
for fileNum = 1:numFiles
fileName = sprintf('myfile%02d.xlsx',fileNum);
myData{fileNum} = importfile3(fileName,sheet,range);
end
data = cell2mat(myData);
The actual data import is performed by importfile3 which is, for the most part, a wrapper for the xlsread function that returns a matrix corresponding to the specified range of excel data.
function data = importfile3(workbookFile, sheetName, range)
% If no sheet is specified, read first sheet
if nargin == 1 || isempty(sheetName)
sheetName = 1;
end
% If no range is specified, read all data
if nargin <= 2 || isempty(range)
range = '';
end
%% Import the data
[~, ~, raw] = xlsread(workbookFile, sheetName, range);
%% Replace non-numeric cells with 0.0
R = cellfun(#(x) ~isnumeric(x) || isnan(x),raw); % Find non-numeric cells
raw(R) = {0.0}; % Replace non-numeric cells
%% Create output variable
data = cell2mat(raw);
The issue that you are running in to is a result of cell2mat concatenating all of the data in your cells in to one large 2-dimensional matrix. If you were to import two excel files with 20 rows and 17 columns, each, this would result in a 2-dimensional matrix of size [20 x 34]. The doc for cell2mat has a nice visual describing this.
I see that your importfile3 function returns a matrix, and based on your use of cell2mat in your final line of code, it looks like you would like to have your final result be in the form of a matrix. So I think the easiest way to go about this is to just bypass the intermediate myData cell array.
In the example code below, the resulting data is a 3-dimensional matrix. The 1st dimension indicates row number, 2nd dimension is column number, and 3rd dimension is file number. Cell arrays are very useful for "jagged" data, but based on the code you provided, each excel data set that you import will have the same number of rows and columns.
numFiles = 2;
range = 'A2:Q21';
sheet = 1;
% Number of rows and cols known before data import
numRows = 20;
numCols = 17;
data = zeros(numRows,numCols,numFiles);
for fileNum = 1:numFiles
fileName = sprintf('myfile%02d.xlsx',fileNum);
data(:,:,fileNum) = importfile3(fileName,sheet,range);
end
Accessing this data is now very straight-forward.
data(:,:,1) returns the data imported from your first excel file.
data(:,:,2) returns the data imported from your second excel file.
etc.