I have a function which generates yout_new(5000,1) at every iteration and I want to store this data to a netcdf file and append the new data generated at every iteration into this existing file . At the 2nd iteration the stored variable size should be yout_new(5000,2) . Here is my try which doesn't work. Is there is any nice way to do it ?
neq=5000;
filename='thrust.nc';
if ~exist(filename, 'file')
%% create file
ncid=netcdf.create(filename,'NC_WRITE');
%%define dimension
tdimID = netcdf.defDim(ncid,'t',...
netcdf.getConstant('NC_UNLIMITED'));
ydimID = netcdf.defDim(ncid,'y',neq);
%%define varibale
varid = netcdf.defVar(ncid,'yout','NC_DOUBLE',[ydimID tdimID]);
netcdf.endDef(ncid);
%%put variables from workspace ( i is the iteration)
netcdf.putVar(ncid,varid,[ 0 0 ],[ neq 0],yout_new);
%%close the file
netcdf.close(ncid);
else
%% open the existing file
ncid=netcdf.open(filename,'NC_WRITE');
%Inquire variables
[varname,xtype,dimids,natts] = netcdf.inqVar(ncid,0);
varid = netcdf.inqVarID(ncid,varname);
%Enquire current dimension length
[dimname, dimlen] = netcdf.inqDim(ncid,0);
% Append new data to existing variable.
netcdf.putVar(ncid,varid,dimlen,numel(yout_new),yout_new);
netcdf.close(ncid);
There are more easy functions in MATLAB, to deal with netCDF. You read about ncdisp, ncinfo,nccreate,ncread,ncwrite. Coming to the question, you said you have to write two columns, I will take number of columns as variable (infinity), every time you can append the columns. Check the below code:
N = 3 ; % number of columns
rows = 5000 ; % number of rows
ncfile = 'myfile.nc' ; % my ncfile name
nccreate(ncfile,'yout_new','Dimensions',{'row',rows,'col',Inf},'DeflateLevel',5) ; % creat nc file
% generate your data in loop and write to nc file
for i = 1:N
yout_new = rand(rows,1) ;
ncwrite(ncfile,'yout_new',yout_new,[1,i]) ;
end
Please not that, it is not mandatory to make number of columns as unlimited, you can fix it to your desired number instead of inf.
Related
This function reads the data from multiple mat files and save them in multiple txt files. But the data (each value) are saved one value in one column and so on. I want to save the data in a form of three columns (coordinates) in the text files, so each row has three values separated by space. Reshape the data before i save them in a text file doesn't work. I know that dlmwrite should be modified in away to make newline after three values but how?
mat = dir('*.mat');
for q = 1:length(mat)
load(mat(q).name);
[~, testName, ~] = fileparts(mat(q).name);
testVar = eval(testName);
pos(q,:,:) = testVar.Bodies.Positions(1,:,:);
%pos=reshape(pos,2,3,2000);
filename = sprintf('data%d.txt', q);
dlmwrite(filename , pos(q,:,:), 'delimiter','\t','newline','pc')
end
My data structure:
These data should be extracted from each mat file and stored in the corresponding text files like this:
332.68 42.76 42.663 3.0737
332.69 42.746 42.655 3.0739
332.69 42.75 42.665 3.074
A TheMathWorks-trainer once told me that there is almost never a good reason nor a need to use eval. Here's a snippet of code that should solve your writing problem using writematrix since dlmwrite is considered to be deprecated.
It further puts the file-handling/loading on a more resilient base. One can access structs dynamically with the .(FILENAME) notation. This is quite convenient if you know your fields. With who one can list variables in the workspace but also in .mat-files!
Have a look:
% path to folder
pFldr = pwd;
% get a list of all mat-files (returns an array of structs)
Lst = dir( fullfile(pFldr,'*.mat') );
% loop over files
for Fl = Lst.'
% create path to file
pFl = fullfile( Fl.folder, Fl.name );
% variable to load
[~, var2load, ~] = fileparts(Fl.name);
% get names of variables inside the file
varInfo = who('-file',pFl);
% check if it contains the desired variables
if ~all( ismember(var2load,varInfo) )
% display some kind of warning/info
disp(strcat("the file ",Fl.name," does not contain all required varibales and is therefore skipped."))
% skip / continue with loop
continue
end
% load | NO NEED TO USE eval()
Dat = load(pFl, var2load);
% DO WHATEVER YOU WANT TO DO
pos = squeeze( Dat.(var2load)(1,:,1:2000) );
% create file name for text file
pFl2save = fullfile( Fl.folder, strrep(Fl.name,'.mat','.txt') );
writematrix(pos,pFl2save,'Delimiter','\t')
end
To get your 3D-matrix data into a 2D matrix that you can write nicely to a file, use the function squeeze. It gets rid of empty dimensions (in your case, the first dimension) and squeezes the data into a lower-dimensional matrix
Why don't you use writematrix() function?
mat = dir('*.mat');
for q = 1:length(mat)
load(mat(q).name);
[~, testName, ~] = fileparts(mat(q).name);
testVar = eval(testName);
pos(q,:,:) = testVar(1,:,1:2000);
filename = sprintf('data%d.txt', q);
writematrix(pos(q,:,:),filename,'Delimiter','space');
end
More insight you can find here:
https://www.mathworks.com/help/matlab/ref/writematrix.html
I have a set of 501 XYZ files which I load in as
for k = 1:501
% AIS SEC data
AIS_SEC{k} = importdata(['AIS_SEC(' num2str(k) ').xyz']);
end
This generates an 1x501 cell array in which all data are stored (I uploaded this file as in attachment at https://nl.mathworks.com/matlabcentral/answers/486579-how-to-merge-multiple-xyz-files-into-1-large-array). How can I merge all these data to have 1 large XYZ file? The ultimate goal is the have a nx3 array where all of the data from the seperate xyz files are merged in to 1.
For example, to concentrate X data, I tried:
for k = 1:501
my_field = sprintf('X%d', k);
variable.(my_field) = ([AIS_SEC{1,k}.data(:,1)]);
end
BUT: Dot indexing is not supported for variables of this type.
Thanks!
There are several wrong things with your code:
First thing, the error Struct contents reference from a non-struct array object. shows up first at index k=33 because the structure imported has no data field (the import was probably empty or failed).
Checking for the presence of the field let the code run through to completion. You'll then notice that you have 8 rows which are empty.
load('AIS_SEC.mat')
n=numel(AIS_SEC) ;
EmptyRows = false(n,1) ;
for k = 1:n
my_field = sprintf('X%03d', k);
if isfield( AIS_SEC{1,k} , 'data')
variable.(my_field) = AIS_SEC{1,k}.data;
else
variable.(my_field) = [] ;
EmptyRows(k) = true ;
end
end
fprintf('Number of empty rows encountered: %u\n',sum(EmptyRows))
I took the liberty to also remove unnecessary parenthesis, added an empty row index counter, and adjusted the sprintf output format so all your field will have the same length (even the first field will have the leading zeros. 'X001' to 'X500' instead of 'X1' to 'X500').
Also, this was only retrieving the first column out of each structure, so I modified it to retrieve the 3 x,y,z columns. If you really wanted only the first column, just replace variable.(my_field) = AIS_SEC{1,k}.data by variable.(my_field) = AIS_SEC{1,k}.data(:,1).
Now this gives you a long structure with 500 fields (each representing one imported variable). Your question is not clear enough on that point but if you want to have a single array where all the values are merged, then you have 2 options:
1) Directly after the code above, convert your structure to a merged array:
vararray = cell2mat(struct2cell(variable)) ;
2) If the steps above (the final variable structure) is not something you need to keep, then you can avoid it in the first place:
load('AIS_SEC.mat')
n = numel(AIS_SEC) ; % number of field to import
% first pass we count how many data point (if any) each structure has
EmptyRows = false(n,1) ;
npts = zeros(n,1) ;
for k = 1:n
if isfield( AIS_SEC{1,k} , 'data')
npts(k) = size( AIS_SEC{1,k}.data , 1 ) ;
else
EmptyRows(k) = true ;
end
end
% fprintf('Number of empty rows encountered: %u\n',sum(EmptyRows))
% The above allows us to preallocate the output matrix at the right size
cumpts = cumsum(npts) ;
totalNumberOfPoints = cumpts(end) ;
vararray = zeros(totalNumberOfPoints,3) ;
% first field to import
vararray( 1:cumpts(1) , : ) = AIS_SEC{1,1}.data ;
% now all the remaining ones
for k = 2:n
idx = (cumpts(k-1)+1):cumpts(k) ;
if ~isempty(idx)
vararray(idx,:) = AIS_SEC{1,k}.data ;
end
end
In this version there are 2 passes of the loop accross the structure. Counter intuitively this is done for better performances. The first pass is only to count the number of data points in each structure (and also to flag the empty ones). Thanks to the number returned by the first pass, we can preallocate the output matrix before the second pass and assign each structure data to the merged array in the right place without having to resize the output array at each iteration.
I'm trying to make some changes in a list of precipitation data.
The data are in a *.txt file in this format:
50810,200301010600,0.0
50810,200301010601,0.0
50810,200301010800,0.0
50810,200301010938,0.1
50810,200301010947,0.1
50810,200301010957,0.1
Each file I have contains precipitation data per minute from 1 year.
The first number is just the station number, and this is equal for each line in the file.
In my original file, I have one line with data for every minute.
I want to make a new file that contains:
Only lines with precipitation that is not zero, so I want to remove
all the lines where the last number is zero. I've figured this out.
If there is one whole hour with no precipitation, I want to remove all
of the zero-lines and create a new line which says:
50810,200301010600,0.0. If there was no precipitation between 6 and 7
am. at the 01.01.
clear all
data = load('jan-31des_2003.txt'); %opens file with data
fid=fopen('50810_2003','w'); %opens empty file to write
[nrow, ncol] = size(data); %size of data
fprintf(fid,'%5s %12s %5s \r\n','Snr','Dato - Tid','RR_01') %Header
for row = 1:nrow
y = data(row,2); %year
m = data(row,3); % month
d = data(row,4); % date
h = data(row,5); % hour
M = data(row,6); % minute
p = data(row,8); % precipitation
if p > 0
fprintf(fid,'50810,%04d%02d%02d%02d%02d,%.1f \r\n',[y,m,d,h,M,p]);
end
if p==0
HERE I NEED SOME HELP
end
end
fclose(fid);
What is the code for my desired formatting within the if p==0 condition?
Assuming you've already imported your data as an M*3 array named infile and M is divisible by 60 and your data begins at the first minute of an hour, and you have an array outfile that will be written to a new file:
outfile = []
while ~isempty(infile)
block = infile(1:60,:); % Take one hour's worth of data
if sum (block(:,3)==0) == 60 % Check if the last column are all zero
outfile = cat(1, outfile, [block(1,1:2), 0]); % If so, put one line to new array
else
filter = block(:,3)~=0; % Otherwise, pick out only rows that the last column is not zeros
outfile = cat(1, outfile, block(filter,:)); % Put these rows into the new array
end
infile(1:60,:) = []; % Remove already processed data from original array
end
Then write the entire outfile array to file.
Can someone help me to understand how I can save in matlab a group of .csv files, select only the columns in which I am interested and get as output a final file in which I have the average value of the y columns and standard deviation of y axes? I am not so good in matlab and so I kindly ask if someone to help me to solve this question.
Here what I tried to do till now:
clear all;
clc;
which_column = 5;
dirstats = dir('*.csv');
col3Complete=0;
col4Complete=0;
for K = 1:length(dirstats)
[num,txt,raw] = xlsread(dirstats(K).name);
col3=num(:,3);
col4=num(:,4);
col3Complete=[col3Complete;col3];
col4Complete=[col4Complete;col4];
avgVal(K)=mean(col4(:));
end
col3Complete(1)=[];
col4Complete(1)=[];
%columnavg = mean(col4Complete);
%columnstd = std(col4Complete);
% xvals = 1 : size(columnavg,1);
% plot(xvals, columnavg, 'b-', xvals, columnavg-columnstd, 'r--', xvals, columnavg+columstd, 'r--');
B = reshape(col4Complete,[5000,K]);
m=mean(B,2);
C = reshape (col4Complete,[5000,K]);
S=std(C,0,2);
Now I know that I should compute mean and stdeviation inside for loop, using mean()function, but I am not sure how I can use it.
which_column = 5;
dirstats = dir('*.csv');
col3Complete=[]; % Initialise as empty matrix
col4Complete=[];
avgVal = zeros(length(dirstats),2); % initialise as columnvector
for K = 1:length(dirstats)
[num,txt,raw] = xlsread(dirstats(K).name);
col3=num(:,3);
col4=num(:,4);
col3Complete=[col3Complete;col3];
col4Complete=[col4Complete;col4];
avgVal(K,1)=mean(col4(:)); % 1st column contains mean
avgVal(K,2)=std(col4(:)); % 2nd column contains standard deviation
end
%columnavg = mean(col4Complete);
%columnstd = std(col4Complete);
% xvals = 1 : size(columnavg,1);
% plot(xvals, columnavg, 'b-', xvals, columnavg-columnstd, 'r--', xvals, columnavg+columstd, 'r--');
B = reshape(col4Complete,[5000,K]);
meanVals=mean(B,2);
I didn't change much, just initialised your arrays as empty arrays so you do not have to delete the first entry later on and made avgVal a column vector with the mean in column 1 and the standard deviation in column 1. You can of course add two columns if you want to collect those statistics for your 3rd column in the csv as well.
As a side note: xlsread is rather heavy for reading files, since Excel is horribly inefficient. If you want to read a structured file such as a csv, it's faster to use importdata.
Create some random matrix to store in a file with header:
A = rand(1e3,5);
out = fopen('output.csv','w');
fprintf(out,['ColumnA', '\t', 'ColumnB', '\t', 'ColumnC', '\t', 'ColumnD', '\t', 'ColumnE','\n']);
fclose(out);
dlmwrite('output.csv', A, 'delimiter','\t','-append');
Load it using csvread:
data = csvread('output.csv',1);
data now contains your five columns, without any headers.
I want to import a sequence of excel files with a large amount of data in them. The problem that I have is I want to process the data in each file at a time and store the output from this into a variable, but each time I try to process a different file the variable gets overwritten in the variable workspace. Is there anyway I could store these files and process each file at a time?
numFiles = 1;
range = 'A2:Q21';
sheet = 1;
myData = cell(1,numFiles); % Importing data from Excel
for fileNum = 1:numFiles
fileName = sprintf('myfile%02d.xlsx',fileNum);
myData{fileNum} = importfile3(fileName,sheet,range);
end
data = cell2mat(myData);
The actual data import is performed by importfile3 which is, for the most part, a wrapper for the xlsread function that returns a matrix corresponding to the specified range of excel data.
function data = importfile3(workbookFile, sheetName, range)
% If no sheet is specified, read first sheet
if nargin == 1 || isempty(sheetName)
sheetName = 1;
end
% If no range is specified, read all data
if nargin <= 2 || isempty(range)
range = '';
end
%% Import the data
[~, ~, raw] = xlsread(workbookFile, sheetName, range);
%% Replace non-numeric cells with 0.0
R = cellfun(#(x) ~isnumeric(x) || isnan(x),raw); % Find non-numeric cells
raw(R) = {0.0}; % Replace non-numeric cells
%% Create output variable
data = cell2mat(raw);
The issue that you are running in to is a result of cell2mat concatenating all of the data in your cells in to one large 2-dimensional matrix. If you were to import two excel files with 20 rows and 17 columns, each, this would result in a 2-dimensional matrix of size [20 x 34]. The doc for cell2mat has a nice visual describing this.
I see that your importfile3 function returns a matrix, and based on your use of cell2mat in your final line of code, it looks like you would like to have your final result be in the form of a matrix. So I think the easiest way to go about this is to just bypass the intermediate myData cell array.
In the example code below, the resulting data is a 3-dimensional matrix. The 1st dimension indicates row number, 2nd dimension is column number, and 3rd dimension is file number. Cell arrays are very useful for "jagged" data, but based on the code you provided, each excel data set that you import will have the same number of rows and columns.
numFiles = 2;
range = 'A2:Q21';
sheet = 1;
% Number of rows and cols known before data import
numRows = 20;
numCols = 17;
data = zeros(numRows,numCols,numFiles);
for fileNum = 1:numFiles
fileName = sprintf('myfile%02d.xlsx',fileNum);
data(:,:,fileNum) = importfile3(fileName,sheet,range);
end
Accessing this data is now very straight-forward.
data(:,:,1) returns the data imported from your first excel file.
data(:,:,2) returns the data imported from your second excel file.
etc.