Save a sparse array in csv - matlab

I have a huge sparse matrix a and I want to save it in a .csv. I can not call full(a) because I do not have enough ram memory. So, calling dlmwrite with full(a) argument is not possible. We must note that dlmwrite is not working with sparse formatted matrices.
The .csv format is depicted below. Note that the first row and column with the characters should be included in the .csv file. The semicolon in the (0,0) position of the .csv file is necessary too.
;A;B;C;D;E
A;0;1.5;0;1;0
B;2;0;0;0;0
C;0;0;1;0;0
D;0;2.1;0;1;0
E;0;0;0;0;0
Could you please help me to tackle this problem and finally save the sparse matrix in the desired form?

You can use csvwrite function:
csvwrite('matrix.csv',a)

You could do this iteratively, as follows:
A = sprand(20,30000,.1);
delimiter = ';';
filename = 'filecontaininghugematrix.csv';
dims = size(A);
N = max(dims);
% create names first
idx = 1:26;
alphabet = dec2base(9+idx,36);
n = ceil(log(N)/log(26));
q = 26.^(1:n);
names = cell(sum(q),1);
p = 0;
for ii = 1:n
temp = repmat({idx},ii,1);
names(p+(1:q(ii))) = num2cell(alphabet(fliplr(combvec(temp{:})')),2);
p = p + q(ii);
end
names(N+1:end) = [];
% formats for writing
headStr = repmat(['%s' delimiter],1,dims(2));
headStr = [delimiter headStr(1:end-1) '\n'];
lineStr = repmat(['%f' delimiter],1,dims(2));
lineStr = ['%s' delimiter lineStr(1:end-1) '\n'];
fid = fopen(filename,'w');
% write header
header = names(1:dims(2));
fprintf(fid,headStr,header{:});
% write matrix rows
for ii = 1:dims(1)
row = full(A(ii,:));
fprintf(fid, lineStr, names{ii}, row);
end
fclose(fid);
The names cell array is quite memory demanding for this example. I have no time to fix that now, so think about this part yourself if it is really a problem ;) Hint: just write the header element wise, first A;, then B; and so on. For the rows, you can create a function that maps the index ii to the desired character, in which case the complete first part is not necessary.

Related

Matlab: Error using readtable (line 216) Input must be a row vector of characters or string scalar

I gave the error Error using readtable (line 216) Input must be a row vector of characters or string scalar when I tried to run this code in Matlab:
clear
close all
clc
D = 'C:\Users\Behzad\Desktop\New folder (2)';
filePattern = fullfile(D, '*.xlsx');
file = dir(filePattern);
x={};
for k = 1 : numel(file)
baseFileName = file(k).name;
fullFileName = fullfile(D, baseFileName);
x{k} = readtable(fullFileName);
fprintf('read file %s\n', fullFileName);
end
% allDates should be out of the loop because it's not necessary to be in the loop
dt1 = datetime([1982 01 01]);
dt2 = datetime([2018 12 31]);
allDates = (dt1 : calmonths(1) : dt2).';
allDates.Format = 'MM/dd/yyyy';
% 1) pre-allocate a cell array that will store
% your tables (see note #3)
T2 = cell(size(x)); % this should work, I don't know what x is
% the x is xlsx files and have different sizes, so I think it should be in
% a loop?
% creating loop
for idx = 1:numel(x)
T = readtable(x{idx});
% 2) This line should probably be T = readtable(x(idx));
sort = sortrows(T, 8);
selected_table = sort (:, 8:9);
tempTable = table(allDates(~ismember(allDates,selected_table.data)), NaN(sum(~ismember(allDates,selected_table.data)),size(selected_table,2)-1),'VariableNames',selected_table.Properties.VariableNames);
T2 = outerjoin(sort,tempTable,'MergeKeys', 1);
% 3) You're overwriting the variabe T2 on each iteration of the i-loop.
% to save each table, do this
T2{idx} = fillmissing(T2, 'next', 'DataVariables', {'lat', 'lon', 'station_elevation'});
end
the x is each xlsx file from the first loop. my xlsx file has a different column and row size. I want to make the second loop process for all my xlsx files in the directory.
did you know what is the problem? and how to fix it?
Readtable has one input argument, a filename. It returns a table. In your code you have the following:
x{k} = readtable(fullFileName);
All fine, you are reading the tables and storing the contents in x. Later in your code you continue with:
T = readtable(x{idx});
You already read the table, what you wrote is basically T = readtable(readtable(fullFileName)). Just use T=x{idx}

MATLAB: export scalars within for loop to text file

I have a large number of text files that I have to read, find the max value for a certain column, and the corresponding time. The for loop for finding these values works fine, but my problem is writing a text file that shows the three variables I need (thisfilename, M, and wavetime) for each iteration of the for loop.
Output_FileName_MaxWaveHeights = ['C:\Users\jl44459\Desktop\QGIS_and_Basement\BASEMENT\Mesh_5_2045\Run_A\','MaxWaveHeights.txt'];
writefile = fopen(Output_FileName_MaxWaveHeights,'a');
dinfo = dir('*.dat');
for K = 1 : length(dinfo)
thisfilename = dinfo(K).name; %just the name of the file
fileID = fopen(thisfilename); %creates numerical ID for the file name
thisdata = textscan(fileID,'%f64%f64%f64%f64%f64%f64%f64',500,'HeaderLines',1); %load just this file
thisdataM = cell2mat(thisdata); %transforms file from cell array to matrix
[M,I] = max(thisdataM(:,5)); %finds max WSE and row it's in
wavetime = 2*(I-1); %converts column of max WSE to time
fprintf(writefile,'%s %8.4f %4.0f \r\n',thisfilename,M,wavetime);
fclose(fileID); %closes file to make space for next one
end
The text file ends up just giving me the values for one iteration instead of all of them. I was able to use displaytable as a workaround, but then I have problems writing "thisfilename", which includes non-numerical characters.
Although I am not able to reproduce the issue with the code provided, a possible solution might be to write to the file outside of the loop and to close the file afterwards:
Output_FileName_MaxWaveHeights = ['C:\Users\jl44459\Desktop\QGIS_and_Basement\BASEMENT\Mesh_5_2045\Run_A\','MaxWaveHeights.txt'];
writefile = fopen(Output_FileName_MaxWaveHeights,'a');
s = [];
dinfo = dir('*.dat');
for K = 1 : length(dinfo)
thisfilename = dinfo(K).name; %just the name of the file
fileID = fopen(thisfilename); %creates numerical ID for the file name
thisdata = textscan(fileID,'%f64%f64%f64%f64%f64%f64%f64',500,'HeaderLines',1); %load just this file
thisdataM = cell2mat(thisdata); %transforms file from cell array to matrix
[M,I] = max(thisdataM(:,5)); %finds max WSE and row it's in
wavetime = 2*(I-1); %converts column of max WSE to time
s = [s, fprintf(writefile,'%s %8.4f %4.0f \r\n',thisfilename,M,wavetime)];
fclose(fileID); %closes file to make space for next one
end
fprintf(writefile,s);
fclose(writefile);
Solved--it was simply me forgetting to close the output file after the loop. Thanks for the help!

Loading data to vector rather than cell in MATlab

I load data from text files into my MATlab function using this code:
data = cell(h.numDirs, numDataFilesInFirstDir);
for d = 1:h.numDirs
% Code to set fileNames, iDir
for t = 1:size(fileNames,1)
fId = fopen([iDir, '/', fileNames{t}]);
% Drop the first two lines (column headers)
for skip = 1:2
fgets(fId);
end
U_temp = fscanf(fId, '%f %f', [2, Inf]);
U_temp = U_temp'; % ' transpose (syntax highlighting on SO)
data(d, t) = {U_temp(:,2)};
fclose(fId);
end
end
The files should each have the same length (at least for varying t, usually for varying d or else I have problems later)
Should I be (/ How can I) simplify the code here to avoid (unnecessary?) cells?
I could scan the first data set, then use something like
data = zeros(h.numDirs, numDataFilesInFirstDir, lengthOfFirstFile)
but I don't know if that would be any better. Is that a 'better' solution/method?
I would use dlmread instead of fscanf. Data type is hard since your dimensions vary. I wouldn't pad arrays... any benefit from not using cells would be overcome by the extra complexity and memory hit. Cell arrays are a reasonable choice. I wouldn't worry about preallocation too much in this case actually. Below is a similar option using structs with dynamic field names that embed the source directory and filename, for later reference.
data = struct();
for d = 1: ...
for t = 1: ...
file = fullfile(iDir, fileNames{t});
range = [3, 1, inf, 2];
dlm = ' ';
Utemp = dlmread(file, dlm, range);
data.(iDir).(fileNames{t}) = Utemp(:, 2);

How to store .csv data and calculate average value in MATLAB

Can someone help me to understand how I can save in matlab a group of .csv files, select only the columns in which I am interested and get as output a final file in which I have the average value of the y columns and standard deviation of y axes? I am not so good in matlab and so I kindly ask if someone to help me to solve this question.
Here what I tried to do till now:
clear all;
clc;
which_column = 5;
dirstats = dir('*.csv');
col3Complete=0;
col4Complete=0;
for K = 1:length(dirstats)
[num,txt,raw] = xlsread(dirstats(K).name);
col3=num(:,3);
col4=num(:,4);
col3Complete=[col3Complete;col3];
col4Complete=[col4Complete;col4];
avgVal(K)=mean(col4(:));
end
col3Complete(1)=[];
col4Complete(1)=[];
%columnavg = mean(col4Complete);
%columnstd = std(col4Complete);
% xvals = 1 : size(columnavg,1);
% plot(xvals, columnavg, 'b-', xvals, columnavg-columnstd, 'r--', xvals, columnavg+columstd, 'r--');
B = reshape(col4Complete,[5000,K]);
m=mean(B,2);
C = reshape (col4Complete,[5000,K]);
S=std(C,0,2);
Now I know that I should compute mean and stdeviation inside for loop, using mean()function, but I am not sure how I can use it.
which_column = 5;
dirstats = dir('*.csv');
col3Complete=[]; % Initialise as empty matrix
col4Complete=[];
avgVal = zeros(length(dirstats),2); % initialise as columnvector
for K = 1:length(dirstats)
[num,txt,raw] = xlsread(dirstats(K).name);
col3=num(:,3);
col4=num(:,4);
col3Complete=[col3Complete;col3];
col4Complete=[col4Complete;col4];
avgVal(K,1)=mean(col4(:)); % 1st column contains mean
avgVal(K,2)=std(col4(:)); % 2nd column contains standard deviation
end
%columnavg = mean(col4Complete);
%columnstd = std(col4Complete);
% xvals = 1 : size(columnavg,1);
% plot(xvals, columnavg, 'b-', xvals, columnavg-columnstd, 'r--', xvals, columnavg+columstd, 'r--');
B = reshape(col4Complete,[5000,K]);
meanVals=mean(B,2);
I didn't change much, just initialised your arrays as empty arrays so you do not have to delete the first entry later on and made avgVal a column vector with the mean in column 1 and the standard deviation in column 1. You can of course add two columns if you want to collect those statistics for your 3rd column in the csv as well.
As a side note: xlsread is rather heavy for reading files, since Excel is horribly inefficient. If you want to read a structured file such as a csv, it's faster to use importdata.
Create some random matrix to store in a file with header:
A = rand(1e3,5);
out = fopen('output.csv','w');
fprintf(out,['ColumnA', '\t', 'ColumnB', '\t', 'ColumnC', '\t', 'ColumnD', '\t', 'ColumnE','\n']);
fclose(out);
dlmwrite('output.csv', A, 'delimiter','\t','-append');
Load it using csvread:
data = csvread('output.csv',1);
data now contains your five columns, without any headers.

Reading data from a Text File into Matlab array

I am having difficulty in reading data from a .txt file using Matlab.
I have to create a 200x128 dimension array in Matlab, using the data from the .txt file. This is a repetitive task, and needs automation.
Each row of the .txt file is a complex number of form a+ib, which is of form a[space]b. A sample of my text file :
Link to text file : Click Here
(0)
1.2 2.32222
2.12 3.113
.
.
.
3.2 2.22
(1)
4.4 3.4444
2.33 2.11
2.3 33.3
.
.
.
(2)
.
.
(3)
.
.
(199)
.
.
I have numbers of rows (X), inside the .txt file surrounded by brackets. My final matrix should be of size 200x128. After each (X), there are exactly 128 complex numbers.
Here is what I would do. First thing, delete the "(0)" types of lines from your text file (could even use a simple shells script for that). This I put into the file called post2.txt.
# First, load the text file into Matlab:
A = load('post2.txt');
# Create the imaginary numbers based on the two columns of data:
vals = A(:,1) + i*A(:,2);
# Then reshape the column of complex numbers into a matrix
mat = reshape(vals, [200,128]);
The mat will be a matrix of 200x128 complex data. Obviously at this point you can put a loop around this to do this multiple times.
Hope that helps.
You can read the data in using the following function:
function data = readData(aFilename, m,n)
% if no parameters were passed, use these as defaults:
if ~exist('aFilename', 'var')
m = 128;
n = 200;
aFilename = 'post.txt';
end
% init some stuff:
data= nan(n, m);
formatStr = [repmat('%f', 1, 2*m)];
% Read in the Data:
fid = fopen(aFilename);
for ind = 1:n
lineID = fgetl(fid);
dataLine = fscanf(fid, formatStr);
dataLineComplex = dataLine(1:2:end) + dataLine(2:2:end)*1i;
data(ind, :) = dataLineComplex;
end
fclose(fid);
(edit) This function can be improved by including the (1) parts in the format string and throwing them out:
function data = readData(aFilename, m,n)
% if no parameters were passed, use these as defaults:
if ~exist('aFilename', 'var')
m = 128;
n = 200;
aFilename = 'post.txt';
end
% init format stuff:
formatStr = ['(%*d)\n' repmat('%f%f\n', 1, m)];
% Read in the Data:
fid = fopen(aFilename);
data = fscanf(fid, formatStr);
data = data(1:2:end) + data(2:2:end)*1i;
data = reshape(data, n,m);
fclose(fid);