Read a complex, and long text file in Matlab - matlab

Question

I have a very long text file which contains the data from 4 different stations with different time steps:
1:00
station 1
a number 1 (e.g.0.6E-06)
matrix1 (41x36)
station 2
number 2 (e.g.0.1E-06)
matrix2 (41x36)
station 3
number 3 (e.g.0.2E-06)
matrix3 (41x36)
station 4
number 4 (e.g.0.4E-06)
matrix4 (41x36)
2:00
station 1
a number (e.g.0.24E-06)
matrix5 (41x36)
station 2
a number (e.g.0.3E-06)
matrix6 (41x36)
station 3
number (e.g.0.12E-06)
matrix7 (41x36)
station 4
number (e.g.0.14E-06)
matrix8 (41x36)
.....
and so on
I need to read this data by each station and each step, and noted that each matrix should be scaled by multiplying with a number above it. An example is here: https://files.fm/u/sn447ttc#/view/example.txt
Could you please help?
Thank you a lot.

Answer 1

My idea here would be to read the textfile using fopen and textscan. Afterwards you can search for appearances of the Keyword FACTOR to subdivide the output. Here's the code:
fid=fopen('example.txt'); % open the document
dataRaw=textscan(fid,'%s','Delimiter',''); % read the file with no delimiter to achieve a cell array with 1 cell per line of the text file
fclose(fid); % close the document
rows=cellfun(#(x) strfind(x,'FACTOR'),dataRaw,'uni',0); % search for appearances of 'FACTOR'
hasFactor=find(~cellfun(#isempty,rows{1})); % get rownumbers of the lines that contain the word FACTOR
dataRaw=dataRaw{1}; % convert array for easier indexing
for ii=1:(numel(hasFactor)-1) % loop over appearances of the word FACTOR
array=cellfun(#str2num,dataRaw(hasFactor(ii)+2:hasFactor(ii+1)-1),'uni',0); % extract numerical data
output{ii}=str2num(dataRaw{hasFactor(ii)+1})*cat(1,array{:}); % create output scaled by the factor
end
array=cellfun(#str2num,dataRaw(hasFactor(end)+2:end),'uni',0);
output{end+1}=str2num(dataRaw{hasFactor(end)+1})*cat(1,array{:}); % These last 2 lines add the last array to the ouput
outputMat=cat(3,output{:}); % convert to a 3-dimensional matrix
outputStations=[{output(1:4:end)} {output(2:4:end)} {output(3:4:end)} {output(4:4:end)}]; % Sort the output to have 1 cell for each station
outputColumnSums=cellfun(#(x) cellfun(#sum,x,'uni',0),outputStations,'uni',0); % To sum up all the columns of each matrix
outputRowSums=cellfun(#(x) cellfun(#(y) sum(y,2),x,'uni',0),outputStations,'uni',0);
This approach is pretty slow and probably can be vectorized, but if you don't need it to be fast it should do the job. I created a cell-output with 1 cell per array and a 3 dimensional array as optional output. Hope that's fine with you

Answer 2

I have looked into your situation and it seems that the problem not trivial as anticipated. Keep in mind that if I have made mistakes on the assumption of the location of the data, you can let me know so I can edit it, or you can just change the numbers to that which suits your case. In this case, I initially loaded the delimited file into an Excel spreadsheet, just to visualize it.
After reading up on dlmread, I found that one can specify the exact rows and columns to pull from example.txt, as shown here:
data = dlmread('example.txt', ' ', [4 1 45 37]); % [r1 c1 r2 c2]
data2 = dlmread('example.txt', ' ', [47 1 88 37]);
The result of which is two matrices that are 41-by-37, containing only numbers. I started data at row 4 to bypass the header information/strings. Noticing the pattern, I set it up as a loop:
No_of_matrices_expected = 4;
dataCell = cell(No_of_matrices_expected, 1);
iterations = length(dataCell)
% Initial Conditions
rowBeginning = 4;
col1 = 1; % Constant
rowEnd = rowBeginning + 40; % == 44, right before next header information
col2 = 36; % Constant
for n = 1 : iterations
dataCell{n} = dlmread('example.txt', ' ', [rowBeginning, col1, rowEnd, col2]);
rowBeginning = rowBeginning + 41 + 2; % skip previous matrix and skip header info
rowEnd = rowBeginning + 40;
end
However, I stumbled across what you stated earlier which was that there are four different stations, each with their own time stamps. So running this loop more than 4 times led to unexpected results and MATLAB crashed. The reason is that the new timestamp creates an extra row for the date. Now, you could change the loop above to compensate for this extra row, or you can make multiple for loops for each station. This will be your decision to make.
Now if you wanted to save the header information, I would recommend taking a look into textscan. You can simply use this function to pull the first column of all the data into a cell array of strings. Then you can pull out the header information that you want. Keep in mind, use fopen if you want to use textscan.
I'll let you use what I have found thus far, but let me know if you need more help.
Numbers