matlab reading variables with varying lengths into the workspace - matlab

This is a follow up question to
Reading parameters from a text file into the workspace
I am wondering, how would I read the following:
% ---------------------- details --------------------------
%---------------------------------------------------------------
% location:
lat = 54.35
lon = -2.9833
%
Eitan T suggested using:
fid = fopen(filename);
C = textscan(fid, '%[^= ]%*[= ]%f', 'CommentStyle', '%')
fclose(fid);
to obtain the information from the file and then
lat = C{2}(strcmp(C{1}, 'lat'));
lon = C{2}(strcmp(C{1}, 'lon'));
to obtain the relevant parameters. How could I alter this to read the following:
% ---------------------- details --------------------------
%---------------------------------------------------------------
% location:
lat = 54.35
lon = -2.9833
heights = 0, 2, 4, 7, 10, 13, 16, 19, 22, 25, 30, 35
Volume = 6197333, 5630000, 4958800, 4419400, 3880000, 3340600,
3146800, 2780200, 2413600, 2177000, 1696000, 811000
%
where the variable should contain all of the data points following the equal sign (up until the start of the next variable, Volume in this case)?
Thanks for your help

Here's one method, which uses some filthy string hacking and eval to get the result. This works on your example, but I wouldn't really recommend it:
fid = fopen('data.txt');
contents = textscan(fid, '%s', 'CommentStyle', '%', 'Delimiter', '\n');
contents = contents{1};
for ii = 1:length(contents)
line = contents{ii};
eval( [strrep(line, '=', '=['), '];'] ) # convert to valid Matlab syntax
end
A better method would be to read each of the lines using textscan
for ii = 1:length(contents)
idx = strfind(contents{ii}, ' = ');
vars{ii} = contents{ii}(1:idx-1);
vals(ii) = textscan(contents{ii}(idx+3:end), '%f', 'Delimiter', ',');
end
Now the variables vars and vals have the names of your variables, and their values. To extract the values you could do something like
ix = strmatch('lat', vars, 'exact');
lat = vals{ix};
ix = strmatch('lon', vars, 'exact');
lon = vals{ix};
ix = strmatch('heights', vars, 'exact');
heights = vals{ix};
ix = strmatch('Volume', vars, 'exact');
volume = vals{ix};

This can be accomplished using a 2-step approach:
Read the leading string (first element), equals sign (ignored), and the rest of the line as a string (second element)
Convert these strings-of-the-rest-of-the-lines to floats (second element)
There is however a slight drawback here; your lines seem to follow two formats; one is the one described in step 1), the other is a continuation of the previous line, which contains numbers.
Because of this, an extra step is required:
Read the leading string (first element), equals sign (ignored), and the rest of the line as a string (second element)
This will fail when the "other format" is encountered. Detect this, correct this, and continue
Convert these strings-of-the-rest-of-the-lines to floats (second element)
I think this will do the trick:
fid = fopen('data.txt');
C = [];
try
while ~feof(fid)
% Read next set of data, assuming the "first format"
C_new = textscan(fid, '%[^= ]%*[= ]%s', 'CommentStyle', '%', 'Delimiter', '');
C = [C; C_new]; %#ok
% When we have non-trivial data, check for the "second format"
if ~all(cellfun('isempty', C_new))
% Second format detected!
if ~feof(fid)
% get the line manually
D = fgetl(fid);
% Insert new data from fgetl
C{2}(end) = {[C{2}{end} C{1}{end} D]};
% Correct the cell
C{1}(end) = [];
end
else
% empty means we've reached the end
C = C(1:end-1,:);
end
end
fclose(fid);
catch ME
% Just to make sure to not have any file handles lingering about
fclose(fid);
throw(ME);
end
% convert second elements to floats
C{2} = cellfun(#str2num, C{2}, 'UniformOutput', false);

If you can get rid of the multi-line Volume line, what you have written is valid matlab. So, just invoke the parameter file as a matlab script using the run command.
run(scriptName)
Your only other alternative, as others have shown, is to write what will end up looking like a bastardized Matlab parser. There are definitely better ways to spend your time than doing that!

Related

Matlab: func2str from a function in a m-file

In a main m-file I have
conformal = maketform('custom', 2, 2, [], #conformalInverse_0001, []);
used in imtransform that refers to the function defined in conformalInverse_0001.m:
function U = conformalInverse_0001(X, ~)
%#codegen
U = [zeros(size(X))];
Z = complex(X(:,1),X(:,2));
W = 1./(4.*Z.^2-1);
U(:,2) = imag(W);
U(:,1) = real(W);
How can I get the string '1./(4.*Z.^2-1)' in the main program?
I found a way to solve it, but it's not so elegant...
Assume conformalInverse_0001.m is a file in your folder.
You can parse the file as a text file, and search for your formula.
Example:
Assume you know the location is 5'th line in file, and start with W =.
You can use something like the following code to read '1./(4.*Z.^2-1)' in the main program:
%Open file for reading.
fid = fopen('conformalInverse_0001.m', 'r');
%Read 5 lines.
s = textscan(fid, '%s', 5, 'delimiter', '\n');
fclose(fid);
%Get the 5'th line.
s = s{1}(5);
%Convert cell array to string.
s = s{1};
%Get characters from the 5'th character to one char before end of string.
s = s(5:end-1)
Result: s = 1./(4.*Z.^2-1)
You can check textscan documentation for finding more elegant solution.
I'm not sure I fully understand the problem here, but what about adding into your conformalInverse_0001 function something like:
str = '1./(4.*Z.^2-1)';
save('temp_str','str') % or whatever data that you want to save from it
and then adding in your main file:
load('str.mat')% or you can use 'impordata'
where you want to extract it.
I have hacked two solutions with textscan: first knowing the line number and second searching the line that starts with substring 'W = '
% read line line_num = 5 and process string
f_id = fopen(conformalInverse_m_path);
conformalInverse_cell = textscan(f_id,'%s','delimiter','\n'); %disp(conformalInverse_cell); % {68×1 cell}
func_string = conformalInverse_cell{1}{line_num}; disp(func_string); % W = 1./(4.*Z.^2-1); OK
func_string_2=func_string(5:end-1); disp(func_string_2); % 1./(4.*Z.^2-1); OK
% read first line that starts with substring 'W = ' and process string
W_string = 'W = ';
for i=1:100
func_string = conformalInverse_cell{1}{i};
Firt4=func_string(1:4); %disp(['i = ', num2str(i), ': First4 = ', Firt4]);
if strcmp(Firt4,W_string) == 1; line_nr = i; break; end;
end
func_string_2 = conformalInverse_cell{1}{line_nr};
func_string_3=func_string_2(5:end-1);

Loading data to vector rather than cell in MATlab

I load data from text files into my MATlab function using this code:
data = cell(h.numDirs, numDataFilesInFirstDir);
for d = 1:h.numDirs
% Code to set fileNames, iDir
for t = 1:size(fileNames,1)
fId = fopen([iDir, '/', fileNames{t}]);
% Drop the first two lines (column headers)
for skip = 1:2
fgets(fId);
end
U_temp = fscanf(fId, '%f %f', [2, Inf]);
U_temp = U_temp'; % ' transpose (syntax highlighting on SO)
data(d, t) = {U_temp(:,2)};
fclose(fId);
end
end
The files should each have the same length (at least for varying t, usually for varying d or else I have problems later)
Should I be (/ How can I) simplify the code here to avoid (unnecessary?) cells?
I could scan the first data set, then use something like
data = zeros(h.numDirs, numDataFilesInFirstDir, lengthOfFirstFile)
but I don't know if that would be any better. Is that a 'better' solution/method?
I would use dlmread instead of fscanf. Data type is hard since your dimensions vary. I wouldn't pad arrays... any benefit from not using cells would be overcome by the extra complexity and memory hit. Cell arrays are a reasonable choice. I wouldn't worry about preallocation too much in this case actually. Below is a similar option using structs with dynamic field names that embed the source directory and filename, for later reference.
data = struct();
for d = 1: ...
for t = 1: ...
file = fullfile(iDir, fileNames{t});
range = [3, 1, inf, 2];
dlm = ' ';
Utemp = dlmread(file, dlm, range);
data.(iDir).(fileNames{t}) = Utemp(:, 2);

Reading parameters from a text file into the workspace

I have a file which has the following information:
% ---------------------- location details --------------------------
%
% lat : latitude [minimum = -90, maximum = 90, unit =
% degrees north]
% lon : longitude [ minimum = -360, maximum = 360, unit =
% deg east]
% z: altitude (above sea level, m)
%---------------------------------------------------------------
% location:
lat = 54.35
lon = -2.9833
This is a small section of the file.
I would like to read some of this information into MATLAB, where the information can then be used to perform some calculations. The part of the file that I would like to read into MATLAB are those in the text file that are not commented, i.e have a % at the start of the line, and the variable should then be saved in the workspace. For example, I would like to have:
lat = 54.35
lon = -2.9833
in the workspace.
How would I go about this? I have read about textscan and fopen, although these don't really seem to help me in this instance.
The quick-and-dirty approach
The simplest solution I could think of to read this file indeed employs textscan :) and since the lines are conveniently written in valid MATLAB syntax, you could use eval later to evaluate them. Start by reading each line as one string (ignoring the comments in the header)
fid = fopen(filename);
C = textscan(fid, '%s', 'Delimiter', '', 'CommentStyle', '%')
fclose(fid);
Then feed the lines one by one into eval to create the variables in the MATLAB workspace:
cellfun(#eval, C{1});
What this does is interpret the line as a MATLAB command, i.e create variables as named in the file and assign the appropriate values. If you want to suppress the output of eval, you can use evalc instead to "absorb the output":
cellfun(#evalc, C{1}, 'UniformOutput', false);
This should work for your basic example, but it would fail if you have more than one instance of any parameter. Also note that the eval family is notoriously slow.
A more robust approach
If the lines in your file structure have the parameter name = number pattern, you can read the lines more intelligently:
fid = fopen(filename);
C = textscan(fid, '%[^= ]%*[= ]%f', 'CommentStyle', '%')
fclose(fid);
The %[^= ] in the pattern matches the first characters until the first space or equality sign. The %*[ =] ignores the equality sign and any trailing spaces, and then the numerical value is matched with %f. The resulting cell array C stores the parameter names in the first cell and their corresponding values in the second cell.
Now it's up to you to manipulate the parsed data. For instance, to extract all values of lat and lon, you can do this:
lat = C{2}(strcmp(C{1}, 'lat'));
lon = C{2}(strcmp(C{1}, 'lon'));
If you have more than one "lat" line, lat will be an array holding all these values.
Here's another quick and dirty way:
fp = fopen('foo.txt');
found = 1;
while ~feof(fp)
line = fgetl(fp);
if (line(1) ~= '%') && ischar(line)
value(found) = sscanf(line,'%*s %*s %f');
found = found + 1;
end
end
The %*s skips the 'lat' or 'long' and the '='.
The example you provided is kinda well-behaved, therefore the following solution might need some tailoring. However, I would recommend it against any eval():
% Read whole file ignoring lines that start with '%' and using '=' as delimiter
fid = fopen('test.txt');
s = textscan(fid,'%s%f', 'CommentStyle','%','Delimiter','=');
fclose(fid);
% Identify lines with latitude and those with longitude
idxLat = strncmpi('lat',s{1},3);
idxLon = strncmpi('lon',s{1},3);
% Store all latitudes and longitudes
lat = s{2}(idxLat);
lon = s{2}(idxLon);
Gets you a structure with field names matching parameter names, accepts comma-separated lists. List any parameters that should stay as strings in char_params
char_params={};
fid = fopen(filename);
% Load lines into cell (1x1) containing cell array s (Nx1),
% skipping lines starting with % and cutting off anything after % in a line
s = textscan(fid,'%s', 'CommentStyle','%','Delimiter','%');
fclose(fid);
% access the lines strings s{1}, split across '=' and remove whitespace on both sides
s=strtrim(split(s{1},'='));
% Interpret parameters and save to structure
for ind=1:length(s)
% User says which parameters are strings
if any(strcmpi(s{ind,1},char_params))
file_struct.(s{ind,1})=s{ind,2};
% Otherwise, assume they are numbers or numeric row arrays
else
% remove parentheses and brackets
trim_s=regexprep(s(ind,2),'[[]()]','');
% convert comma-separated lists into row arrays
file_struct.(s{ind,1})=str2double(split(trim_s{1},',')).';
end
end

Reading a text file and plotting it in 3D

I want to read in a text file that contains some strings but mostly numbers. I want to be able to ignore the strings and only look at the numbers. I want to plot those values on a 3D plane. The data looks like this:
Tech4:<152.266724,173.189377,27.995975>
<117.880638,156.116531,27.999983>
<129.849899,59.195660,27.999983>
<249.321121,60.605404,27.999983>
<224.120361,139.072739,28.000668>
<171.188950,143.490921,56.933430>
<171.188950,143.490921,83.548088>
<171.188950,143.490921,27.999985>
I believe to read in a file is just:
File = textread('testFile.txt');
How can I only look at those values and then plot it.
Thanks!
fid = fopen([pathname,filename]);
tline = fgetl(fid);
CX = [];
CY = [];
CZ = [];
while ischar(tline)
% skip < and >
tline = substr(tline, 1, length(tline)-2)
% extract numbers
temp = textscan(tline,'%n%n%n', 'delimiter',',');
CX(end+1,:) = [temp(1)];
CY(end+1,:) = [temp(2)];
CZ(end+1,:) = [temp(3)];
tline = fgetl(fid);
end
fclose(fid);
and then plot it using
plot3(CX, CY, CZ)
function call.
Add the check for "Tech4:" at the beginning however...
I think you can also directly use textscan in a one-liner:
fid = fopen('testFile.txt');
data = textscan(fid,'%*s%f,%f,%f');
fclose(fid);
this loads the values from all rows with the specified format into the variable data.
no matlab around to test it out though.
fscanf is an option to, the same kind of parameters as textscan.
EDIT: typo, you want to detect floats (%f) of course, and not integers (%d)
EDIT2: got matlab and tested it out, this works here for your sample input ^^
fid = fopen('testFile.txt');
data = textscan(fid,'%*s%f%f%f','Delimiter',',<>')
fclose(fid);

Problem (bug?) loading hexadecimal data into MATLAB

I'm trying to load the following ascii file into MATLAB using load()
% some comment
1 0xc661
2 0xd661
3 0xe661
(This is actually a simplified file. The actual file I'm trying to load contains an undefined number of columns and an undefined number of comment lines at the beginning, which is why the load function was attractive)
For some strange reason, I obtain the following:
K>> data = load('testMixed.txt')
data =
1 50785
2 58977
3 58977
I've observed that the problem occurs anytime there's a "d" in the hexadecimal number.
Direct hex2dec conversion works properly:
K>> hex2dec('d661')
ans =
54881
importdata seems to have the same conversion issue, and so does the ImportWizard:
K>> importdata('testMixed.txt')
ans =
1 50785
2 58977
3 58977
Is that a bug, am I using the load function in some prohibited way, or is there something obvious I'm overlooking?
Are there workarounds around the problem, save from reimplementing the file parsing on my own?
Edited my input file to better reflect my actual file format. I had a bit oversimplified in my original question.
"GOLF" ANSWER:
This starts with the answer from mtrw and shortens it further:
fid = fopen('testMixed.txt','rt');
data = textscan(fid,'%s','Delimiter','\n','MultipleDelimsAsOne','1',...
'CommentStyle','%');
fclose(fid);
data = strcat(data{1},{' '});
data = sscanf([data{:}],'%i',[sum(isspace(data{1})) inf]).';
PREVIOUS ANSWER:
My first thought was to use TEXTSCAN, since it has an option that allows you to ignore certain lines as comments when they start with a given character (like %). However, TEXTSCAN doesn't appear to handle numbers in hexadecimal format well. Here's another option:
fid = fopen('testMixed.txt','r'); % Open file
% First, read all the comment lines (lines that start with '%'):
comments = {};
position = 0;
nextLine = fgetl(fid); % Read the first line
while strcmp(nextLine(1),'%')
comments = [comments; {nextLine}]; % Collect the comments
position = ftell(fid); % Get the file pointer position
nextLine = fgetl(fid); % Read the next line
end
fseek(fid,position,-1); % Rewind to beginning of last line read
% Read numerical data:
nCol = sum(isspace(nextLine))+1; % Get the number of columns
data = fscanf(fid,'%i',[nCol inf]).'; % Note '%i' works for all integer formats
fclose(fid); % Close file
This will work for an arbitrary number of comments at the beginning of the file. The computation to get the number of columns was inspired by Jacob's answer.
New:
This is the best I could come up with. It should work for any number of comment lines and columns. You'll have to do the rest yourself if there are strings, etc.
% Define the characters representing the start of the commented line
% and the delimiter
COMMENT_START = '%%';
DELIMITER = ' ';
% Open the file
fid = fopen('testMixed.txt');
% Read each line till we reach the data
l = COMMENT_START;
while(l(1)==COMMENT_START)
l = fgetl(fid);
end
% Compute the number of columns
cols = sum(l==DELIMITER)+1;
% Split the first line
split_l = regexp(l,' ','split');
% Read all the data
A = textscan(fid,'%s');
% Compute the number of rows
rows = numel(A{:})/cols;
% Close the file
fclose(fid);
% Assemble all the data into a matrix of cell strings
DATA = [split_l ; reshape(A{:},[cols rows])']; %' adding this to make it pretty in SO
% Recognize each column and process accordingly
% by analyzing each element in the first row
numeric_data = zeros(size(DATA));
for i=1:cols
str = DATA(1,i);
% If there is no '0x' present
if isempty(findstr(str{1},'0x')) == true
% This is a number
numeric_data(:,i) = str2num(char(DATA(:,i)));
else
% This is a hexadecimal number
col = char(DATA(:,i));
numeric_data(:,i) = hex2dec(col(:,3:end));
end
end
% Display the data
format short g;
disp(numeric_data)
This works for data like this:
% Comment 1
% Comment 2
1.2 0xc661 10 0xa661
2 0xd661 20 0xb661
3 0xe661 30 0xc661
Output:
1.2 50785 10 42593
2 54881 20 46689
3 58977 30 50785
OLD:
Yeah, I don't think LOAD is the way to go. You could try:
a = char(importdata('testHexa.txt'));
a = hex2dec(a(:,3:end));
This is based on both gnovice's and Jacob's answers, and is a "best of breed"
For files like:
% this is my comment
% this is my other comment
1 0xc661 123
2 0xd661 456
% surprise comment
3 0xe661 789
4 0xb661 1234567
(where the number of columns within the file MUST be the same, but not known ahead of time, and all comments denoted by a '%' character), the following code is fast and easy to read:
f = fopen('hexdata.txt', 'rt');
A = textscan(f, '%s', 'Delimiter', '\n', 'MultipleDelimsAsOne', '1', 'CollectOutput', '1', 'CommentStyle', '%');
fclose(f);
A = A{1};
data = sscanf(A{1}, '%i')';
data = repmat(data, length(A), 1);
for ctr = 2:length(A)
data(ctr,:) = sscanf(A{ctr}, '%i')';
end