import data from files with unknown format in matlab - matlab

Question

I have some files with about 3000 entries including text and numeric data similar to this:
1 0 23 'x' 'x' 'x' 0 0 0 1 1 10.3 54 123.45678 'x' 'x' 'x' ...
i want to import each file data in a separate 1x3000 vector in MatLab but when i use 'importdata' function, it creates a 1x1 struct with two fields (data and textdata).
file_path = '/home/my/file/path';
list_of_files = dir(file_path);
for i = 3:end_
new_data = importdata(fullfile(file_path,list_of_files(i).name));
end
Also i tried to use 'textscan' function but it requires format specification but format of files is unknown (length of each file is constant but it's not clear where we have 'x' or a number)
does anybody have suggestions what to do?

Answer 1

You can not have an array containing both numeric values and strings.
If you want to have both numeric values and strings you have to use cellarray.
Since, as you wrote, you do not like having a struct with two fields, using textscan seems a promising way even if it will be a little complicated.
You can overcame the problem of the format specification by specifying:
the format as string
the delimiter as ' (in you example the text is included within two ')
Your input file will them be stored in a cellarray as sset of strings.
Now you can extract both the numeric values and the strings by scanning the elements of the cellarray.
To identify the numeric values you can try to convert the string in a numeric array with the function str2num:
if the string only contains nueric values, you can store cumulatively the values in an array
if the conversin fails, that means that it was a string, then you can store it cumulatively in a string
You can also set a flag and use it to allows inserting in the numeric output array a value (e. g. NaN) when a string is found; this can allows you to understand where the strings were in the input file.
Also for both the above conditions, you can evaluate the length of the partial arrays of numbers or string and store it in another array.
This allows you to understand where a specific number or string was in the input file.
In the folowing you cn find a possible implementatino of the above described approach.
% Open the input file
fp=fopen('mix_n_s.dat','r');
% Read the input file as a string in a cell array using "'" as a
% delimitator
% c=textscan(fp,'%s','delimiter','''');
c=textscan(fp,'%s','delimiter','''');
% Close the input file
fclose(fp);
% Extract the cell array
a=c{1};
% Initialize the output variables
% Array with the numeric values
numeric_array=[];
% String with the string in the input file
the_strings=[];
% Array with the number of numeric values and strings
the_cnt=[];
% Define the flag for enabling the isertion of NaN in the output numeric
% array in case a string is found
insert_nan=1;
% Scan the cellarray to extract the numbers and the strings
for i=1:length(a)
x=a{i}
% If the i-th element is empty (this occurs when there are at least two
% consecutive string in the input file, do nothing
if(~isempty(x))
% If the i-th element is not empty try to convert it into a numeric array
m=str2num(x);
% If the output is not empty you have read one or more than one
% numeric values
if(~isempty(m));
% Then store them into an array
numeric_array=[numeric_array m];
% The lengh of the array gives you the number of numeric values;
% store it the array
the_cnt=[the_cnt length(m)];
else
% If the conversin failed, you have read a string; store it in a
% string
the_strings=[the_strings ' ' x];
% Store the length of the string in the array; if you store it as
% a negative value, you can recognise it later on
the_cnt=[the_cnt -length(x)];
% if the flag is on, then insert NaN in the numeric array
if(insert_nan)
numeric_array=[numeric_array NaN];
end
end
end
end
numeric_array
the_strings
the_cnt
Based in the input example you've provided (I've slightly modified the strings):
1 0 23 'x' 'x' 'x' 0 0 0 1 1 10.3 54 123.45678 'x' 'x' 'x'
the output is the following (the flag for inssert NaN is on):
numeric_array =
Columns 1 through 7
1.0000 0 23.0000 NaN NaN NaN 0
Columns 8 through 14
0 0 1.0000 1.0000 10.3000 54.0000 123.4568
Columns 15 through 17
NaN NaN NaN
the_strings =
x abcd efghilm x x x
the_cnt =
3 -1 -4 -7 8 -1 -1 -1
It can be interpreted as follows:
looking at the numeric_array array: in the input file
three numeric values, then three strings, then eight numeric values and three strings
looking at the the_cnt array, you can understand the length (discard the - sign) of each string.
Hope this helps.
Qapla'