Matlab: Edit values in text file without changing the file format - matlab

I have a following parameter file in which I want to change values on left hand side starting with gam.dat till 1 1 1 (against -tail variable, head variable, variogram type) without changing the format of the file.
This parameter file will be called inside the loop such that each iteration of the loop would require changing the values inside this parameter file.
Reading and writing from a file has always been my weak point. Any help on how this can be done easily? Thanks!
Parameters
**********
START OF PARAMETERS:
gam.dat -file with data
1 1 - number of variables, column numbers
-1.0e21 1.0e21 - trimming limits
gam.out -file for output
1 -grid or realization number
100 1.0 1.0 -nx, xmn, xsiz
100 1.0 1.0 -ny, ymn, ysiz
20 1.0 1.0 -nz, zmn, zsiz
4 30 -number of directions, number of h
1 0 1 -ixd(1),iyd(1),izd(1)
1 0 2 -ixd(2),iyd(2),izd(2)
1 0 3 -ixd(3),iyd(3),izd(3)
1 1 1 -ixd(4),iyd(4),izd(4)
1 -standardize sill? (0=no, 1=yes)
1 -number of gamma
1 1 1 -tail variable, head variable, gamma type

Something like this might help. Then again it might not be exactly what you're looking for.
fid = fopen(filename as a string);
n = 1;
textline = [];
while( ~feof(fid) ) // This just runs until the end of the file is reached.
textline(n) = fgetl(fid)
// some operations you want to perform?
// You can also do anything you want to the lines here as you are reading them in.
// This will read in every line in the file as well.
n = n + 1;
end
fwrite(fid, textline); // This writes to the file and will overwrite what is already there.
// You always write to a new file if you want to though!
fclose(fid);
The only reason I am suggesting the use of fgetl here is because it looks like there are specific operations/changes you want to make based on the line or the information in the line. You can also use fread which will do the same thing but you'll then have to operate on the matrix as a whole after it's built rather than making any modifications to it while reading the data in and building the matrix.
Hope that helps!
More complete example based on the comments below.
fid = fopen('gam.txt');
n = 1;
textline = {};
while( ~feof(fid) ) % This just runs until the end of the file is reached.
textline(n,1) = {fgetl(fid)}
% some operations you want to perform?
% You can also do anything you want to the lines here as you are reading them in.
% This will read in every line in the file as well.
if ( n == 5 ) % This is just an operation that will adjust line number 5.
temp = cell2mat(textline(n));
textline(n,1) = {['newfile.name', temp(regexp(temp, '\s', 'once'):end)]};
end
n = n + 1;
end
fclose(fid)
fid = fopen('gam2.txt', 'w') % this file has to already be created.
for(n = 1:length(textline))
fwrite(fid, cell2mat(textline(n));
end
fclose(fid)

Related

MATLAB - Read Textfile (lines with different formats) line by line

I have a text file (lets call it an input file) of this type:
%My kind of input file % Comment 1 % Comment 2
4 %Parameter F
2.745 5.222 4.888 1.234 %Parameter X
273.15 373.15 1 %Temperature Initial/Final/Step
3.5 %Parameter Y
%Matrix A
1.1 1.3 1 1.05
2.0 1.5 3.1 2.1
1.3 1.2 1.5 1.6
1.3 2.2 1.7 1.4
I need to read this file and save the values as variables or even better as part of different arrays. For example by reading I should obtain Array1.F=4; then Array1.X should be a vector of 3 real numbers, Array2.Y=3.5 then Array2.A is a matrix FxF. There are tons of functions to read from text file but I don't know how to read these kind of different formats. I've used in the past fgetl/fgets to read lines but it reads as strings, I've used fscanf but it reads the whole text file as if it is formatted all equally. However I need something to read sequentially with predefined formats. I can easily do this with fortran reading line by line because read has a format statement. What is the equivalent in MATLAB?
This actually parses the file you posted in your example. I could've done better, but I'm tired today:
res = struct();
fid = fopen('test.txt','r');
read_mat = false;
while (~feof(fid))
% Read text line by line...
line = strtrim(fgets(fid));
if (isempty(line))
continue;
end
if (read_mat) % If I'm reading the final matrix...
% I use a regex to capture the values...
mat_line = regexp(line,'(-?(?:\d*\.)?\d+)+','tokens');
% If the regex succeeds I insert the values in the matrix...
if (~isempty(mat_line))
res.A = [res.A; str2double([mat_line{:}])];
continue;
end
else % If I'm not reading the final matrix...
% I use a regex to check if the line matches F and Y parameters...
param_single = regexp(line,'^(-?(?:\d*\.)?\d+) %Parameter (F|Y)$','tokens');
% If the regex succeeds I assign the values...
if (~isempty(param_single))
param_single = param_single{1};
res.(param_single{2}) = str2double(param_single{1});
continue;
end
% I use a regex to check if the line matches X parameters...
param_x = regexp(line,'^((?:-?(?:\d*\.)?\d+ ){4})%Parameter X$','tokens');
% If the regex succeeds I assign the values...
if (~isempty(param_x))
param_x = param_x{1};
res.X = str2double(strsplit(strtrim(param_x{1}),' '));
continue;
end
% If the line indicates that the matrix starts I set my loop so that it reads the final matrix...
if (strcmp(line,'%Matrix A'))
res.A = [];
read_mat = true;
continue;
end
end
end
fclose(fid);

Open specific file with the specific words (16 bits) structure

I have a specific binary? file format containing datas about the configuration used to take a picture with a custom camera. This file format is named DAI and contains for example values of offset/gain/etc...
I am using a black-box script in java to turn this file into a .csv and I want to perform the same thing in Matlab. I've got a config file describing in ascii format how this file is built (name of the field, type of the data, first_word, last_word, low_bit, high_bit). For example I know that the first field in the DAI file will be :
spare1; PCHAR; first_word=0; low_bit=0; high_bit=7
But right now I have no clue of how to use this information. My first thought were to fopen() the file and use fread() to read the binary data from the file and turn it into the format I want but I don't know how to use the values of "last_word,high_bit,..." to do so. I have a limited understanding of binary files.
To sum up everything :
file.dai contains datas /
file.cfg contains the structure :
mband_1_start_line; PCHAR; first_word=12; low_bit=6; high_bit=15
mband_1_length; PCHAR; first_word=12; low_bit=0; high_bit=5
mband_1_gain; PCHAR; first_word=13; low_bit=0; high_bit=7
mband_1_offset; PCHAR; first_word=13; low_bit=8; last_word=14; high_bit=7
and I want to recover the datas corresponding to the fields like mband_1_offset.
If someone can help me to figure the good way of doing that I will be very thankful !
[EDIT : SOLVED] So thanks to your very helpful help I've manage to get the values for each field even when the header changes !!
Here's the final code :
Here's the final code :
...code to retrieve the content of the .cfg file....
%% Open and read the DAI file
fid = fopen(dai_file,'r','l');
% First thing is to skip the header
% We read a first time the file
dat=fread(fid,inf,'*uint8');
% We search for the position of the end of the header : NUL NUL ETX
% In decimal it gives :
skip = findstr(dat',[000,000,003]);
% We define the wordsize : 2 bytes (2 words)
wordsize = 2;
% We rewind the file to start over to get the values for each field
frewind(fid);
% We initiate the structure camdat containing the datas of the camera
camdat=struct;
% We start the loop for each field of the layout config file
for ct = 1:length(layout)
% Defining the words/bits
first_word = layout{ct,3};
last_word = layout{ct,5};
low_bit = layout{ct,4};
high_bit = layout{ct,6};
% We position to the "skip value + the position of the first_word in bytes"
fseek(fid,skip+first_word*wordsize,-1);
% We compute the number of words (last - first +1)
datasize=last_word-first_word+1;
% We read the datas as uint16 (words are 16bits)
data=fread(fid,datasize,'*uint16');
% We convert it to bits
% Case of 1 word
bits=bitget(data(1),[1:16]);
% Case of 2 words
if length(data) > 1
bits=[bits,bitget(data(2),1:16)];
high_bit = high_bit+16;
end
% We take only the bits that define the field (between low_bit and
% high_bit)
bits_used = bits(low_bit+1:high_bit+1);
% We convert the bits to dec
data = sum(bits_used.*uint16(2).^uint16([0:length(bits_used)-1]));
% We store it in the camdat.field struct
camdat.(layout{ct,1})=data;
end
% We close the DAI file
fclose(fid);
% Displaying for test
camdat
My approach in this case is to find the part of the file that matches your data.
fid = fopen('dai_file.dai','r','l');
dat=fread(fid,inf,'*uint8');
findstr(dat',[74,210,129,93]);
>> 891 1159 1427 1695 ....
Strange enough this happens 100 times.
If byte 891 is right than bios_1 is NOT in the 4th word from bit 0 to 7, but in the 445th word bit 0 to 7.
Let's try
fid = fopen(dai_file,'r','l');
fseek(fid,445*2,-1)
data=fread(fid,1,'*uint16');
bits=bitget(data(1),[1:16]);
bits = bits(1:8);
data = sum(bits.*uint16(2).^uint16([0:7]))
>> data = 74
Yep, there it is. So I would suggest to add 441 to each word entry and see if it works.
Oke, so you get information about the layout of the file.
I would first store this in a more accessabel format
layout{1,1} = 'mband_1_start_line';
layout{1,2} = 'PCHAR';
layout{1,3} = 12;
layout{1,4} = 6;
layout{1,5} = 12;
layout{1,6} = 15;
Then you loop over the layout
wordsize = 2; %bytes / word
fid = fread(filename,'r','l')
camdat=struct;
for ct = 1:size(layout,1)
fseek(fid,-1,layout{1,3}/wordsize) %go to byte position
datsize=layout{1,5}-layout{1,3}+1; %number of words
data=fread(fid,datsize,'*uint16') %get words
bits=bitget(data(1),[1:16]); %convert to bits
for ct = 2:datasize
bits=[bits,bitget(data(ct),[1:16])];
end
bits = bits(layout{1,4}:(datasize-1)*16+layout{1,6};%get bits
data = sum(bits.*uint16(2).^uint16([0:(length(bits)-1)])) %convert back
camdat.(layout{1,1})=data; %store
end
fclose(fid)
There will be problems with values that are longer than 16 bits ofcourse.
If the wordsize is different, you can change it to 4 for 32 bit, or 8 for 64 bit, but then you have to also change that in the loop.
So I've been using your help to figure a way to do what I wanted.
The idea is to go to the bytes of the "first_word", take the bits between the first and last word (and low_bit and high_bit), turn them into decimals. With your code I've done the following that gives results but not the one I was waiting for (in the .csv) (attached file).
First I'm not sure I'm handling well the case where the last_word is not the same as the first_word.
Then I'm not sure that my fseek() sends me at the correct bytes of the file...
%% Name of the files
%% Open and read the .cfg file
%% Open and read the DAI file
...So here I've got my .cfg opened and store in layout{i,j}
wordsize = 2; %bytes / word
fid = fopen(dai_file,'r','l');
camdat=struct;
for ct = 1:length(layout)
first_word = layout{ct,3};
last_word = layout{ct,5};
low_bit = layout{ct,4};
high_bit = layout{ct,6};
fseek(fid,first_word*wordsize,-1); %go to bytes
datasize=last_word-first_word+1; %number of words
data=fread(fid,datasize,'*uint16'); %get words
bits=bitget(data(1),[1:16]); %convert to bits
if length(data) > 1 % case of 2 words
bits=[bits,bitget(data(2),1:16)];
high_bit = high_bit+16;
end
bits = bits(low_bit+1:high_bit+1);%get bits
data = sum(bits.*uint16(2).^uint16([0:length(bits)-1])); %convert back
camdat.(layout{ct,1})=data; %store
end
camdat
fclose(fid);
So if you have ideas of where I'm wrong, I'll be very grateful !!!!

Performing logical OR between multiple CSVs with 32 bit hex values using MATLAB

I am trying to read multiple (50+) CSV files within the same folder using MATLAB. These CSVs contain 3 32 bit hex values and the format of the data is the same for all files. Each CSV contains the data within 2 rows and 3 columns with no headers. For e.g.
00000800,D404002C,4447538F
000008FF,D404002C,4447538F
After ORing the 2 rows from all files, the final 2 rows of 3 32 bit hex values need to be written out to a CSV.
Now, before jumping in the deep end trying to process multiple files, I have started by just trying to OR Row 1 with Row 2 of the same file. So, 00000800| 000008FF , D404002C | D404002C.. I have been able to convert them to binary and do a logical OR between the 3 values however currently have the following issues:
1) If the MSB of the hex value starts with 3 or 4 (binary 0011 or 0100) then the leading 0's are missed or if the second hex value happens to be 800 then the leading 00000's are missed.
2) I cannot convert the integer cell array back to hex
I have seen many posts about just reading CSVs using MATLAB or separating the data and etc on stackoverflow and matlabcentral however not been able to interpret any of them to sort my issue. Any help would be much appreciated.Below is what I have so far:
fid = fopen('File1.csv');
c = textscan(fid,'%s','Delimiter','\n');
fclose(fid);
contents = c{1};
row1 = strsplit(contents{1},',','CollapseDelimiters',0);
row2 = strsplit(contents{2},',','CollapseDelimiters',0);
x = 1;
y = 1;
while x <= length(row1)
column1{x} = hex2dec(row1(x));
column2{x} = hex2dec(row2(x));
x = x + 1;
end
while y <= length(column1)
bin1{y} = zeros(1,32);
bin2{y} = zeros(1,32);
bin1{y} = dec2bin(column1{y});
bin2{y} = dec2bin(column2{y});
result{y} = bitor(bitget(uint8(bin1{y}),1),bitget(uint8(bin2{y}),1));
y = y+ 1 ;
end
Also, eventually need to be able to do this process with multiple CSVs so I have attached link to File1.csv and File2.csv if someone wants to try to OR row 1 of File1 with row 2 of File2.csv and so on.
CSV Files
Apologies if I have missed anything, Please leave a comment and I'll try to explain it further.
Thanks!
EDIT: Hope the image below explains what I am trying to do better.
You can try the following approach:
use the dir function to get the list of files to be processed
create a loop to go through the files to be processed. In the loop
read the input files
convert the hexadecimal values read from the files into a matrix of characters using the char function
convert the data stored in the char matrinx from hex to dec and then to uint32 using the functions hex2dec and uint32
perform the or using the bitor function
go to next iteration
at the end of the loop, write the output
The above described approach has been implemented in the folowing code:
% Get the list of CSV files
hex_files=dir('O_File*.csv');
% Open the outpur file
fp_out=fopen('new_hex_file.csv','wt');
% Loop over the CSV files
for i=1:length(hex_files)
% Read the i-th CSV file
fid = fopen(hex_files(i).name);
c = textscan(fid,'%s','Delimiter','\n');
fclose(fid);
% Get the 2 rows
contents = c{1};
row_1=char(strsplit(contents{1},',','CollapseDelimiters',0));
row_2=char(strsplit(contents{2},',','CollapseDelimiters',0));
% Convert from hex to uint32
row_d_1=uint32(hex2dec(row_1));
row_d_2=uint32(hex2dec(row_2));
if(i == 1)
% Store the row of the first file and continue
tmp_var_1=row_d_1;
tmp_var_2=row_d_2;
continue
else
% OR the two rows
tmp_var_1=bitor(tmp_var_1,row_d_1);
tmp_var_2=bitor(tmp_var_2,row_d_2);
end
end
% Write the OR values into the new file
fprintf(fp_out,'%08X,%08X,%08X\n',tmp_var_1);
fprintf(fp_out,'%08X,%08X,%08X\n',tmp_var_2);
% Close the output file
fclose(fp_out);
The following input files have been used to test it:
File1.csv
00000800,D404002C,4447538F
000008FF,D404002C,4447538F
File2.csv
000008FF,D404DD2C,49475115
11100800,D411EC2C,3ACD1266
File3.csv
123456FF,ABCDEF2C,369ABC15
01012369,00110033,36936966
The output is:
12345EFF,FFCDFF2C,7FDFFF9F
11112BFF,D415EC3F,7EDF7BEF
Hope this helps.
Qapla'

How to sparsely read a large file in Matlab?

I ran a simulation which wrote a huge file to disk. The file is a big matrix v. I can't read it all, but I really only need a portion of the matrix, say, 1:100 of the columns and rows. I'd like to do something like
vtag = dlmread('v',1:100:end, 1:100:end);
Of course, that doesn't work. I know I should have only done the following when writing to the file
dlmwrite('vtag',v(1:100:end, 1:100:end));
But I did not, and running everything again would take two more days.
Thanks
Amir
Thankfully the dlmread function supports specifying a range to read as the third input. So if you wan to read all N columns for the first 100 rows, you can specify that with the following command
startRow = 1;
startColumn = 1;
endRow = 100;
endColumn = N;
rng = [startRow, startColumn, endRow, endColumn] - 1;
vtag = dlmread(filename, ',', rng);
EDIT Based on your clarification
Since you don't want 1:100 rows but rather 1:100:end rows, the following approach should work better for you.
You can use textscan to read chunks of data at a time. You can read a "good" row and then read in the next "chunk" of data to ignore (discarding it in the process), and continue until you reach the end of the file.
The code below is a slight modification of that idea, except it utilizes the HeaderLines input to textscan which instructs the function how many lines to ignore before reading in the data. The first time through the loop, no lines will be skipped, however all other times through the loop, rows2skip lines will be skipped. This allows us to "jump" through the file very rapidly without calling any additional file opertions.
startRow = 1;
rows2skip = 99;
columns = 3000;
fid = fopen(filename, 'rb');
% For now, we'll just assume you're reading in floating-point numbers
format = repmat('%f ', [1 columns]);
count = 1;
lines2discard = startRow - 1;
while ~feof(fid)
% Use "HeaderLines" to skip data before reading in data we care about
row = textscan(fid, format, 1, 'Delimiter', ',', 'HeaderLines', lines2discard);
data{count} = [row{:}];
% After the first time through, set the "HeaderLines" (i.e. lines to ignore)
% to be the # we want to skip between lines (much faster than alternatives!)
lines2discard = rows2skip;
count = count + 1;
end
fclose(fid);
data = cat(1, data{:});
You may need to adjust your format specifier for your own type of input.

Problem (bug?) loading hexadecimal data into MATLAB

I'm trying to load the following ascii file into MATLAB using load()
% some comment
1 0xc661
2 0xd661
3 0xe661
(This is actually a simplified file. The actual file I'm trying to load contains an undefined number of columns and an undefined number of comment lines at the beginning, which is why the load function was attractive)
For some strange reason, I obtain the following:
K>> data = load('testMixed.txt')
data =
1 50785
2 58977
3 58977
I've observed that the problem occurs anytime there's a "d" in the hexadecimal number.
Direct hex2dec conversion works properly:
K>> hex2dec('d661')
ans =
54881
importdata seems to have the same conversion issue, and so does the ImportWizard:
K>> importdata('testMixed.txt')
ans =
1 50785
2 58977
3 58977
Is that a bug, am I using the load function in some prohibited way, or is there something obvious I'm overlooking?
Are there workarounds around the problem, save from reimplementing the file parsing on my own?
Edited my input file to better reflect my actual file format. I had a bit oversimplified in my original question.
"GOLF" ANSWER:
This starts with the answer from mtrw and shortens it further:
fid = fopen('testMixed.txt','rt');
data = textscan(fid,'%s','Delimiter','\n','MultipleDelimsAsOne','1',...
'CommentStyle','%');
fclose(fid);
data = strcat(data{1},{' '});
data = sscanf([data{:}],'%i',[sum(isspace(data{1})) inf]).';
PREVIOUS ANSWER:
My first thought was to use TEXTSCAN, since it has an option that allows you to ignore certain lines as comments when they start with a given character (like %). However, TEXTSCAN doesn't appear to handle numbers in hexadecimal format well. Here's another option:
fid = fopen('testMixed.txt','r'); % Open file
% First, read all the comment lines (lines that start with '%'):
comments = {};
position = 0;
nextLine = fgetl(fid); % Read the first line
while strcmp(nextLine(1),'%')
comments = [comments; {nextLine}]; % Collect the comments
position = ftell(fid); % Get the file pointer position
nextLine = fgetl(fid); % Read the next line
end
fseek(fid,position,-1); % Rewind to beginning of last line read
% Read numerical data:
nCol = sum(isspace(nextLine))+1; % Get the number of columns
data = fscanf(fid,'%i',[nCol inf]).'; % Note '%i' works for all integer formats
fclose(fid); % Close file
This will work for an arbitrary number of comments at the beginning of the file. The computation to get the number of columns was inspired by Jacob's answer.
New:
This is the best I could come up with. It should work for any number of comment lines and columns. You'll have to do the rest yourself if there are strings, etc.
% Define the characters representing the start of the commented line
% and the delimiter
COMMENT_START = '%%';
DELIMITER = ' ';
% Open the file
fid = fopen('testMixed.txt');
% Read each line till we reach the data
l = COMMENT_START;
while(l(1)==COMMENT_START)
l = fgetl(fid);
end
% Compute the number of columns
cols = sum(l==DELIMITER)+1;
% Split the first line
split_l = regexp(l,' ','split');
% Read all the data
A = textscan(fid,'%s');
% Compute the number of rows
rows = numel(A{:})/cols;
% Close the file
fclose(fid);
% Assemble all the data into a matrix of cell strings
DATA = [split_l ; reshape(A{:},[cols rows])']; %' adding this to make it pretty in SO
% Recognize each column and process accordingly
% by analyzing each element in the first row
numeric_data = zeros(size(DATA));
for i=1:cols
str = DATA(1,i);
% If there is no '0x' present
if isempty(findstr(str{1},'0x')) == true
% This is a number
numeric_data(:,i) = str2num(char(DATA(:,i)));
else
% This is a hexadecimal number
col = char(DATA(:,i));
numeric_data(:,i) = hex2dec(col(:,3:end));
end
end
% Display the data
format short g;
disp(numeric_data)
This works for data like this:
% Comment 1
% Comment 2
1.2 0xc661 10 0xa661
2 0xd661 20 0xb661
3 0xe661 30 0xc661
Output:
1.2 50785 10 42593
2 54881 20 46689
3 58977 30 50785
OLD:
Yeah, I don't think LOAD is the way to go. You could try:
a = char(importdata('testHexa.txt'));
a = hex2dec(a(:,3:end));
This is based on both gnovice's and Jacob's answers, and is a "best of breed"
For files like:
% this is my comment
% this is my other comment
1 0xc661 123
2 0xd661 456
% surprise comment
3 0xe661 789
4 0xb661 1234567
(where the number of columns within the file MUST be the same, but not known ahead of time, and all comments denoted by a '%' character), the following code is fast and easy to read:
f = fopen('hexdata.txt', 'rt');
A = textscan(f, '%s', 'Delimiter', '\n', 'MultipleDelimsAsOne', '1', 'CollectOutput', '1', 'CommentStyle', '%');
fclose(f);
A = A{1};
data = sscanf(A{1}, '%i')';
data = repmat(data, length(A), 1);
for ctr = 2:length(A)
data(ctr,:) = sscanf(A{ctr}, '%i')';
end