I have a text file (5 columns "\t" separated) that's being written to by another software. I need to take the readings from the file and do some calculations. Is there a was to read the new lines added to the file and process it then repeat again for every new set of lines. I don't mind a bit of delay as long as it does the job.
My idea is to start reading the file line by line until the end of file, then it will read from where it stopped last until the new end of file ...etc.
Can this be done in Matlab? Can I specify the starting line for the file reading? can I also update the end of file point?
To prevent the loop from breaking at the eof point, I think I should set my loop to be controlled by time or anything else, while it should check for eof at the end of every iteration.
I've mostly worked with Matlab, but if there is a better option to use for this purpose (that I can reasonably learn) please feel free to guide me.
Edit1: I've tried using dlmread as you suggested, when I read the file outside the loop it reads the file correctly even when I change R1 and with the other software updating the text. However, when I put it in a loop I get this error:
Error using dlmread (line 143)
Empty format string is not supported at the end of a file.
Here is my code to read it multiple times:
clear all
x=0;
R1=0; C1=0;
while(x<10)
M = dlmread('tst_4.txt','\t',R1,C1);
R1=length(M);
x=x+1;
end
Thanks
You can used dlmread(filename,delimiter,R1,C1). Where R1 and C1 are the row and column offset respectively. By setting the row offset to the last row that you have read, you can read the file content excluding what you would have already read.
Related
I have a file that is nth lines long and I want to extract line 10 from the file and read it in as a string. I don't want to import the file, I don't want to search for a string in the file, and I don't want to skip nth lines, I just want to read in line 10. I'm having trouble scripting this up, how can I do this?
fileID = fopen(test.txt','r');
fclose(fileID)
If you knew exactly how many bytes into the file line 10 was, you could use fseek to skip to that offset in file. If you do not know this, then you have no other option than to read line by line using fgetl and ignore lines until you get to line 10.
Matlab can't find the nth line without linearly searching for eol characters. Even if a function did exist to go to line 10, that function would still need to read every line and check for eol. You have to either skip n lines use fgets/fgetl or use fseek if you know how many bytes precede the line.
Is there any way or command in Tcl for writing in the middle of {data.txt} and also specific line number ... ?
for example after writing data in text file, when I'm writing in line number 1000, is there any way for turning back to line number 20 and adding the data in this line for output. (something look like llappend & append for list variables, but in puts command)
You can use seek to move the current location in a channel opened on a file (it's not meaningful for pipes, sockets and serial lines); the next read (gets, read) or write (puts) will happen at that point. Except in append mode, when writes always go to the end.
seek $channel $theNewLocation
However, the location to seek to is in bytes; the only locations that it is trivially easy to go to are the start of the file and the end of the file (the latter using end-based indexing). Thus you need to either remember where “line 20” really is from the first time, or go to the start and read forward a few lines.
seek $channel 0
for {set i 0} {$i < 20} {incr i} {gets $channel}
Also be aware that the data afterwards does not shuffle up or down to accommodate what you've written the second time. If you don't write exactly the same length of data as was already there, you end up with partial lines in the file. Truncating the file with chan truncate might help, but might also be totally the wrong thing to do. If you're going to go back and patch a text file, writing an extra long line where you're going to do the patch (e.g., with lots of spaces on the end) can make it much easier. It's a cheesy hack, but simple to do.
I am new to MATLAB programming and some of the syntax escapes me. So I need a little help. Plus I need some complex looping ideas.
Here's the breakdown of what I have:
12 seperate .dat files, each titled something like output_1_x.dat, output_2_x.dat, etc.
each file is actually one piece of a whole that was seperated and processed
each .dat file is approx. 3.9 GB
Here's what I need to do:
create a single file containing all the data from each seperate file, i.e. I need to recreate the original file.
call this complete output file something like output_final.dat
it has to be done in MATLAB, there are no other alternatives (actually there maybe; see note below)
What is implied:
I will have to fread each 3.9 GBfile into chunks or packets, probably 100 mb at a time (using an imbedded loop?)
these packets will have to be read then written sequentially
after one file is read then written into output_final.dat, the next file is automatically read & written (the master loop).
Well, that's pretty much it. I did a search for 'merging mulitple files' and found this. That isn't exactly what I need to do...I don't need to take part of a file, or data from files, and write it to a new one. I'm simply...concatenating...? This would be simple in Java or Perl, but I only have MATLAB as a tool.
Note: I am however running KDE in OpenSUSE on a pretty powerful box. Maybe someone who is also an expert in terminal knows a command/script to do this from the kernel?
So on this site we usually would point you to whathaveyoutried.com but this question is well phrased.
I wont write the code but i will give you how I would do it. So first I am a bit confused about why you need to fread the file. Are you just appending one file onto the end of another?
You can actually use unix commands to achieve what you want:
files = dir('*.dat');
for i = 1:length(files)
string = sprintf('cat %s >> output_final.dat.temp', files(i).name);
unix(string);
end
That code should loop through all the files and pipe all of the content into output_final.dat.temp (then just rename it, we didn't want it to be included in anything);
But if you really want to use fread because you want to parse the lines in some manner then you can use the same process:
files = dir('*.dat');
fidF = fopen('output_final.dat', 'w');
for i = 1:length(files)
fid = fopen(files(i).name);
while(~feof(fid))
string = fgetl(fid) %You may choose to parse the string in some manner here
fprintf(fidF, '%s', string)
end
end
Just remember, if you are not parsing the lines this will take much much longer.
Hope this helps.
I suggest using a matlab.io.matfileclass objects on two of the files:
matObj1 = matfile('datafile1.mat')
matObj2 = matfile('datafile2.mat')
This does not load any data into memory. Then you can use the objects' methods to sequentialy save a variable from one file to another.
matObj1.varName = matObj2.varName
You can get all the variables in one file with fieldnames(mathObj1) and loop through to copy contents from one file to another. You can then clear some space by removing the copied fields. Or you can use a bit more risky procedure by directly moving the data:
matObj1.varName = rmfield(matObj2,'varName')
Just a disclaimer: haven't tried it, use at own risk.
I have a large MATLAB file (150MB) in matrix form (i.e. 4070x4070). I need to work on this file in MATLAB but I can't seem to load this file. I am getting an "out of memory" error. Is there any other way I can load this size of file? I am using a 32bit processor and have 2GB of RAM. Please help me, I am getting exhausted from dealing with this problem.
Starting from release R2011b (ver.7.13) there is a new object matlab.io.MatFile with MATFILE as a constructor. It allows to load and save parts of variables in MAT-files. See the documentation for more details. Here is a simple example to read part of a matrix:
matObj = matfile(filename);
a = matObj.a(100:500, 200:600);
If your original file is not a MAT file, but some text file, you can read it partially and use matfile to save those parts to the same variable in a MAT file for later access. Just remember to set Writable property to true in the constructor.
Assuming your text file is tab-delimited and contains only numbers, here is a sample script to read the data by blocks and save them to MAT file:
blocksize = 100;
startrow = 0;
filename = 'test.mat';
matObj = matfile(filename,'Writable',true);
while true
try
a = dlmread(filename,'\t',startrow,0); %# depends on your file format
startrow = startrow + blocksize;
matObj.a(startrow+(1:blocksize),:) = a;
catch
break
end
end
I don't have the latest release now to test, but hope it should work.
If it is an image file, and you want to work with it, try the matlab block processing. By using it, you will load small parts of the file. Your function fun will be applied to each block individually.
B = blockproc(src_filename,[M N],fun)
In case it is an xml file, try the XML DOM Node mode together with SAX - (Thanks to #Nzbuu for pointing that out), but that seems to be an undocumented functionality.
Also, if it is a textual file of any kind (Unlikely, due to the amount of data), try external tool to split.
You can also user MATLAB's Memory-Mapping of Data Files to read in a block of the file, process it, and proceed to the next block without having to load the entire file into memory at once.
For instance, see this example, which "maps a file of 100 double-precision floating-point numbers to memory."
In my scenario, I have 100 nodes. Each time a random node out of them generates a data. I wish to record them in previously created files.
I have been using switch-case style to open the particular file associated with a node. However, it's clumsy for 100 nodes already and I need to increase the number of nodes. I was looking for a straight forward manner of opening a file based on node. I found bit hint here:
Stackoverflow_a_year_ago
But I'm unable to pick and open a particular file, say if the random node is 125, I'll open n125.txt file. Any help is appreciated. Here goes the code:
number_of_nodes=100;
for i=1:number_of_nodes
rand_node=ceil(rand(1,1)*100);
rand_output=ceil(rand(1,1)*10);
switch(rand_node)
case{1}
f1=fopen('n1.txt', 'a+');
fprintf(f1, rand_output);
fclose(f1);
case{2}
f2=fopen('n2.txt', 'a+');
fprintf(f2, rand_output);
fclose(f2) ;
end
end
Also, tried,
%..........................................
Names = dir('myprog*.TXT');
Names.name; %returns all file names.
Maybe I'm misunderstanding your question but the answer seems obvious:
fid=fopen(sprintf('n%d.txt',rand_node), 'a+');
fprintf(fid, rand_output);
fclose(fid);