I have large .bin files (10GB 60GB) that I want to import to MATLAB; each binary file represents the output of two sensors, thus there are too columns of data. Here is a more manageable sized example of my data.
You will notice that there is a .txt version of the data; I need to upload the .bin files directly to MATLAB, I can't use the .txt version because it takes hours to convert with larger files.
The problem I have is that the .bin file has header information that I can't seem to interpret properly, and thus I cannot extract the data in MATLAB every time I try I seem to get gibberish values.
This is all the information I have about the binary header:
Loading Labview Binary Data into Matlab
LabVIEW Data Logger: Binary Header File Format
Any help/advice would be much appreciated I have been trying to solve this problem for days now.
P.S. Someone has already written a function to solve this problem but it does not seem to work with my binary data (could be something to do with the dimensions/size of my data): http://www.mathworks.co.uk/matlabcentral/fileexchange/27195-load-labview-binary-data
Below is the code that I am using to import my data, I believe that that d1 and d2 are the dimensions of my binary data. D2 is probably incorrect for the example file in the dropbox because it has been truncated.
The problem I have is that the code extracts my data and I know it is correct because I can check it with the .txt file (also in the drop box) however there are seaming random bad values between the good data points. These bad values result from the following strings following strings: "NI_ChannelName", "Sensor A", "Sensor B", "NI_UnitDescription", and "Volts" scatted throughout the binary file.
clear all
clc
fname = 'RTL5_57.bin';
fid = fopen(fname,'r','ieee-be');
d1 = fread(fid,4);
trash=fread(fid,2,'double');
d2 = fread(fid,4);
trash=fread(fid,1,'double');
data=fread(fid,'double');
I suppose you will need to change the data-format. See Matlab help.
https://decibel.ni.com/content/docs/DOC-39038
Scope:
1) Write a binary file in matlab and read into labview. 2) Write a binary file in labview and read into matlab.
Background:
IMPORTANT:
You must know (3) things about the binary data in the file before you can read the data:
1) what binary format (precision) was used to store the data
2) the exact number of values in the file to read.
3) Endianness
There is no row or column in binary files. Think of a long row/or a long column that needs to be mapped to a 2D array.
Resources on data in binary format.
http://cse.unl.edu/~sincovec/Matlab/Lesson%2024/Binary/CS211%20Lesson%2024%20-%20Binary%20File%20Input-Output.htm
Related
I need to read text files into Matlab. In the text files there are numbers like 5.875489^*-6, which is indeed 0.000005875489. Matlab cannot read this format and since there are too many files, I cannot change the format in all the files manually. So, I wonder if there is any tips to make Matlab reading the files as they are?
Any help and guide is highly appreciated.
Marilla.
As pointed out by #vu1p3n0x, it would probably be easier to replace ^* by e using a replace-all. Alternatively, if that is unpractical, you could read in the the mantissa and exponent separately and perform the exponentiation in Matlab:
Raw = textscan(fid, '%f^*%f');
Result = Raw{1}.*10.^Raw{2};
I have spent the last couple hours scouring the Internet for a solution to my problem, and while I have seen some "answers" on other forums, none of them suit my needs...
I have a binary file, which I am creating in Matlab using fwrite (although, if someone has a better way to generate a binary file in Matlab, I'm open to suggestions). Back to my problem - I have this binary file, and I want to convert it to a jpeg. Nevermind where the binary data comes from, I just want to generate a jpeg image of the binary data.
Is this even possible? - Like I said, lots of "solutions" out there to similar problems, but none match up to my needs.
I can write code in C++, if necessary, but for simplicity, I'd like to stay in Matlab.
Any help will be appreciated.
EDIT:
from binary to array:
fid = fopen('yourfilename.bin');
% read the entire file as characters
% transpose so that F is a row vector
B = fread(fid, '*char')'
fclose(fid);
Reshape to array according to image dimensions
C=reshape(B,512,512); % or whatever dimension you have
Upon having the array of strings just use:
D=int32(str2num(C));
I have a data file having 50 2-D data points written in Notepad. I want to use it in clustering algorithm to cluster these 50 points. How can I import this file? Is there any other way to use it in program?
You can save the data as a .csv file or you can save it to an excel spreadsheet and use xlsread(). See here for more info: http://www.mathworks.com/help/techdoc/ref/xlsread.html
For the .csv case, this post should prove helpful: Fastest way to import CSV files in MATLAB
Imagine you had the following data:
X = [randn(100,2)-1 ; randn(100,2)];
save data.mat X
Then its as simple as doing:
%# load data from MAT-file
load data.mat
%# cluster into K=2 clusters
C = kmeans(X,2);
%# show cluster assignment
gscatter(X(:,1), X(:,2), C)
It depends how you have formatted the data file. You say it is saved on notepad but that is not too helpful. Depending on what you have used as the data delimiter you can import the datafile into an array using the dlmread function. For example if your file is called filename.dat and have used a ; character to separate each data item within this file you could read the data into a matrix A using
A = dlmread("filename.dat",';');
I would suggest reading the help documentation on the dlmread function in matlab.
I have 20 text files each containing a vector of size 180. How can I access each text file and assign the vector to a variable?
If you want to do it manually use Import Data under File (or use uiimport).
If you want to automate it use fid = fopen(filename) and then use var = textscan(fid, 'format') where format depends on how your vectors are structured. Spend some time reading doc textscan and doc fopen everything you probably need to know is in those two files. If your data is nicely structured look at doc importdata.
My knowledge of matlab is merely on a need to know basis, so this is probably an elementary question. Nevertheless here it comes:
I have got a file containing data (16-bit integers) stored in binary format. How do I read it into a vector /an array in matlab? How do I write this data to a file in matlab? Is there any smart tweak to increase the performance speed when reading/writing a huge amount of data (gigabytes)?
As Bill the Lizard wrote you can use fread to load the data into a vector. I just want to expand a little on his answer.
Reading Data
>> fid=fopen('data.bin','rb') % opens the file for reading
>> A = fread(fid, count, 'int16') % reads _count_ elements and stores them in A.
The commands fopen and fread default to Little-endian[1] encoding for the integers. If your file is Big-endian encoded you will need to change the fread to
>> A = fread(fid, count, 'int16', 'ieee-be');
Also, if you want to read the whole file set
>> count=inf;
and if you want to read the data into matrix with n columns use
>> count=[n inf];
Writing Data
As for witting the data to a file. The command, fwrite, in Bill's answer will write to a binary file. If you want to write the data to a text file you can use dlmwrite
>> dlmwrite('data.csv',A,',');
References
[1] http://en.wikipedia.org/wiki/Endianness
Update
The machine format (IE, ieee-be,
ieee-le, vaxd etc.) of the binary data can be specified in either the
fopen or the fread commands in Matlab. Details of the supported
machine format can be found in
Matlab's documentation of fopen.
Scott French's comment to Bill's
answer
suggests reading the data into an
int16 variable. To do this use
>> A = int16(fread(fid,count,precision,machineFormat));
where count is the size/shape of
the data to be read, precision is
the data format, and machineformat
is the encoding of each byte.
See commands fseek to move around the file. For example,
>> fseek(fid,0,'bof');
will rewind the file to the beginning where bof stands for beginning of file.
Assuming you know how many values you have stored in the file, you can do something like this to read the data into an array.
fid = fopen('data.bin','rb')
A = fread(fid, count, 'int16')
To write data to a file do this:
fid = fopen('data.bin','w')
count = fwrite(fid, A, 'int16')
The fwrite function returns the number of elements (not bytes) written to the file.
As far as performance tuning goes, you can read data in chunks to only use as much as you need to process. This is the same in any language, and there's no way to speed it up that's specific to Matlab.
I usually hate seeing links in a response, but this looks pretty close:
http://www.mathworks.com/support/tech-notes/1400/1403.html
As to the second part of performance tuning, it's been 6 years since I've used Matlab, so I don't know.
HTH