Reading labview binary files in Matlab? - matlab

Question

I have large .bin files (10GB-60GB) created by Labview software, the .bin files represent the output of two sensors used from experiments that I have done.
The problem I have is importing the data into Matlab, the only way I have achieved this so far is by converting the .bin files to .txt files in Labview software then Importing the data into MATLAB using the following code:
Nlines = 1e6; % set number of lines to sample per cycle
sample_rate = (1); %sample rate
DECE= 1000;% decimation factor
TIME = (0:sample_rate:sample_rate*((Nlines)-1));%first inctance of time vector
format = '%f\t%f';
fid = fopen('H:\PhD backup\Data/ONK_PP260_G_text.txt');
while(~feof(fid))
C = textscan(fid, format, Nlines, 'CollectOutput', true);
d = C{1}; % immediately clear C at this point you need the memory!
clearvars C ;
TIME = ((TIME(end)+sample_rate):sample_rate:(sample_rate*(size(d,1)))+(TIME(end)));%shift Time along
plot((TIME(1:DECE:end)),(d(1:DECE:end,:)))%plot and decimate
hold on;
clearvars d;
end
fclose(fid);
The basic idea behind my code is to conserve RAM by reading Nlines of data from .txt on disk to Matlab variable C in RAM, plotting C then clearing C. This process occurs in loop so the data is plotted in chunks until the end of the .txt file is reached.
I want to read the .bin file directly into MATLAB rather than converting it to .txt first because it takes hours for the conversion to complete and I have a lot of data. Here are some examples of my data but in manageable sizes:
https://www.dropbox.com/sh/rzut4zbrert9fm0/q9SiZYmrdG
Here is a description of the binary data:
http://forums.ni.com/t5/LabVIEW/Loading-Labview-Binary-Data-into-Matlab/td-p/1107587
Someone has all ready written a Matlab script to import Labveiw .bin files but their script will only work with very small files:
% LABVIEWLOAD Load Labview binary data into Matlab
%
% DATA = LABVIEWLOAD(FNAME,DIM); % Loads the Labview data in binary
% format from the file specified by FNAME, given the NUMBER of
% dimensions (not the actual dimensions of the data in the binary
% file) of dimensions specified by DIM.
%
% LABVIEWLOAD will repeatedly grab data until the end of the file.
% Labview arrays of the same number of dimensions can be repeatedly
% appended to the same binary file. Labview arrays of any dimensions
% can be read.
%
% DATA = LABVIEWLOAD(FNAME,DIM,PREC); % Loads the data with the specified
% precision, PREC.
%
% Note: This script assumes the default parameters were used in the
% Labview VI "Write to Binary File". Labview uses the Big Endian
% binary number format.
%
% Examples:
% D = labviewload('Data.bin',2); % Loads in Data.bin assuming it
% contains double precision data and two dimensions.
%
% D = labviewload('OthereData.bin',3,'int8'); % Loads in
% OtherData.bin assuming it contains 8 bit integer values or
% boolean values.
%
% Jeremiah Smith
% 4/8/10
% Last Edit: 5/6/10
function data = labviewload(fname,dim,varargin)
siz = [2^32 2^16 2^8 1]'; % Array dimension conversion table
% Precision Input
if nargin == 2
prec = 'double';
elseif nargin == 3
prec = varargin{1};
else
error('Too many inputs.')
end
%% Initialize Values
fid = fopen(fname,'r','ieee-be'); % Open for reading and set to big-endian binary format
fsize = dir(fname); % File information
fsize = fsize.bytes; % Files size in bytes
%% Preallocation
rows = [];
columns = [];
I = 0;
while fsize ~= ftell(fid)
dims = [];
for i=1:1:dim
temp = fread(fid,4);
temp = sum(siz.*temp);
dims = [dims,temp];
end
I = I + 1;
% fseek(fid,prod(dims)*8,'cof'); % Skip the actual data
temp = fread(fid,prod(dims),prec,0,'ieee-be'); % Skip the actual data (much faster for some reason)
end
fseek(fid,0,'bof'); % Reset the cursor
data = repmat({NaN*ones(dims)},I,1); % Preallocate space, assumes each section is the same
%% Load and parse data
for j=1:1:I
dims = []; % Actual array dimensions
for i=1:1:dim
temp = fread(fid,4);
temp = sum(siz.*temp);
dims = [dims,temp];
end
clear temp i
temp = fread(fid,prod(dims),prec,0,'ieee-be'); % 11 is the values per row,
% double is the data type, 0 is the bytes to skip, and
% ieee-be specified big endian binary data
%% Reshape the data into the correct array configuration
if dim == 1
temp = reshape(temp,1,dims);
else
evalfunc = 'temp = reshape(temp';
for i=1:1:dim
evalfunc = [evalfunc ',' num2str(dims(dim-i+1))];
end
if dim ~= 2
eval([evalfunc ');'])
else
eval([evalfunc ')'';'])
end
end
data{j} = temp; % Save the data
end
fclose(fid); % Close the file
The code has the following error message, when you try to process even relatively small .bin files:
Error using ones
Maximum variable size allowed by the program is exceeded.
Error in labviewload (line 65)
data = repmat({NaN*ones(dims)},I,1); % Preallocate space, assumes each section is the same
Can you help me modify the code so that I can open large .bin files? Any help will be much appreciated.
Cheers,
Jim