Matlab, textscan error - matlab

I get an error when I run a Matlab code (attached), that continuosly check and opens two .txt files at a given time interval, (2 seconds) and depending on the result a value comparison between those two files, chose one or another and use it....
Those two files are constantly saved and updated by a Java script at the same time interval of 2 seconds.
All the files are located in the same path.
The error I get is: "invalid file identifier. use fopen to generate a valid file identifier
Error in KinectDEMelevation_with_filecomparison2 (line 36)
DEM1 =
textscan(fid2,formatSpec,'HeaderLines',6,'Delimiter','\b'); "
The code is:
DEM = GRIDobj('kinectDEM0.txt');
clims = [-30 140];
imagesc(DEM,clims);
colormap(jet);
hold on
while(1)
tic
% clear all
% clf
% load('MyColormaps','mycmap');
%% Get kinectDEM0 data
% Open 'kinectDEM0.txt'
fid1 = fopen('kinectDEM0.txt', 'r+');
% Read data in from the .txt file
formatSpec = '%n';
DEM0 = textscan(fid1,formatSpec,'HeaderLines',6,'Delimiter','\b');
% Extract data from DEM0
DEM0Data = DEM0{1,1}(200:100:287800,1);
% Close 'kinectDEM0.txt'
fclose(fid1);
%% Get kinectDEM1 data
% Open 'kinectDEM1.txt'
fid2 = fopen('kinectDEM1.txt', 'r+');
% Read data in from the .txt file
formatSpec = '%n';
DEM1 = textscan(fid2,formatSpec,'HeaderLines',6,'Delimiter','\b');
% Extract data from DEM1
DEM1Data = DEM1{1,1}(200:100:287800,1);
% Close 'kinectDEM1.txt'
fclose(fid2);
%% Compare data, a logical array return 1 for true (points that has been changed), 0 for false
test = eq(DEM0Data,DEM1Data);
num = sum(test); % numbers of point in the scene that has been changed
threshold = 2900; % after this threshold update the image
if num > threshold
DEM = GRIDobj('kinectDEM1.txt');
clf
clims = [-30 140];
imagesc(DEM,clims);
colormap(jet);
hold on
end
T = toc;
pause(2 - T);
end
How can I fix it?
Thanks

You should add some error checking to make sure the file:
exists
has been successfully opened
For the first, you can use exist:
r = exist('kinectDEM0.txt','file');
% will return 2 if file exists
For the second, you need to check fid is valid. If fopen cannot open the file, this will be -1. You may also use the second output for fopen, errmsg, for additional checking; errmsg should be an empty string if the file was opened successfully, otherwise it will return the system error message, such as "No such file or directory".
A simple example:
fid = -1
while fid == -1
fid = fopen('myfile.txt'); % attempt to open until success
end

Related

Matlab: Error using readtable (line 216) Input must be a row vector of characters or string scalar

I gave the error Error using readtable (line 216) Input must be a row vector of characters or string scalar when I tried to run this code in Matlab:
clear
close all
clc
D = 'C:\Users\Behzad\Desktop\New folder (2)';
filePattern = fullfile(D, '*.xlsx');
file = dir(filePattern);
x={};
for k = 1 : numel(file)
baseFileName = file(k).name;
fullFileName = fullfile(D, baseFileName);
x{k} = readtable(fullFileName);
fprintf('read file %s\n', fullFileName);
end
% allDates should be out of the loop because it's not necessary to be in the loop
dt1 = datetime([1982 01 01]);
dt2 = datetime([2018 12 31]);
allDates = (dt1 : calmonths(1) : dt2).';
allDates.Format = 'MM/dd/yyyy';
% 1) pre-allocate a cell array that will store
% your tables (see note #3)
T2 = cell(size(x)); % this should work, I don't know what x is
% the x is xlsx files and have different sizes, so I think it should be in
% a loop?
% creating loop
for idx = 1:numel(x)
T = readtable(x{idx});
% 2) This line should probably be T = readtable(x(idx));
sort = sortrows(T, 8);
selected_table = sort (:, 8:9);
tempTable = table(allDates(~ismember(allDates,selected_table.data)), NaN(sum(~ismember(allDates,selected_table.data)),size(selected_table,2)-1),'VariableNames',selected_table.Properties.VariableNames);
T2 = outerjoin(sort,tempTable,'MergeKeys', 1);
% 3) You're overwriting the variabe T2 on each iteration of the i-loop.
% to save each table, do this
T2{idx} = fillmissing(T2, 'next', 'DataVariables', {'lat', 'lon', 'station_elevation'});
end
the x is each xlsx file from the first loop. my xlsx file has a different column and row size. I want to make the second loop process for all my xlsx files in the directory.
did you know what is the problem? and how to fix it?
Readtable has one input argument, a filename. It returns a table. In your code you have the following:
x{k} = readtable(fullFileName);
All fine, you are reading the tables and storing the contents in x. Later in your code you continue with:
T = readtable(x{idx});
You already read the table, what you wrote is basically T = readtable(readtable(fullFileName)). Just use T=x{idx}

how to save multiple Cell array values in one .csv file

I have been working on making a database which contains images and their preset values and other important parameters. But unfortunately, I'm not being able to save the initial data of say 10 images in one .csv file. I have made the code that runs fine with creating .csv file but saving the last value and overwriting all the previous values. I gave also once modified that is comment down in the code using sprintf but it make .csv file for every iteration separately. But i want to make one .csv file containing 7 column with all the respective values.
My code is below and output of my code is attached Output.
Please someone guide me how to make single .csv file with 10 values for instance (could be increased to hundreds in final database) to save in 1 .csv file.
clc
clear all
myFolder = 'C:\Users\USER\Desktop\PixROIDirectory\PixelLabelData_1';
filePattern = fullfile(myFolder, '*.png'); % Change to whatever pattern you need
theFiles = dir(filePattern);
load('gTruthPIXDATA.mat','gTruth')
gTruth.LabelDefinitions;
for i=1:10
%gTruth.LabelData{i,1};
baseFileName = theFiles(i).name;
fullFileName = fullfile(myFolder, baseFileName);
fprintf(1, 'Now reading %s\n', fullFileName);
imageArray = imread(fullFileName);
oUt = regionprops(imageArray,'BoundingBox');
Y = floor(oUt.BoundingBox);
X_axis = Y(1);
Y_axis = Y(2);
Width = Y(3);
Height = Y(4);
CLASS = gTruth.LabelDefinitions{1,1};
JPG = gTruth.DataSource.Source{i,1};
PNG = gTruth.LabelData{i,1};
OUTPUT = [JPG X_axis Y_axis Width Height CLASS PNG]
% myFile = sprintf('value%d.csv',i);
% csvwrite(myFile,OUTPUT);
end
Try fprintf (https://www.mathworks.com/help/matlab/ref/fprintf.html).
You will need to open your output file to be written, then you can append lines to it through each iteration
Simple example:
A = [1:10]; % made up a matrix of numbers
fid = fopen('test.csv','w'); % open a blank csv and set as writable
for i = 1:length(A) % loop through the matrix
fprintf(fid,'%i\n',A(i)); % print each integer, then a line break \n
end
fclose(fid); % close the file for writing

Undefined function 'relabel' for input arguments of type 'int32'. Error in import_experiment_label (line 22) runs = relabel(run);

I got an error when I run a matlab code online. The error is:
Warning: Name is nonexistent or not a directory: ......\toolbox_misc
Undefined function 'relabel' for input arguments of type 'int32'.
Error in import_experiment_label (line 22)
runs = relabel(run);
I tried to download matlab toolbox misc online, but still cannot fix
the problem. Anyone can help me? Thank you so much!
The following is the original code:
% Load the text experiment-label file to cell
% Kinalizer: /share/Bot/Research/mvpa_data/Haxby_6_subjects
% mac: /Users/kittipat/Downloads/Research/Haxby_7_subjects
% subjID = 2;
inDir = ['/Users/kittipat/Downloads/Research/Haxby_7_subjects/subj',num2str(subjID),'/'];
inFile = 'labels.txt';
outDir = ['/Users/kittipat/Downloads/Research/Haxby_7_subjects/subj',num2str(subjID),'/matlab_format'];
fileID = fopen(fullfile(inDir,inFile));
% !!!!! Must remove "labels chunks" at the top of the txt file first
myCell = textscan(fileID, '%s %d');
fclose(fileID);
category_name = myCell{1};
run = myCell{2};
% Make sure the run numbers are well-ordered from 1 to R
addpath('../../../toolbox_misc/');
runs = relabel(run);
num_run = length(unique(runs));
num_time_stamp = length(runs);
% Make associate labels (needs input from user)
category_name_list = {'rest','face','house','cat','bottle','scissors','shoe','chair','scrambledpix'};
assoc_label_list = [0,1,2,3,4,5,6,7,8];
num_category = length(assoc_label_list); % including 'rest'
assoc_label = zeros(num_time_stamp,1);
regs = zeros(num_category,num_time_stamp);
for i = 1:num_time_stamp
assoc_label(i) = assoc_label_list(strcmp(category_name{i},category_name_list));
regs(assoc_label(i)+1,i) = 1; % 'rest' is column 1
end
regs_with_rest = regs;
regs = regs(2:end,:); % exclude "rest" in the 1-st column
num_category = num_category - 1; % exclude the "rest"
save(fullfile(outDir,'experiment_design'),...
'category_name',...% the category name for each time stamp
'assoc_label',...% the number label for each time stamp
'assoc_label_list',...% the mapping between category_name and assoc_label
'category_name_list',...% list of the category name
'num_category',...% number of categories excluding "rest"
'regs',...% the category matrix excluding "rest"
'num_run',...% number of runs in well-ordered integer
'runs'... % the run# for each time stamp
);
%% plot the figure
h1 = figure;
subplot(4,1,2); plot(assoc_label,'b.-');
xlim([1, num_time_stamp]);
set(gca,'YTick',0:max(assoc_label(:))); set(gca,'YTickLabel',category_name_list);
subplot(4,1,1); plot(runs,'r.-');
title('run number after relabeling --> runs'); xlim([1, num_time_stamp]);
subplot(4,1,3); imagesc(regs_with_rest);
title('original design matrix --> regs\_with\_rest');
set(gca,'YTick',1:(num_category+1)); set(gca,'YTickLabel',category_name_list);
subplot(4,1,4); imagesc(regs);
title('after "rest" is removed --> regs');
xlabel('time stamps');
set(gca,'YTick',1:num_category); set(gca,'YTickLabel',category_name_list(2:end));
print(h1,'-djpeg',fullfile(outDir,'experiment_design.jpg'));
I fix the relabel problem. Then I change part of the code as follows.
inDir = ['/D disk/MATLAB/R2014a/subjX/beta_extraction_for_Haxby_matlab_toolbox_v1_8/subj',num2str(subjID),'/'];
inFile = 'labels.txt';
outDir = ['/D disk/MATLAB/R2014a/subjX/beta_extraction_for_Haxby_matlab_toolbox_v1_8/subj',num2str(subjID),'/matlab_format'];
Another error comes:
import_experiment_label
Undefined function or variable 'subjID'.
Error in import_experiment_label (line 7)
inDir = ['/D disk/MATLAB/R2014a/subjX/beta_extraction_for_Haxby_matlab_toolbox_v1_8/subj',num2str(subjID),'/'];
How to fix this problem? I do not know what is the wrong here. Thank you guys!

Find which line of a .dat file MATLAB is working on

I have a MATLAB script that reads a line from a text file. Each line of the text file contains the filename of a CSV. I need to keep track of what line MATLAB is working on so that I can save the data for that line in a cell array. How can I do that?
To illustrate, the first few lines of my .dat file looks like this:
2006-01-003-0010.mat
2006-01-027-0001.mat
2006-01-033-1002.mat
2006-01-051-0001.mat
2006-01-055-0011.mat
2006-01-069-0004.mat
2006-01-073-0023.mat
2006-01-073-1003.mat
2006-01-073-1005.mat
2006-01-073-1009.mat
2006-01-073-1010.mat
2006-01-073-2006.mat
2006-01-073-5002.mat
2006-01-073-5003.mat
I need to save the variable site_data from each of these .mat files into a different cell of O3_data. Therefore, I need to have a counter so that O3_data{1} is the data from the first line of the text file, O3_data{2} is the data from the second line, etc.
This code works, but it's done without using the counter so I only get the data for one of the files I'm reading in:
year = 2006:2014;
for y = 1:9
flist = fopen(['MDA8_' num2str(year(y)) '_mat.dat']); % Open the list of file names - CSV files of states with data under consideration
nt = 0; % Counter will go up one for each file loaded
while ~feof(flist) % While end of file has not been reached
fname = fgetl(flist);
disp(fname); % Stores name as string in fname
fid = fopen(fname);
while ~feof(fid)
currentLine = fgetl(fid);
load (fname, 'site_data'); % Load current file. It is all the data for one site for one year
O3_data = site_data;
% Do other stuff
end
fclose(fid);
end
fclose(flist);
end
If I add the time index part, MATLAB is telling me that Subscript indices must either be real positive integers or logicals. nt is an integer so I don't know what I'm doing wrong. I need the time index so that I can have O3_data{i} in which each i is one of the files I'm reading in.
year = 2006:2014;
for y = 1:9
flist = fopen(['MDA8_O3_' num2str(year(y)) '_mat.dat']); % Open the list of file names - CSV files of states with data under consideration
nt = 0;
while ~feof(flist) % While end of file has not been reached
fname = fgetl(flist);
fid = fopen(fname);
while ~feof(fid)
currentLine = fgetl(fid);
nt = nt+1; % Time index
load (fname, 'site_data'); % Load current file. It is all the data for one site for one year
O3_data{nt} = site_data;
% Do other stuff
end
fclose(fid);
end
fclose(flist);
end
Try the following:
years = 2006:2014;
for y=1:numel(years)
% read list of filenames for this year (as a cell array of strings)
fid = fopen(sprintf('MDA8_O3_%d_mat.dat',years(y)), 'rt');
fnames = textscan(fid, '%s');
fnames = fnames{1};
fclose(fid);
% load data from each MAT-file
O3_data = cell(numel(fnames),1);
for i=1:numel(fnames)
S = load(fnames{i}, 'site_data');
O3_data{i} = S.site_data;
end
% do something with O3_data cell array ...
end
Try the following - note that since there is an outer for loop, the nt variable will have to be initialized outside of that loop so that we don't overwrite data from previous years (or previous j's). We can avoid the inner while loop since the just read file is a *.mat file and we are using the load command to load its single variable into the workspace.
year = 2006:2014;
nt = 0;
data_03 = {}; % EDIT added this line to initialize to empty cell array
% note also the renaming from 03_data to data_03
for y = 1:9
% Open the list of file names - CSV files of states with data under
% consideration
flist = fopen(['MDA8_O3_' num2str(year(y)) '_mat.dat']);
% make sure that the file identifier is valid
if flist>0
% While end of file has not been reached
while ~feof(flist)
% get the name of the *.mat file
fname = fgetl(flist);
% load the data into a temp structure
data = load(fname,'site_data');
% save the data to the cell array
nt = nt + 1;
data_03{nt} = data.site_data;
end
fclose(flist); % EDIT moved this in to the if statement
end
end
Note that the above assumes that each *.dat file contains a list of *.mat files as illustrated in your above example.
Note the EDITs in the above code from the previous posting.

Reading labview binary files in Matlab?

I have large .bin files (10GB-60GB) created by Labview software, the .bin files represent the output of two sensors used from experiments that I have done.
The problem I have is importing the data into Matlab, the only way I have achieved this so far is by converting the .bin files to .txt files in Labview software then Importing the data into MATLAB using the following code:
Nlines = 1e6; % set number of lines to sample per cycle
sample_rate = (1); %sample rate
DECE= 1000;% decimation factor
TIME = (0:sample_rate:sample_rate*((Nlines)-1));%first inctance of time vector
format = '%f\t%f';
fid = fopen('H:\PhD backup\Data/ONK_PP260_G_text.txt');
while(~feof(fid))
C = textscan(fid, format, Nlines, 'CollectOutput', true);
d = C{1}; % immediately clear C at this point you need the memory!
clearvars C ;
TIME = ((TIME(end)+sample_rate):sample_rate:(sample_rate*(size(d,1)))+(TIME(end)));%shift Time along
plot((TIME(1:DECE:end)),(d(1:DECE:end,:)))%plot and decimate
hold on;
clearvars d;
end
fclose(fid);
The basic idea behind my code is to conserve RAM by reading Nlines of data from .txt on disk to Matlab variable C in RAM, plotting C then clearing C. This process occurs in loop so the data is plotted in chunks until the end of the .txt file is reached.
I want to read the .bin file directly into MATLAB rather than converting it to .txt first because it takes hours for the conversion to complete and I have a lot of data. Here are some examples of my data but in manageable sizes:
https://www.dropbox.com/sh/rzut4zbrert9fm0/q9SiZYmrdG
Here is a description of the binary data:
http://forums.ni.com/t5/LabVIEW/Loading-Labview-Binary-Data-into-Matlab/td-p/1107587
Someone has all ready written a Matlab script to import Labveiw .bin files but their script will only work with very small files:
% LABVIEWLOAD Load Labview binary data into Matlab
%
% DATA = LABVIEWLOAD(FNAME,DIM); % Loads the Labview data in binary
% format from the file specified by FNAME, given the NUMBER of
% dimensions (not the actual dimensions of the data in the binary
% file) of dimensions specified by DIM.
%
% LABVIEWLOAD will repeatedly grab data until the end of the file.
% Labview arrays of the same number of dimensions can be repeatedly
% appended to the same binary file. Labview arrays of any dimensions
% can be read.
%
% DATA = LABVIEWLOAD(FNAME,DIM,PREC); % Loads the data with the specified
% precision, PREC.
%
% Note: This script assumes the default parameters were used in the
% Labview VI "Write to Binary File". Labview uses the Big Endian
% binary number format.
%
% Examples:
% D = labviewload('Data.bin',2); % Loads in Data.bin assuming it
% contains double precision data and two dimensions.
%
% D = labviewload('OthereData.bin',3,'int8'); % Loads in
% OtherData.bin assuming it contains 8 bit integer values or
% boolean values.
%
% Jeremiah Smith
% 4/8/10
% Last Edit: 5/6/10
function data = labviewload(fname,dim,varargin)
siz = [2^32 2^16 2^8 1]'; % Array dimension conversion table
% Precision Input
if nargin == 2
prec = 'double';
elseif nargin == 3
prec = varargin{1};
else
error('Too many inputs.')
end
%% Initialize Values
fid = fopen(fname,'r','ieee-be'); % Open for reading and set to big-endian binary format
fsize = dir(fname); % File information
fsize = fsize.bytes; % Files size in bytes
%% Preallocation
rows = [];
columns = [];
I = 0;
while fsize ~= ftell(fid)
dims = [];
for i=1:1:dim
temp = fread(fid,4);
temp = sum(siz.*temp);
dims = [dims,temp];
end
I = I + 1;
% fseek(fid,prod(dims)*8,'cof'); % Skip the actual data
temp = fread(fid,prod(dims),prec,0,'ieee-be'); % Skip the actual data (much faster for some reason)
end
fseek(fid,0,'bof'); % Reset the cursor
data = repmat({NaN*ones(dims)},I,1); % Preallocate space, assumes each section is the same
%% Load and parse data
for j=1:1:I
dims = []; % Actual array dimensions
for i=1:1:dim
temp = fread(fid,4);
temp = sum(siz.*temp);
dims = [dims,temp];
end
clear temp i
temp = fread(fid,prod(dims),prec,0,'ieee-be'); % 11 is the values per row,
% double is the data type, 0 is the bytes to skip, and
% ieee-be specified big endian binary data
%% Reshape the data into the correct array configuration
if dim == 1
temp = reshape(temp,1,dims);
else
evalfunc = 'temp = reshape(temp';
for i=1:1:dim
evalfunc = [evalfunc ',' num2str(dims(dim-i+1))];
end
if dim ~= 2
eval([evalfunc ');'])
else
eval([evalfunc ')'';'])
end
end
data{j} = temp; % Save the data
end
fclose(fid); % Close the file
The code has the following error message, when you try to process even relatively small .bin files:
Error using ones
Maximum variable size allowed by the program is exceeded.
Error in labviewload (line 65)
data = repmat({NaN*ones(dims)},I,1); % Preallocate space, assumes each section is the same
Can you help me modify the code so that I can open large .bin files? Any help will be much appreciated.
Cheers,
Jim