Following this example:
http://twiecki.github.io/blog/2016/07/05/bayesian-deep-learning/, the network can be trained on the gpu using float32 values for each variable.
Strangely though, if more layers get passed to the neural net (for example 5 layers with 800 neurons each) or when the number of neurons is set to higher values using less layers, I get an error message like this:
0%| | 1/50000 [00:00<6:47:16, 2.05it/s]Traceback (most recent >call last):
File "/opt/pycharm-community-2016.3.2/helpers/pydev/pydevd.py", line 1596, >in
globals = debugger.run(setup['file'], None, None, is_module)
File "/opt/pycharm-community-2016.3.2/helpers/pydev/pydevd.py", line 974, in >run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/d1211/PycharmProjects/BNN/withLasagne.py", line 342, in
v_params, trace, ppc, y_pred = run_advi(likelihood)
File "/home/d1211/PycharmProjects/BNN/withLasagne.py", line 299, in >run_advi
total_size=total_size, learning_rate=1e-2, epsilon=1.0
File "/home/d1211/pythonpath/lib/python/Theano-0.9.0-py2.7.egg/theano/configparser.py", line 117, in res
return f(*args, **kwargs)
File "/home/d1211/pythonpath/lib/python/pymc3-3.1rc3-py2.7.egg/pymc3/variational/advi_minibatch.py", line 528, in advi_minibatch
raise FloatingPointError('NaN occurred in ADVI optimization.')
FloatingPointError: NaN occurred in ADVI optimization.
On the other hand, when using float64 variables there is no error.
So I guess the number of variables that have to be stored is too big, but I don't understand how this is connected to float32/float64-values, which are just a mean of precision.
Can you please help me understand this?
Shame on me, the weights from my huge layers to the output of 10 neurons were too small to be computed by float32 integers. So there's no problem at all
Related
Background Information
I am newer to coding, and have been working on mask rcnn (modified faster rcnn) project to identify Stryker Miltary Vehicles in images. I trained the maskrcnn network with no problems with the help of matlab's mask rcnn training example and matlab answers, but when I tried to run the function detectMaskRCNN I am getting errors. The main error is that after running lines 39-45 in the Matlab's detectMaskRCNN funciton the YRCNNReg variable and the bboxes are not the same size. I know that it is the YRCNNReg variable that is not the correct size after reading the documentation YRCNNReg is supposed to be [numClasses*4 numObs]. I tried walking through the code of how YRCNNReg was created in the function but I couldn't understand what was happening in the multiple layers. What I do know is that both variables are using numClasses but YRCNNReg is including the background class while bboxes doesn't. For example YRCNNReg size is 8x1000 single and bboxes is a 1000x4 single. The params parameter that is passed in indicates that the numClasses to detect is 1. So I am not sure how to fix this size discrepancy. Thanks so much for your help!
Question
So my question is "how can I get YRCNNReg to be the correct size? Where in the process caused this discrepancy happen?"
Reference Link:
https://www.mathworks.com/help/deeplearning/ug/instance-segmentation-using-mask-rcnn.html
This is similar to the training example that I used to train my neural network but I changed what classes are trying to be identified.
Error:
Arrays have incompatible
sizes for this operation.
Error in
helper.applyRegression
(line 24)
gx = boxIn(:,3).*x +
px; % center position
Error in detectMaskRCNN
(line 93)
bboxes =
helper.applyRegression(bboxes,
reg, params.MinSize,
params.MaxSize);
detectMaskRCNN Code that is being ran:
line 38: % Run prediction on the inputs
line 39: [bboxes, YRCNNClass, YRCNNReg, featureMap] = predict(...
line 40: dlnet, X, 'Outputs', outputNodes);
line 41:
line 42 % Extract data from the output dlarrays
line 43: bboxes = extractdata(bboxes)';
line 44: YRCNNClass = extractdata(YRCNNClass);
line 45: YRCNNReg = extractdata(YRCNNReg);
...
line 93: bboxes = helper.applyRegression(bboxes, reg, params.MinSize,params.MaxSize);
https://github.com/matlab-deep-learning/mask-rcnn/blob/main/detectMaskRCNN.m
i have a very big size of text file(about 11GB) that needs to load in matlab.but when i use "textread" function,"out of memory" error occurs.and There is no way to reduce the file size. when i type memory, show this to me.
memory
Maximum possible array: 24000 MB (2.517e+10 bytes) *
Memory available for all arrays: 24000 MB (2.517e+10 bytes) *
Memory used by MATLAB: 1113 MB (1.167e+09 bytes)
Physical Memory (RAM): 16065 MB (1.684e+10 bytes)
* Limited by System Memory (physical + swap file) available.
Does anyone have a solution to this problem?
#Anthony suggested a way to read the file line-by-line, which is perfectly fine, but more recent (>=R2014b) versions of MATLAB have datastore functionality, which is designed for processing large data files in chunks.
There are several types of datastore available depending on the format of your text file. In the simplest cases (e.g. CSV files), the automatic detection works well and you can simply say
ds = datastore('myCsvFile.csv');
while hasdata(ds)
chunkOfData = read(ds);
... compute with chunkOfData ...
end
In even more recent (>=R2016b) versions of MATLAB, you can go one step further and wrap your datastore into a tall array. tall arrays let you operate on data that is too large to fit into memory all at once. (Behind the scenes, tall arrays perform computations in chunks, and give you the results only when you ask for them via a call to gather). For example:
tt = tall(datastore('myCsvFile.csv'));
data = tt.SomeVariable;
result = gather(mean(data)); % Trigger tall array evaluation
According to your clarification of the purpose of your code:
it is a point cloud with XYZRGB column in txt file and i needs to add another column to this.
What I suggest you to do is read the text file one line at a time, modify the line and write the modified line straight to a new text file.
To read one line at a time:
% Open file for reading.
fid = fopen(filename, 'r');
% Get the first line.
line = fgetl(fid);
while ~isnumeric(line)
% Do something.
% get the next line
line = fgetl(fid);
end
fclose(fid);
To write the line, you can use fprintf.
Here is a demonstration:
filename = 'myfile.txt';
filename_new = 'myfile_new.txt';
fid = fopen(filename);
fid_new = fopen(filename_new,'w+');
line = fgetl(fid);
while ~isnumeric(line)
% Make sure you add \r\n at the end of the string;
% otherwise, your text file will become a one liner.
fprintf(fid_new, '%s %s\r\n', line, 'new column');
line = fgetl(fid);
end
fclose(fid);
fclose(fid_new);
I need to load and do some maths with 32 files (extension .mat) at the same time. So, after running the code, I expect to have 32 results of the maths.
The problem is that all the codes I'm trying just load the first or the last file.
The name of my files are: 21 pcb 11_01.mat; 21 pcb 11_02 ....21 pcb 11_32. I've tried this:
for i=1:32
filename=strcat("21 pcb 11_",sprintf("%02d",i),".mat")
load(filename)
endfor
As a result, the code only shows the last file in the workspace.
I expected the code to load the 32 files.
Can you help me?
If your Picoscope files are all the same length, say Lpico, then this ought to work:
Pico=NaN*ones(32,Lpico);
for k=1:32
filename=strcat("21 pcb 11_",sprintf("%02d",i),".mat")
load(filename)
Lthisrun=length(A);
Pico(k,1:Lthisrun)=A;
endfor
If they have different length, then make Lpico as long as the longest A. Shorter scope ouputs will be padded with NaN's
I'm batch processing statistical analyses and need to save down graphs as part of the output. I want to include some parameters settings as part of the graphs' filenames, to associate them easier at a later stage.
However, if I the FileName is 'too long' the print command throws errors. MATLAB Help is silent on this issue.
h = figure;
plot(somedata, moredata);
FileName = strcat(Name(1),'_',Name(2),'_',num2str(a),'_',num2str(b),'_','.jpeg');
print(h,'-djpeg',FileName);
Any idea how many chars 'too long' is?
I am working on importing some data interpretation of binary files from Fortran to MATLAB and have come across a bit of an issue.
In the Fortran file I am working with the following check is performed
CHARACTER*72 PICTFILE
CHARACTER*8192 INLINE
INTEGER NPX
INTEGER NLN
INTEGER BYTES
c This is read from another file but I'll just hard code it for now
NPX = 1024
NLN = 1024
bytes=2
open(unit=10, file=pictfile, access='direct', recl=2*npx, status='old')
read(10,rec=nln, err=20) inline(1:2*npx)
go to 21
20 bytes=1
21 continue
close(unit=10)
where nln is the number of lines in the file being read, and npx is the number of integers contained in each line. This check basically determines whether each of those integers is 1 byte or 2 bytes. I understand the Fortran code well enough to figure that out, but now I need to figure out how to perform this check in MATLAB. I have tried using the fgetl command on the file and then reading the length of the characters contained but the length never seems to be more than 4 or 5 characters, when even if each integer is 1 byte the length should be somewhere around 1000.
Does someone know a way that I can automatically perform this check in MATLAB?
So what we figured out was that the check is simply to see if the file is the correct size. In Matlab this can be done as
fullpath=which(file); %extracting the full file path
s=dir(fullpath); %extracting information about hte file
fid=fopen(file_name,'r'); %opening image file
if s.bytes/NLN==2*NPX %if the file is NLN*NPX*2 bytes
for n=1:NLN %for each line
dn(n,:) = (fread(fid, NPX, '*uint16','b'))'; %reading in lines into DN
end
elseif s.bytes/NLN==NPX %Else if the file is NLN*NPX bytes
for n=1:NLN %for each line
dn(n,:) = (fread(fid, NPX, '*uint8','b'))'; %reading in lines into DN
end
else %If the file is neither something went wrong
error('Invalid file. The file is not the correct size specified by the SUM file')
end
where file contains the filename, nln contains the number of lines, and npx contains the number of columns. Hope this helps anyone who may have a similar answer, but be warned because this will only work if your file only contains data that has the same number of bytes for each entry, and if you know the total number of entries there should be!
Generally speaking, binary files don't have line lengths; only text files have line lengths. MATLAB's getl will read until it finds the binary equivalent of newline characters. It then removes them and returns the result. The binary file, on the other hand, should read a block of length 2*npx and return the result. It looks like you want to use fread to get a block of data like this:
inline = fread(fileID,2*npx)
Your fortran code is requesting to read record nln. If the code you have shared reads all the records starting at the first one and working up, then can just put the above code in a loop for nln=1:maxValue. However, if you really do want to yank out record nln you need to fseek to that position first:
fseek(fileID, nln*2*npx, -1);
inline = fread(fileID,2*npx)
so you get something like the following:
Either reading them all in a loop:
fileID = fopen(pictfile);
nln = 0;
while ~feof(fileID)
nln = nln+1;
inline = fread(fileID,2*npx);
end
fclose(fileID);
or picking out only the number `nln record:
fileID = fopen(pictfile);
nln = 7;
fseek(fileID, nln*2*npx, -1);
inline = fread(fileID,2*npx);
fclose(fileID);