I am following this tutorial to build cat-dog classifier using caffe. But while training I got this error; why?
The error you got "std::bad_alloc" usually refers to CPU memory allocation: you are out of memory.
Judging by the stack trace, you got the error when trying to read an element from the LMDB file. You need to check your LMDB: is it possible you store HUGE images there? is it possible the LMDB is corrupt? can you read it with different software?
Related
I am running a classification script in GEE and I have about 2100 training data since my AOI is a region in Italy and have many classes. I receive the following error while I try save my script:
Script error File too large (larger than 512KB).
I tried cancelling some of the training data and it saves. I thought there is no limit in GEE to choose training points. How can I know what is the limit so I adjust my training points or if there is a way to save the script without deleting any points.
Here is the link to my code
The Earth Engine Code Editor “drawing tools” are a convenient, but not very scalable, way to create geometry. The error you're getting is because “under the covers” they actually create additional code that is part of your script file. Not only is this fairly verbose (hence the error you received), it's not very efficient to run, either.
In order to use large training data sets, you will need to create your point data in another tool and upload it (using CSV or SHP files) to become one or more Earth Engine “table” assets, and use those from your script.
I'm writing neural networks using torch. Here's a little problem I can't solve.
I've put both network and network inputs into the GPU, but there was an error in training.
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same
I have solved the problem.
After reviewing the training function, I confirm that the model and input data have been loaded onto the GPU. Then the problem must be in the network model. When I checked the network model, I found that I wasn't using structures like List. It is likely that the model was not loaded into the GPU when other custom network models were used.
So I checked each layer of the network model, which is to force each layer onto the GPU (.cuda()). Depending on where the error prompt is located, the problem was identified after each layer's attempt, which was that the model was not loaded onto the GPU when a custom model was called.
Finally, simply find the location to call the custom model in the network-defined forward function and force it into the GPU(.cuda()).
I'm trying to train a network on my own data. Whole dataset consists of 256x256 jpeg images. There is 236 objects for classification. Training and validation sets have ~247K and ~61K images, respectively. I've made LMDBs from them using $CAFFE_ROOT/build/tools/convert_imageset utility.
Just for starting I'm using caffenet's topology for my model. During training I come across the weird message "Data layer prefetch queue empty" that I never seen before.
Moreover, initially, network has an abnormal accuracy (~0.00378378) and during next 1000 iterations, it reaches max ~0.01 and further does not increase (just fluctuates).
What I'm doing wrong and how can I improve the accuracy?
Runtime log:
http://paste.ubuntu.com/15568421/
Model:
http://paste.ubuntu.com/15568426/
Solver:
http://paste.ubuntu.com/15568430/
P.S. I'm using the latest version of Caffe, Ubuntu Server 14.04 LTS and g2.2xlarge instance on AWS.
this question may come as being too broad, but I will try to make every sub-topic to be as specific as possible.
My setting:
Large binary input (2-4 KB per sample) (no images)
Large binary output of the same size
My target: Using Deep Learning to find a mapping function from my binary input to the binary output.
I have already generated a large training set (> 1'000'000 samples), and can easily generate more.
In my (admittedly limited) knowledge of Neural networks and deep learning, my plan was to build a network with 2000 or 4000 input nodes, the same number of output nodes and try different amounts of hidden layers.
Then train the network on my data set (waiting several weeks if necessary), and checking whether there is a correlation between in- and output.
Would it be better to input my binary data as single bits into the net, or as larger entities (like 16 bits at a time, etc)?
For bit-by-bit input:
I have tried "Neural Designer", but the software crashes when I try to load my data set (even on small ones with 6 rows), and I had to edit the project save files to set Input and Target properties. And then it crashes again.
I have tried OpenNN, but it tries to allocate a matrix of size (hidden_layers * input nodes) ^ 2, which, of course, fails (sorry, no 117GB of RAM available).
Is there a suitable open-source framework available for this kind of
binary mapping function regression? Do I have to implement my own?
Is Deep learning the right approach?
Has anyone experience with these kind of tasks?
Sadly, I could not find any papers on deep learning + binary mapping.
I will gladly add further information, if requested.
Thank you for providing guidance to a noob.
You have a dataset containing pairs of binary valued vectors, with a max length of 4,000 bits. You want to create a mapping function between the pairs. On the surface, that doesn't seem unreasonable - imagine a 64x64 image with binary pixels – this only contains 4,096 bits of data and is well within the reach of modern neural networks.
As your dealing with binary values, then a multi-layered Restricted Boltzmann Machine would seem like a good choice. How many layers you add to the network really depends on the level of abstraction in the data.
You don’t mention the source of the data, but I assume you expect there to be a decent correlation. Assuming the location of each bit is arbitrary and is independent of its near neighbours, I would rule out a convolutional neural network.
A good open source framework to experiment with is Torch - a scientific computing framework with wide support for machine learning algorithms. It has the added benefit of utilising your GPU to speed up processing thanks to its CUDA implementation. This would hopefully avoid you waiting several weeks for a result.
If you provide more background, then maybe we can home in on a solution…
I tried to train my own neural net using my own imagedatabase as described in
http://caffe.berkeleyvision.org/gathered/examples/imagenet.html
However when I want to check the neural net after training on some standard images using the matlab wrapper I get the following output / error:
Done with init
Using GPU Mode
Done with set_mode
Elapsed time is 3.215971 seconds.
Error using caffe
Invalid input size
I used the matlab wrapper before to extract cnn features based on a pretrained model. It worked. So I don't think the input size of my images is the problem (They are converted to the correct size internally by the function "prepare_image").
Has anyone an idea what could be the error?
Found the solution: I was referencing the wrong ".prototxt" file (Its a little bit confusing because the files are quite similar.
So for computing features using the matlab wrapper one needs to reference the following to files in "matcaffe_demo.m":
models/bvlc_reference_caffenet/deploy.prototxt
models/bvlc_reference_caffenet/MyModel_caffenet_train_iter_450000.caffemodel
where "MyModel_caffenet_train_iter_450000.caffemodel" is the only file needed which is created during training.
In the beginning I was accidently referencing
models/bvlc_reference_caffenet/MyModel_train_val.prototxt
which was the ".prototxt" file used for training.