Nvidia digits on TX2 Error code 1 - neural-network

Question

I am new to Digits and TX2. I am trying to create object detection model using the tutorial from: https://github.com/dusty-nv/jetson-inference
I created dataset sucessfully. The issue is with the model
While creating a model, I am getting the following error.
Memory required for data: 3268934784
creating layer bbox_loss
Creating Layer bbox_loss
bbox_loss <- bboxes-obj-masked-norm
bbox_loss <- bbox-obj-label-norm
bbox_loss -> loss_bbox
Setting up bbox_loss
Top shape: (1)
with loss weight 2
Memory required for data: 3268934788
Creating layer coverage_loss
Creating Layer coverage_loss
coverage_loss <- coverage_coverage/sig_0_split_0
coverage_loss <- coverage-label_slice-label_4_split_0
coverage_loss -> loss_coverage
Setting up coverage_loss
Top shape: (1)
with loss weight 1
Memory required for data: 3268934792
Creating layer cluster
The job directory information on the left is:
Job Directory
/home/nvidia/DIGITS/digits/jobs/20180816-161051-e67a
Disk Size
0 B
Network (train/val)
train_val.prototxt
Network (deploy)
deploy.prototxt
Network (original)
original.prototxt
Solver
solver.prototxt
Raw caffe output
caffe_output.log
Pretrained Model
/home/nvidia/bvlc_googlenet.caffemodel.4
Visualizations
Tensorboard
The error on the server is
2018-08-16 16:10:53 [20180816-161051-e67a] [INFO ] Task subprocess args: "/home/nvidia/Caffe/caffe/build/tools/caffe train --solver=/home/nvidia/DIGITS/digits/jobs/20180816-161051-e67a/solver.prototxt --gpu=0 --weights=/home/nvidia/bvlc_googlenet.caffemodel.4"
2018-08-16 16:11:00 [20180816-161051-e67a] [ERROR] Train Caffe Model task failed with error code 1
I have no idea on how to free up memory as I have more than 2 gb available in the job directory.
Please help me. Thanks in advance.

Answer 1

Had the same issue for the last few days, maybe it will help someone in the future. Firstly, make sure that you have the right version of protobuf. You can check it with:
protoc --version
If it's 2.* you have to update to 3.*, for example to build it as listed here https://github.com/NVIDIA/DIGITS/blob/digits-6.0/docs/BuildProtobuf.md, and then rebuild the Caffe. Also, make sure that you have the compatible version of pip package of protobuf. For me the following version is working well right now for Digits and Caffe from the tutorial https://github.com/dusty-nv/jetson-inference :
pip install --user --upgrade protobuf==3.1.0.post1