While attempting to compile darknet in the build command of a docker container I constantly run into the exception include/darknet.h:11:30: fatal error: cuda_runtime.h: No such file or directory.
I am building the container from the instructions here: https://github.com/NVIDIA/nvidia-docker/wiki/Deploy-on-Amazon-EC2. I have a simple Dockerfile I am testing with - the relevant parts:
FROM nvidia/cuda:9.2-runtime-ubuntu16.04
...
WORKDIR /
RUN apt-get install -y git
RUN git clone https://github.com/pjreddie/darknet.git
WORKDIR /darknet
# Set OpenCV makefile flag
RUN sed -i '/OPENCV=0/c\OPENCV=1' Makefile
RUN sed -i '/GPU=0/c\GPU=1' Makefile
#RUN ln -s /usr/local/cuda-9.2 /usr/local/cuda
# HERE I have been playing with commands to show me the state of the docker image to try to troubleshoot the problem
RUN find / -name "cuda_runtime.h"
RUN ls /usr/local/cuda/lib64/
RUN less /usr/local/cuda/README
RUN make
Most of the documentation I see references using the nvidia libraries when running a container, but the darknet compiles differently when built with gpu support so I need cuda_runtime.h available at build time.
Perhaps I misunderstand what nvidia-docker is doing - I'm assuming that nvidia-docker exists because the Nvidia code must be installed on the actual host machine and not inside the container & they use some mechanism to share the "native" code with the containers so the GPU can be managed - is that correct?
Should I even be trying to build darknet when building my container or should I be installing it on the host machine, then making it available somehow to the container? This seems to go against the portability of the containers but I can live with some constraints to get access to the GPU.
FROM nvidia/cuda:9.2-runtime-ubuntu16.04
Your image only has bits and pieces of CUDA-9.2 needed to run a CUDA app, but does not have the bits needed to build one.
You need to use -devel variant.
Related
I need to install some libraries in my pod before it starts working as expected.
My use case: I need some libraries that will support SMB (samba), and the image that I have to use does not have it installed.
Unfortunately, exec'ing into the actual pod and running commands do not seem to be a very good idea.
Is there a way by which I can use an init-container to install libsmbclient-dev in my ubuntu pod?
Edit: Some restrictions in my case.
I use helm chart to install my app (nextcloud). So I guess I cannot use a custom image (as far as I know, we cannot use our own images in an existing helm chart). This would have been the best solution.
I cannot run commands in kubernetes value.yaml since I do not use kubectl to install my app. Also I need to restart apache2 after I install the library, and unfortunately, restarting apache2 results in restarting the pod, effectively making the whole installation meaningless.
Since nextcloud helm allows the use of initcontainers, I wondered if that could be used, but as far as I understand the usability of initcontainers, this is not possible (?).
You should build your own container image - e.g. with docker - and push it to a container repository that is suitable for your cluster, e.g. Docker Hub, AWS ECR, Google Artifact registry ...
First install docker (https://docs.docker.com/get-docker/)
create an empty directory and change into it.
Then create a file Dockerfile with the following content:
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y libsmbclient-dev \
&& rm -rf /var/lib/apt/lists/*
Execute
docker build -t myimage:latest .
This will download Ubuntu and build a new container image where the commands from the RUN statement will be executed. The image name will be myimage and the version will be latest.
Then push your image with docker push to your appropriate repository.
See also Docker best practices:
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
I tried doing the following. First, I install a relocatable perl, e.g.:
% perlbrew -Duserelocatableinc -j4 -n --as perl-5.28.1-fresh_src 5.28.1
Then, whenever I need a new throw-away fresh installation of perl to test things out, I do:
% cp -la ~/perl5/perlbrew/perls/perl-5.28.1-fresh{_src,1}
% perlbrew switch 5.28.1-fresh1
(then would later create fresh2, fresh3 and so on as needed; and when I want to throw these away I just rm -rf ~/perl5/perlbrew/perls/*fresh1 and so on.)
However, something (cpanm? EUMM?) is still confused. Scripts installed from CPAN still has the original perl path (in this case, /home/USER/perl5/perlbrew/perls/perl-5.28.1-fresh_src/bin) in their shebang line.
Any hint to the problem above, or an alternative way to quickly create a fresh throw-away perl installation is appreciated. Also, would the -l (--link) option of cp in the above command cause issue (other than perllocal.pod being appended and will contain installation records of perl installations, which is fine for me)?
It's not quite the same, but I find docker works if I'm needing a clean install for certain testing scenarios - build up a base image and a clean perl install, fire up the container and run 'whatever'.
You'll need root access and be able to install packages (e.g. docker) so it's not going to be suitable for every use case.
https://buildlogs.centos.org/centos/7/docker/ has a base image you can use to build a 'clean' CentOS (or use whatever OS you prefer frankly - most have container versions)
FROM scratch
ADD CentOS-7-20140625-x86_64-docker_01.img.tar.xz
LABEL name="CentOS Base Image"
CMD ["/bin/bash"]
In the working directory:
docker build -t mycentos .
This will give you a very basic image:
Then your perl dockerfile:
FROM mycentos
COPY local.repo /etc/yum.repos.d/local.repo
RUN yum clean all
RUN yum install --nogpgcheck -y make gcc tar
ADD perl-5.28.1.tar.gz /build/
RUN cd /build/perl-5.28.1 && ./Configure -de
RUN cd /build/perl-5.28.1 && make && make test && make install
COPY MyConfig.pm /root/.cpan/CPAN/MyConfig.pm
RUN cpan install Bundle::CPAN
CMD ["/bin/bash"]
You'll be able to spin this up, with e.g. docker run and invoke a script using /usr/local/bin/perl.
It's not quite what you asked for, but your base image can be started and discarded as you wish - you can maintain image for a few different OS and perl combinations too.
I want to run ZFS inside of a virtual machine using Alpine Linux. The linux-virt kernel is much smaller and does not have the 200+MB of firmware files listed as dependencies, so that is the kernel I selected for the VM. However, I now find that there is no zfs-virt package, only zfs-vanilla which installs the vanilla kernel and all the firmware as dependencies.
Is there a zfs-virt package available in perhaps a third party repository? If not, I am not against building the package myself, but I'm relatively new to Alpine so have not yet figured out how its build system works nor if it is possible to build against an already-compiled kernel (in my own past experience, the only way I've successfully built kernel modules is using the source tree of the target kernel)
Just copy paste latest zfx-lts to zfs-virt, then abuild checksum and abuild -r. There is a prebuild package in my repository.
Install by
(cd /etc/apk/keys; sudo curl -LO https://repo.wener.me/alpine/wenermail#gmail.com-5dc8c7cd.rsa.pub )
echo https://repo.wener.me/alpine/v3.11/community | sudo tee -a /etc/apk/repositories
sudo apk add zfs-virt
I tried to install postgresql-9.3-postgis-2.1 or postgresql-9.1-postgis-2.1 for a cloned app, but I can only get postgresql-9.4-postgis-2.1 on my Ubuntu docker image which is build from python:2.7 image.
I looked into the image and found it's on a Ubuntu 14.04.1 image. I tries to install postgis on my Xubuntu 14.04.2 VM, everything is OK.
How could I get the installation works OK?
Dockerfile is pretty easy:
FROM python:2.7
RUN mkdir /workspace
RUN mkdir /data
WORKDIR /workspace
RUN apt-get update
RUN apt-get install postgresql postgresql-common postgresql-9.3-postgis-2.1
Error code is very normal too:
E: Unable to locate package postgresql-9.3-postgis-2.1
E: Couldn't find any package by regex 'postgresql-9.3-postgis-2.1'
Please provide more information, like the dockerfile and the errors you get.
From your comment it appears you load the python libraries before the postgresql libraries. I assume that your python app needs postgresql access and that it uses one of the python wrappers around the postgresql C libraries.
If that is the case then install the postgresql libraries before installing the python libraries, and do not forget to add the -dev libraries.
What I do in such a case is to make a minimal docker image, start a root shell in the container and do the install manually, take notes and use them to update the docker file. Alternatively you can run
$ docker exec -t -i bash -i
to get a shell in the container and try out what needs to be done.
Thanks for everyone who tried to help me! Though I finally fix this myself, there is nothing wrong with the Dockerfile which is pretty simple, but the image I chose is not a typical Ubuntu image, the docker office use buildpack-deps:jessie instead of ubuntu:14.04 image:
https://github.com/docker-library/python/blob/master/2.7/Dockerfile
It caused different behavior in docker and Ubuntu VM.
Finally, I build a Python image from Ubuntu:12.04 and fixed this issue.
I'm trying to imagine a workflow that could be applied on a scientific work environment. My work involves doing some scientific coding, basically with Python, pandas, numpy and friends. Sometimes I have to use some modules that are not common standards in the scientific community and sometimes I have to integrate some compiled code in my chain of simulations. The code I run is most of the time parallelised with IPython notebook.
What do I find interesting about docker?
The fact that I could create a docker containing my code and its working environment. I can then send the docker to my colleges, without asking them to change their work environment, e.g., install an outdated version of a module so that they can run my code.
A rough draft of the workflow I have in mind goes something as follows:
Develop locally until I have a version I want to share with somebody.
Build a docker, possibly with a hook from a git repo.
Share the docker.
Can somebody give me some pointers of what I should take into account to develop further this workflow? A point that intrigues me: code running on a docker can lunch parallel process on the several cores of the machine? e.g., an IPython notebook connected to a cluster.
Docker can launch multiple process/thread on multiple core. Multiple processes may need the use of a supervisor (see : https://docs.docker.com/articles/using_supervisord/ )
You should probably build an image that contain the things you always use and use it as a base for all your project. (Would save you the pain of writing a complete Dockerfile each time)
Why not develop directly in a container and use the commit command to save your progress on a local docker registry? Then share the final image to your colleague.
How to make a local registry : https://blog.codecentric.de/en/2014/02/docker-registry-run-private-docker-image-repository/
Even though you'll have a full container, I think a package manager like conda can still be a solid part of the base image for your workflow.
FROM ubuntu:14.04
RUN apt-get update && apt-get install curl -y
# Install miniconda
RUN curl -LO http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh
RUN bash Miniconda-latest-Linux-x86_64.sh -p /miniconda -b
RUN rm Miniconda-latest-Linux-x86_64.sh
ENV PATH=/miniconda/bin:${PATH}
RUN conda update -y conda
* from nice example showing docker + miniconda + flask
Wrt doing source activate <env> in the Dockerfile you need to:
RUN /bin/bash -c "source activate <env> && <do something in the env>"