I think I don't get it. First, I created docker-machine:
$ docker-machine create -d virtualbox dev
$ eval $(docker-machine env dev)
Then I wrote Dockerfile and docker-compose.yml:
FROM python:2.7
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
ADD requirements.txt /code/
RUN pip install -r requirements.txt
ADD . /code/
version: '2'
services:
db:
image: postgres
web:
build: .
restart: always
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
links:
- db
Finally, I built and started the image:
$ docker-compose build --no-cache
$ docker-compose start
I checked ip of my virtual machine
$ docker-machine ip dev
and successfully opened the site in my browser. But when I made some changes in my code - nothing happened. So I logged to the "dev" machine:
$ docker-machine ssh dev
and I didn't find my code! So I logged to the docker "web" image:
$ docker exec -it project_web_1 bash
and there was a code, but unchanged.
What is the docker-machine for? What is the sense? Why docker doesn't syncing files after changes? It looks like docker + docker-machine + docker-compose are pain in the a...s for local development :-)
Thanks.
Docker is the command-line tool that uses containerization to manage multiple images and containers and volumes and such -- a container is basically a lightweight virtual machine. See https://docs.docker.com/ for extensive documentation.
Until recently Docker didn't run on native Mac or Windows OS, so another tool was created, Docker-Machine, which creates a virtual machine (using yet another tool, e.g. Oracle VirtualBox), runs Docker on that VM, and helps coordinate between the host OS and the Docker VM.
Since Docker isn't running on your actual host OS, docker-machine needs to deal with IP addresses and ports and volumes and such. And its settings are saved in environment variables, which means you have to run commands like this every time you open a new shell:
eval $(docker-machine env default)
docker-machine ip default
Docker-Compose is essentially a higher-level scripting interface on top of Docker itself, making it easier (ostensibly) to manage launching several containers simultaneously. Its config file (docker-compose.yml) is confusing since some of its settings are passed down to the lower-level docker process, and some are used only at the higher level.
I agree that it's a mess; my advice is to start with a single Dockerfile and get it running either with docker-machine or with the new beta native Mac/Windows Docker, and ignore docker-compose until you feel more comfortable with the lower-level tools.
Related
I want to create two Docker volumes and have their data be persistent. I run sudo docker compose up -d, post some data to my website (text that is stores in a sqlite database and an image stored in the filesystem), then run sudo docker compose down. When I run sudo docker compose up -d again, all the data I posted is gone. With the following configs, I expect the data to still be present.
Dockerfile:
FROM python:3.9.16-buster
RUN pip install --upgrade pip
# The Debian/Buster default is to disable the password.
RUN adduser nonroot
RUN mkdir /home/site/ && chown -R nonroot:nonroot /home/site
RUN chown -R nonroot:nonroot /var/log/site
# two volumes created
VOLUME /home/site/db /home/site/static
WORKDIR /home/site
USER nonroot
# folders ./site/static and ./site/db exist in my host directory
COPY --chown=nonroot:nonroot . .
CMD ["python", "./site/main.py"]
compose.yaml:
services:
site:
build: flask
restart: always
ports:
- '8081:8081'
volumes:
- site_db:/home/site/db # same path as the volumes created in the Dockerfile
- site_static:/home/site/static
command: gunicorn -w 1 -t 3 -b 0.0.0.0:8081 --chdir ./site main:app
volumes:
site_db: # I find it odd these volumes keys don't have values, but that's what I have see other people do
site_static:
docker compose up and docker compose down delete my volumes.
docker compose start and docker compose stop do NOT delete my volumes.
Through the Flask app, check where you are uploading the files to, as well as where the sqlite3 db file is. If these paths do not align with the volumes paths, data will not persist.
I have a docker-compose file that looks like this
version: "3.7"
services:
app:
stdin_open: true
tty: true
build:
context: .
dockerfile: app.Dockerfile
volumes:
- ${HOST_SAVE_DIRC}:${CONTAINER_SAVE_DIRC}
depends_on:
- postgres
postgres:
image: 'postgres'
environment:
- POSTGRES_DB=${POSTGRES_DB}
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
- POSTGRES_HOST_AUTH_METHOD=trust
restart: always
expose:
- "5432"
where variables like POSTGRES_USER are entries from a env file. app.Dockerfile looks like
FROM python:3.8.3-slim-buster
COPY src /src/
COPY init.sql .
COPY .env .
COPY run.sh run.sh
COPY requirements.txt .
RUN ls -a
RUN pip install --no-cache-dir -r requirements.txt
The containers are created, then the user is logged into the app container w/ the main function of the program being called - this is when the database calls
From the app container I am attempting to connect to the postgres container via psycopg2. However when I attempt to do so, I receive the following error:
psycopg2.OperationalError: could not connect to server: No route to host
Is the server running on host "postgres" (172.22.0.2) and accepting
TCP/IP connections on port 5432?
using a psycopg2 call that looks like
with psy.connect(host='postgres', port=5432, user='postgres', password='postgres') as conn:
...
the entries of this psycopg2 call match the env file given to the docker-compose file.
My understanding is that Postgres uses port 5432 by default. Also that when docker-compose creates the two containers - it creates a docker network for those containers name DIR_default where DIR is the name of the directory the docker-compose file lives in, where each container can be accessed with using the name listed in the docker-compose file ('postgres' and 'app' in these cases).
Among various tries:
I've checked and the database isn't going down between the container being created and the user being exec'd in.
I've tried various little changes like changing the container names, postgres login info, etc.
I've tried linking the postgres container name explicitly with link: "postgres:postgres".
Other solutions suggested here
Any help would be greatly appreciated! I see no reason why something as simple as this should be occurring, but also here I am.
Edit:
Pinging the Postgres container from the app container appears to be working when running docker exec app ping postgres_container_name. Is this a sign that the Docker network is set up correctly and the issue is something of mine?
Edit 2:
Tried clearing all images and containers, then restarting the Docker daemon and afterwards my PC. No change in either case.
For reference, the ping command looked like
docker exec python-app ping name_given_to_postgres_container
returning various statements which looked like
64 bytes from name_given_to_postgres_container.project_name_default (172.18.0.3): icmp_seq=1 ttl=64 time=0.090 ms
which unless I am mistaken, I believe is signalling a succesful ping.
The top level .env file provided to docker-compose
HOST_SAVE_DIRC=~/python_projects/project_directory/directory_in_project
CONTAINER_SAVE_DIRC=/pdfs
POSTGRES_DB=project_name # same as project_directory
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_PORT=5432
Here is the requirements.txt file for the Python app as well
certifi==2020.4.5.1
chardet==3.0.4
idna==2.9
psycopg2-binary==2.8.5
read-env==1.1.0
requests==2.23.0
urllib3==1.25.9
Exec-ing into the Postgres container with docker exec -it container_id bash and running psql -U postgres appears to be successful - even with restart: always removed. I can also see the database named in the docker-compose file is also created. I feel confident in saying this container isn't dying spontaneously.
However, hitting the 5432 port on the Postgres container with netcat via nc name_given_to_postgres_container 5432-5433 returns an error similar to the one returned by psycopg2
arxivist_postgres_1 [172.22.0.3] 5433 (?) : No route to host
arxivist_postgres_1 [172.22.0.3] 5432 (postgresql) : No route to host
The same error is also returned with curl. So my guess the issue isn't with the Postgres container directly, psycopg2, or the host-name - but something with the port?
Edit 3:
As a last attempt to fix this project, the full project this post is referring to is posted at this link. If anyone would like to download the repo and try building the docker containers themselves via ./start.sh - that might be just what is needed to find a solution!
I thought I had Docker setup on my machine, which runs Fedora 32. However as I came to realize from this article, setting up Docker on Fedora 32 requires some extra steps I was not previously aware of.
Specifically for this issue, the command listed in the article to add Docker to whitelist Docker on the local network's firewall with the command
sudo firewall-cmd --permanent --zone=FedoraWorkstation --add-masquerade
So I believe the root cause of my issue was simply my app container being blocked from accessing the postgres container by the firewall. Making the above change made the program work finally!
I want to be able to run a simple bash script within a container service on the hour using cron. I'm using Alpine Linux via docker-compose with a custom Dockerfile to produce a php-fpm based image, on which I hope to get crond running as well - except I can't.
Executing ps aux | grep cron on the container once built, returns nothing.
From what I understand, the usual Linux startup processes don't exist in Docker containers - fine - so how do I auto-start crond? Its dirs under /etc/periodic/ are created automatically, so I don't understand why the applicable process that consumes those dirs, isn't also running.
I tried creating a dedicated service definition within docker-compose.yml, which actually worked but the shell script to be run hourly needs access to a php binary which is running in a different container, so this isn't a viable solution.
If I shell into the container and run rc-service crond start I get this - but it never "finishes":
/var/www/html # rc-service crond start
* WARNING: crond is already starting
#> docker --version
Docker version 19.03.8, build afacb8b7f0
#> docker-compose --version
docker-compose version 1.23.2, build 1110ad01
I need a solution that I can place into my Dockerfile or docker-compose.yml files.
Dockerd is running on Ubuntu Xenial FWIW.
to run a cronjob container (Alpine), you need to make sure sure that the command of your docker container is
exec crond -f
if you want to add this to a docker file
CMD ["exec", "crond", "-f"]
you also may need to update the corn files before running the above command
Update based on the docker file and compose
To be able to solve your issues you need to update your docker-compose to have two containers one for cron and one for web
service_php_cron:
build:
context: .
dockerfile: .docker/services/php/Dockerfile.dev
container_name: base_service_php
command: 'cron_jobs'
volumes:
- ./app:/var/www/html/public
env_file:
- ./.env
# Low level container logging
logging:
driver: "json-file"
options:
max-size: "1m"
max-file: "5"
service_php:
build:
context: .
dockerfile: .docker/services/php/Dockerfile.dev
ports:
- "9000:9000"
command: 'web_server'
container_name: base_service_php
volumes:
- ./app:/var/www/html/public
env_file:
- ./.env
# Low level container logging
logging:
driver: "json-file"
options:
max-size: "1m"
max-file: "5"
you also need to update your docker file to be able to handle multiple commands using docker entry points
Add the below line to your docker file + remove the CMD one
COPY ./docker-entrypoint.sh /
RUN chmod a+x /docker-entrypoint.sh
ENTRYPOINT ["/docker-entrypoint.sh"]
and finally, create the entry point (make sure it hash execute permissions)
#!/bin/sh -e
case $1 in
web_server)
YOUR WEB SERVER COMMAND
;;
cron_jobs)
exec crond -f
;;
*)
exec "$#"
;;
esac
exit 0
you can check this link for more info about entrypoints
I have built a RESTful API web service using Flask framework, Redis as main database, MongoDB as a backup store and Celery as task queue to store data into MongoDB in background
Then I dockerize my application using docker-compose. Here is my docker-compose.yml:
version: '3'
services:
web:
build: .
ports:
- "5000:5000"
volumes:
- .:/app
redis:
image: "redis:alpine"
ports:
- "6379:6379"
mongo:
image: "mongo:3.6.5"
ports:
- "27017:27017"
environment:
MONGO_INITDB_DATABASE: syncapp
Here is my Dockerfile:
# base image
FROM python:3.5-alpine
MAINTAINER xhoix <145giakhang#gmail.com>
# copy just the requirements.txt first to leverage Docker cache
# install all dependencies for Python app
COPY ./requirements.txt /app/requirements.txt
WORKDIR /app
# install dependencies in requirements.txt
RUN pip install -r requirements.txt
# copy all content to work directory /app
COPY . /app
# specify the port number the container should expose
EXPOSE 5000
# run the application
CMD ["python", "/app/app.py"]
After run command docker-compose up, the app server, Redis and Mongo server just run well. But when I use Postman or curl to call the API, for example http://127.0.0.1:5000/sync/api/v1.0/users, which should return JSON format of all users, but the result is Could not get any response: There was an error connecting to http://127.0.0.1:5000/sync/api/v1.0/users.
I have no idea why this happens.
Thanks for any help and suggestion!
I found the cause of the issue:
After an hour debug, it turns out that I only need to change the app host to 0.0.0.0. Maybe when mapping port, docker default will be 0.0.0.0, since when I run command docker-compose ps, the PORTS column of each container has format 0.0.0.0:<port> -> <port>. I don't know this is the cause of the issue or not, but I did it and the problem is solved
If operating system Linux then use :
ifconfig -a
If operating system Windows then use :
ipconfig /all
Then check the interface like docker or something with virtualization, and use the ipv4 or inet
Or Just use the docker command:
docker network inspect bridge
Then use the gateway ip on IPAM
I have this (custom) container which runs a java program which requires mongo locally. Now, with docker I would like to setup mongo in its own container. So I guess, in order to expose this 27017 port locally in this java-container I need to setup an SSH-tunnel, right ? If there is a easier way please let me know.
So, there is this official mongo image image, but I get the impression ssh is not install or running. What would be the best approach to do this?
UPDATE: I've rephrased the question more focussed on port-forwarding here
You have to make your container run on the same network. No need to ssh into your mongo or app container.
https://docs.docker.com/engine/userguide/networking/
First define a network
docker network create --driver bridge isolated_nw
Start you containers using that newly network
docker run -p 27017:27017 --network=isolated_nw -itd --name=mongo-cont mongo
docker run --network=isolated_nw -itd --name=app your_image
The image of mongo includes EXPOSE 27017 so from your app container, you should be able to access to the mongo container using its name mongo-cont
You can build your custom image on top of mongodb official image, which gives you the flexibility to install additional required packages.
FROM mongo:latest
RUN apt-get install ssh
Also try to use docker-compose to build and link your containers together, it will ease the process greatly.
version: '2'
services:
mongo:
image: mongo:latest
ports:
- "27017"
custom_project:
build:
context: . # Parent directory address of Dockerfile
dockerfile: Dockerfile-Custom # Name of Dockerfile
command: /root/docker-entrypoint.sh
This is the image used for mongodb official image.
You are trying to SSH into your container to gain access to it, but that isn't how you connect. Docker provides functionality to securely connect via the following methods.
Connect into a running container - Docs:
docker exec -it <container name> bash
$ root#665b4a1e17b6:/#
Start a container from image, and connect to it - Docs:
docker run -it <image name> bash
$ root#665b4a1e17b6:/#
Note: If it is an Alpine based image, it may not have Bash installed. In that case using sh instead of bash in your commands should work. Mongo's Dockerfile looks to use debian:jessie which will have bash support.