I have a docker environment with 5 containers that are composed via docker compose. Now only on mac machines and only sometimes (seems completely random) 1 of these 5 container doesn't start.
The weird thing about it is, that docker ps says the container is running and I can connect to it. Inside the container is a JBoss server and ps says that there is a process that runs the JBoss. BUT in fact the JBoss is not up and running. There is no logging in the docker compose console and JBoss not accessible.
There is also the problem that if this happens the whole docker-compose process cannot be canceled properly anymore. All containers shutdown and also can be forced to shutdown but the JBoss container. Then the docker-machine hangs up.
I didn't find any hint in the interwebs ... please help !
It seems that the process running inside the container is in a weird state.
Try killing it without providing a grace period, or removing the container.
stop : Stop a container by sending SIGTERM and then SIGKILL after a
grace period
--help=false Print usage
-t, --time=10 Seconds to wait for stop before killing it
kill : Kill a running container using SIGKILL or a specified signal
--help=false Print usage
-s, --signal="KILL" Signal to send to the container
rm : Remove one or more containers
-f, --force=false Force the removal of a running container (uses SIGKILL)
--help=false Print usage
-l, --link=false Remove the specified link
-v, --volumes=false Remove the volumes associated with the container
Moreover try checking the logs of the container:
docker logs --follow <container_name or container_id>
After updating to docker v1.10 the problem didn't occur anymore :)
Related
I have been using this MUP config for the deployment until recently. When I encountered an issue and I had to stop, reboot the Instance multiple times.
Then, This causes the meteor app container to shut down and the MongoDB container is running just fine but wasn't accessible through an SSH tunnel on a MongoDB GUI (but running systemctl status mongo shows active status: activating.
I troubleshoot and run docker ps -a. It shows the MongoDB container only as a running container and the meteor app container completely shutdown.
I tried running the MUP deployment in an attempt to get the meteor app container up and running.
However, I got an error Removing docker containers. Errors about nonexistent endpoints and containers are normal.
I run the mup setup command successfully and then I tried running mup reconfig and I got the same above error, I have attached the screenshot of the error below.
To Reproduce this error
Create a meteor app with Iron-meteor.
Setup an Instance (Ec2).
Setup Deployment with Meteor-up
Deploy your app with Meteor-up.
SSH into the instance and run cmd docker ps. Should see at least two running containers, app and mongo respectively.
Run a cmd to stop the app container while the mongo container is running.
Finally, Goto your project and redeployed with mup
Should see a similar error as above. for step 6 restarting the instance in my case shut down the two containers and I was able to get the mongo container back up and running.
However, I couldn't get the app container running, so I tried redeploying with the expectation that a new app container would be created if it doesn't exist on the instance.
UPDATED!
I don't know if this will help, but in my experience, mup likes a fresh instance better than an existing one.
My first step would be a mup stop command. This will shut down the docker instances. Then you can remove them with docker rm, and you can remove the images with docker rmi. Then do a mup setup again, followed by a mup deploy.
If the first one doesn't work, you can basically start with a fresh vm, as in the droplet or ec2 instance. This is generally quite successful.
I am using Postgres with repmgr, one of the small problems I am having is that sometimes repmgr will have to stop and start the Postgres service and that will just kill the container, I tried some of the solutions online in the Dokcerfile but none seems to work, is there something I can add in the docker-compose file to prevent docker from exiting immediately, I don't want to stay alive forever, but maybe couple minutes?
Remember that docker-composer is mostly development thing. For production there are other ways, like kubernetes.
The only solution i know is to run our own .sh script as main process, which would have infinite loop with necessary checks in it.
This way you can control how to check - like ps aux and grep what you need. and exit main process if you need to by doing logic.
sh scrip would look something like:
while sleep 180; do
ps aux |grep postgres_service_name |grep -v grep
POSTGRESS=$?
if [ $POSTGRESS -ne 0 ]; then
# do what you need before exiting whole container
exit 1
fi
done
make sure you replace postgres_service_name with real name of Postgres service on linux.
Use that script as a startup script in docker compose or whatever you would use in prod.
if you really need 2 minutes before it is off - i would implement logic which would measure time after first time process is not there
The way docker is designed it will start a new container by starting the command specified as entrypoint/command to it and when this process terminates docker will kill all remaining processes in that container and shut it down.
So to keep the container running while the Postgres process is restarted you need to have another command running as the root process in the container.
You can achieve this by writing a simple shell script as a wrapper which will only exit when no Postgres process is running anymore or by using a dedicated init tool such as supervisord.
I cant find more information about those.
Should we use docker stop for containers which we started with docker start
Or same for docker-compose up?
What is the difference between stop and down?
In docker-compose help
stop Stop services
down Stop and remove containers and networks (optionally images and volumes as well)
# Stop services only
docker-compose stop
# Stop and remove containers, networks..
docker-compose down
# Down and remove volumes
docker-compose down --volumes
# Down and remove images
docker-compose down --rmi <all|local>
Following are the differences among various docker-compose command options:
docker-compose up - start and restart all the services defined in docker-compose.yml
docker-compose down - command will stop running containers, but it also removes the stopped containers as well as any networks that were created. You can take down one step further and add the -v flag to remove all volumes too. This is great for doing a full blown reset on your environment by running docker-compose down -v.
docker-compose start - command will only restart containers stopped previously
docker-compose stop - command will stop running containers but won’t remove them
Just to answer the other part of the question:
Use docker-compose up to start or restart all the services defined in a docker-compose.yml.
The docker-compose start command is useful only to restart containers that were previously created, but were stopped. It never creates new containers.
The docker-compose run command is for running “one-off” or “adhoc” tasks.
For further information visit this page.
I tried to restart Postgres in Docker using 'docker restart ' command. It got stopped but I'm not able to start it. When I run the command 'ps -a' it says the status as 'Exited'. Is there any way to start it again? I don't want to loose any data in that database.
The container had one active connection during restarting. Is that creating a problem?
If the container crashed due to a bug or something, you may not be able to restart it. However, you should still be able to recover at least part of your data by making a new image out of the container that you want to recover. Here's how you do it:
First, list all the containers that have run in your machine:
docker ps -a
Find out which one is the container that run with all the data you want to recover. You should be able to figure out from the CREATED field (you know when you started it).
Grab the hash (CONTAINER_ID) of the container, and execute the following command:
docker commit <hash> <a_new_name:tag>
This will save the container as an image that you can execute.
Execute the container with a bash or sh session, depending on what our base image offers:
docker run --entrypoint sh/bash -it <a_new_name:tag>
This will give you access to the state of the container at the time of exiting, which will allow you to inspect its conditions, find bugs, and possibly recover some data. Good luck!
My question aims at verifying and maybe rectifying my idea of the reliability of Docker containers. I read both, the Docker documentation and several articles on VOLUME in the Dockerfile and --v as an argument when running a container as means to persist data outside a Docker container. Be it in a data container or on the host system. As would like to keep the complexity of my setup simple, I would prefer not to copy/save/store data round and about but keep it in the Docker container itself.
There are several cases through which I discovered the behaviour of Docker containers. I'd like to know if I missed a scenario where a container can be 100% lost unpurposely, i.e. NOT doing $ docker rm -f mycontainer
docker commands to pause, stop and kill a container
-> restartable by $ docker restart mycontainer or $ docker run mycontainer
Host system reboot
-> docker container exits with 0 or 255
Host system unexpected power off
-> What happens?
Application exception
-> docker container exits with -1
Updating or restarting docker (as pointed out by Greg)
-> expected behavior: like on system reboot (?)
In all those cases, the docker container is still existent in the end. So is there any other scenario that can cause a docker container to be lost like with $ docker rm -f mycontainer?
The background is, that I read a lot about mounted volumes and external datastorage on the host system for Postgres but I'd like to avoid storing data outside my containers on the host system if possible. On the other hand, I don't want to wake up and have all data lost. (I do perform regular SQL-dumps, but I don't want to do this every 5 minutes). If a docker container itself is not reliable for persistant data, I don't see why I should create a second container to hold the data for a first one and increase the complexity of my system by adding a new container but not gaining anything in terms of reliability.
Edit: There are two points in the Docker userguide on Volumes which do not explicitly explain which behaviour to expect and therefore making me question if these concepts provide extra reliability:
Changes to a data volume will not be included when you update an
image
-> Does that mean that they get lost or that the content of the volume won't be changed?
Volumes persist until no containers use them
-> What's the definition of 'use'? As long as a container is not stopped, killed, removed? Does that mean that the volume Docker created on the host system will get removed? Or does volume only refer to a virtual bridge between a directory inside Docker and one on the host system?
If you store all your data in the container, what are you going to do when you need to update the image? Updates to images are normally done by changing the Dockerfile and rebuilding the image. If my data is kept separate to my container, I can start a new version of the image, mount the data with --volumes-from or -v and kill the old container. In your case, you have to keep the container running and try to patch in place with something like puppet.
Also, I'm not sure what you think you're saving. If you run the official postgres image, it will have declared volumes in the Dockerfile. Those volumes exist as normal directories on your host system whether you ran the container with -v or not. Even if your Dockerfile has no volumes, clearly the UFS is being stored on your host anyway.
In general, you should consider containers to be temporary and stateless. Whilst you don't have to do this, you will find most of the tooling and support services are designed around this idiom.
Regarding your scenarios, there are a few you're missing:
A bug could make it impossible to restart a stopped container
The updating issue mentioned above
If you want to change storage driver. This will cause a great deal of problems, as you need to migrate your images.
Just for clarity on the commands, docker start will restart stopped or exited containers and docker unpause will unpause paused containers.