Preventing the Docker container from exiting when the main process dies - postgresql

I am using Postgres with repmgr, one of the small problems I am having is that sometimes repmgr will have to stop and start the Postgres service and that will just kill the container, I tried some of the solutions online in the Dokcerfile but none seems to work, is there something I can add in the docker-compose file to prevent docker from exiting immediately, I don't want to stay alive forever, but maybe couple minutes?

Remember that docker-composer is mostly development thing. For production there are other ways, like kubernetes.
The only solution i know is to run our own .sh script as main process, which would have infinite loop with necessary checks in it.
This way you can control how to check - like ps aux and grep what you need. and exit main process if you need to by doing logic.
sh scrip would look something like:
while sleep 180; do
ps aux |grep postgres_service_name |grep -v grep
POSTGRESS=$?
if [ $POSTGRESS -ne 0 ]; then
# do what you need before exiting whole container
exit 1
fi
done
make sure you replace postgres_service_name with real name of Postgres service on linux.
Use that script as a startup script in docker compose or whatever you would use in prod.
if you really need 2 minutes before it is off - i would implement logic which would measure time after first time process is not there

The way docker is designed it will start a new container by starting the command specified as entrypoint/command to it and when this process terminates docker will kill all remaining processes in that container and shut it down.
So to keep the container running while the Postgres process is restarted you need to have another command running as the root process in the container.
You can achieve this by writing a simple shell script as a wrapper which will only exit when no Postgres process is running anymore or by using a dedicated init tool such as supervisord.

Related

postgresql docker ctrl C seems to be better than docker kill

I am productionalizing using docker postgresql on a very large project. Sometimes, I would need to restart postgresql manually. I tried 3 approaches to shut it down and then restarted later.
The 1st approach is 'I go to the screen container postgresql (I used screens to manage my orchestration). and press Ctrl-C multiple times to shut it down' This approach seems the best. Restarting seems to be smooth as well. The shutdown usually completes 1 minute or two but I have to be there manually.
2nd approach is
DOCKER_CONTAINER_NAME="timescaledb"
docker stop $DOCKER_CONTAINER_NAME
However, it seems it never completes.
3rd approach
docker kill $DOCKER_CONTAINER_NAME
However, the restart seems to be pretty long with a large recovery process....
What's the best I can do, mimicking method 1 where I keep pressing Ctrl-C to terminate it? While I could smoothly restart it later?
This solved my problem! Thanks!
docker stop -t 120 $DOCKER_CONTAINER_NAME
docker kill $DOCKER_CONTAINER_NAME
screen -S i2 -X stuff 'docker run -ti --user 1000:1000 -p 5432:5432 --name timescaledb --volume=/home/ubuntu/pgdata3:/home/postgresql/pgdata --rm -e POSTGRES_PASSWORD=sahisahikqhwwkejkqwjehjhwqjh -e PGDATA=/home/postgresql/pgdata timescale/timescaledb-ha:pg13-latest;\n'

Docker container stops right after starting Postgres DB

I am working on setting up Postgres 9.5 AS in Docker, and got everything installed. The issue however is, when I start the Docker Container, it appears that Postgres starts at first, but then the Container stops right away. (it does not show up with a docker container ls.). When I overwrite the Container startup with --entrypoint sh, and manually start Postgres, it all works fine.
I also checked with docker logs <container-id>, but that does not give me any info at all.
The setup is like this :
Dockerfile :
ENTRYPOINT ["/opt/edb/9.5AS/bin/init.sh"]
init.sh :
su enterprisedb -c '/opt/edb/9.5AS/bin/pg_ctl start -D /opt/edb/9.5AS/data'
From my command prompt I run :
docker run -it -v pgdata:/opt/edb/9.5AS/data <image_name>
It almost looks like it does start, but as soon as the start process is done, the shell stops, and as a result the Container stops as well.
So how to get it so the Container starts, Postgres starts and everything stays running, preferable in detached mode of course?
After researching some more, I found the answer in part also by finding clues on Stackoverflow.
Anyway, I modified my init.sh script to look like this :
/opt/edb/9.5AS/bin/pg_ctl start -D /opt/edb/9.5AS/data exec
"$#"
And the Dockerfile now ends like below :
USER enterprisedb
ENTRYPOINT ["/opt/edb/9.5AS/bin/init.sh"]
CMD ["/bin/bash"]
The core of the solution is the last line in the init.sh script as well as the last line in the Dockerfile. Both combined make it so that once the DB started, a new shell (/bin/bash) gets started. This will run in the foreground, thus keeping the Container alive. By starting the container in detached mode, it now does exactly what we need it to do.

Restart Postgres in a docker

I tried to restart Postgres in Docker using 'docker restart ' command. It got stopped but I'm not able to start it. When I run the command 'ps -a' it says the status as 'Exited'. Is there any way to start it again? I don't want to loose any data in that database.
The container had one active connection during restarting. Is that creating a problem?
If the container crashed due to a bug or something, you may not be able to restart it. However, you should still be able to recover at least part of your data by making a new image out of the container that you want to recover. Here's how you do it:
First, list all the containers that have run in your machine:
docker ps -a
Find out which one is the container that run with all the data you want to recover. You should be able to figure out from the CREATED field (you know when you started it).
Grab the hash (CONTAINER_ID) of the container, and execute the following command:
docker commit <hash> <a_new_name:tag>
This will save the container as an image that you can execute.
Execute the container with a bash or sh session, depending on what our base image offers:
docker run --entrypoint sh/bash -it <a_new_name:tag>
This will give you access to the state of the container at the time of exiting, which will allow you to inspect its conditions, find bugs, and possibly recover some data. Good luck!

Docker Compose - one specific container randomly doesn't start properly

I have a docker environment with 5 containers that are composed via docker compose. Now only on mac machines and only sometimes (seems completely random) 1 of these 5 container doesn't start.
The weird thing about it is, that docker ps says the container is running and I can connect to it. Inside the container is a JBoss server and ps says that there is a process that runs the JBoss. BUT in fact the JBoss is not up and running. There is no logging in the docker compose console and JBoss not accessible.
There is also the problem that if this happens the whole docker-compose process cannot be canceled properly anymore. All containers shutdown and also can be forced to shutdown but the JBoss container. Then the docker-machine hangs up.
I didn't find any hint in the interwebs ... please help !
It seems that the process running inside the container is in a weird state.
Try killing it without providing a grace period, or removing the container.
stop : Stop a container by sending SIGTERM and then SIGKILL after a
grace period
--help=false Print usage
-t, --time=10 Seconds to wait for stop before killing it
kill : Kill a running container using SIGKILL or a specified signal
--help=false Print usage
-s, --signal="KILL" Signal to send to the container
rm : Remove one or more containers
-f, --force=false Force the removal of a running container (uses SIGKILL)
--help=false Print usage
-l, --link=false Remove the specified link
-v, --volumes=false Remove the volumes associated with the container
Moreover try checking the logs of the container:
docker logs --follow <container_name or container_id>
After updating to docker v1.10 the problem didn't occur anymore :)

strace to monitor Dockerized application activity

My goal is to monitor which ports are opened and closed by a multi-process application.
My plan is to run the application in a Docker container, in order to isolate it, and then use strace to report the application activity.
I've tried with Apache server dockerized :
strace -f -o /tmp/docker.out docker run -D -P apache
I don't see any line in the report file that shows that the application accept a connection in a socket.
Can strace report the activity of processes inside the container?
The issue with your command+strace combination is that docker has a client/server model, and your docker run represents the client side of a REST API transaction to ask the docker daemon to run the Apache container on your behalf. Depending on how your client is configured, that container may not even run on the same system on which you type your docker run command.
However, to take the simplest case where the Docker client and daemon are on the same system, you can use ps find the PID of the running Apache server and use strace to join and trace the already-started process, as long as that is sufficient for your tracing needs.
Given I had to debug several early-start issues with "runc", the executor for containers in docker version 1.11 and above, I also created a small wrapper for docker-runc which strace's the container process from the start (from the outside system, so strace is not required in the container filesystem). You can find it here on GitHub, although fair warning that it is somewhat buggy for regular use as I believe the shell+strace invocation gets in the way of some signaling between containerd and the real docker-runc and associated processes. A more elegant solution might be to create a variant of runc which knows how to prepend the actual start of the contained process with an strace wrapper rather than intercepting the entire invocation of runc in an strace.
Take a look at the solution described at https://medium.com/#rothgar/how-to-debug-a-running-docker-container-from-a-separate-container-983f11740dc6 which tells you how to fire up a container with strace installed, which is in the same pid and network namespace as the container/process you wish to run strace against.
This is nice since it means you don't need to install strace in the container you wish to debug.
The guts of it is that when debugging a container (caddy in the example below) you run a docker container called strace and with appropriate tools installed:
docker run -t --pid=container:caddy \
--net=container:caddy \
--cap-add sys_admin \
--cap-add sys_ptrace \
strace
Assuming you make it so when building your strace container, you'll now have a shell with appropriate tools, from which you can run ps and see the process in the caddy container, and you can run strace against it.
You will be in a different container, with a different file system, but you can see the file-space of the target container at /prof/$PID/root.
I just managed to strace a docker container using these steps:
Work out what distro image the container is based on, then obtain the strace binary from that distro, e.g. by installing the corresponding distro package from a container created for that purpose.
Copy the strace binary into a volume you can mount into the container.
Also create a small wrapper shell script called entry.sh which contains your strace invocation. In my case I wrote it like this:
#!/bin/sh
exec /path/to/strace -ff -o /path/to/dumps /bin/bash /original/entrypoint
This is assuming that the original entry point, which you read from the Dockerfile of the image you want to debug, started with #!/bin/bash. Make sure you set the execute bit of this script, and place it where you also placed the strace binary.
Launch docker using a command like this:
docker run -v $PWD/shared:/path/to \
--entrypoint="/path/to/entry.sh" \
--cap-add SYS_PTRACE \
image-name
The mounted volume will make the strace and entry.sh available inside the docker. The entry point will do the strace invocation before calling the actual entry point. This might potentially cause some trouble with strace itself becoming pid 1 in the container, and failing to reap children. If that's a problem, a different approach like the one Phil suggested would be better. Finally that added capability tells docker that it's OK to start tracing processes. Otherwise you'd get error messages like
strace: …: PTRACE_TRACEME doesn't work: Operation not permitted
Actually pointing out this capability setting is the reason I'm writing my answer. I had already done the steps except for this flag, and while searching for a solution there I found both this question here and a blog post by John Goulah containing that bit of information. For the sake of completeness I think the flag should be mentioned here, too. Haven't tried Phil's approach yet, so I definitely don't claim my approach to be superior to what he suggested. I guess it might work more easily on systems where you don't want to mess with the docker daemon.
we can add param
--security-opt=seccomp:unconfined
I have try it ,it work well!!
docker run -it --security-opt=seccomp:unconfined centos:7 /bin/bash
yum install strace
strace ls
execve("/usr/bin/ls", ["ls"], [/* 8 vars */]) = 0
brk(NULL) = 0x1d0a000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7ffb588da000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or
directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
refer:http://johntellsall.blogspot.com/2016/10/tip-use-strace-to-debug-issues-inside.html
try to launch Apache docker run -D -P apache and connect inside docker exec -it container_id bash and then strace your Apache process.