Docker disk space issue with space left on host - postgresql

I have PostgreSQL running in a Docker container (Docker 17.09.0-ce-mac35 on OS X 10.11.6) and I'm inserting data from a Python application on the host. After a while I consistently get the following error in Python while there is still plenty of disk space available on the host:
psycopg2.OperationalError: could not extend file "base/16385/24599.49": wrote only 4096 of 8192 bytes at block 6543502
HINT: Check free disk space.
This is my docker-compose.yml:
version: "2"
services:
rabbitmq:
container_name: rabbitmq
build: ../messaging/
ports:
- "4369:4369"
- "5672:5672"
- "25672:25672"
- "15672:15672"
- "5671:5671"
database:
container_name: database
build: ../database/
ports:
- "5432:5432"
The database Dockerfile looks like this:
FROM ubuntu:17.04
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ zesty-pgdg main" > /etc/apt/sources.list.d/pgdg.list
RUN apt-get update && apt-get install -y --allow-unauthenticated python-software-properties software-properties-common postgresql-10 postgresql-client-10 postgresql-contrib-10
USER postgres
RUN /etc/init.d/postgresql start &&\
psql --command "CREATE USER ****** WITH SUPERUSER PASSWORD '******';" &&\
createdb -O ****** ******
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/10/main/pg_hba.conf
RUN echo "listen_addresses='*'" >> /etc/postgresql/10/main/postgresql.conf
EXPOSE 5432
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
CMD ["/usr/lib/postgresql/10/bin/postgres", "-D", "/var/lib/postgresql/10/main", "-c", "config_file=/etc/postgresql/10/main/postgresql.conf"]
df -k output:
Filesystem 1024-blocks Used Available Capacity iused ifree %iused Mounted on
/dev/disk2 1088358016 414085004 674017012 39% 103585249 168504253 38% /
devfs 190 190 0 100% 658 0 100% /dev
map -hosts 0 0 0 100% 0 0 100% /net
map auto_home 0 0 0 100% 0 0 100% /home
Update 1:
It seems like the container has now shut down. I'll start over and try to df -k in the container before it shuts down.
2017-11-14 14:48:25.117 UTC [18] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-11-14 14:48:25.120 UTC [17] WARNING: terminating connection because of crash of another server process
2017-11-14 14:48:25.120 UTC [17] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2017-11-14 14:48:25.120 UTC [17] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2017-11-14 14:48:25.132 UTC [1] LOG: all server processes terminated; reinitializing
2017-11-14 14:48:25.175 UTC [1] FATAL: could not access status of transaction 0
2017-11-14 14:48:25.175 UTC [1] DETAIL: Could not write to file "pg_notify/0000" at offset 0: No space left on device.
2017-11-14 14:48:25.181 UTC [1] LOG: database system is shut down
Update 2:
This is df -k on the container, /dev/vda2 seems to be filling up quickly:
$ docker exec -it database df -k
Filesystem 1K-blocks Used Available Use% Mounted on
none 61890340 15022448 43700968 26% /
tmpfs 65536 0 65536 0% /dev
tmpfs 1023516 0 1023516 0% /sys/fs/cgroup
/dev/vda2 61890340 15022448 43700968 26% /etc/postgresql
shm 65536 8 65528 1% /dev/shm
tmpfs 1023516 0 1023516 0% /sys/firmware
Update 3:
This seems to be related to the ~64 GB file size limit on Docker.qcow2. Solved using qemu and gparted as follows:
cd ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/
qemu-img info Docker.qcow2
qemu-img resize Docker.qcow2 +200G
qemu-img info Docker.qcow2
qemu-system-x86_64 -drive file=Docker.qcow2 -m 512 -cdrom ~/Downloads/gparted-live-0.30.0-1-i686.iso -boot d -device usb-mouse -usb

Related

PostgrSQL custom Docker Container

I am trying to write a docker entrypoint script, that’s able to start postgreSQL and creating a new named database.
I feel like there is a much better way, than mine. I am happy for every input.
Issues:
a few times I have to restart the container befor it creates the databas and does not crash. Some times I don't know why it workes. I can't recreate it right now.
Freequently I get messages like [unknown]#[unknown] LOG: incomplete startup packet but there shouldn't be any.
Reason for custom docker image: I need PostgreSQL to use ssl for every connection. The official image is sadly not able to allow ssl with real certificates.
Dockerfile:
FROM debian:10.9
RUN apt update && apt install -y \
postgresql postgresql-contrib \
openssl \
net-tools netcat \
vim
RUN mkdir database_stuff
COPY --chown=postgres:postgres ./entrypoint.sh /database_stuff/entrypoint.sh
RUN chmod 777 /database_stuff/entrypoint.sh
COPY --chown=postgres:postgres ./certificates/cert.pem /database_stuff/cert.pem
RUN chmod 600 /database_stuff/cert.pem
COPY --chown=postgres:postgres ./certificates/key.pem /database_stuff/key.pem
RUN chmod 600 /database_stuff/key.pem
COPY --chown=postgres:postgres ./configuration/postgresql.conf /etc/postgresql/11/main/postgresql.conf
COPY --chown=postgres:postgres ./configuration/pg_hba.conf /etc/postgresql/11/main/pg_hba.conf
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
USER postgres
CMD ["/usr/lib/postgresql/11/bin/postgres", "-D", "/var/lib/postgresql/11/main/", "-c", "config_file=/etc/postgresql/11/main/postgresql.conf"]
ENTRYPOINT ["/database_stuff/entrypoint.sh"]
entrypoint.sh script
#!/usr/bin/env bash
echo "START entrypoint.sh"
echo "start postgrSQL"
/usr/lib/postgresql/11/bin/postgres -D /var/lib/postgresql/11/main/ -c config_file=/etc/postgresql/11/main/postgresql.conf &
until nc -z db 5432; do
sleep 0.1
done
echo "create Database"
psql -U postgres -c 'create database part_screening;'
psql -U postgres -c "create user alstom with encrypted password 'Laser765_f';"
echo "END entrypoint.sh .... Stay open"
tail -f /dev/null
just a log with the unknown#unknown issue
START entrypoint.sh
start postgrSQL
create Database
CREATE DATABASE
CREATE ROLE
END entrypoint.sh .... Stay open
2021-08-02 17:08:52.280 UTC [7] LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-02 17:08:52.280 UTC [7] LOG: listening on IPv6 address "::", port 5432
2021-08-02 17:08:52.284 UTC [7] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-08-02 17:08:52.300 UTC [10] LOG: database system was shut down at 2021-08-02 17:06:42 UTC
2021-08-02 17:08:52.305 UTC [7] LOG: database system is ready to accept connections
2021-08-02 17:08:52.330 UTC [18] [unknown]#[unknown] LOG: incomplete startup packet
2021-08-02 17:08:53.145 UTC [38] [unknown]#[unknown] LOG: incomplete startup packet

Postgresql 10 pg_basebackup not backing up pg_control file

I'm new to administrating postgresql 10 database so please bear with me (I'm from Oracle background). I was hoping a kind soul could point me to the right direction with our current problem.
We use pg_base to backup our current production system and for some reason since a couple of days ago pg_base has stopped backing up the database's pg_control file as well as postgres.conf which ofcourse renders the backup useless for restoration.
Below is the command that we use to initiate the backup
/usr/bin/docker exec -i --user postgres asf-db1 pg_basebackup --wal-method=fetch -D /var/lib/postgresql/data/pgdata/master_backup/latest -P -v --format=tar
The logs does not indicate anything suspicious and the issue only became apparent when I did a test restore.
Below is an extract of the backup logs
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 6/BF000178 on timeline 1
0/1519047 kB (0%), 0/1 tablespace (...ta/master_backup/latest/base.tar)
397408/1519047 kB (26%), 0/1 tablespace (...ta/master_backup/latest/base.tar
)^M1019400/1519047 kB (67%), 0/1 tablespace (...ta/master_backup/latest/base.
tar)^M1565927/1565927 kB (100%), 0/1 tablespace (...ta/master_backup/latest/b
t/base.tar)^M2164615/2164615 kB (100%), 0/1 tablespace (...ta/master_backup/l
p/latest/base.tar)^M2751015/2751015 kB (100%), 0/1 tablespace (...ta/master_b
r_backup/latest/base.tar)^M3061585/3061585 kB (100%), 0/1 tablespace (...ta/m
pg_basebackup: write-ahead log end point: 6/C0000050
pg_basebackup: base backup completed
Backup completed at: Wed 14 Nov 18:00:32 GMT 2018

Resizing disk on google cloud kubernetes

Hi im trying to resize a disk for a pod in my kubernetes cluster,following the steps on the docs, i ssh in to the instance node to follow the steps, but its giving me an error
sudo growpart /dev/sdb 1
WARN: unknown label
failed [sfd_dump:1] sfdisk --unit=S --dump /dev/sdb
/dev/sdb: device contains a valid 'ext4' signature; it is strongly recommended to wipe the device with wipefs(8)
if this is unexpected, in order to avoid possible collisions
sfdisk: failed to dump partition table: Success
FAILED: failed to dump sfdisk info for /dev/sdb
i try running the commands from inside the pod but doesnt even locate the disk even tho its there
root#rc-test-r2cfg:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 59G 2.5G 56G 5% /
/dev/sdb 49G 22G 25G 47% /var/lib/postgresql/data
root#rc-test-r2cfg:/# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 96G 0 disk /var/lib/postgresql/data
sda 8:0 0 60G 0 disk
└─sda1 8:1 0 60G 0 part /etc/hosts
root#rc-test-r2cfg:/# growpart /dev/sdb 1
FAILED: /dev/sdb: does not exist
where /dev/sdb is the disk location
This can now be easily done by updating the storage specification directly of the Persistent Volume Claim. See these posts for reference:
https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/
https://dev.to/bzon/resizing-persistent-volumes-in-kubernetes-like-magic-4f96 (GKE example)

Ran out of Docker disk space

I have this Docker command:
docker run -d mongo
this will build and run a mongodb server running in a docker container
However, I get an error:
no space left on device
I am on MacOS, and using the newer versions of Docker which use hyper-v instead of VirtualBox (I think that's correct).
Here is the exact error message from the mongo container:
$ docker logs efee16702c5756659d563b98d4ae0f58ecf1f1bba8a54f63443c0ae4b520ab4e
about to fork child process, waiting until server is ready for connections.
forked process: 21
2017-05-04T20:23:51.412+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2017-05-04T20:23:51.430+0000 I CONTROL [main] ERROR: Cannot write pid file to /tmp/tmp.Lo035QkbfL: No space left on device
ERROR: child process failed, exited with error number 1
Any idea how to fix this and prevent it from happening in future?
As suggested, the output of df -h is:
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk1 465Gi 116Gi 349Gi 25% 1963838 4293003441 0% /
devfs 183Ki 183Ki 0Bi 100% 634 0 100% /dev
map -hosts 0Bi 0Bi 0Bi 100% 0 0 100% /net
map auto_home 0Bi 0Bi 0Bi 100% 0 0 100% /home
Output of docker info is:
$ docker info
Containers: 5
Running: 0
Paused: 0
Stopped: 5
Images: 741
Server Version: 17.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.13-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952 GiB
Name: moby
ID: OR4L:WYWW:FFAP:IDX3:B6UK:O2AN:UVTO:EPH6:GYSV:4GV4:L5WP:BQTH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 17
Goroutines: 30
System Time: 2017-05-04T20:45:27.056157913Z
EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
As you state in the comments to the question, ls -altrh ~/Library/Containers/com.docker.docker/Data/com.docker.driver.‌​amd64-linux/Docker.q‌​cow2 returns the following:
-rw-r--r--# 1 alexamil staff 53G
This is a known bug on MacOS (actually, not only) and an official dev comment could be found here. Except for one thing: I read, that different people get different size limit. In the comment it is 64Gb, but for another person it was 20Gb.
There are a couple walkarounds, but no definite solution that I could find.
The manual one
Run docker ps -a and manually remove all unused containers. Then run docker images and remove manually all the intermediate and unused images.
The simplest one
Delete the Docker.qcow2 file entirely. But you will lose all images and containers. Completely.
The less simple
Another way is to run docker volume prune, which will remove all unused volumes
The resizing one (keeps the data)
Another idea that comes to me is to expand the disk image size with QEMU or something like it:
$ brew install qemu
$ /Applications/Docker.app/Contents/MacOS/qemu-img resize ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 +5G
After you expanded the image, you will need to run a VM in which you should run GParted against Docker.qcow2 and expand the partition to use added space. You could use GParted Live ISO for that:
$ qemu-system-x86_64 -drive file=Docker.qcow2 -m 512 -cdrom ~/Downloads/gparted-live.iso -boot d -device usb-mouse -usb
Some people report this either doesn't work or doesn't help.
Yet another resizing one (wipes the data)
Create a substitute image with desired size (120G):
$ qemu-img create -f qcow2 ~/data.qcow2 120G
$ cp ~/data.qcow2 /Application/Docker.app/Contents/Resources/moby/data.qcow2
$ rm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
data.qcow2 is copied to ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 when you restart docker.
This walkaround comes from this comment.
Hope this helps. Good luck!

PostgreSQL 9.6 won't connect on Ubuntu 16

I installed PostgreSQL 9.6 on my Ubuntu 16.04 system. It seems like it's up and running, but when I try to make a connection it just fails. When I run the status command I see that it's always saying exited for some reason. I see lots of things in a google search but none seem to be helping.
I changed the first entry in the pg_hba.conf to be local all postgres trust
% ps augxw | grep postg
postgres 769 0.0 0.3 303964 24384 ? S 22:13 0:00 /usr/lib/postgresql/9.6/bin/postgres -D /var/lib/postgresql/9.6/main -c config_file=/etc/postgresql/9.6/main/postgresql.conf
postgres 772 0.0 0.0 303964 3956 ? Ss 22:13 0:00 postgres: 9.6/main: checkpointer process
postgres 773 0.0 0.0 303964 3956 ? Ss 22:13 0:00 postgres: 9.6/main: writer process
postgres 774 0.0 0.0 303964 3956 ? Ss 22:13 0:00 postgres: 9.6/main: wal writer process
postgres 775 0.0 0.0 304408 6572 ? Ss 22:13 0:00 postgres: 9.6/main: autovacuum launcher process
postgres 776 0.0 0.0 158964 3204 ? Ss 22:13 0:00 postgres: 9.6/main: stats collector process
ubuntu 1492 0.0 0.0 12944 936 pts/0 S+ 22:14 0:00 grep postg
%
% sudo su - postgres
$ psql -h localhost
psql: could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
$ exit
logout
% systemctl status postgresql
● postgresql.service - PostgreSQL RDBMS
Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
Active: active (exited) since Thu 2017-03-30 22:13:57 PDT; 1min 19s ago
Process: 901 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 901 (code=exited, status=0/SUCCESS)
Tasks: 0
Memory: 0B
CPU: 0
CGroup: /system.slice/postgresql.service
Mar 30 22:13:57 ip-172-31-9-223 systemd[1]: Starting PostgreSQL RDBMS...
Mar 30 22:13:57 ip-172-31-9-223 systemd[1]: Started PostgreSQL RDBMS.
You are connecting using TCP/IP socket, but local refers to unix-domain socket.
The first field is the connection type: "local" is a Unix-domain socket, "host" is either a plain or SSL-encrypted TCP/IP socket, "hostssl" is an SSL-encrypted TCP/IP socket, and "hostnossl" is a plain TCP/IP socket.
I did install postgresql-9.6 from that repo and:
host all postgres 127.0.0.1/32 trust does what you want to do
postgres#lkaminski-ubuntu-desk:~$ psql works out of the box, no need to change config. It is using unix-domain socket that is already trusted. So you can just drop -h localhost and no need to edit configs.
$ sudo grep -e "^[^#]" /etc/postgresql/9.6/main/pg_hba.conf
local all postgres peer
host all postgres 127.0.0.1/32 trust
local all all peer
host all all 127.0.0.1/32 md5
host all all ::1/128 md5
The issue ended up being that version 9.5 had the normal 5432 port and so when 9.6 got installed/started, it pointed to 5433. Deleting the 9.5 install and editing the config file to point at 5432 fixed the issue.