Docker container can’t startup with a 166G(nfs/efs) volume - docker-compose

my docker container can’t startup with a 166G (nfs/efs) volume, I use the EFS mounted to the EC2 with amazon-efs-utils tool, and then volume to the docker, below is my docker-compose file
services:
app:
image: xxxx
restart: always
environment:
- ASPNETCORE_ENVIRONMENT=Production
ports:
- "8002:8002"
volumes:
- "./BinaryObjects/Static:/app/App_Data/BinaryObjects/Static"
- "./BinaryObjects/Temp:/app/App_Data/BinaryObjects/Temp"
the /BinaryObjects is the root which EFS mounted to, and /Static folder has the 166G files, and /Temp folder has 10M files, if I remove the /Static in volumes, and the docker container can startup. so I think it is not the permission issue. I did the same thing in our staging env, and it works well, the different between staging and production is the /Static folder in staging has only 2G files. I have tried that created the nfs volumes first,
docker volume create \
--driver local \
--opt type=nfs \
--opt o=addr=xxxx.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport \
--opt device=:/Static
efs-static
docker volume create \
--driver local \
--opt type=nfs \
--opt o=addr=xxxx.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport \
--opt device=:/Temp
efs-temp
and no luck, the docker container can't startup either, and works if I remove the efs-static volume, it is just stuck, and no error message, please refer to below screenshots.
Does someone know why? I want the docker can startup with my 166G efs volume. Thanks.
my docker info
Client:
Context: default
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 31
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version:
init version:
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-1059-aws
Operating System: Ubuntu 18.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.18GiB
Name: ip-172-31-50-32
ID: XWJL:SUKJ:DVZL:SVII:JFHJ:W7ND:LZJ5:2LKU:ASBA:ILEB:HF54:KLCV
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Related

File permission issue with mongo docker image

I'm trying to instanciate a mongodb docker image using the following commmand:
docker run -e MONGO_INITDB_ROOT_USERNAME=root -e MONGO_INITDB_ROOT_PASSWORD=password mongo
The command fails instantly because of a Persmission denied:
2019-11-12T20:16:29.503+0000 I CONTROL [main] ERROR: Cannot write pid file to /tmp/docker-entrypoint-temp-mongod.pid: Permission denied
The weird thing is the same command works fine on some other machines that has the same users, groups... The only thing that differs is the docker version.
I don't understand why the mongo instance does not run as I do not have any volumes or user specified on the command line.
Here is my docker info
Client:
Debug Mode: false
Server:
Containers: 29
Running: 1
Paused: 0
Stopped: 28
Images: 87
Server Version: 19.03.4
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0-11-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 25.52GiB
Name: jenkins-vm
ID: YIGQ:YOVJ:2Y7F:LM77:VHK6:ICMY:QDGA:5EFD:ZYDD:EQM5:DR77:DANT
Docker Root Dir: /data/var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
And as suggested by #jan-garaj, here is the result of docker run -e MONGO_INITDB_ROOT_USERNAME=root -e MONGO_INITDB_ROOT_PASSWORD=password mongo id: uid=0(root) gid=0(root) groups=0(root)
What could be the reason of this failure ?
You may have a problem with some security configuration. Check and compare docker info outputs. You may have enabled user namespaces (userns-remap), some special seccomp, selinux policies, weird storage driver, full disk space, ...
Could be because of the selinux policy :
edit config file at /etc/selinux/config :
SELINUX:disabled
reboot your system and try to run the image after.

Docker [for mac] file system became read-only which breaks almost all features of docker

My Docker ran into an error state, where I cannot use it anymore.
output of docker system info:
Containers: 14
Running: 2
Paused: 0
Stopped: 12
Images: 61
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: error
NodeID:
Error: open /var/lib/docker/swarm/worker/tasks.db: read-only file system
Is Manager: false
Node Address: 192.168.65.3
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.87-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952GiB
Name: linuxkit-025000000001
ID: MCSC:SFXH:R3JC:NU4D:OJ5V:K4B5:LPMJ:2BFL:LHT3:LYCI:XKY2:DTE6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
HTTP Proxy: docker.for.mac.http.internal:3128
HTTPS Proxy: docker.for.mac.http.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
This behaviour occured, after I built the following Dockerfile:
FROM perl:5.20
RUN apt-get update && apt-get install -y libsoap-lite-perl \
&& rm -rf /var/lib/apt/lists/*
RUN cpan SOAP::LITE
the error message when I try to build an image or run a container or remove an image is always similar to this:
Error: open /var/lib/docker/swarm/worker/tasks.db: read-only file system
for example if I try to execute this command:
docker container run -it perl:5.20 bash
I get this error:
docker: Error response from daemon: mkdir /var/lib/docker/overlay2/1b966e163e500a8c78a64e8d0f14984b091c1c5fe188a60b8bd030672d3138d9-init: read-only file system.
How can I reset my docker so these errors go away?
Go to your docker for mac Icon in the top right, click on it and then click Restart.
After that Docker works as expected.
This seems to be an temporary issue since I cannot reproduce it after restarting docker. My guess is that I had an network communication breakdown while docker tried to download and install the packages in the Dockerfile.

How to move Postresql to RAM disk in Docker?

I want to run the Docker image postgres:9, stop Postgres, move it to /dev/shm, and restart it, so I can run my application tests really fast.
But when I try to stop Postgres in the container using postgres or pg_ctl I get told cannot be run as root.
Since all Docker containers log you in as the root user what can I do to run the Postgres commands I need?
And which folders do I need to move to /dev/shm before restarting it?
Command to start the container if you want to try this:
docker run -it postgres:9 bash
cd /usr/lib/postgresql/9.6/bin
./pg_ctl stop
Mount a tmpfs in the container and point the PostgreSQL data at it
docker run --tmpfs=/pgtmpfs -e PGDATA=/pgtmpfs postgres:15
Use size=Nk to set a size limit (rather than all free memory).
--tmpfs /pgtmpfs:size=131072k
The same can be done for MySQL
docker run --tmpfs=/var/lib/mysql -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:8
Kubernetes
An emptyDir volume can set the medium property to Memory
apiVersion: v1
kind: Pod
metadata:
name: tmpfs-pd
spec:
containers:
- image: docker.io/postgres:15
name: tmpdb
env:
- name: PGDATA
value: /pgtmpfs
volumeMounts:
- mountPath: /pgtmpfs
name: tmpdata-volume
volumes:
- name: tmpdata-volume
emptyDir:
medium: Memory
sizeLimit: 131072k
Docker Compose
And in a docker compose 3.6+ definition (not supported by stack)
version: "3.6"
services:
db:
image: docker.io/postgres:15
environment:
- PGDATA=/pgtmpfs
tmpfs:
- /pgtmpfs
Compose can define shared volumes of tmpfs as well.
By adding --user=postgres
to your docker parameter list, you'll get to be user=postgres directly:
docker --user=postgres run -it postgres:9 bash
cd /usr/lib/postgresql/9.6/bin
./pg_ctl stop

The rbd volumes config of rbd issue in kubernetes

I want to use the volumes rbd config to mount the folder on ceph images .
But it seems the container mount a host path.
I used the daemon of "https://github.com/kubernetes/kubernetes/tree/master/examples/rbd".
The pod and container start successfully.
I use the "docker exec " login the container and watch the /mnt folder.
root#test-rbd-read-01:/usr/local/tomcat# findmnt /mnt
TARGET SOURCE FSTYPE OPTIONS
/mnt /dev/vda1[/var/lib/kubelet/pods/****/volumes/kubernetes.io~rbd/rbd] xfs rw,relatime,attr2,inode64,noquota
root#test-rbd-read-01:/usr/local/tomcat# ls /mnt/
root#test-rbd-read-01:/usr/local/tomcat#
And then I watch the host path that mount on the ceph. The 1.txt had existed on ceph image.
[20:52 root#mongodb:/home] # mount |grep kubelet
/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/wujianlin-image-zlh_test type ext4 (ro,relatime,stripe=1024,data=ordered)
/dev/rbd0 on /var/lib/kubelet/pods/****/volumes/kubernetes.io~rbd/rbd type ext4 (ro,relatime,stripe=1024,data=ordered)
[20:53 root#mongodb:/home] # ll /var/lib/kubelet/pods/****/volumes/kubernetes.io~rbd/rbd
total 20K
drwx------ 2 root root 16K Mar 18 09:49 lost+found
-rw-r--r-- 1 root root 4 Mar 18 09:53 1.txt
[20:53 root#mongodb:/home] # rbd showmapped
id pool image snap device
0 wujianlin zlh_test - /dev/rbd0
It should except that the container folder /mnt is same as the host path /var/lib/kubelet/pods/ * * * */volumes/kubernetes.io~rbd/rbd, but it was not.`
And I try to write file to /mnt, it also can not see any changes in /var/lib/kubelet/pods/* * * */volumes/kubernetes.io~rbd/rbd
So is my some config wrong, or someting misunderstand ?
k8s version: Release v1.2.0
Here is my config:
apiVersion: v1
kind: Pod
metadata:
name: test-rbd-read-01
spec:
containers:
- name: tomcat-read-only-01
image: tomcat
volumeMounts:
- name: rbd
mountPath: /mnt
volumes:
- name: rbd
rbd:
monitors:
- 10.63.90.177:6789
pool: wujianlin
image: zlh_test
user: wujianlin
secretRef:
name: ceph-client-admin-keyring
keyring: /etc/ceph/ceph.client.wujianlin.keyring
fsType: ext4
readOnly: true
what did you do when you restart docker? Are you able to reproduce this issue after the docker is restarted and pod is recreated?

can not connect to docker container mapping port

I use mongo image in docker, but I can not connect to 20217 port.
docker#default:~$ docker ps
prot info show: 0.0.0.0:20217->20217/tcp, 27017/tcp
but,
gilbertdeMacBook-Pro:~ gilbert$ lsof -i tcp:20217
there is no PID,
gilbertdeMacBook-Pro:~ gilbert$ docker info
Containers: 3
Images: 43
Server Version: 1.9.1
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 50
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: default
ID: MRAZ:ZG5E:HDMY:EJNQ:HFL4:PW6Y:AXIS:6JFL:PFI5:GBAY:5SMF:NYQR
Debug mode (server): true
File Descriptors: 25
Goroutines: 44
System Time: 2016-01-27T14:53:52.005531869Z
EventsListeners: 0
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Username: gilbertgan
Registry: https://index.docker.io/v1/
Labels:
provider=virtualbox
I found this is because on MAC docker-machine is running on VM,so we need add the VM IP when connect to container.
the ip can be show by: docker-machine ls
Your docker container maps port 20217 which isn't the MongoDB default port. The correct port is 27017. And gilbert_gan is right as well. When running docker on docker-machine the docker host is not localhost but rather the virtual machine under docker-machine control.