I followed this guide to start airflow with docker-compose. However, I cannot bring it down gracefully. The containers keep running. When I try to stop/kill them I get a permission denied error, even when run as sudo.
sudo docker container stop $(docker ps -q)
>>>
Error response from daemon: cannot stop container: f5949450056b: Cannot kill container f5949450056b850da78e9485e27765bfa088fd9e182ee621866c684a30239445: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 2a051877d204: Cannot kill container 2a051877d2046c212d3405ad39b09112613a2b56c5a350e823991c0e07650460: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 226b5e46010d: Cannot kill container 226b5e46010d4aff63941624deb0afc5aeca01826decf93ea4969f333d2e9d5c: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 09385b720d31: Cannot kill container 09385b720d3110d0511504839f0f3dc03ae122832a8b162b75c886c6e0a4d3e5: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 6cd9b8213531: Cannot kill container 6cd9b821353175e7416859f8fac2910ff499a94d581741fbdc7cce24d43cebc9: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 602eec1a9b1b: Cannot kill container 602eec1a9b1bb0c7134ca95663695b20572e3d4d7f42d9e22158001134b6d02b: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
running on Ubuntu 20.04.
The only solution I found so far is to enter each container with docker exec -it ... bash and run kill 1.
I had to run
sudo aa-remove-unknown
to be able to stop the docker process.
According to documentation, you should use:
docker-compose down --volumes --rmi all
You go to the directory where your docker-compose.yaml file is located that you used to start your docker services. There you type:
sudo docker-compose stop
It's as simple as that!
I am creating a egress-operator. I have a pod egress-operator-controller-manager created from makefile by command make deploy IMG=my_azure_repo/egress-operator:v0.1. The pod is failing , its showing error in pod description as:
State: Waiting
Reason: RunContainerError
Last State: Terminated
Reason: StartError
Message: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/manager": stat /manager: no such file or directory: unknown
Failed 44s (x3 over 59s) kubelet Error: failed to create containerd task:
OCI runtime create failed: container_linux.go:380: starting container process caused:
exec: "/manager": stat /manager: no such file or directory: unknown
I suspect that in manager.yaml, under command: /manager is executed.Can someone let me know what is going wrong in this manager.yaml and whether /manager is valid under command: in deployment.yaml
Debugging UPDATE
Instead of running Dockerfile, now I just build and run image_azure_repo/egress-operator:v0.1 locally (on same ubuntu 18 VM), When I try to login with: docker exec -it <container_id> /bin/bash, I am getting error :
OCI runtime exec failed: exec failed: container_linux.go:380:
starting container process caused: exec: "/bin/bash": stat /bin/bash:
no such file or directory: unknown
This is similar error what I see in pod description. Instead of /bin/bash , I also tried docker exec with /bin/sh and only sh ; Its giving same error
This should be a mismatch for libc.
Golang uses glibc by default while alpine uses musl.
You could try those to fix it:
Use a base image using glibc
Install glibc in your alpine image
In Dockerfile , the Entrypoint you have mentioned is ENTRYPOINT ["/manager"] but shouldnt it be ENTRYPOINT ["./manager"]
regarding "starting container process caused: exec: "/bin/bash": stat /bin/bash: " , it seems base image does not have /bin/bash it might have just /bin/sh
Env:amazon linux2
kernel version: 4.14.198-152.320.amzn2.x86_64
concourse version:v6.6.0
Issue:
After I setup concourse on amazon EC2, I can login concourse web and create pipeline. But when I trigger the pipeline build, I got below error message:
runc run: exit status 1: container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF
Checking the backend log:
{"timestamp":"2020-10-15T03:14:44.622775204Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.completing","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624243772Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"nsexec:601 nsexec started","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624281598Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"try_mapping_tool:240 nsenter: mapping tool not present: Operation not permitted","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624308669Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"nsexec:889 nsenter: failed to sync with parent: read(SYNC_USERMAP_ACK): Invalid argument","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624328533Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"log pipe has been closed: EOF","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624349468Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624379918Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.finished","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624412285Z","level":"error","source":"guardian","message":"guardian.create.containerizer-create.runtime-create-failed","data":{"error":"runc run: exit status 1: container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3"}}
{"timestamp":"2020-10-15T03:14:44.624429680Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.finished","data":{"handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3"}}
I already highlight the message which may cause this issue. Thank you
to help me to find the root cause.
I found the root cause is system missing some tools. I replace it using a new system image, then working fine.
when using kubeadm join token to join worker node to a k8 master. Iam receiving following errors.
[preflight] running pre-flight checks
[preflight] WARNING: Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": executable file not found in $PATH
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_sh nf_conntrack_ipv4 ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support
[preflight] Some fatal errors occurred:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
*********************
error 2 : when run modprobe_nrfilter
modprobe: FATAL: Module br_netfilter not found.
It seems like docker is not installed or not in your PATH:
Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": executable file not found in $PATH
This can be fixed by installing docker and ensuring the docker executable is in your PATH.
Getting this error when i ran ./byfn.sh -m up command : OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"scripts/script.sh\": stat scripts/script.sh: no such file or directory
You’re missing part of the byfn.sh script. Have you cloned the Hyperledger examples? There should be a directory called scripts containing script.sh and utils.sh. Byfn is dependent on both scripts