Kubernetes pod failing because of incorrect container command - kubernetes

I am creating a egress-operator. I have a pod egress-operator-controller-manager created from makefile by command make deploy IMG=my_azure_repo/egress-operator:v0.1. The pod is failing , its showing error in pod description as:
State: Waiting
Reason: RunContainerError
Last State: Terminated
Reason: StartError
Message: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/manager": stat /manager: no such file or directory: unknown
Failed 44s (x3 over 59s) kubelet Error: failed to create containerd task:
OCI runtime create failed: container_linux.go:380: starting container process caused:
exec: "/manager": stat /manager: no such file or directory: unknown
I suspect that in manager.yaml, under command: /manager is executed.Can someone let me know what is going wrong in this manager.yaml and whether /manager is valid under command: in deployment.yaml
Debugging UPDATE
Instead of running Dockerfile, now I just build and run image_azure_repo/egress-operator:v0.1 locally (on same ubuntu 18 VM), When I try to login with: docker exec -it <container_id> /bin/bash, I am getting error :
OCI runtime exec failed: exec failed: container_linux.go:380:
starting container process caused: exec: "/bin/bash": stat /bin/bash:
no such file or directory: unknown
This is similar error what I see in pod description. Instead of /bin/bash , I also tried docker exec with /bin/sh and only sh ; Its giving same error


This should be a mismatch for libc.
Golang uses glibc by default while alpine uses musl.
You could try those to fix it:
Use a base image using glibc
Install glibc in your alpine image


In Dockerfile , the Entrypoint you have mentioned is ENTRYPOINT ["/manager"] but shouldnt it be ENTRYPOINT ["./manager"]
regarding "starting container process caused: exec: "/bin/bash": stat /bin/bash: " , it seems base image does not have /bin/bash it might have just /bin/sh

Related

Issue while Entering a kubernetes pod

I am trying to enter to a running pod using command -
kubectl exec pod1-temp-6b6b4b75c6-c9hn5 -- /bin/bash
but its throwing me Error saying-
error: Internal error occurred: error executing command in container:
failed to exec in container: failed to start exec
"ee928c2cbe5699f60603abd7a0d94bae57ee2eb4bb3d3a74bc90aef7dbd1a4d8":
OCI runtime exec failed: exec failed: container_linux.go:380: starting
container process caused: exec: "C:/Program Files/Git/usr/bin/bash":
stat C:/Program Files/Git/usr/bin/bash: no such file or directory:
unknown
Its working for others but in my system its causing issue that means its local issue.
I have also checked the path and bash exec. file is also there as shown below -
enter image description here

How to stop airflow running in docker?

I followed this guide to start airflow with docker-compose. However, I cannot bring it down gracefully. The containers keep running. When I try to stop/kill them I get a permission denied error, even when run as sudo.
sudo docker container stop $(docker ps -q)
>>>
Error response from daemon: cannot stop container: f5949450056b: Cannot kill container f5949450056b850da78e9485e27765bfa088fd9e182ee621866c684a30239445: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 2a051877d204: Cannot kill container 2a051877d2046c212d3405ad39b09112613a2b56c5a350e823991c0e07650460: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 226b5e46010d: Cannot kill container 226b5e46010d4aff63941624deb0afc5aeca01826decf93ea4969f333d2e9d5c: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 09385b720d31: Cannot kill container 09385b720d3110d0511504839f0f3dc03ae122832a8b162b75c886c6e0a4d3e5: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 6cd9b8213531: Cannot kill container 6cd9b821353175e7416859f8fac2910ff499a94d581741fbdc7cce24d43cebc9: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 602eec1a9b1b: Cannot kill container 602eec1a9b1bb0c7134ca95663695b20572e3d4d7f42d9e22158001134b6d02b: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
running on Ubuntu 20.04.
The only solution I found so far is to enter each container with docker exec -it ... bash and run kill 1.

I had to run
sudo aa-remove-unknown
to be able to stop the docker process.

According to documentation, you should use:
docker-compose down --volumes --rmi all

You go to the directory where your docker-compose.yaml file is located that you used to start your docker services. There you type:
sudo docker-compose stop
It's as simple as that!

concourse start container failed: getting the final child's pid from pipe caused: EOF

Env:amazon linux2
kernel version: 4.14.198-152.320.amzn2.x86_64
concourse version:v6.6.0
Issue:
After I setup concourse on amazon EC2, I can login concourse web and create pipeline. But when I trigger the pipeline build, I got below error message:
runc run: exit status 1: container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF
Checking the backend log:
{"timestamp":"2020-10-15T03:14:44.622775204Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.completing","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624243772Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"nsexec:601 nsexec started","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624281598Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"try_mapping_tool:240 nsenter: mapping tool not present: Operation not permitted","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624308669Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"nsexec:889 nsenter: failed to sync with parent: read(SYNC_USERMAP_ACK): Invalid argument","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624328533Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"log pipe has been closed: EOF","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624349468Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624379918Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.finished","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624412285Z","level":"error","source":"guardian","message":"guardian.create.containerizer-create.runtime-create-failed","data":{"error":"runc run: exit status 1: container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3"}}
{"timestamp":"2020-10-15T03:14:44.624429680Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.finished","data":{"handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3"}}
I already highlight the message which may cause this issue. Thank you
to help me to find the root cause.

I found the root cause is system missing some tools. I replace it using a new system image, then working fine.

unable to join worker node to master node in k8

when using kubeadm join token to join worker node to a k8 master. Iam receiving following errors.
[preflight] running pre-flight checks
[preflight] WARNING: Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": executable file not found in $PATH
[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_sh nf_conntrack_ipv4 ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support
[preflight] Some fatal errors occurred:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
*********************
error 2 : when run modprobe_nrfilter
modprobe: FATAL: Module br_netfilter not found.

It seems like docker is not installed or not in your PATH:
Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": executable file not found in $PATH
This can be fixed by installing docker and ensuring the docker executable is in your PATH.

OCI runtime execution failed

Getting this error when i ran ./byfn.sh -m up command : OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"scripts/script.sh\": stat scripts/script.sh: no such file or directory

You’re missing part of the byfn.sh script. Have you cloned the Hyperledger examples? There should be a directory called scripts containing script.sh and utils.sh. Byfn is dependent on both scripts