Env:amazon linux2
kernel version: 4.14.198-152.320.amzn2.x86_64
concourse version:v6.6.0
Issue:
After I setup concourse on amazon EC2, I can login concourse web and create pipeline. But when I trigger the pipeline build, I got below error message:
runc run: exit status 1: container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF
Checking the backend log:
{"timestamp":"2020-10-15T03:14:44.622775204Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.completing","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624243772Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"nsexec:601 nsexec started","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624281598Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"try_mapping_tool:240 nsenter: mapping tool not present: Operation not permitted","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624308669Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"nsexec:889 nsenter: failed to sync with parent: read(SYNC_USERMAP_ACK): Invalid argument","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624328533Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"log pipe has been closed: EOF","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624349468Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624379918Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.finished","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624412285Z","level":"error","source":"guardian","message":"guardian.create.containerizer-create.runtime-create-failed","data":{"error":"runc run: exit status 1: container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3"}}
{"timestamp":"2020-10-15T03:14:44.624429680Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.finished","data":{"handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3"}}
I already highlight the message which may cause this issue. Thank you
to help me to find the root cause.
I found the root cause is system missing some tools. I replace it using a new system image, then working fine.
Related
I am trying to enter to a running pod using command -
kubectl exec pod1-temp-6b6b4b75c6-c9hn5 -- /bin/bash
but its throwing me Error saying-
error: Internal error occurred: error executing command in container:
failed to exec in container: failed to start exec
"ee928c2cbe5699f60603abd7a0d94bae57ee2eb4bb3d3a74bc90aef7dbd1a4d8":
OCI runtime exec failed: exec failed: container_linux.go:380: starting
container process caused: exec: "C:/Program Files/Git/usr/bin/bash":
stat C:/Program Files/Git/usr/bin/bash: no such file or directory:
unknown
Its working for others but in my system its causing issue that means its local issue.
I have also checked the path and bash exec. file is also there as shown below -
enter image description here
I followed this guide to start airflow with docker-compose. However, I cannot bring it down gracefully. The containers keep running. When I try to stop/kill them I get a permission denied error, even when run as sudo.
sudo docker container stop $(docker ps -q)
>>>
Error response from daemon: cannot stop container: f5949450056b: Cannot kill container f5949450056b850da78e9485e27765bfa088fd9e182ee621866c684a30239445: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 2a051877d204: Cannot kill container 2a051877d2046c212d3405ad39b09112613a2b56c5a350e823991c0e07650460: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 226b5e46010d: Cannot kill container 226b5e46010d4aff63941624deb0afc5aeca01826decf93ea4969f333d2e9d5c: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 09385b720d31: Cannot kill container 09385b720d3110d0511504839f0f3dc03ae122832a8b162b75c886c6e0a4d3e5: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 6cd9b8213531: Cannot kill container 6cd9b821353175e7416859f8fac2910ff499a94d581741fbdc7cce24d43cebc9: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
Error response from daemon: cannot stop container: 602eec1a9b1b: Cannot kill container 602eec1a9b1bb0c7134ca95663695b20572e3d4d7f42d9e22158001134b6d02b: unknown error after kill: runc did not terminate sucessfully: container_linux.go:392: signaling init process caused "permission denied"
: unknown
running on Ubuntu 20.04.
The only solution I found so far is to enter each container with docker exec -it ... bash and run kill 1.
I had to run
sudo aa-remove-unknown
to be able to stop the docker process.
According to documentation, you should use:
docker-compose down --volumes --rmi all
You go to the directory where your docker-compose.yaml file is located that you used to start your docker services. There you type:
sudo docker-compose stop
It's as simple as that!
I am creating a egress-operator. I have a pod egress-operator-controller-manager created from makefile by command make deploy IMG=my_azure_repo/egress-operator:v0.1. The pod is failing , its showing error in pod description as:
State: Waiting
Reason: RunContainerError
Last State: Terminated
Reason: StartError
Message: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/manager": stat /manager: no such file or directory: unknown
Failed 44s (x3 over 59s) kubelet Error: failed to create containerd task:
OCI runtime create failed: container_linux.go:380: starting container process caused:
exec: "/manager": stat /manager: no such file or directory: unknown
I suspect that in manager.yaml, under command: /manager is executed.Can someone let me know what is going wrong in this manager.yaml and whether /manager is valid under command: in deployment.yaml
Debugging UPDATE
Instead of running Dockerfile, now I just build and run image_azure_repo/egress-operator:v0.1 locally (on same ubuntu 18 VM), When I try to login with: docker exec -it <container_id> /bin/bash, I am getting error :
OCI runtime exec failed: exec failed: container_linux.go:380:
starting container process caused: exec: "/bin/bash": stat /bin/bash:
no such file or directory: unknown
This is similar error what I see in pod description. Instead of /bin/bash , I also tried docker exec with /bin/sh and only sh ; Its giving same error
This should be a mismatch for libc.
Golang uses glibc by default while alpine uses musl.
You could try those to fix it:
Use a base image using glibc
Install glibc in your alpine image
In Dockerfile , the Entrypoint you have mentioned is ENTRYPOINT ["/manager"] but shouldnt it be ENTRYPOINT ["./manager"]
regarding "starting container process caused: exec: "/bin/bash": stat /bin/bash: " , it seems base image does not have /bin/bash it might have just /bin/sh
Getting this error when i ran ./byfn.sh -m up command : OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"scripts/script.sh\": stat scripts/script.sh: no such file or directory
You’re missing part of the byfn.sh script. Have you cloned the Hyperledger examples? There should be a directory called scripts containing script.sh and utils.sh. Byfn is dependent on both scripts
I was trying to run this command
#mongod --fork --logpath /var/log/mongod.log
but I got this message
about to fork child process, waiting until server is ready for connections.
forked process: 10750
all output going to: /var/log/mongod.log
log file [/var/log/mongod.log] exists; copied to temporary file
[/var/log/mongod.log.2017-05-23T14-57-31]
ERROR: child process failed, exited with error number 4
Could you please help me resolve the problem?
Per the MongoDB docs:
4
The version of the database is different from the version supported by the mongod (or mongod.exe) instance. The instance exits cleanly.