I am trying to migrate an existing node application to Diego on IBM Bluemix, but I am getting these error messages, I have tried to migrate using the "Migrate to Diego" link in the bluemix console.
2017-04-24T16:20:40.27+0800 [CELL/0] ERR Copying into the container failed: stream-in: nstar: error streaming in: exit status 2. Output: tar: ./app/client/n
ode_modules/node-modules-path/.travis.yml: Cannot open: No space left on device
2017-04-24T16:20:40.27+0800 [CELL/0] ERR tar: ./app/client/node_modules/node-modules-path/fixtures: Cannot mkdir: No space left on device
2017-04-24T16:20:40.27+0800 [CELL/0] ERR tar: ./app/client/node_modules/node-modules-path/fixtures: Cannot mkdir: No space left on device
2017-04-24T16:20:40.27+0800 [CELL/0] ERR tar: ./app/client/node_modules/node-modules-path/fixtures/foo: Cannot mkdir: No such file or directory
2017-04-24T16:20:40.27+0800 [CELL/0] ERR tar: ./app/client/node_modules/node-modules-path/fixtures: Cannot mkdir: No space left on device
2017-04-24T16:20:40.27+0800 [CELL/0] ERR tar: ./app/client/node_modules/node-modules-path/fixtures/foo/node_modules: Cannot mkdir: No such file or directory
2017-04-24T16:20:40.27+0800 [CELL/0] ERR tar: ./app/client/node_modules/node-modules-path/fixtures: Cannot mkdir: No space left on device
2017-04-24T16:20:40.27+0800 [CELL/0] ERR tar: ./app/client/node_modules/node-modules-path/fixtures/foo/node_modules/.gitkeep: Cannot open: No such file or d
irectory
Related
I am trying to enter to a running pod using command -
kubectl exec pod1-temp-6b6b4b75c6-c9hn5 -- /bin/bash
but its throwing me Error saying-
error: Internal error occurred: error executing command in container:
failed to exec in container: failed to start exec
"ee928c2cbe5699f60603abd7a0d94bae57ee2eb4bb3d3a74bc90aef7dbd1a4d8":
OCI runtime exec failed: exec failed: container_linux.go:380: starting
container process caused: exec: "C:/Program Files/Git/usr/bin/bash":
stat C:/Program Files/Git/usr/bin/bash: no such file or directory:
unknown
Its working for others but in my system its causing issue that means its local issue.
I have also checked the path and bash exec. file is also there as shown below -
enter image description here
I have a MongoDB StatefulSet running on OpenShift v3.11. The PersistentVolume is using NFSv4.
On our environment, I set up so that the directory in the NFS server are owned by nfsnobody:nfsnobody. SELinux has also been set to Permissive. All the inner directories and files are also granted with chmod ug+rwx,o-rwx.
This is done so that at runtime, when the Pod accesses to the shared path using the user with group root (gid=0), due to NFS by default squashes user and group root to nfsnobody, the Pod will be able to read and write to the shared path.
$> ls -halZ /srv/share/openshift/mongo/
drwxrwx---. nfsnobody nfsnobody unconfined_u:object_r:default_t:s0 data
This set up has been working for months. But then it starts to fail.
However, when I deploy the Pod, it fails to start with the following error:
021-01-26T16:12:48.163+0000 W STORAGE [initandlisten] Detected unclean shutdown - /var/lib/mongodb/data/mongod.lock is not empty.
2021-01-26T16:12:48.163+0000 I STORAGE [initandlisten] Detected data files in /var/lib/mongodb/data created by the 'wiredTiger' storage engine, so setting theactive storage engine to 'wiredTiger'.
2021-01-26T16:12:48.163+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2021-01-26T16:12:48.164+0000 I STORAGE [initandlisten] wiredtiger_open config:create,cache_size=31220M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2021-01-26T16:12:48.688+0000 E STORAGE [initandlisten] WiredTiger error (1) [1611677568:688148][457:0x7f9b59cc1ca8], file:WiredTiger.wt, connection: __posix_open_file, 715: /var/lib/mongodb/data/WiredTiger.wt: handle-open: open: Operationnot permitted Raw: [1611677568:688148][457:0x7f9b59cc1ca8], file:WiredTiger.wt,connection: __posix_open_file, 715: /var/lib/mongodb/data/WiredTiger.wt: handle-open: open: Operation not permitted
2021-01-26T16:12:48.708+0000 E STORAGE [initandlisten] WiredTiger error (1) [1611677568:708810][457:0x7f9b59cc1ca8], file:WiredTiger.wt, connection: __posix_open_file, 715: /var/lib/mongodb/data/WiredTiger.wt: handle-open: open: Operationnot permitted Raw: [1611677568:708810][457:0x7f9b59cc1ca8], file:WiredTiger.wt,connection: __posix_open_file, 715: /var/lib/mongodb/data/WiredTiger.wt: handle-open: open: Operation not permitted
2021-01-26T16:12:48.728+0000 E STORAGE [initandlisten] WiredTiger error (1) [1611677568:728860][457:0x7f9b59cc1ca8], file:WiredTiger.wt, connection: __posix_open_file, 715: /var/lib/mongodb/data/WiredTiger.wt: handle-open: open: Operationnot permitted Raw: [1611677568:728860][457:0x7f9b59cc1ca8], file:WiredTiger.wt,connection: __posix_open_file, 715: /var/lib/mongodb/data/WiredTiger.wt: handle-open: open: Operation not permitted
2021-01-26T16:12:48.744+0000 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version.
2021-01-26T16:12:48.744+0000 F STORAGE [initandlisten] Reason: 1: Operation not permitted
2021-01-26T16:12:48.744+0000 F - [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 638
2021-01-26T16:12:48.744+0000 F - [initandlisten]
At first glance, one could say "it's probably the mongod process does not have the permission to read the file". However, when I run in debug mode to access to the Terminal, I can totally access to the path /var/lib/mongo/data.
$> id
id=1000230000 gid=0(root) groups=0(root),1000230000
$> cd /var/lib/mongodb/data
/var/lib/mongodb/data$> echo "This is a test" >new_file
/var/lib/mongodb/data$> rm new_file
/var/lib/mongodb/data$> cat WiredTiger.wt | wc -l
23
/var/lib/mongodb/data$> mongod --dbpath $(pwd)
....failed...
The above commands show that I can read the /var/lib/mongod/data/WiredTiger.wt to count the lines but the mongod process cannot.
Only if I do
# 1000230000 is the random UID and GID granted by OpenShift for the Pod.
$> chown -R 1000230000:nfsnobody /srv/share/openshift/mongo/
...the Pod is able to read the files.
Is there anything else I should check to resolve this problem?
UPDATE:
The MongoDB version is 4.0.5.
Add more log which could pinpoint where the error happened. The wiredtiger_kv_engine.cpp.
By reading the MongoDB source code at the tag r4.0.5, I can now understand why I got the error.
Thanks to #Alex Blex for hinting the source code!
Summary
When mongod attempts to read the WiredTiger.wt (or any other files), it tries to not update the file's last access time (st_time in the inode). This reason for doing this is to increase performance. Under the hood, it uses system call open() with a flag O_NOATIME.
According to open() man page:
This flag can be employed only if one of the following
conditions is true:
The effective UID of the process matches the owner UID
of the file.
The calling process has the CAP_FOWNER capability in
its user namespace and the owner UID of the file has a
mapping in the namespace.
The call fails with the error
EPERM The O_NOATIME flag was specified, but the effective user
ID of the caller did not match the owner of the file and
the caller was not privileged.
In my case, the file is being owned by the nfsnobody, not the current UID, hence the error. That explains only by doing chown $UID:nfsnobody the problem goes away.
Some further details
The error comes from posix/os_fs.c when it tries to open a file. At line 693, the flag NO_ATIME is set if the __posix_open_file is called with WT_FS_OPEN_FILE_TYPE_DATA.
Env:amazon linux2
kernel version: 4.14.198-152.320.amzn2.x86_64
concourse version:v6.6.0
Issue:
After I setup concourse on amazon EC2, I can login concourse web and create pipeline. But when I trigger the pipeline build, I got below error message:
runc run: exit status 1: container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF
Checking the backend log:
{"timestamp":"2020-10-15T03:14:44.622775204Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.completing","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624243772Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"nsexec:601 nsexec started","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624281598Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"try_mapping_tool:240 nsenter: mapping tool not present: Operation not permitted","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624308669Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"nsexec:889 nsenter: failed to sync with parent: read(SYNC_USERMAP_ACK): Invalid argument","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624328533Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"log pipe has been closed: EOF","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624349468Z","level":"debug","source":"guardian","message":"guardian.create.containerizer-create.create.runc","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","message":"container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624379918Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.finished","data":{"bundle":"/home/concourse/worker/depot/2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3.2"}}
{"timestamp":"2020-10-15T03:14:44.624412285Z","level":"error","source":"guardian","message":"guardian.create.containerizer-create.runtime-create-failed","data":{"error":"runc run: exit status 1: container_linux.go:367: starting container process caused: process_linux.go:338: getting the final child's pid from pipe caused: EOF","handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3"}}
{"timestamp":"2020-10-15T03:14:44.624429680Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.finished","data":{"handle":"2b4ce51a-c28d-4aac-7122-b82cd1cbb99e","session":"99.3"}}
I already highlight the message which may cause this issue. Thank you
to help me to find the root cause.
I found the root cause is system missing some tools. I replace it using a new system image, then working fine.
my jod id is
2019-02-01_06_50_27-10838491598599390366
this is dataflow batch job. with template.
here is the dataflow error code.
2019-02-01 23:51:02.647 JST
EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities
2019-02-01 23:51:02.659 JST
EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities
2019-02-01 23:51:02.699 JST
EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities
2019-02-01 23:51:02.699 JST
EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities
2019-02-01 23:51:02.700 JST
EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities
2019-02-01 23:51:02.710 JST
EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities
2019-02-01 23:51:02.937 JST
EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities
2019-02-01 23:51:03.387 JST
EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities
2019-02-01 23:51:10.509 JST
Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system
2019-02-01 23:51:10.511 JST
Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
{
insertId: "s=51b724ba020b4384acc382e634e62cbc;i=568;b=879cba75f5cd4eff82751e8f30ef312b;m=9a91b9;t=580d6461241e4;x=6549465094b7bc54"
jsonPayload: {…}
labels: {…}
logName: "projects/fluted-airline-109810/logs/dataflow.googleapis.com%2Fkubelet"
receiveTimestamp: "2019-02-01T14:51:18.883283433Z"
resource: {…}
severity: "ERROR"
timestamp: "2019-02-01T14:51:10.511494Z"
}
2019-02-01 23:51:10.560 JST
[ContainerManager]: Fail to get rootfs information unable to find data for container /
2019-02-01 23:51:10.577 JST
Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system
2019-02-01 23:51:10.580 JST
Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
2019-02-01 23:51:10.608 JST
[ContainerManager]: Fail to get rootfs information unable to find data for container /
2019-02-01 23:51:10.645 JST
Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system
2019-02-01 23:51:10.646 JST
Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
2019-02-01 23:51:10.694 JST
[ContainerManager]: Fail to get rootfs information unable to find data for container /
2019-02-01 23:51:10.749 JST
Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system
2019-02-01 23:51:10.751 JST
Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
2019-02-01 23:51:10.775 JST
Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system
2019-02-01 23:51:10.777 JST
Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
2019-02-01 23:51:10.785 JST
[ContainerManager]: Fail to get rootfs information unable to find data for container /
2019-02-01 23:51:10.809 JST
Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system
2019-02-01 23:51:10.811 JST
Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
2019-02-01 23:51:10.816 JST
[ContainerManager]: Fail to get rootfs information unable to find data for container /
2019-02-01 23:51:10.857 JST
[ContainerManager]: Fail to get rootfs information unable to find data for container /
2019-02-01 23:51:10.929 JST
Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system
2019-02-01 23:51:10.931 JST
Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
2019-02-01 23:51:10.966 JST
[ContainerManager]: Fail to get rootfs information unable to find data for container /
2019-02-01 23:51:11.214 JST
Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system
2019-02-01 23:51:11.216 JST
Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
2019-02-01 23:51:11.254 JST
[ContainerManager]: Fail to get rootfs information unable to find data for container /
2019-02-01 23:51:15.619 JST
PercpuUsage had 0 cpus, but the actual number is 2; ignoring extra CPUs
2019-02-01 23:51:15.793 JST
PercpuUsage had 0 cpus, but the actual number is 2; ignoring extra CPUs
2019-02-01 23:51:15.974 JST
PercpuUsage had 0 cpus, but the actual number is 2; ignoring extra CPUs
2019-02-01 23:51:16.264 JST
PercpuUsage had 0 cpus, but the actual number is 2; ignoring extra CPUs
Is the gs:// bucket accessible by the service account for this job?
I am not able to start mongodb when I execute "/etc/init.d/mongod status
" I got message "mongod dead but pid file exists" i am new to mongodb Please help me out.
To get more info on why mongod is not starting successfully, you can check the mongodb log file for any errors. On CentOS, this file is most likely in /var/log/mongodb/mongodb.log.
When this occurred for us once, it pointed to a permissions issue:
ERROR: listen(): bind() failed errno:13 Permission denied for socket: /tmp/mongodb-27017.sock
This error can be resolved by setting the correct permissions on the /tmp directory, which is already answered in this Stackoverflow question: MongoDB - Permission denied for socket: /tmp/mongodb-27017.sock
Hope that helps!