GCE "create-with-container --container-mount-disk" flag mounts disk as read-only - mongodb

Question

I am trying to use a Percona Docker image for MongoDB on GCE, however I'm running into an issue with Mongo saying the mounted path is read-only. I looked around as much as I could, but im stumped at what could be the issue.
gcloud compute instances create-with-container mongo-svr \
--create-disk name=disk-1,size=1GB \
--container-mount-disk mount-path="/data/mongodb",mode=rw \
--container-image=docker.io/percona/percona-server-mongodb:4.2
I used the above command and it creates my instance. I then SSH into the server, connect to the running mongo instance to shutdown, then I run: docker exec -it [NAME] mongod --configsvr --replSet rs0 --dbpath=/data/mongodb --bind_ip localhost
This spits out an error stating:
CONTROL [initandlisten] options: { net: { bindIp: "localhost" }, replication: { replSet: "rs0" }, sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/mongodb" } }
STORAGE [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/mongodb, terminating
At this point, I've been recreating instances with different params, but nothing has worked so far. Anyone have an idea what I'm missing?
Updated with command output
gcloud compute instances create-with-container mongo-config-f --zone us-central1-f --create-disk name=disk-1,size=1GB --container-mount-disk mount-path="/data/mongodb" --container-image=docker.io/percona/percona-server-mongodb:4.2 --machine-type=f1-micro
WARNING: Default device-name for disk name [disk-1] will be [disk-1] because it is being mounted to a container with [`--container-mount-disk`]
Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_NAME]/zones/us-central1-f/instances/mongo-config-f].
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
mongo-config-f us-central1-f f1-micro xx.xx.xx.xx xx.xx.xx.xx RUNNING

Answer 1

I've tried to replicate your issue on my test project and found that:
persistent disk was created and mounted in read-write mode as expected;
bash-4.2$ mount
...
/dev/sdb on /data/mongodb type ext4 (rw,relatime)
docker runs containers inside our VM properly;
the cause of the error while running docker exec -it [NAME] mongod --configsvr --replSet rs0 --dbpath=/data/mongodb --bind_ip localhost is permissions inside mongodb container:
bash-4.2$ ls -l /data/
...
drwxr-xr-x 3 root root 4096 Feb 19 15:33 mongodb
As a workaround commands could be executed with root permissions:
$ docker exec -it --user root klt-mongo-svr-upd-wowt mongod --configsvr --replSet rs0 --db path=/data/mongodb
Please find more details and my steps below:
create VM:
$ gcloud compute instances create-with-container mongo-svr \
--create-disk name=disk-1,size=1GB \
--container-image docker.io/percona/percona-server-mongodb:4.2 \
--container-mount-disk mount-path="/data/mongodb"
WARNING: Default device-name for disk name [disk-1] will be [disk-1] because it is being mounted to a container with [`--container-mount-disk`]
Created [https://www.googleapis.com/compute/v1/projects/test-prj/zones/europe-west3-a/instances/mongo-svr].
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
mongo-svr-upd europe-west3-a n1-standard-1 10.156.0.9 35.XXX.155.XXX RUNNING
SSH to instance;
check if container is running:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dfad9c10235d percona/percona-server-mongodb:4.2 "/entrypoint.sh mong…" About a minute ago Up About a minute klt-mongo-svr-upd-wowt
bbe02c8e8621 gcr.io/stackdriver-agents/stackdriver-logging-agent:0.2-1.5.33-1-1 "/entrypoint.sh /usr…" About a minute ago Up About a minute stackdriver-logging-agent
everything looks good at this point;
try to run command as user:
$ docker exec -it klt-mongo-svr-upd-wowt mongod --configsvr --replSet rs0 --dbpath=/data/mongodb --bind_ip localhost
and observe the same error:
2020-02-19T15:37:53.176+0000 I STORAGE [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/mongodb, terminating
here key read-only directory: /data/mongodb;
check mounts and permissions inside of the container:
$ docker exec -it klt-mongo-svr-upd-wowt /bin/bash
bash-4.2$ mount
...
/dev/sdb on /data/mongodb type ext4 (rw,relatime)
...
as we expected disk was created and mounted in read-write mode to the container
bash-4.2$ ls -l /data/
total 8
drwxr-xr-x 4 mongodb root 4096 Feb 19 15:36 db
drwxr-xr-x 3 root root 4096 Feb 19 15:33 mongodb
bash-4.2$
but to work with /data/mongodb you need root permissions;
try to run command as root:
$ docker exec -it --user root klt-mongo-svr-upd-wowt mongod --configsvr --replSet rs0 --dbpath=/data/mongodb
2020-02-19T15:45:24.970+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-02-19T15:45:24.973+0000 I CONTROL [initandlisten] MongoDB starting : pid=119 port=27019 dbpath=/data/mongodb 64-bit host=mongo-svr-upd
2020-02-19T15:45:24.974+0000 I CONTROL [initandlisten] db version v4.2.2-3
2020-02-19T15:45:24.974+0000 I CONTROL [initandlisten] git version: 2cdb6e50913583f627acc5de35dc4e04dbfe196f
2020-02-19T15:45:24.974+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
2020-02-19T15:45:24.974+0000 I CONTROL [initandlisten] allocator: tcmalloc
2020-02-19T15:45:24.974+0000 I CONTROL [initandlisten] modules: none
2020-02-19T15:45:24.974+0000 I CONTROL [initandlisten] build environment:
2020-02-19T15:45:24.975+0000 I CONTROL [initandlisten] distarch: x86_64
2020-02-19T15:45:24.975+0000 I CONTROL [initandlisten] target_arch: x86_64
2020-02-19T15:45:24.975+0000 I CONTROL [initandlisten] options: { replication: { replSet: "rs0" }, sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/mongodb" } }
2020-02-19T15:45:24.976+0000 I STORAGE [initandlisten] Detected data files in /data/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
...
and it's working with root permissions.