ECS + EFS - files never make their way out of the container - amazon-ecs

Question

I'm trying to do a POC with ECS + EFS (MySQL for a personal site), but the file changes to the mounted volume within docker don't make their way to EFS.
I have it mounted on the container host:
us-east-1a.fs-#####.efs.us-east-1.amazonaws.com:/ on /mnt/mysql-data-docker type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.128,local_lock=none,addr=10.0.0.90)
My task definition (relevant parts) shows:
"mountPoints": [
{
"containerPath": "/var/lib/mysql",
"sourceVolume": "mysql-data",
"readOnly": null
}
]
and
"volumes": [
{
"host": {
"sourcePath": "/mnt/mysql-data-docker"
},
"name": "mysql-data"
}
],
I can write a file there, terminate the host, have the new host come up via scaling group and get mounted, and the file is still there, so I know that's working (and EFS shows 12kb on that FS instead of 6kb).
Looking at the running MySQL container:
[ec2-user#ip-10-0-0-128 ~]$ docker inspect e96a7 | jq '.[0].Mounts'
[
{
"Source": "/mnt/mysql-data-docker",
"Destination": "/var/lib/mysql",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
}
]
/mnt/mysql-data-docker on the host only shows my test file I verified with. In the container, there's a bunch of stuff in /var/lib/mysql but it never makes its way to the host or to EFS.

Answer 1

Turns out its because:
If you're using the Amazon ECS-Optimized AMI or Amazon Linux AMI's docker packages, the Docker daemon's mount namespace is unshared from the host's at launch. Some other AMIs might also have this behaviour.
On any of those, a filesystem mount will not propagate to the Docker daemon until the next time it's restarted.
So running sudo service docker restart && sudo start ecs solved it.

Answer 2

When you startup an EC2 instance you can specify a "user data" script that is run at boot time, BEFORE the docker start up
In your case, something like this should work:
#cloud-boothook
#!/bin/bash
# Install nfs-utils
sudo yum install -y nfs-utils
# Mount EFS, writing the volume to fstab ensures it will automatically mount even on reboot
sudo mkdir -p /mnt/mysql-data-docker
sudo echo "us-east-1a.fs-#####.efs.us-east-1.amazonaws.com:/ /mnt/mysql-data-docker nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2" >> /etc/fstab
sudo mount -a