There are some similar posts but this is specifically related to running Postgres with WSL2 backend on Docker desktop. WSL2 brings full Linux experience on Windows. Volumes can be mounted to both Windows and Linux file systems. But the best practice is to use Linux file system for performance reasons see docker documentation.
Performance is much higher when files are bind-mounted from the Linux filesystem, rather than remoted from the Windows host. Therefore avoid docker run -v /mnt/c/users:/users (where /mnt/c is mounted from Windows).
Instead, from a Linux shell use a command like docker run -v ~/my-project:/sources where ~ is expanded by the Linux shell to $HOME.
My WSL distro is Ubuntu 20.04 LTS. I'm bind mounting Postgres data directory to a directory on Linux filesystem and I'm also configuring the Postgres PGDATA to use a sub-directory because this is instructed on the official Docker image docs:
PGDATA
This optional variable can be used to define another location - like a subdirectory - for the database files. The default is /var/lib/postgresql/data. If the data volume you're using is a filesystem mountpoint (like with GCE persistent disks) or remote folder that cannot be chowned to the postgres user (like some NFS mounts), Postgres initdb recommends a subdirectory be created to contain the data.
So this is how I start Postgres with the volume mounting to WSL2 Ubuntu file system:
docker run -d \
--name some-postgres -e POSTGRES_PASSWORD=root \
-e PGDATA=/var/lib/postgresql/data/pgdata \
-v ~/custom/mount:/var/lib/postgresql/data \
postgres
I can exec into the running container and verify that the data folder exists and it's configured correctly:
Now from the host machine (WSL2 Linux) if I try to access that folder I get the permission denied:
I would appreciate if anyone can provide a solution. None of the existing posts worked to resolve the issue.
This has got nothing to do with PostgreSQL. Docker containers run as root and so any directory created by Docker will also belong to root.
When you attach to the container and list the directory under /var/lib/postgresql/data it shows postgres as the owner.
Check "Arbitrary --user Notes" section in the official documentation here
The second option "bind-mount /etc/passwd read-only from the host" worked for me.
Two things that were blocking us working with WSL2 on Windows were:
Folder c:\Program files\WindowsApps didn't have admin account listed as owner
McAfee was blocking the WSL. In order to disable blocking we had to remove following rule: Open McAfee -> Threat Prevention -> Show Advanced (button in Right upper corner) -> scroll down to Rules -> name of the rule is "Executing Subsystem for Linux"
Related
For automated testing we can't use a DB Docker container with a defined volume. Just wondering if there would be available an "offical" Postgres image with no mounted volume or volume definitions.
Or if someone has a Dockerfile that would create a container without any volume definitions, that would be very helpful to see or try to use one.
Or is there any way to override a defined volume mount and just use datafile inside of to be created Docker container with running DB.
I think you are mixing up volumes and bind mounts.
https://docs.docker.com/storage/
VOLUME Dockerfile command: A volume with the VOLUME command in a Dockerfile is created into the docker area on the host that is /var/lib/docker/volumes/.
I don't think it is possible to run docker without it having access to this directory or it would be not advisable to restrict permission of docker to these directories, these are dockers own directories after all.
So postgres dockerfile has this command in dockerfile, for example: https://github.com/docker-library/postgres/blob/master/15/bullseye/Dockerfile
line 186: VOLUME /var/lib/postgresql/data
This means that the /var/lib/postgresql/data directory that is inside the postgres container will be a VOLUME that will be stored on the host somewhere in /var/lib/docker/volumes/somerandomhashorguid..... in a directory with a random name.
You can also create a volume like this with docker run:
docker run --name mypostgres -e POSTGRES_PASSWORD=password -v /etc postgres:15.1
This way the /etc directory that is inside the container will be stored on the host in the /var/lib/docker/volumes/somerandomhashorguid.....
This volume solution is needed for containers that need extra IO, because the files of the containers (that are not in volumes) are stored in the writeable layer as per the docs: "Writing into a container’s writable layer requires a storage driver to manage the filesystem. The storage driver provides a union filesystem, using the Linux kernel. This extra abstraction reduces performance as compared to using data volumes, which write directly to the host filesystem."
So you could technically remove the VOLUME command from the postgres dockerfile and rebuild the image for yourself and use that image to create your postgres container but it would have lesser performance.
Bind mounts are the type of data storage solution that can be mounted to anywhere on the host filesystem. For example if you would run:
docker run --name mypostgres -e POSTGRES_PASSWORD=password -v /tmp/mypostgresdata:/var/lib/postgresql/data postgres:15.1
(Take not of the -v flag here, there is a colon between the host and the container directory while previously in the volume version of this flag there was no host directory and no colon either.)
then you would have a directory created on your docker host machine /tmp/mypostgresdata and the directory of the container of /var/lib/postgresql/data would be mapped here instead of the docker volumes internal directory /var/lib/docker/volumes/somerandomhashorguid.....
My general rule of thumb would be to use volumes - as in /var/lib/docker/volumes/ - whenever you can and deviate only if really necessary. Bind mounts are not flexible enough to make an image/container portable and the writable container layer has less performance than docker volumes.
You can list docker volumes with docker volume ls but you will not see bind mounted directories here. For that you will need to do docker inspect containername
"You could just copy one of the dockerfiles used by the postgres project, and remove the VOLUME statement. github.com/docker-library/postgres/blob/… –
Nick ODell
Nov 26, 2022 at 18:05"
answered Nick abow.
And that edited Dockerfile would build "almost" Docker Official Image.
I wish to store my persists data in my local D:\dockerData\postgres9.6. Below is my docker command
docker pull postgres
docker run -d -v /d/dockerData/postgres9.6:/var/lib/postgresql/data -p 5432:5432 postgres
It successful create a container and I can use pgAdmin to access and create database.
But I found out that there is no file in my D:\dockerData\postgres9.6. I exec bash into the container, there is at least 20+ files inside /var/lib/postgresql/data.
Anyone can point out which part goes wrong?
It depends what kind of Docker you are using on Windows:
Docker Toolbox with VirtualBox: only C:\Users\mylogin is shared by default. D:\ is not mounted.
Docker for Windows with HyperV: only C:\ is mounted by default. Make sure D:\ is a shared drive: see image
I am learning docker and deploying some sample images from the docker hub. One of them requires postgresql. When I deploy without specifying a volume, it works beautifully. When I specify the volume 'path on host', it fails with inability to fsync properly. My question is when I inspect the volumes, I cannot find where docker is storing those volumes. I'd like to be able to specify a volume so I can move the data if/when needed. Where does Docker store this on a windows machine? I tried enabling volume through Kinematic but the container became unusable.
> docker volume inspect 0622ff3e0de10e2159fa4fe6b7cd7407c6149067f138b72380a5bbe337df8f62
[
{
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/0622ff3e0de10e2159fa4fe6b7cd7407c6149067f138b72380a5bbe337df8f62/_data",
"Name": "0622ff3e0de10e2159fa4fe6b7cd7407c6149067f138b72380a5bbe337df8f62",
"Options": {},
"Scope": "local"
}
]
I can create a volume through docker but am not sure where it is stored on the harddisk.
If you're on windows 10 and use Docker For Windows, docker will create a VM and run it on your local hyper-v, the volumes you create are then located inside of this VM which is stored in something called MobyLinuxVM.vhdx (you can check it in the settings of docker).
One way to have your data on your host computer for now is to share a drive on docker settings and then map your postgres data folder to your windows hard drive.
Something like docker run -it -v /c/mypgdata:/var/lib/postgresql/data postgres
Another way would be to create a volume with a specific driver, take a look at existing volume drivers if one can do what you want.
This one could be of interest for you : https://github.com/CWSpear/local-persist
You can also enter the MobyLinux VM with this "kind of" hack
#get a privileged container with access to Docker daemon
docker run --privileged -it --rm -v /var/run/docker.sock:/var/run/docker.sock -v /usr/bin/docker:/usr/bin/docker alpine sh
#run a container with full root access to MobyLinuxVM and no seccomp profile (so you can mount stuff)
docker run --net=host --ipc=host --uts=host --pid=host -it --security-opt=seccomp=unconfined --privileged --rm -v /:/host alpine /bin/sh
#switch to host FS
chroot /host
#and then go to the volume you asked for
cd /var/lib/docker/volumes/0622ff3e0de10e2159fa4fe6b7cd7407c6149067f138b72380a5bbe337df8f62/_data
Found here : http://docker-saigon.github.io/post/Docker-Beta/
I'm trying to run a docker-compose on my Windows machine spinning up a MongoDB instance and a data-only container which proxies an attached volume containing the database files.
mongodata:
image: mongo:2.6.8
volumes:
- ./data/db:/data/db
command: --break-mongo
mongo:
image: mongo:2.6.8
volumes_from:
- mongodata
ports:
- "27017:27017"
command: --smallfiles --rest
*p.s. the --break-mongo command is there on purpose as it just needs to create the volume
To my understanding, using a data-only volume pattern would handle permission issues but I can see the following error during the Mongo container startup:
[0m2016-01-26T00:23:52.340+0000 [initandlisten] info preallocateIsFaster couldn't run due to: couldn't open file /data/db/journal/tempLatencyTest for writing errno:1 Operation not permitted; returning false
[0m2016-01-26T00:23:52.341+0000 [initandlisten] Unable to remove temporary file due to: boost::filesystem::remove: Text file busy: "/data/db/journal/tempLatencyTest"
[0m2016-01-26T00:23:52.344+0000 [initandlisten] exception in initAndListen: 13516 couldn't open file /data/db/journal/j._0 for writing errno:1 Operation not permitted, terminating
Therefore I'm unable to use MongoDb with an attached volume from my local machine. Is there any way around this issue?
The documentation states
If you are using Docker Machine on Mac or Windows, your Docker daemon
has only limited access to your OS X or Windows filesystem. Docker
Machine tries to auto-share your /Users (OS X) or C:\Users (Windows)
directory. So, you can mount files or directories on OS X using.
docker run -v /Users/<path>:/<container path> ...
On Windows, mount directories using:
docker run -v /c/Users/<path>:/<container path> ...
All other paths come from your virtual machine’s filesystem. For example, if you are
using VirtualBox some other folder available for sharing, you need to
do additional work. In the case of VirtualBox you need to make the
host folder available as a shared folder in VirtualBox. Then, you can
mount it using the Docker -v flag.
Basically, either try to give a full path beginning from your C:\Users folder as shown above, or if you can't have that, make the host folder a shared folder in Virtualbox.
Update
No need to give a full path. docker-compose will handle that. You have to make sure that your docker-compose.yml is inside (somewhere down the line) of your Users folder. It can't be in some root folder. If you are already doing that, then you will have to adjust your permissions. Just give full permissions to that folder.
Update: Check out the latest Docker for Windows and MacOS X.
Faster and more reliable: no more VirtualBox! The Docker engine is
running in an Alpine Linux distribution on top of an xhyve Virtual
Machine on Mac OS X or on a Hyper-V VM on Windows, and that VM is
managed by the Docker application. You don’t need docker-machine to
run Docker for Mac and Windows.
Note: if Windows, you need Windows 10 Pro to make it work as Hyper-V is not included in other releases.
For Docker Toolbox previously, it seems there is no solution at all on Windows and OS X due to VirtualBox. The image documentation indeed states:
WARNING (Windows & OS X): The default Docker setup on Windows and OS X
uses a VirtualBox VM to host the Docker daemon. Unfortunately, the
mechanism VirtualBox uses to share folders between the host system and
the Docker container is not compatible with the memory mapped files
used by MongoDB (see vbox bug, docs.mongodb.org and related
jira.mongodb.org bug). This means that it is not possible to run a
MongoDB container with the data directory mapped to the host
As an workaround I just copy from a folder before mongo deamon starts. Also, in my case I don't care of journal files, so i only copy database files.
I've used this command on my docker-compose.yml
command: bash -c "(rm /data/db/*.lock && cd /prev && cp *.* /data/db) && mongod"
And everytime before stoping the container I use:
docker exec <container_name> bash -c 'cd /data/db && cp $(ls *.* | grep -v *.lock) /prev'
Note: /prev is set as a volume. path/to/your/prev:/prev
Another workaround is to use mongodump and mongorestore.
in docker-compose.yml: command: bash -c "(sleep 30; mongorestore
--quiet) & mongod"
in terminal: docker exec <container_name> mongodump
Note: I use sleep because I want to make sure that mongo started, and it takes a while.
I know this involves manual work etc, but I am happy that at least I got mongo with existing data running on my Windows 10 machine, and still can work on my Macbook when I want.
(croscopy https://stackoverflow.com/a/42044756/1894856)
I am running Docker on windows and I have a database with some entries on disk at C:\data\db.
I want to add this database to my container. I have tried numerous ways to do this but failed.
I tried: docker run -p 27017:27017 -v //c/data/db:/data/db --name mongodb devops-mongodb
In my dockerfile I have:
RUN mkdir -p /data/db
VOLUME /data/db
But this doesn't add my current database on disk to the container. It creates a fresh /data/db directory and persists the data I add to it.
The docs here https://docs.docker.com/userguide/dockervolumes/ under 'Mount a host directory as a data volume' specifically told me to execute the -v //c/data/db:/data/db but this isn't working.
Any ideas?
You're using Boot2Docker (which runs inside a Virtual Machine). Boot2Docker uses VirtualBox guest additions to make directories on your Windows machine available to Docker running inside the Virtual Machine.
By default, only the C:\Users directory (on Windows), or /Users/ directory (on OS X) is shared with the virtual machine. Anything outside those directories is not shared with the Virtual Machine, which results in Docker creating an empty directory at the specified location for the volume.
To share directories outside C:\Users\ with the Virtual Machine, you have to manually configure Boot2Docker to share those. You can find the steps needed in the VirtualBox guest addition section of the README;
If some other path or share is desired, it can be mounted at run time by doing something like:
$ mount -t vboxsf -o uid=1000,gid=50 your-other-share-name /some/mount/location
It is also important to note that in the future, the plan is to have any share which is created in VirtualBox with the "automount" flag turned on be mounted during boot at the directory of the share name (ie, a share named home/jsmith would be automounted at /home/jsmith).
Please be aware that using VirtualBox guest additions have a really bad impact on performance (reading/writing to the volume will be really slow). Which could be fine for development, but should be used with caution.