I've been trying to mount a volume to a docker container with clickhouse, specifically on docker desktop windows 10. Following the documentation:
https://hub.docker.com/r/yandex/clickhouse-server/
I have no problem setting up the docker container on my C drive which is in my $HOME path and loading data into etc. I want to now mount a custom volume, my E/ drive which is larger as the database will continue to grow. I am getting an error when I run this:
docker run -d -p 8123:8123 --name clickhousedb --ulimit nofile=262144:262144 --volume=/E:/ch/clickhousedb:/var/lib/clickhouse yandex/clickhouse-server
specifically this:
Error response from daemon: invalid mode: /var/lib/clickhouse.
Any ideas what might be the issue?
The issue is the "/" character right after " --volume=", which tells the docker CLI to split the string as:
empty string (directory to be mounted)
E:/ch/clickhousedb (mounting point inside the container)
/var/lib/clickhouse (mounting mode)
Docker thought "/var/lib/clickhouse" was the mode for the volume mount, hence the error message.
Seemed to be a permission issue. Was able to access the root of the E drive:
docker run -d -p 8134:8123 --name clickhousedb --ulimit nofile=262144:262144 --volume=E:/:/var/lib/clickhouse yandex/clickhouse-server
Related
I'm very new to using docker and I've created a postgres container using
docker run --name mytrainingdb -e POSTGRES_PASSWORD=mysecretpassword -d postgres. Then I connected to it with docker exec -it <container-id> bash and then psql.
Then I stop the container.
My query is, what do I do reconnect to the same database? I tried to run same docker run command, but it says the name 'mytrainingdb' is used, which means it is trying to create it afresh, which is not what I want. Hope my expectation is right, as in when I restart my laptop or resume work I can just restart the same container and my data/config would be preserved?
The documentation also mentions that we can link a host directory to volume of pg container to have the stored data accessible to us, but I'm ok with docker managing my storage for that database.
You will have error when you try to re-run the same command, because docker is trying to create a new container with same name as the previous one "mytrainingdb". If you close docker and reopen it you will still find your container , but its not running , you can start it again with docker start mytrainingdb or you can remove it with docker rm mytrainingdb .
However , dont restart docker because you want to create a new container with the same name! If you want to start a new container with the same name and your container is still running you can first stop it with docker stop mytrainingdb and docker rm mytrainingdb or you can just do docker rm -f mytrainingdb (this will remove you running container with force ) and then create a new container..
As for the volumes ,you just created one by default which is named is kind of hash , and its found at volumes/var/lib/docker/volumes/ .Because generally containers such PostgreSQL, or databases in general persists volumes. The volume gets created when running the container and is handy to save persistent data, whether you start the container with -v or not.
The volume you talked about in your question , is called mounted volume , is when you basically just bind a certain directory or file from the host (outside) to inside the container
docker run -v /hostdir:/containerdir in your case docker run -v /hostdir:/var/lib/postgresql/data
If you restart docker or your computer running containers won't be automatically restarted. You can start your container again with docker start mytrainingdb (related question), then connect with your docker exec command.
(one tip: instead of running bash, then psql, you can directly run psql, e.g. docker exec -it mytrainingdb psql --user postgres)
Your understanding of data persistence is correct, docker will manage the data and it will still be around.
From the postgres image documentation
There are several ways to store data used by applications that run in Docker containers. We encourage users of the postgres images to familiarize themselves with the options available, including:
Let Docker manage the storage of your database data by writing the database files to disk on the host system using its own internal volume management. This is the default and is easy and fairly transparent to the user. The downside is that the files may be hard to locate for tools and applications that run directly on the host system, i.e. outside containers.
You can add --rm argument so that whenever you stop the container manually, or container stops for any reasons (his task is done or it fails), it will remove that container.
In your case, you can use this:
docker run --name mytrainingdb --rm -e POSTGRES_PASSWORD=mysecretpassword -d postgres
I am creating postgreSQL container using following command
sudo docker run -d --name=pg -p 5432:5432 -e POSTGRES_PASSWORD=secret -e PGDATA=/pgdata -v pg:/pgdata postgres
After running this container when I check volumes by running following command
sudo docker volume ls
DRIVER VOLUME NAME
local 6d283475c6fe923155018c847f2c607c464244cb6767dd37a579824cf8c7e612
local pg
I get two volumes. pg volume is created in the command but what the second volume is??
If you look at the Docker Hub decomposition of the postgres image you will notice it has a declaration
VOLUME ["/var/lib/postgresql/data"]
If you don't explicitly mount something else on that directory, Docker will create an anonymous volume and mount it there for you. This behaves identically to a named volume except that it doesn't have a specific name.
docker inspect mostly dumps out low-level diagnostic information, but it should include the mount information, and you should see two volume mounts, one with the anonymous volume on the default PostgreSQL data directory and a second matching the explicit mount on /pgdata.
I have a running postgreSQL docker container and need to add a volume mount.
I followed the steps from How can I add a volume to an existing Docker container?, (ran docker commit on the container to save it as an image, and spun up another container based on that image with a named volume mounted in). All the data files from the first container are present in /var/lib/postgres/data of the second container.
However, when I try to query this second postgres database, I cannot see any tables that are in the first container. Been trying to fix this for a few days with no luck, am I missing something here (does mounting a volume obscure the existing data in /var/lib/postres/data)?
Commit will not work as there is the volume defined in the Dockerfile.
Volumes are useful in many cases, for example, for running
database-storage. However, since volumes are not 'part' of a
container, it makes containers no longer portable - which seems in
direct conflict with the slogan "Build once... Run anywhere.."
docker commit data container with VOLUME
One option that you can try is copying data folder to host from an existing container and then launch the container with mount path.
docker cp my_db_cotainer:/var/lib/postgresql/data db_data
then start a new container with this path so it will contain the same data as the previous one
docker run -d --name some-postgres -v $PWD/db_data/:/var/lib/postgresql/data postgres
same for mysql
docker cp some-mysql-old:/var/lib/mysql db_backup
docker run --rm --name some-mysql-new -v $PWD/db_backup:/var/lib/mysql -it mysql
I was trying to run postgres 12.0 alpine with arbitrary user in an attempt to have easier acces to the mounted drives. However, I get the following error. I was following the instructions from official docker hub here
docker run -it --user "$(id -u):$(id -g)" -v /etc/passwd:/etc/passwd:ro postgres:12.0-alipne
I get: initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
Then I tried initializing the target directory separately which needs a restart in between. This is also not working and gives me the same error. But this time, the container starts as a root user.
Has anyone had success running the postgres alpine container with an arbitrary user?
I wish to store my persists data in my local D:\dockerData\postgres9.6. Below is my docker command
docker pull postgres
docker run -d -v /d/dockerData/postgres9.6:/var/lib/postgresql/data -p 5432:5432 postgres
It successful create a container and I can use pgAdmin to access and create database.
But I found out that there is no file in my D:\dockerData\postgres9.6. I exec bash into the container, there is at least 20+ files inside /var/lib/postgresql/data.
Anyone can point out which part goes wrong?
It depends what kind of Docker you are using on Windows:
Docker Toolbox with VirtualBox: only C:\Users\mylogin is shared by default. D:\ is not mounted.
Docker for Windows with HyperV: only C:\ is mounted by default. Make sure D:\ is a shared drive: see image