I tried this approach from here: https://stackoverflow.com/a/42239760/1455384
So I have this kind of db setup:
db:
# 15.1-alpine3.16
image: postgres#sha256:44b4073d487d0f9baf82bc95d10e2b1f161b18dcc52cfbd0eb2a894b5e2cd513
environment:
<<: *common-variables
PGDATA: /pgtmpfs
tmpfs:
- /run:size=131072k
- /var/cache:size=65536k
networks:
- backend
When I run my application with this kind of postgres setup, I don't really notice any difference in speed (for example when I run various tests that hit database, they finish in about 2 minutes, whether tmpfs is used or not).
Also if I run: docker exec -it demo-db-1 df /run
I see:
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 131072 4 131068 0% /run
Does 0% Use mean, postgres does not actually use that memory "block" from tmpfs?
Update
Changing to:
tmpfs:
- /pgtmpfs:size=131072k
makes it properly use memory, but speed is not changed. Not sure if thats, because postgres does not really benefit from this or because something else is wrong?
Related
I am trying to start a cassandra container using docker-compose . I am issueing the following command from my macbook terminal.
docker-compose -f src/main/docker/cassandra.yml up
my cassandra.yml file is below
version: '2'
services:
primecast-cassandra:
image: cassandra:3.9
# volumes:
# - ~/volumes/jhipster/primecast/cassandra/:/var/lib/cassandra/data
ports:
- 7000:7000
- 7001:7001
- 7199:7199
- 9042:9042
- 9160:9160
primecast-cassandra-migration:
extends:
file: cassandra-migration.yml
service: primecast-cassandra-migration
environment:
- CREATE_KEYSPACE_SCRIPT=create-keyspace.cql
However when i run the docker-compose command to start the cassandra services i get some warning on the terminal & eventually it stops.
primecast-cassandra_1 | WARN 14:33:45 Unable to lock JVM memory (ENOMEM).
This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
appreciate if you can help
thank you
Essentially, this message is telling you that you need to specify memlock unlimited for the Linux system resource limit on the locked-in memory for a specific user. As per the documentation on recommended production settings, the normal (non-Docker) way to solve for this was to adjust the /etc/security/limits.conf file (or equivalent) as such:
cassandra - memlock unlimited
In the above example, cassandra is the user that Cassandra is running under. If you're running Cassandra as another user, you'll need to either change that accordingly or just use asterisk (*) to enforce that setting for all users. Ex:
* - memlock unlimited
In this case with Docker Compose v2, ulimits should be a valid entry under the service configuration. Set memlock to -1 so that both the soft and hard limits are "unlimited":
primecast-cassandra:
image: cassandra:3.9
ulimits:
memlock: -1
ports:
I set up a Github codespaces environment using devcontainer.json and docker-compose.yaml. Everything works fine, but the postgres database defined in docker-compose.yml loses its data every time the container needs to be re-built.
Here's the bottom part of the docker-compose.yml
db:
image: postgres:latest
restart: unless-stopped
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_USER: test_user
POSTGRES_DB: test_db
POSTGRES_PASSWORD: test_pass
volumes:
postgres-data:
as you can see, I am trying to map the postgres data volume into a postgres-data volume, but this doesn't work for some reason.
What am I doing wrong that's preventing postgres data from persisting between container builds?
Another option would be to look into using Spawn. (Disclaimer - I'm one of the devs working on it).
We've written some documentation about exactly how to use Spawn-hosted databases with GitHub codespaces here: https://docs.spawn.cc/blog/2021/08/01/spawn-and-codespaces
This will allow you to provision a database thats independent from the GitHub codespace and preserve data between restarts.
You get some extra features with Spawn like arbitrary save points, resets and loading back to saved revisions with Spawn - but the key functionality of spinning up a database for a GitHub codespace and preserving data is one of the things it works extremely well for.
according to https://docs.github.com/en/codespaces/customizing-your-codespace/configuring-codespaces-for-your-project#dockerfile ,
only docker images can be pulled from source and set-up, nowhere they mention that volume persistence is guaranteed.
and after going through this https://code.visualstudio.com/docs/remote/devcontainerjson-reference looks like mounts and few other features related to volumes are not supported for codespaces.
workspaceMount : Not yet supported in Codespaces or when using Clone Repository in Container Volume.
workaround :
in .devcontainer folder where your dockerfile is present add a line like this
RUN curl https://<your_public_cloud>/your_volume.vol -O
here <your_public_cloud> can be google drive, aws or any endpoint where you have access to download the volume. its also the volume you needed to be persist.
and once its downloaded you can mount the volume to postgres service or make a hotswap.
and when you want to save, just upload the volume to your cloud storage provider.
repeat the process every time you build, and save and upload before "unbuild" or dismissing your codespace whatever you like to call.
hope that eases your issue, happy coding!
As long as you don't remove the volume with docker-compose down --volumes as an example, the data should persist.
I had the same issue; and it turned out that I had put a crontab running docker system prune -af every 15 minutes!
You could just mount a host directory, instead of using a docker volume:
volumes:
- /home/me/postgres_data:/var/lib/postgresql/data
This guarantees that no volume cleanup (accidental or deliberate) nukes your database.
Indeed the postgres docs do this in their examples. See the PGDATA environment variable.
As you don't have access to VM, maybe the directory containing your docker-compose.yml changes.
In that case, volume name may change too.
Indeed, by default, your volume name would be the following :
<directory_name>_postgres-data
Could you try a named volume (starting with compose 3.4):
db:
image: postgres:latest
restart: unless-stopped
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_USER: test_user
POSTGRES_DB: test_db
POSTGRES_PASSWORD: test_pass
volumes:
postgres-data:
external: false
name: postgres-data
documentation of docker-compose can be found here :
https://docs.docker.com/compose/compose-file/compose-file-v3/#name
EDIT 1
If your VM is created at each build, docker dependencies too.
volumes, networks, etc...
A persistent volume is needed somewhere (surviving VM builds).
You may have to create a directory in your local workspace, like:
/local/workspace/postgres-data/
which become in codespaces according to my understanding :
./postgres-data
Check permissions, your user may not exist in the container.
As a result your compose file become:
db:
image: postgres:latest
restart: unless-stopped
volumes:
- ./postgres-data:/var/lib/postgresql/data
environment:
POSTGRES_USER: test_user
POSTGRES_DB: test_db
POSTGRES_PASSWORD: test_pass
Let's say I have the following setup in my docker-compose.yml.
services:
postgres:
image: postgres:11.6
env_file:
- .local.env
volumes:
- ./database/:/docker-entrypoint-initdb.d
ports:
- 5432:5432
...
where ./database contains some SQL files that initialize the database. Here's my question... is initdb run every single time the stopped postgres container starts running again (via $ docker-compose up).
Thus, is it fair to say that every time I restart my postgres container, it builds the entire database from scratch all over again?
My guess is 'yes' as in the documentation it says
The default postgres user and database are created in the entrypoint with initdb.
The answer is no, when you stop your container it is not deleted, only stopped, you can start it when it is stopped the same when you stop your computer it will not vanish from your desk :)
You can even restart it when it is running, same as you would do with your computer.
However when you remove/delete the container with
docker rm -f containername
or
docker-compose rm
then it is truly deleted, equivalent of making your computer vanish from your desk.
But even than you can still persist your data with volume mounts, for example in your compose file your ./database directory will not be deleted from your host machine even when you delete the containers using it. It is the equivalent of using an external usb drive in your computer, so when you make your computer vanish from your desk with deleting it, you still have your usb drive with the data on it that was there when you still had your computer.
So you can persist your database files with the same technique in a volume mount like this:
services:
postgres:
image: postgres:11.6
env_file:
- .local.env
volumes:
- ./database/:/docker-entrypoint-initdb.d
- ./postgres-data/data:/var/lib/postgresql/data
ports:
- 5432:5432
...
This way when you delete your container(s) and do "docker-compose up" again for the same compose file, postgres will not run its init scirpt because the /var/lib/postgresql directory is already populated in it.
However, my computer analogy is valid only in this context, please do not think of containers being mini computers or mini virtual machines, they are not! But that's an other discussion.
I had assumed that docker-compose volumes are mounted before the container's service is started. Is this the case?
I ask, since I've got a docker-compose.yml that, amongst other things, fires up a parse-server container, and mounts a host directory in the container, which contains some code the service should run (Cloud Code).
I can see the directory is mounted correctly after docker-compose up by shelling into the container; the expected files are in the expected place. But the parse-server instance doesn't trigger the code in the mounted directory (I checked it by adding some garbage; no errors).
Is it possible the volume is being mounted after the parse-server service starts?
This is my docker-compose.yml:
version: "3"
volumes:
myappdbdata:
myappconfigdata:
services:
# MongoDB
myappdb:
image: mongo:3.0.8
volumes:
- myappdbdata:/data/db
- myappconfigdata:/data/configdb
# Parse Server
myapp-parse-server:
image: parseplatform/parse-server:2.7.2
environment:
- PARSE_SERVER_MASTER_KEY=someString
- PARSE_SERVER_APPLICATION_ID=myapp
- VERBOSE=1
- PARSE_SERVER_DATABASE_URI=mongodb://myappdb:27017/dev
- PARSE_SERVER_URL=http://myapp-parse-server:1337/parse
- PARSE_SERVER_CLOUD_CODE_MAIN = /parse-server/cloud/
depends_on:
- myappdb
ports:
- 5000:1337
volumes:
- ./cloud:/parse-server/cloud
I'm not sure of the response as I can't find this information in the docs. But I had problems with volumes when I needed them mounted before the container was really running. Sometimes the configuration files were not loaded for example.
The only way to deal with it, is to create a Dockerfile, copy what you want, and use this image for your container.
Hth.
Sadly enough, the biggest issue here was whitespace
PARSE_SERVER_CLOUD_CODE_MAIN = /parse-server/cloud/
should have been
PARSE_SERVER_CLOUD=/parse-server/cloud/
Used 1.5 days chasing this; fml.
I have one VM host in one physical server with many docker containers inside.
Here one fragment of my fig.yml
pg:
image: pg...
redis:
image: redis...
mongodb:
image: mongodb...
app:
image: myapp...
I wish set pg container use only 25% of host cpu and app use only 50% of host cpu and so on.
Could I do it with fig or with docker run and manage link by hand?
In my case when one of this container is running a costly task it affect the cpu performance of the others ones. But when in the same physical server I have others VM with similar deploy inside the problem increase dramatically.
For now, Fig doesn't support setting CPU and memory limitation. Maybe it will support in the future.
I encourage you to experiment with using docker run -m for memory limit, and docker run -c for CPU shares. These flags will allow you to set memory and CPU values when starting a container. Read more about the flags you can use with docker run here:
https://docs.docker.com/reference/commandline/cli/#run
But it can only set when you are create a new container.
After creating container, you cannot change the value.