What is the simplest way to configure parameter max_prepared_transactions=100 in docker kubernetes?
I am using image:
https://hub.docker.com/_/postgres/
Which has postgresql.conf file at /var/lib/postgresql/data
In my kubernetes deployment, that directory is externally mounted so I can't copy postgresql.conf using Dockerfile so I need to specify that parameter as a ENV parameter in Kubernetes .yml file, or changing the location of postgresql.conf file to, for example, /etc/postgresql.conf (how can I do this as a ENV parameter too?)
Thanks
You can set this config as runtime flag when you start your docker container. Something like this:
$ docker run -d postgres --max_prepared_transactions=100
If you're using the Postgres chart from Bitnami, you can add this to your values.yaml.
postgresqlExtendedConf:
{ "maxPreparedTransactions" : 100 }
Related
For automated testing we can't use a DB Docker container with a defined volume. Just wondering if there would be available an "offical" Postgres image with no mounted volume or volume definitions.
Or if someone has a Dockerfile that would create a container without any volume definitions, that would be very helpful to see or try to use one.
Or is there any way to override a defined volume mount and just use datafile inside of to be created Docker container with running DB.
I think you are mixing up volumes and bind mounts.
https://docs.docker.com/storage/
VOLUME Dockerfile command: A volume with the VOLUME command in a Dockerfile is created into the docker area on the host that is /var/lib/docker/volumes/.
I don't think it is possible to run docker without it having access to this directory or it would be not advisable to restrict permission of docker to these directories, these are dockers own directories after all.
So postgres dockerfile has this command in dockerfile, for example: https://github.com/docker-library/postgres/blob/master/15/bullseye/Dockerfile
line 186: VOLUME /var/lib/postgresql/data
This means that the /var/lib/postgresql/data directory that is inside the postgres container will be a VOLUME that will be stored on the host somewhere in /var/lib/docker/volumes/somerandomhashorguid..... in a directory with a random name.
You can also create a volume like this with docker run:
docker run --name mypostgres -e POSTGRES_PASSWORD=password -v /etc postgres:15.1
This way the /etc directory that is inside the container will be stored on the host in the /var/lib/docker/volumes/somerandomhashorguid.....
This volume solution is needed for containers that need extra IO, because the files of the containers (that are not in volumes) are stored in the writeable layer as per the docs: "Writing into a container’s writable layer requires a storage driver to manage the filesystem. The storage driver provides a union filesystem, using the Linux kernel. This extra abstraction reduces performance as compared to using data volumes, which write directly to the host filesystem."
So you could technically remove the VOLUME command from the postgres dockerfile and rebuild the image for yourself and use that image to create your postgres container but it would have lesser performance.
Bind mounts are the type of data storage solution that can be mounted to anywhere on the host filesystem. For example if you would run:
docker run --name mypostgres -e POSTGRES_PASSWORD=password -v /tmp/mypostgresdata:/var/lib/postgresql/data postgres:15.1
(Take not of the -v flag here, there is a colon between the host and the container directory while previously in the volume version of this flag there was no host directory and no colon either.)
then you would have a directory created on your docker host machine /tmp/mypostgresdata and the directory of the container of /var/lib/postgresql/data would be mapped here instead of the docker volumes internal directory /var/lib/docker/volumes/somerandomhashorguid.....
My general rule of thumb would be to use volumes - as in /var/lib/docker/volumes/ - whenever you can and deviate only if really necessary. Bind mounts are not flexible enough to make an image/container portable and the writable container layer has less performance than docker volumes.
You can list docker volumes with docker volume ls but you will not see bind mounted directories here. For that you will need to do docker inspect containername
"You could just copy one of the dockerfiles used by the postgres project, and remove the VOLUME statement. github.com/docker-library/postgres/blob/… –
Nick ODell
Nov 26, 2022 at 18:05"
answered Nick abow.
And that edited Dockerfile would build "almost" Docker Official Image.
Our project uses TimescaleDB on PostgreSQL in Azure Container Instance. But when I change the settings, for example, max_connections in the var/lib/postgresql/data/postgresql.conf file, the value returns to the previous state after the reboot (I used vi editor to modify). I am building a container via Dockerfile.
FROM timescale/timescaledb:latest-pg12
ENV POSTGRES_USER=admin \
POSTGRES_DB=timescaledb \
POSTGRES_PASSWORD=test123# \
Is there some environment variable to set these values? What is the best way to store the database, is it is possible to transfer DB to the Azure Storage?
If you are managing Dockerfile yourself, then you can fix modify postgresql.conf there. For example, add the following line in the Dockerfile, which will set max_connections to 100:
sed -r -i "s/[#]*\s*(max_connections)\s*=\s*0/\1 = 100/" /usr/local/share/postgresql/postgresql.conf.sample
Also I suggest to check Azure documentation on configuring PostgreSQL server. For example, to update max_connections server configuration parameter of server mydemoserver.postgres.database.azure.com under resource group myresourcegroup will be:
az postgres server configuration set --name max_connections --resource-group myresourcegroup --server mydemoserver --value 100
I'm trying to override a single item in the Postgres configuration of the official Docker Postgres image. Namely, I want to override the log_statement property and set it to all.
Tried it with:
docker run -d -e POSTGRES_PASSWORD=postgres postgres -c 'log_statement=all'
After enter the docker container and execute:
cat /var/lib/postgresql/data/postgresql.conf | grep log_statement
is still get the default value which is none.
but without success.
There are a few answers/questions regarding Postgres Docker configuration but they suggest replacing the complete postgresql.conf file.
From the docs:
Command-line options override any conflicting settings in postgresql.conf. Note that this means you won't be able to change the value on-the-fly by editing postgresql.conf, so while the command-line method might be convenient, it can cost you flexibility later.
As far as I understand that doesn't change the postgresql.conf but it actually runs the postgres command with the desired options. That's probably why you don't see the value set to all in the configuration file.
Hope it helps.
My solution is to mount Postgres config file somewhere on the server and edit it.
I test your question in my pc.
In the postgresql.conf log_statement="none" is commented.
Go manually and discomment it and it solve your problems.
docker exec -it postgres bash
Into the container:
apt update
apt install vim
vim /var/lib/postgresql/data/postgresql.conf
search what do you are looking for and change it manually.
Stop/start the container.
That is my solution.
I deploy a service on a standard Docker for AWS stack (using this template).
I deploy using docker stack deploy -c docker-compose.yml pos with this compose file:
version: "3.2"
services:
postgres_vanilla:
image: postgres
volumes:
- db-data:/var/lib/postgresql
volumes:
db-data:
driver: "cloudstor:aws"
driver_opts:
size: "6"
ebstype: "gp2"
backing: "relocatable"
I then change some data in the db and force an update of the service with docker service update --force pos_postgres_vanilla
Problem is that the data I change doesn't persist after the update.
I've noticed that postgres initdb script runs every time I update, so I assume it's related.
Is there something i'm doing wrong?
Issue was that cloudstor:aws creates the volume with a lost+found under it, so when postgres starts it finds that the data directory isn't empty and complains about it. To fix that I changed the volume to be mounted one directory above the data directory, at /var/lib/postgresql, but that caused postgres to not find the PGVERSION file, which in turn caused it to run initdb every time the container starts (https://github.com/docker-library/postgres/blob/master/11/docker-entrypoint.sh#L57).
So to work around it, instead of changing the volume to be mounted one directory above the data directory, I changed the data directory to be one level below the volume mount by overriding environment variable PGDATA (to something like /var/lib/postgresql/data/db/).
Is there any chance to pass variables from docker-compose into apache.conf file?
I have Dockerfile with variables
ENV APACHE_SERVER_NAME localhost
ENV APACHE_DOCUMENT_ROOT /var/www/html
I have apache.conf which I copy into /etc/apache2/sites-available/ while building image
ServerName ${APACHE_SERVER_NAME}
DocumentRoot ${APACHE_DOCUMENT_ROOT}
I have docker-compose.yml
environment:
- APACHE_SERVER_NAME=cms
- APACHE_DOCUMENT_ROOT=/var/www/html/public
When I run docker-compose, nothing happened and apache.conf in container is unchanged.
Am I completely wrong and is this imposible or am I missing any step or point?
Thank you
Let me explain some little differences among ways to pass environment variables:
Environment variables for building time and for entrypoint in running time
ENV (Dockerfile): Once specified in Dockerfile before building, containers, will have environment variables for entrypoints.
environment: The same than ENV but for docker-compose.
docker run -e VAR=value... The same than before but for cli.
Environment variables only for building time
ARG (Dockerfile): They won't appear in deployed containers.
Environment variables accesibled for every container and builds
.env file defined in your working dir where executes docker run or docker-compose.
env_file: section in docker-compose.yml file to define another .env file.
As you're trying to define variables for apache conf file, maybe you should try to use apache as entrypoint, or just define them in .env file.