postgres -c <parameter>=<value> not working for Postgres 11 in docker - postgresql

I'm running a Postgres inside a docker container. I want to change the default config of Postgres so I'm running:
docker container run -d postgres -c max_connections=200 -c shared_buffers=1GB -c effective_cache_size=3GB -c maintenance_work_mem=256MB -c checkpoint_completion_target=0.7 -c wal_buffers=16MB
But when I'm connecting to Postgres running:
docker exec -it container_name psql
And then the result of :
SHOW max_connections;
is
max_connections
-----------------
100
(1 row)
And it's not just max_connections. None of the parameters are changed. And I don't know what is the problem with what I'm doing?
Update: the result of
root=# SELECT *
root-# FROM pg_settings
root-# WHERE name = 'max_connections';
is
name | setting | unit | category | short_desc | extra_desc | context | vartype | source | min_val | max_val | enumvals | boot_val | reset_val | sourcefile | sourceline | pending_restart
-----------------+---------+------+------------------------------------------------------+----------------------------------------------------+------------+------------+---------+--------------------+---------+---------+----------+----------+-----------+------------------------------------------+------------+-----------------
max_connections | 100 | | Connections and Authentication / Connection Settings | Sets the maximum number of concurrent connections. | | postmaster | integer | configuration file | 1 | 262143 | | 100 | 100 | /var/lib/postgresql/data/postgresql.conf | 64 | f
(1 row)

If you cannot get it to work while starting the server, try ALTER SYSTEM:
psql -c "ALTER SYSTEM SET max_connections=200; SELECT pg_reload_conf()"
That changes the setting in postgresql.auto.conf.
Changing shared_buffers, wal_buffersandmax_connections` require a restart of the PostgreSQL server, the other parameters can be changed on the fly.

Is it possible that you are connecting to the wrong container? When I try to run psql as you have, I get:
$ docker exec -it boring_hermann psql
psql: FATAL: role "root" does not exist
...because the standard user is root and has no access to the container's DB. When I run as user 999, group 999 (the one listed as postgres in that image), it works correctly:
$ docker exec -u 999:999 -it boring_hermann psql
psql (11.1 (Debian 11.1-1.pgdg90+1))
Type "help" for help.
postgres=# show max_connections;
max_connections
-----------------
200
(1 row)

Related

pgpool/postgres - replication_delay is too high, how to reset?

in our setup the show pool_nodes shows a very high replication_delay and it keeps increasing, becuase of which any new queries are not replicated in the slave
following is the output of show pool_nodes command, is there a way to reset this, data loss if fine as this is not a live/production system.
[root#DB2 ~]# psql -h DB-HA-Hostname -U postgres -p 5432 -c 'show pool_nodes'
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | last_status_change
---------+--------------------------------------+------+--------+-----------+---------+------------+-------------------+-------------------+---------------------
0 | DB1-hostname | 5432 | up | 0.500000 | primary | 0 | true | 0 | 2021-01-11 19:32:00
1 | DB2-hostname | 5432 | up | 0.500000 | standby | 0 | false | 54986528 | 2021-01-11 19:32:00
(2 rows)
I have tried, restarting nodes, restarting pgpool, restarting postgresql , deleting database etc.. but no luck. As soon as the slave gets attached the replication_delay is high again..
You can run this command to check status of replication:
psql -h DB-HA-Hostname -U postgres -p 5432 -c "select * from pg_stat_replication" -x
if it shows:
if not, the configuration has failed.
You can show your configuration ?
Check the replication is running or not if it's not running re-configure the standby then attached the nodes
select * from pg_stat_replication;
after taking basebackup start the postgresql server then pcp-attach-node on pgpool

Upgrading Postgres from 10 to 12 -- problem with encodings

I'm trying to upgrade a postgres server from 10 12, and am having problems with the encodings. I'm following what I believe to be established recipes.
Behold:
postgres#serverbot:~$ psql -l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-------------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | postgres=CTc/postgres+
| | | | | =c/postgres
thingsboard | postgres | SQL_ASCII | C | C | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | nagios=c/postgres
(4 rows)
Notice the encodings on the 10 database. Time to create the database for 12.
postgres#serverbot:~$ sudo service postgresql stop
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/initdb -E SQL_ASCII --locale=C -D /var/lib/postgresql/12/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /var/lib/postgresql/12/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/New_York
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
Excellent! Let's fire up the new server.
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
waiting for server to start.... done
server started
And verify the encodings...
postgres#serverbot:~$ psql -l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
Everything matches... time to upgrade!
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_upgrade \
--old-datadir=/var/lib/postgresql/10/main \
--new-datadir=/var/lib/postgresql/12/main \
--old-bindir=/usr/lib/postgresql/10/bin \
--new-bindir=/usr/lib/postgresql/12/bin \
--old-options '-c config_file=/etc/postgresql/10/main/postgresql.conf' \
--new-options '-c config_file=/etc/postgresql/12/main/postgresql.conf' \
--link --check
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for tables WITH OIDS ok
Checking for invalid "sql_identifier" user columns ok
encodings for database "postgres" do not match: old "SQL_ASCII", new "UTF8"
Failure, exiting
postgres#serverbot:~$
Doh!
What is wrong here? I assert that the encodings do match, and I'm stuck.
Can anyone offer any advice?
What I see is:
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
waiting for server to start.... done
server started
and then:
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_upgrade \
--old-datadir=/var/lib/postgresql/10/main \
--new-datadir=/var/lib/postgresql/12/main \
[...]
Note the $PGDATA directories. The database cluster you did the initdb for is different from the one you are doing the pg_upgrade to.
UPDATE. Since you seem to be using a Debian based OS and it's Postgres packaging, it might work better to stick with the packaging tools:
sudo pg_createcluster --locale=C 12 ascii
Creating new PostgreSQL cluster 12/ascii ...
/usr/lib/postgresql/12/bin/initdb -D /var/lib/postgresql/12/ascii --auth-local peer --auth-host md5 --locale C
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/12/ascii ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/Los_Angeles
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctlcluster 12 ascii start
Ver Cluster Port Status Owner Data directory Log file
12 ascii 5434 down postgres /var/lib/postgresql/12/ascii /var/log/postgresql/postgresql-12-ascii.log
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
Also I would use pg_upgradecluster, see here for more information. This keeps everything in the same system.

Docker file postgres initial schemas not executed

I updated my docker file to version 3 but now the inicial schemas are not being created. i already tried with a different volume
I run it with : docker-compose -f docker-compose.yml up
version: '3'
services:
db-service:
image: postgres:11.2
volumes:
- ./dbscripts/:/docker-entrypoint-initdb.d/
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=sample
ports:
- 5432:5432
Check these 2 things:
The init scripts are triggered only on the first deploy. On the subsequent docker-compose ups you probably see as well on the logs the following message:
db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
To force the re-initialization, you can wipe the data volume(This will delete the entries, not only the schema).
To find the path, simply docker inspect <postgres-container-id> and look for the HostConfig.Binds
The script executes till the first type "longvarbinary" does not exist error. You will need to fix this.
db-service_1 | CREATE TABLE
db-service_1 | 2020-07-26 19:29:34.121 UTC [68] ERROR: type "longvarbinary" does not exist at character 68
I opened a console in the postgres container and your sample database is there, as well as a table that is created before the .sql error occurs in the script:
docker exec -ti 22b0bb87561f sh
# psql -U postgres
psql (11.2 (Debian 11.2-1.pgdg90+1))
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
sample | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres=# \c sample
You are now connected to database "sample" as user "postgres".
sample=# \dt
List of relations
Schema | Name | Type | Owner
--------+----------------------+-------+----------
public | oauth_client_details | table | postgres
(1 row)
I ran into the same issue, and I figured it was because I forgot to clear the volume when I take down docker compose with command docker-compose down.
The solution that helps is when you take down docker compose, you specify that all volumes must be removed docker-compose down --volumes. Also, make sure to check and delete any related volumes using docker volume ls.
WARNING: Any subsequent command docker-compose down should not have the flag --volume or else your persistent data will be wiped out. This may or may not be desirable depending on your use case. The idea is to ONLY run the init script on the very first time you run docker-compose up. Any subsequent command up after that will skip initialization because data is already populated, and you don't want to override it.
Hope it helps!

GitLab setup using official PostgreSQL and GitLab docker container not working

I am trying to setup GitLab using separate docker containers for both GitLab and PostgreSQL. I am using RancherOS v1.0.3 with Kubernetes enabled. For now, all i want is to have a single node having both the containers. Later, i will look into configuring them on separate nodes.
[rancher#rancher-agent-2 ~]$ cat postgresql.sh
docker run --name=postgresql -d \
--env 'DB_NAME=gitlabhq_production' \
--env 'DB_USER=gitlab' --env 'DB_PASS=password' \
--env 'DB_EXTENSION=pg_trgm' \
--volume /opt/postgresql:/var/lib/postgresql \
postgres:9.6.3
[rancher#rancher-agent-2 ~]$ cat gitlab.sh
docker run --name=gitlab -d --link postgresql:postgresql \
-v /opt/gitlab/data:/var/opt/gitlab \
gitlab/gitlab-ce:9.3.9-ce.0
Now when i run the PostgreSQL container and try logging in, i get errors.
[rancher#rancher-agent-2 ~]$ docker exec -it postgresql bash
root#a6cef780c594:/# psql -d gitlabhq_production -U gitlab
psql: FATAL: role "gitlab" does not exist
Seems the db (gitlabhq_production) has not been created.
root#a6cef780c594:/# psql -U postgres
psql (9.6.3)
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
Why is it not creating the db despite passing the parameters?
UPDATE:
PostgreSQL container log:
[rancher#rancher-agent-2 ~]$ docker logs postgresql
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
...
...
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
done
server started
ALTER ROLE
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
...
...
You are using the wrong environment vars names:
DB_NAME should be POSTGRES_DB
DB_USER should be POSTGRES_USER
DB_PASS should be POSTGRES_PASSWORD
Check https://hub.docker.com/_/postgres/

Problems accessing databases on remote postgresql sever

I have a postgresql (9.1.4) server running on a remote machine (Ubuntu 12.04), and I'm having trouble accessing it on my local machine. Specifically, on my local machine, I can access the remote server using only the username "postgres" and the database name "postgres", even though I have other user names and databases. First some of the background information. While in the interactive environment invoked with "psql" on the remote machine, I created another user, called "mxtxdb", and set the password for that user. I also created a database called "mxtxdb". To demonstrate their existence, I logged onto the remote machine and ran:
sudo su postgres
Password:
postgres#myhost:~$ psql
psql (9.1.4)
Type "help" for help.
postgres=# \du
List of roles
Role name | Attributes | Member of
-----------+------------------------------------------------+-----------
mxtxdb | | {}
postgres | Superuser, Create role, Create DB, Replication | {}
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
mxtxdb | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | mxtxdb=CTc/postgres
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
Furthermore, while still on the remote machine, I can access the database from the "mxtxdb" postgresql account:
postgres#myhost:~$ psql -h localhost -U mxtxdb -d mxtxdb
Password for user mxtxdb:
psql (9.1.4)
Type "help" for help.
mxtxdb=>
But when I try to do the same on the local machine, I get this:
psql -h <server's address> -U postgres -d "postgres" -p 5432
Password for user postgres:
psql (9.1.4)
Type "help" for help.
postgres=#
and
psql -h <server's address> -U "postgres" -d "mxtxdb" -p 5432
Password for user postgres:
psql: FATAL: database "mxtxdb" does not exist
and
psql -h <server's address> -U "mxtxdb" -d "mxtxdb" -p 5432
Password for user mxtxdb:
psql: FATAL: role "mxtxdb" does not exist
The last two lines of my pg_hba.conf file are:
host all mxtxdb 0.0.0.0/0 trust
host all postgres 0.0.0.0/0 trust
and the listen_addresses is set to '*' (and uncommented) in the postgresql.conf file.
Any idea why I cannot access other accounts or databases remotely, but I can when ssh'd into the remote machine?
First of all, thanks to Craig Richard for your suggestions, and in fact you were both correct. Here's what happened: I installed PostgreSQL using sudo apt-get install postgresql-9.1, and then I followed the instructions for configuring the server at http://www.postgresql.org/docs/9.1/interactive/runtime.html . Unfortunately, what I did not notice was that the apt-get install process creates a postgresql database cluster by default, so when I followed the instructions to create a new one, I then had two clusters, one at /var/lib/postgresql/9.1/main and one at /usr/local/pgsql/data, and I was interacting with different clusters when I logged in locally vs remotely. When I removed the second cluster, the issue was resolved and I was able to connect remotely.
My takeaways from this process: when installing postgresql with apt-get, and then following the instructions in the postgresql manual, be aware that the install process will create a postgres operating system user by default, and will also create a database cluster by default, so those steps in the manual do not need to be performed. If anyone experiences an error like:
"FATAL: role/database '<name>' does not exist"
when you are sure you've created that user or database, check to ensure that you're operating with the correct database cluster.
Thanks again!