When I run the command on the database server it is returning the syntax error. Command:
psql -h xxxxx -p 5432 -U user -W -d "dbname=db_replication replication=true" -c "IDENTIFY_SYSTEM;"
command response
ERROR: syntax error at or near "IDENTIFY_SYSTEM"
LINE 1: IDENTIFY_SYSTEM;
Postgres version is 14.4.
which according to the documentation is a supported version.
running the command on a docker I raised
systemid | timeline | xlogpos | dbname
---------------------+----------+-----------+--------
9999999999999999999 | 1 | 0/FFFFFFF | replic
Related
I want to unit-test my app which uses a postgres database inside a docker.
EDIT: based on the suggested answer I modified the Dockerfile:
FROM postgres
USER postgres
RUN sleep 2 # remark 1
RUN initdb # remark 2
RUN sleep 3 # remark 1
RUN psql --host=localhost -l
The remarks are:
from this reference
Try putting a sleep in there and see if it's still a problem
from the docs:
The default postgres user and database are created in the entrypoint with initdb.
Here is the Dockerfile from the original question:
FROM postgres
COPY input.json .
RUN createdb -h localhost -p 7654 -U moish myLovelyAndTemporaryDb
#
# [ 1 ] run application on input.json
# [ 2 ] check db content after run
#
When I use the above Dockerfile I seem to be missing something:
(The errors from the edited version are the same)
$ docker build --tag host --file Dockerfile .
[+] Building 0.3s (7/7) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 125B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/postgres:latest 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 40B 0.0s
=> CACHED [1/3] FROM docker.io/library/postgres 0.0s
=> [2/3] COPY input.json . 0.0s
=> ERROR [3/3] RUN createdb -h localhost -p 7654 -U moish myLovelyAndTemporaryDb 0.2s
------
> [3/3] RUN createdb -h localhost -p 7654 -U moish myLovelyAndTemporaryDb:
#7 0.188 createdb: error: connection to server at "localhost" (127.0.0.1), port 7654 failed: Connection refused
#7 0.188 Is the server running on that host and accepting TCP/IP connections?
#7 0.188 connection to server at "localhost" (::1), port 7654 failed: Cannot assign requested address
#7 0.188 Is the server running on that host and accepting TCP/IP connections?
------
Postgres database starts only after you create a container based on postgres image. docker build process doesn't start entrypoint script. You might need a bash script or CI pipeline where you firstly start postgres container and then use it in your unit tests
$ docker run --name mypg -p 5432:5432 -e POSTGRES_PASSWORD=mypgpass -d postgres:9
# copy a script to the mypg container
$ docker cp run.sh mypg:/root/run.sh
# run the script
$ docker exec mypg bash /root/run.sh
...
# use postgres client on your host to connect to mypg container
$ PGPASSWORD="mypgpass" psql -U postgres -p 5432 -h localhost -c "select version()"
version
--------------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 9.6.24 on x86_64-pc-linux-gnu (Debian 9.6.24-1.pgdg90+1), compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bit
(1 row)
Postgres container docs
Postgres client authentication
EDIT:
By trying to run initdb, psql etc directly in Dockerfile, you are reinventing the docker-entrypoint.sh
During the build step of the postgres Docker you cannot run postgres commands. Postgres database will only be available after you run the Docker.
As specified in the postgres Docker documentation you can add customization to your postgres instance through scripts placed in docker-entrypoint-initdb.d directory.
Alternatively you could use a RUN directive to start the postgres database and after that run the postgres commands you want (making sure to wait for the DB to accept connections), as mentioned here.
Another side note, I personally avoid using real databases for unit testing applications. To me, it's always better to mock the database for unit tests, in python you can do this with unittest.mock.
Here is a complete answer based on the concepts of slava-kuravsky and mello:
$ docker build --tag host --file Dockerfile .
$ docker run -d -t --name pg host
$ docker exec pg bash run.sh
The script can do what I want, currently it only lists the databases:
$ cat run.sh # <--- copied during docker build
echo "Hello Postgres World"
psql --host=localhost -l
The Dockerfile does only initialization:
$ cat Dockerfile
FROM postgres
USER postgres
COPY run.sh . # <--- the testing script
RUN sleep 2 # <--- probably not needed anymore
RUN initdb
RUN sleep 3 # <--- probably not needed anymore
When I perform the three commands above I get what I expect 🙂 :
$ docker build --tag host --file Dockerfile .
[+] Building 7.2s (10/10) FINISHED
# ... omitted for brevity ...
$ docker run -d -t --name pg host
608ac7324e838924c9a5d0cfe65c8000e33350b86faf9df4511ef5fcf7440597
$ docker exec pg bash run.sh
Hello Postgres World
List of databases
Name | Owner | Encoding | Collate | Ctype | ICU Locale | Locale Provider | Access privileges
-----------+----------+----------+------------+------------+------------+-----------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 | | libc |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | | libc | =c/postgres +
| | | | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | | libc | =c/postgres +
| | | | | | | postgres=CTc/postgres
(3 rows)
Trying to make batch file that will get query from script file and put results into csv.
Batch file code:
psql -h host -d test -U test -p 5432 -a -q -f C:\Users\test\Documents\my_query.sql TO STDOUT WITH CSV HEADER DELIMITER ';' > C:\Users\test\Documents\res.csv
In result file I'm getting result like this:
select *
from public.test
limit 3
id | name | count_01
----------+------------+---------------+
11021555 | a | 1 |
39534568 | b | 2 |
11695210 | c | 3 |
(3 rows)
How to get only script results without rows count and symbols like '|' or '+' and using ';' delimetres as in the usual csv file?
Working script:
psql -h host -d test -U test -p 5432 -q --quiet --no-align --field-separator=';' --file=C:\Users\test\Documents\my_query.sql --output=C:\Users\test\Documents\res.csv
From PostgreSQL v12 on, you can use the CSV output format of psql:
psql --quiet --csv --file=my_query.sql --output=res.csv
--quiet suppresses the psql welcome message.
Should work with
psql -h host -d dbname -U user -p port -a -q -f my_query.sql -o res.csv --record-separator=',' --csv
I'm having an issue getting a database restore using pg_restore to work. I've got Postgres 12.2 installed on my Mac running High Sierra (10.13.6). It was installed with Brew.
I created a simple database "foo" with table "foo" to test this out:
Nates-MacBook-Pro:k8s natereed$ psql -d foo
psql (12.2, server 12.1)
Type "help" for help.
foo=# SELECT table_schema,table_name
FROM information_schema.tables
WHERE table_schema='public' ORDER BY table_schema,table_name;
table_schema | table_name
--------------+------------
public | foo
(1 row)
Generate dump:
pg_dump -Fc foo -f foo.backup
When I try to restore, nothing happens. pg_restore just hangs:
pg_restore -h localhost -p 5432 -U postgres -v -Fc -f foo.backup
I've tried various permutations of this command, but every time it just hangs. In Activity Monitor, I see the process but it is not using any CPU.
Online help says:
pg_restore restores a PostgreSQL database from an archive created by pg_dump.
Usage:
pg_restore [OPTION]... [FILE]
General options:
-d, --dbname=NAME connect to database name
-f, --file=FILENAME output file name
-F, --format=c|d|t backup file format (should be automatic)
-l, --list print summarized TOC of the archive
-v, --verbose verbose mode
-V, --version output version information, then exit
-?, --help show this help, then exit
For pg_restore -f is parameter for the output file: it is not the input file. Remove -f in your example:
pg_restore -h localhost -p 5432 -U postgres -v -Fc foo.backup
I want to build a custom Postgres11 image in which some users are created and some extensions are installed. As I want these to be created at build time, so I don't want to use docker-entrypoint-initdb.d. A next step would be to restore a sql dump as well.
FROM postgres:11
ENV PG_MAJOR 11
ENV POSTGISV 2.5
ENV TZ Europe/Brussels
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
postgresql-$PG_MAJOR-postgis-$POSTGISV \
postgresql-$PG_MAJOR-postgis-$POSTGISV-scripts
USER postgres
RUN initdb && pg_ctl -o "-c listen_addresses='*'" start &&\
psql -h 0.0.0.0 --command "CREATE USER docker WITH SUPERUSER PASSWORD 'docker';" &&\
psql -h 0.0.0.0 --command "CREATE USER akela_test WITH PASSWORD 'akela';" &&\
createdb -E UTF8 -U postgres -h 0.0.0.0 -O akela_test akela_test --template template0 &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c 'CREATE EXTENSION IF NOT EXISTS "hstore";' &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c 'CREATE EXTENSION IF NOT EXISTS "postgis";' &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp";' &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c "CREATE ROLE akela_db WITH LOGIN PASSWORD 'akela'" &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c "GRANT ALL PRIVILEGES ON DATABASE akela_test to akela_db" &&\
psql -U postgres -d akela_test -h 0.0.0.0 -c "CREATE schema db" &&\
pg_ctl stop
# gunzip -c /tmp/dump.sql.gz | psql -U akela -h 0.0.0.0 akela
USER root
seems to work:
...
CREATE SCHEMA
ALTER SCHEMA
CREATE ROLE
GRANT
CREATE SCHEMA
ALTER SCHEMA
waiting for server to shut down....2019-07-08 12:58:06.962 CEST [22] LOG: received fast shutdown request
2019-07-08 12:58:06.964 CEST [22] LOG: aborting any active transactions
2019-07-08 12:58:06.965 CEST [22] LOG: background worker "logical replication launcher" (PID 29) exited with exit code 1
2019-07-08 12:58:06.965 CEST [24] LOG: shutting down
2019-07-08 12:58:07.006 CEST [22] LOG: database system is shut down
done
server stopped
...
running the image however shows no users nor db:
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
postgres=# \du
List of roles
Role name | Attributes | Member of
-----------+------------------------------------------------------------+-----------
postgres | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
What could be the issue?
The Dockerfile for postgres defines a volume which means any changes to this directory by a RUN step will be discarded. To make changes to this directory you need to do one of the following options:
Make the changes at run time rather than doing the build, and save the resulting volume.
Make changes during the build, but in a different directory. This would require changing the postgres configuration to use the different directory.
Save your changes to a different directory and then restore those changes when you start the container (see the save and load volume scripts for an example of this).
Build your own postgres image without the volume definition.
I'm trying to set up two databases on travis but it just stops halfway the before_install stating:
(END)
No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated
eg https://travis-ci.org/B3Partners/brmo/builds/85746119
my yml is the following:
language: java
sudo: false
branches:
only:
- travis-integration
addons:
postgresql: "9.3"
jdk:
# - openjdk6
# - openjdk7
- oraclejdk7
# - oraclejdk8
matrix:
fast_finish: true
cache:
directories:
- $HOME/.m2
before_install:
# STAGING
- psql --version
- psql -U postgres -c 'create database staging'
- psql -U postgres -c 'create database rsgb'
- psql -U postgres --list
# set up RSGB
- psql -U postgres -d rsgb -c 'create extension postgis'
- psql -U postgres -d rsgb -f ./datamodel/generated_scripts/datamodel_postgresql.sql --single-transaction --echo-all
# - psql -f ./datamodel/utility_scripts/111a_update_gemeente_geom.sql -U postgres -d rsgb --single-transaction
# - psql -f ./datamodel/utility_scripts/113a_update_wijk_geom.sql -U postgres -d rsgb --single-transaction
install:
# install all dependencies + artifacts without any testing
- mvn install -Dmaven.test.skip=true -B -V -fae -q
before_script:
# dit dient na afloop van de 'install' gedaan te worden omdat de staging DB sql gegenereerd wordt door Hibernate
- psql -U postgres -d staging -f ./brmo-persistence/target/ddlscripts/create-brmo-persistence-postgresql.sql --single-transaction
- psql -U postgres -d staging -f ./brmo-persistence/db/01_create_indexes.sql
- psql -U postgres -d staging -f ./brmo-persistence/db/02_insert_default_user.sql
- psql -U postgres -d staging -f ./brmo-persistence/db/03_update_status_enum_value.sql
# run tests
script:
# run unit tests
- mvn -e test -B
# run integration tests
- mvn -e verify -B
after_success:
after_failure:
after_script:
notifications:
email: false
# on_success: [always|never|change] # default: change
# on_failure: [always|never|change] # default: always
and as you can see in the log it just stalls after a few psql calls.
0.01s$ psql --version
psql (PostgreSQL) 9.3.5
before_install.2
0.02s$ psql -U postgres -c 'create database staging'
CREATE DATABASE
before_install.3
0.22s$ psql -U postgres -c 'create database rsgb'
CREATE DATABASE
before_install.4
1.04s$ psql -U postgres -d rsgb -c 'create extension postgis'
CREATE EXTENSION
$ psql -U postgres --list
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
rsgb | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
staging | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
travis | travis | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
(6 rows)
(END)
No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated
I just spent about 3 hours troubleshooting this same issue and the problem is pretty simple once you understand why. psql is simply trying to page the output. There are multiple ways to disable the pager, but the solution I went with was to set the PAGER=cat environment variable in .travis.yml like so:
env:
- PGUSER=postgres
PAGER=cat