Remote control on my postgresql container docker works but can't see tables creates - postgresql

I am new in docker, I've created a dockerfile to launch postgresql image and insert data in a new database.
When I run docker it works well in its container, I can see database and tables created.
But When I try to connect from a client postgres, I see the database created but not tables and we dont know why me and my team..
My DockerFile:
# Dockerfile
FROM postgres:9.6.5
RUN localedef -i fr_FR -c -f UTF-8 -A /usr/share/locale/locale.alias fr_FR.UTF-8
ENV LANG fr_FR.utf8
COPY postgresql.conf /
COPY sql/structure.sql /docker-entrypoint-initdb.d/
My file SQL structure.sql:
CREATE DATABASE DB_ALLOT_BOUCHON_NEW;
GRANT ALL PRIVILEGES ON DATABASE DB_ALLOT_BOUCHON_NEW TO postgres;
CREATE Table IF NOT EXISTS public.SUBSCR (
ID BIGINT NOT NULL,
subscriber_name VARCHAR(100) NOT NULL,
sp_array VARCHAR(100) NOT NULL,
PRIMARY KEY (ID)
);
INSERT INTO SUBSCR (ID, subscriber_name, sp_array) VALUES (1, 'client_1', 'sp_array client_1 client_2');
INSERT INTO SUBSCR (ID, subscriber_name, sp_array) VALUES (2, 'client_2', 'sp_array client_2 client_2');
Command to build and run my dockerfile:
$ docker build -t allot_psql .
$ docker run -d --name custom_psql_running -p 5467:5432 allot_psql -c config_file=postgresql.conf
$ docker logs custom_psql_running
$ docker run -it --rm --link custom_psql_running postgres psql -h custom_psql_running -U postgres
# postgres=# SELECT * from subscr;
Result in my container:
postgres=# select * from subscr;
id | subscriber_name | sp_array
----+-----------------+----------------------------
1 | client_1 | sp_array client_1 client_2
2 | client_2 | sp_array client_2 client_2
(2 rows)
postgres=# \d
List of relations
Schema | Name | Type | Owner
--------+--------+-------+----------
public | subscr | table | postgres
(1 row)
postgres=# \d*
Invalid command \d*. Try \? for help.
postgres=# \dt
List of relations
Schema | Name | Type | Owner
--------+--------+-------+----------
public | subscr | table | postgres
(1 row)
postgres=# \d+
List of relations
Schema | Name | Type | Owner | Size | Description
--------+--------+-------+----------+------------+-------------
public | subscr | table | postgres | 8192 bytes |
(1 row)
postgres=# \d
List of relations
Schema | Name | Type | Owner
--------+--------+-------+----------
public | subscr | table | postgres
(1 row)
postgres=# \
Invalid command \. Try \? for help.
postgres=# \q
Everything seems to work well but when I use postgresql client, I can see only the database created but not tables inside.
jdbc:postgresql://localhost:5467/db_allot_bouchon_new
username:postgres

The structure.sql script creates the tables in the default database postgres and not in DB_ALLOT_BOUCHON_NEW. When the container is initialized you are connected to postgres.To quickly fix connect to db_allot_bouchon_new right after you create it
CREATE DATABASE DB_ALLOT_BOUCHON_NEW;
\connect db_allot_bouchon_new
...
This may help as well.

I will suggest to set POSTGRES_DB environment variable in your Dockerfile so this will create DB and Container will treat this DB as a default DB.
POSTGRES_DB
This optional environment variable can be used to define a different
name for the default database that is created when the image is first
started. If it is not specified, then the value of POSTGRES_USER will
be used.
Dockerfile
FROM postgres:latest
RUN localedef -i fr_FR -c -f UTF-8 -A /usr/share/locale/locale.alias fr_FR.UTF-8
ENV LANG fr_FR.utf8
ENV POSTGRES_DB="db_allot_bouchon_new"
# To set user and pass but better to pass at run time
ENV POSTGRES_USER="appdbuser"
ENV POSTGRES_PASSWORD="123123"
COPY sql/structure.sql /docker-entrypoint-initdb.d/
In the above Dockerfile, this will create user appdbuser.
docker exec -it custom_psql_running bash -c " psql -U appdbuser"
So you will not need to mention these two lines in SQL script as Docker image will do that for you. So better to remove these line.
CREATE DATABASE DB_ALLOT_BOUCHON_NEW;
GRANT ALL PRIVILEGES ON DATABASE DB_ALLOT_BOUCHON_NEW TO postgres;
Warning:
scripts in /docker-entrypoint-initdb.d are only run if you start the
container with a data directory that is empty; any pre-existing
database will be left untouched on container startup. One common
problem is that if one of your /docker-entrypoint-initdb.d scripts
fails (which will cause the entrypoint script to exit) and your
orchestrator restarts the container with the already initialized data
directory, it will not continue on with your scripts.

Related

Validating the data inside postgres table

I have launched a postgres container. By injecting script.sql at docker entrypoint, I created database, schema, tables and have inserted data into them. Docker Logs says that all table creation and data insertion is successful .
But How can I validate the data insertion? Below commands didn't help
List of relations
Schema | Name | Type | Owner
------------+----------------------+-------+----------
my_db | users | table | postgres
my_db | audit_log | table | postgres
my_db | config | table | postgres
(3 rows)
my_db=# SELECT * FROM my_db.users
my_db-# SELECT * FROM users
my_db-# SELECT * FROM my_db.users;
What is wrong here? Please help.
You should use
\c my_db
to connect to your database. And then:
SELECT * FROM users;
To query your table into your database.
It seems that you understood this part, but everything should be done using postgresql command line, so inside the docker container.

Docker file postgres initial schemas not executed

I updated my docker file to version 3 but now the inicial schemas are not being created. i already tried with a different volume
I run it with : docker-compose -f docker-compose.yml up
version: '3'
services:
db-service:
image: postgres:11.2
volumes:
- ./dbscripts/:/docker-entrypoint-initdb.d/
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=sample
ports:
- 5432:5432
Check these 2 things:
The init scripts are triggered only on the first deploy. On the subsequent docker-compose ups you probably see as well on the logs the following message:
db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
To force the re-initialization, you can wipe the data volume(This will delete the entries, not only the schema).
To find the path, simply docker inspect <postgres-container-id> and look for the HostConfig.Binds
The script executes till the first type "longvarbinary" does not exist error. You will need to fix this.
db-service_1 | CREATE TABLE
db-service_1 | 2020-07-26 19:29:34.121 UTC [68] ERROR: type "longvarbinary" does not exist at character 68
I opened a console in the postgres container and your sample database is there, as well as a table that is created before the .sql error occurs in the script:
docker exec -ti 22b0bb87561f sh
# psql -U postgres
psql (11.2 (Debian 11.2-1.pgdg90+1))
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
sample | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres=# \c sample
You are now connected to database "sample" as user "postgres".
sample=# \dt
List of relations
Schema | Name | Type | Owner
--------+----------------------+-------+----------
public | oauth_client_details | table | postgres
(1 row)
I ran into the same issue, and I figured it was because I forgot to clear the volume when I take down docker compose with command docker-compose down.
The solution that helps is when you take down docker compose, you specify that all volumes must be removed docker-compose down --volumes. Also, make sure to check and delete any related volumes using docker volume ls.
WARNING: Any subsequent command docker-compose down should not have the flag --volume or else your persistent data will be wiped out. This may or may not be desirable depending on your use case. The idea is to ONLY run the init script on the very first time you run docker-compose up. Any subsequent command up after that will skip initialization because data is already populated, and you don't want to override it.
Hope it helps!

Importing postgres database in a docker postgres container

I am trying to import an existing database into a postgres docker container.
This is how I proceed:
docker run --name pg-docker -e POSTGRES_PASSWORD=***** -d -p 5432:5432 -v BCS/postgres_data postgres
Then
docker exec -it pg-docker bash
psql -U postgres
postgres=# CREATE DATABASE myDB;
psql -U postgres myDB < BCS/mydb.sql
but when i execute the command \dt I have this error Did not find any relations.
knowing that my database has already tables.
So waht I am doing wrong?
First thing better to go with the approach that is mentioned by #LinPy.
Or better to copy at build time.
Dockerfile
FROM postgres
COPY mydb.sql /docker-entrypoint-initdb.d/
Another option, you do need to script for only DB creation.
FROM postgres
ENV POSTGRES_DB=mydb
The will create DB for you.
POSTGRES_DB
This optional environment variable can be used to define a different
name for the default database that is created when the image is first
started. If it is not specified, then the value of POSTGRES_USER will
be used.
In the above, the Postgres entrypoint will take care of the SQL script.
Second thing, the current issue with database name, Postgress will not treat them in uppercase simply, unless you did some trick.
Postgresql treats the db name as lowercase, normalising. However, the
field in the postgresapi does not replicate this behaviour, thus
allowing you to create a database with Capital letters. The fix could
be to warn the user that no uppercase letters are allowed in the db
aname and to add in a validation rule to the API to stop a user
creating such a database.
postgres-api
Change your command to
create DB
docker exec -it pg-docker bash
psql -U postgres
postgres=# CREATE DATABASE myDB;
verfiy DB
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
mydb | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
so the import command will be
psql -U postgres mydb < BCS/mydb.sql
or
psql -d mydb -U postgres -f ab.sql
You can also have your sql scripts being redirected to the container like that:
Postgres user:
docker exec -i my-postgres-container psql -U postgres < created-db.sql
Regular user:
docker exec -i my-postgres-container psql -d my-db -U my-user < create-schema.sql
If you are sure that the database installed everything correctly, are you are still not seeing tables, there are 2 things you should double-check:
When you connect, are you connecting to the right database? If you are using psql in a terminal, the database is specified with the -d switch.
psql -h <host> -U <user> -d <dbname>
You can also change your database after you connect using the \connect <dbname> command.
Are you specifying the right schema? \dt will show you tables, but you need to specify a schema first using set schema:
postgres=# \dt
...
<no tables>
...
postgres=# set schema 'my_schema';
postgres=# \dt
...
<my tables>
...

Docker compose postgresql service - can't create user and database during build?

I have wasted an entire day on this, and to say I'm not impressed by the unnecessary complexity of what should be a simple task - would be a gross understatement.
Ok, having got that off my chest, I am building a django application using docker-machine, docker-compose, postgresql and redis - by following this tutorial.
I have managed to get the basic tutorial to work - but it does not suit my needs, as I need to create a user and a database for my application - as opposed to using 'postgres' for both.
I have used the answer from #dnephin to a similar question, and modified my code as follows:
I created a new Dockerfile in a new directory ./database/:
FROM postgres:9.6
COPY . /fixtures
WORKDIR /fixtures
RUN /fixtures/setup.sh
./database/setup.sh contents:
#!/bin/bash
set -e
pg_createcluster 9.6 main --start
/etc/init.d/postgresql start
su - postgres # makes no effing difference ...
psql -f create_fixtures.sql
/etc/init.d/postgresql stop
./database/create_fixtures.sql contents:
CREATE DATABASE mydatabase WITH ENCODING 'UTF8';
CREATE USER webuser ENCRYPTED PASSWORD 'deadbeefsnaf0' NOSUPERUSER NOCREATEDB NOCREATEROLE;
GRANT ALL PRIVILEGES ON mydatabase TO webuser;
and finally my postgres service in the docker_compose.yml is modified to use build:
postgres:
build: ./database/
...
When I run docker-compose build, the build goes through the motions and then barfs at where I'm importing the SQL fixtures file via psql:
frothing#themouth:~/path/to/directory$ docker-compose build
redis uses an image, skipping
Building postgres
Step 1/4 : FROM postgres:9.6
---> ff0943ecbb3c
Step 2/4 : COPY . /fixtures
---> fae19dc88da8
Removing intermediate container 84b860aee55c
Step 3/4 : WORKDIR /fixtures
---> aa88438dc69f
Removing intermediate container b801ddc3b374
Step 4/4 : RUN /fixtures/setup.sh
---> Running in ca3e89ec2460
Creating new cluster 9.6/main ...
config /etc/postgresql/9.6/main
data /var/lib/postgresql/9.6/main
locale en_US.utf8
socket /var/run/postgresql
port 5432
Starting PostgreSQL 9.6 database server: main.
psql: FATAL: role "root" does not exist
ERROR: Service 'postgres' failed to build: The command '/bin/sh -c /fixtures/setup.sh' returned a non-zero code: 2
I tried to solve this using the useless documentation on docker for postgresql service - but got no where.
How can I solve this?
Volumes are not available at build time. You can create /var/lib/postgresql/data in your script but it will be overwritten by VOLUME /var/lib/postgresql/data from postgres:9.6 image.
In your case: just use the following docker file:
FROM postgres:9.6
COPY ./create_fixtures.sql /docker-entrypoint-initdb.d/create_fixtures.sql
They get automatically executed once the container starts. Here is an example:
$ docker run -d --name mydb -p 33306:3306 yourtag
$ docker exec -ti mydb psql -U postgres
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
------------+----------+----------+------------+------------+-----------------------
mydatabase | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | webuser=CTc/postgres
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
Outdated answer:
Your script should work on a container except in the fixture you have to execute psql like this:
su postgres -c "psql -f create_fixtures.sql"
su --login postgres does not work because postgres can't open a bash or shell. You can try around with docker run --rm -ti postgres:9.6 bash.
Sorry I have to tell you there is one more error in your sql script: GRANT ALL PRIVILEGES ON DATABASE mydatabase TO webuser - the keyword DATABASE is necessary here.
Here is a complete log how I tested and can confirm this works:
docker run --rm -ti postgres:9.6 bash
root#be03ab1eb704:/# cat > test.sql <<EOF
> CREATE DATABASE mydatabase WITH ENCODING 'UTF8';
> CREATE USER webuser ENCRYPTED PASSWORD 'asdf123' NOSUPERUSER NOCREATEDB NOCREATEROLE;
> GRANT ALL PRIVILEGES ON DATABASE mydatabase TO webuser;
> EOF
root#be03ab1eb704:/# pg_createcluster 9.6 main --start
Creating new PostgreSQL cluster 9.6/main ...
/usr/lib/postgresql/9.6/bin/initdb -D /var/lib/postgresql/9.6/main --auth-local peer --auth-host md5
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/9.6/main ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/lib/postgresql/9.6/bin/pg_ctl -D /var/lib/postgresql/9.6/main -l logfile start
Ver Cluster Port Status Owner Data directory Log file
9.6 main 5432 online postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log
root#be03ab1eb704:/# /etc/init.d/postgresql start
[ ok ] Starting PostgreSQL 9.6 database server: main.
root#be03ab1eb704:/# su postgres -c "psql -f test.sql"
CREATE DATABASE
CREATE ROLE
GRANT
root#be03ab1eb704:/# /etc/init.d/postgresql stop
[ ok ] Stopping PostgreSQL 9.6 database server: main.
root#be03ab1eb704:/# exit
exit
The official postgresql docker image automatically imports scripts at the first start of a container. So if you mount your directory with your init sql script to container path '/docker-entrypoint-initdb.d/' it should be run.
For example if you have your import script myImport.sql and it's on your host in a directory /opt/import, you can mount the import directory on container start to your postgres image to /docker-entrypoint-initdb.d and the sql file will be executed after initial database setup.
docker run -p $toHostParam:5432 -e POSTGRES_PASSWORD="$dbPwd" \
-e POSTGRES_USER="$dbUser" \
-e POSTGRES_DB="$dbName" \
-e PGDATA=/opt/pgdata \
-v ${dataDir}:/opt/pgdata \
# look here
# mount of local import dir
-v /opt/import:/docker-entrypoint-initdb.d \
postgres:9.6
Take a look at the postgesql image start script here (from line 126):
https://github.com/docker-library/postgres/blob/master/9.6/docker-entrypoint.sh
If you want a specific db user or database you can also customize your postgresql container with environment variables.
Take a look at the 'Environment Variables' section here: https://hub.docker.com/_/postgres/
Try with this setup.sh
#!/bin/bash
set -e
pg_createcluster 9.6 main --start
su postgres sh -c "/etc/init.d/postgresql start && psql -f create_fixtures.sql && /etc/init.d/postgresql stop"
Try a explicit user when running psql:
psql -U postgres -f create_fixtures.sql

Where are PostgreSQL database files in the cluster directory?

On OSX, I recently installed PostgreSQL via Homebrew:
brew install postgresql
I then created a new database cluster:
initdb /usr/local/var/postgres
I confirm that postgresql server is running with the expected database cluster:
$ ps auxwww | grep postgres
0:00.03 /usr/local/opt/postgresql/bin/postgres -D /usr/local/var/postgres
I create a new database:
createdb mynewdb
I see that it exists.
$ psql
<user>=# \l
mynewdb | <user> | UTF8 | en_US.UTF-8 | en_US.UTF-8
But, I see no obvious changes to the cluster directory (e.g., just by checking contents ls -lt).
Where is the database written / stored in the cluster directory (or sub-directories)?
as #ahorsewithnoname says they are in the base directory, (on some other platfotms data/base)
the numeric name is the OID of the database and in the directory will be files named for the OID of the table or index they relate to.
databse oids can be determined with this query. oid is like a hidden unique id column.
select oid,* from pg_catalog.pg_database;
table and index oids can be found like this,
select * from pg_catalog.pg_class;
column relifilenode for the base of the filename (thanks #jjames)