PostgreSQL is being installed with SQL_ASCII using Puppet - postgresql

I'm trying to build Vagrant box provisioned with Puppet.
I use Ubuntu 12.04 LTS box from http://files.vagrantup.com/precise64.box.
I also use http://forge.puppetlabs.com/puppetlabs/postgresql module to install PostgreSQL with the following manifest:
class db {
class { 'postgresql': version => '9.1' }
class { 'postgresql::server': }
}
class { 'db': }
It installs correctly, but databases are created with SQL_ASCII encoding:
$ psql -l -U postgres
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
After googling I've thought it might be a locale problem, but it looks correct to me:
$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE=
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
Do you know what should I do to be able to install PostgreSQL with UTF8 encoding default? It'd be also nice to be able to use Puppet for this.

Apparently, it is Puppet bug http://projects.puppetlabs.com/issues/4695. Added the following code from issue comments as a workaround:
# workaround for http://projects.puppetlabs.com/issues/4695
# when PostgreSQL is installed with SQL_ASCII encoding instead of UTF8
exec { 'utf8 postgres':
command => 'pg_dropcluster --stop 9.1 main ; pg_createcluster --start --locale en_US.UTF-8 9.1 main',
unless => 'sudo -u postgres psql -t -c "\l" | grep template1 | grep -q UTF',
require => Class['postgresql::server'],
path => ['/bin', '/sbin', '/usr/bin', '/usr/sbin'],
}

Related

Can not connect to a postgres database in a docker container from my local machine

I have a strange problem with docker and postgres. I can not connect from my local machine to psql inside the docker container.
First of all I created my container with this command:
docker run -d -p 5432:5432 --name my-postgres -e POSTGRES_PASSWORD=mysecretpassword postgres
After that I executed the container with following command:
docker exec -it my-postgres bash
Inside the container I started psql with the postgres user:
psql -U postgres
And I created a new database:
CREATE DATABASE mytestdb;
Everything works inside the container, but if I want to connect to it from my machine it does only show the database from my machine. I tried this to connect to postgres:
psql -h localhost -p 5432 -U postgres
Which should me show something like this:
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
mytestdb | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
But instead I get this:
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------------+-----------------+----------+---------+---------+----------------------------------
user | user | UTF8 | C | C |
postgres | user | UTF8 | C | C |
template0 | user | UTF8 | C | C | =c/user +
| | | | | user =CTc/user
template1 | user | UTF8 | C | C | =c/user +
| | | | | user =CTc/user
(4 rows)
(END)
In every tutorial, I saw they did it, but for me, it doesn't work. So please help me.
The problem is when you creating the db.
CREATE DATABASE mytestdb you forgot to add ; the command should be like CREATE DATABASE mytestdb;
And then you have your db and you can access it from inside the container or your host with these commands

Upgrading Postgres from 10 to 12 -- problem with encodings

I'm trying to upgrade a postgres server from 10 12, and am having problems with the encodings. I'm following what I believe to be established recipes.
Behold:
postgres#serverbot:~$ psql -l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-------------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | postgres=CTc/postgres+
| | | | | =c/postgres
thingsboard | postgres | SQL_ASCII | C | C | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | nagios=c/postgres
(4 rows)
Notice the encodings on the 10 database. Time to create the database for 12.
postgres#serverbot:~$ sudo service postgresql stop
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/initdb -E SQL_ASCII --locale=C -D /var/lib/postgresql/12/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /var/lib/postgresql/12/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/New_York
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
Excellent! Let's fire up the new server.
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
waiting for server to start.... done
server started
And verify the encodings...
postgres#serverbot:~$ psql -l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
Everything matches... time to upgrade!
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_upgrade \
--old-datadir=/var/lib/postgresql/10/main \
--new-datadir=/var/lib/postgresql/12/main \
--old-bindir=/usr/lib/postgresql/10/bin \
--new-bindir=/usr/lib/postgresql/12/bin \
--old-options '-c config_file=/etc/postgresql/10/main/postgresql.conf' \
--new-options '-c config_file=/etc/postgresql/12/main/postgresql.conf' \
--link --check
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for tables WITH OIDS ok
Checking for invalid "sql_identifier" user columns ok
encodings for database "postgres" do not match: old "SQL_ASCII", new "UTF8"
Failure, exiting
postgres#serverbot:~$
Doh!
What is wrong here? I assert that the encodings do match, and I'm stuck.
Can anyone offer any advice?
What I see is:
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
waiting for server to start.... done
server started
and then:
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_upgrade \
--old-datadir=/var/lib/postgresql/10/main \
--new-datadir=/var/lib/postgresql/12/main \
[...]
Note the $PGDATA directories. The database cluster you did the initdb for is different from the one you are doing the pg_upgrade to.
UPDATE. Since you seem to be using a Debian based OS and it's Postgres packaging, it might work better to stick with the packaging tools:
sudo pg_createcluster --locale=C 12 ascii
Creating new PostgreSQL cluster 12/ascii ...
/usr/lib/postgresql/12/bin/initdb -D /var/lib/postgresql/12/ascii --auth-local peer --auth-host md5 --locale C
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/12/ascii ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/Los_Angeles
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctlcluster 12 ascii start
Ver Cluster Port Status Owner Data directory Log file
12 ascii 5434 down postgres /var/lib/postgresql/12/ascii /var/log/postgresql/postgresql-12-ascii.log
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
Also I would use pg_upgradecluster, see here for more information. This keeps everything in the same system.

PostgreSQL's `initdb` fails with "invalid locale settings; check LANG and LC_* environment variables"

Already found a solution for this (see answer below), but I am not sure that it is the appropriate one; plus this may help someone else too.
Tried to set up PostgreSQL by following the documentation (18.2 Creating a Database Cluster), but got the following error on Ubuntu 18.04 (kernel: 4.15.0-22-generic):
$ initdb -D /usr/local/pgsql/data
(...)
initdb: invalid locale settings; check LANG and LC_* environment variables
Found a couple answers on Stackoverflow (1, 2) that were relevant, but these did not resolve the issue and the one on Serverfault suggested to restart the service, but PostgreSQL wasn't even running.
Tried passing the locale explicitly in every variation that I found on the system, but these failed too,
3617 2018/06/07-08:36 initdb -D ~/Downloads/ --locale=en_US.utf8
3618 2018/06/07-08:36 initdb -D ~/Downloads/ --locale=en_US.UTF8
3621 2018/06/07-08:37 initdb -D ~/Downloads/ --locale=en_US.UTF-8
3622 2018/06/07-08:37 initdb -D ~/Downloads/ --locale="en_US.UTF-8"
3623 2018/06/07-08:37 initdb -D ~/Downloads/ --locale="en_US.utf8"
3645 2018/06/07-09:24 initdb -D ~/Downloads/ --locale="en_US.utf8"
with
initdb: invalid locale name <the_option_value_above>
There was an Arch Linux forum discussion about this, but there were no solution.
2018/06/07 1214 UPDATE
I linked answers above, but perhaps wasn't explicit enough: I did look at locale -a and locale (not listing the former's output because I installed ALL of them in my attempts below):
$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8
What have been tried, but did not work (and terminal has been restarted for every iteration):
https://askubuntu.com/questions/454260/how-to-solve-locale-problem
Selected and configured ALL locales.
$ sudo locale-gen en_US.UTF-8
$ sudo dpkg-reconfigure locales.
https://github.com/singularityware/singularity/issues/11
neither
echo "LC_ALL=en_US.UTF-8" >> /etc/environment
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
echo "LANG=en_US.UTF-8" > /etc/locale.conf
locale-gen en_US.UTF-8
nor
sudo apt clean
sudo apt update
sudo apt upgrade
sudo apt-get install -y locales language-pack-fi language-pack-en
export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
sudo locale-gen en_US.UTF-8
sudo dpkg-reconfigure locales
https://askubuntu.com/questions/162391/how-do-i-fix-my-locale-issue/229512#229512
https://askubuntu.com/questions/114759/warning-setlocale-lc-all-cannot-change-locale
(Basically variations of the github link above, tried it anyway.)
TODO:
https://unix.stackexchange.com/questions/294845/bash-warning-setlocale-lc-all-cannot-change-locale-en-us-utf-8
From this thread:
initdb -D <your_data_location> --no-locale --encoding=UTF8
where
--locale=LOCALE set default locale for new databases
--no-locale equivalent to --locale=C
There are caveats (see warning below), but an all-utf8 database can be created using template0 (see 21.3. Template Databases).
From the client (psql):
postgres=# create database test LC_COLLATE "en_US.UTF-8" LC_CTYPE "en_US.UTF-8" template template0;
Or via createdb:
createdb --lc-collate="en_US.UTF-8" --lc-ctype="en_US.UTF-8" --template="template0" test2
Check:
$ psql
psql (10.3)
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | C | C |
template0 | postgres | UTF8 | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
test | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
test2 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
WARNING: This is probably not the correct solution and the workaround above is just that, a workaround.
Note the "Collate" and "Ctype" fields below in a database created with the above solution and this may cause issues, because "The results of comparisons between strings depends on LC_CTYPE. In practice, the most visible effect is the sort order." (see DBA StackExchange thread). This is also confirmed on the PostgreSQL mailing list (see this thread about this issue on a database in production). Probably the easiest way to solve this would be re-initializing/recreating the database.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+-------+-----------------------
postgres | postgres | UTF8 | C | C |
template0 | postgres | UTF8 | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
Although the question does not mention Nix, the original poster linked to this issue from the Nix discourse site so I believe this is a Nix-related issue.
I ran into this issue when running under Nix shell and found the solution here after much searching. I just had to add glibcLocales to my environment. I.e. either run nix-shell -p glibcLocales or add glibcLocales to buildInputs.
You can get a listing of the locales available in Linux with
locale -a
Use one of these.
You have to choose a locale that matches your encoding, for example
initdb -E UTF8 --locale=en_US.utf8
or
initdb -E LATIN9 --locale=et_EE.iso885915
As far as I know, you can install additional locales with
sudo apt-get install language-pack-XX
Check if the locale is enabled in /etc/locale.gen. On my fresh install of Arch Linux ARM the following line was commented out:
en_US.UTF-8 UTF-8
Run locale-gen without any arguments. It will list all the uncommented locales as it generates them.
Optional: edit /etc/locale.conf to set the system locale:
echo "LANG=en_US.UTF-8" > /etc/locale.conf
Restart the system to make all services pick up the new setting.
Run your initdb command.
for me it was an issue when I tried to upgrade postgres from 9.6 to 15.
locale should be replaced with local-provider and icu locale, e.g.:
POSTGRES_INITDB_ARGS="--locale=nl_NL --encoding=UTF8"
->
POSTGRES_INITDB_ARGS="--locale-provider=icu --icu-locale=nl_NL --encoding=UTF8

How to import database to PostgreSQL as root?

I want to import a database of mine to a database of my server. So, I copied my database dump file to my server's root directory and logged in and did this:
root#iWidgetServer1:~# sudo -u postgres psql -U iwidget -d iwidget -f iwidget_dump2.sql
could not change directory to "/root"
psql: FATAL: Peer authentication failed for user "iwidget"
However, iwidget is a role and has granted all priviliges for this database:
root#iWidgetServer1:~# sudo -u postgres psql -l
could not change directory to "/root"
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
------------------+----------+----------+-------------+-------------+-----------------------
iwidget | iwidget | UTF8 | en_GB.UTF-8 | en_GB.UTF-8 | =Tc/iwidget +
| | | | | iwidget=CTc/iwidget
postgres | postgres | UTF8 | en_GB.UTF-8 | en_GB.UTF-8 |
sample_db | postgres | UTF8 | en_GB.UTF-8 | en_GB.UTF-8 |
template0 | postgres | UTF8 | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template_postgis | postgres | UTF8 | en_GB.UTF-8 | en_GB.UTF-8 |
(6 rows)
What am I doing wrong?
From the docs:
The peer authentication method works by obtaining the client's operating system user name from the kernel and using it as the allowed database user name (with optional user name mapping). This method is only supported on local connections.
You're doing sudo -u postgres, but are trying to connect as iwidget.
You need to create a user named iwidget and login as this user.
This seems to be more a problem with the File permissions of the dump instead of Postgres. Have you tried moving the dump to a folder not owned by root?

Chef ENV settings not working

I am attempting to set up postgresql on a Vagrant box using Chef solo and am running into some problems. I need the default postgres encoding/locale to be UTF8. By default, the locale of the precise64 Ubuntu box is set to "C" so postgres is using LATIN1 for encoding. This is what I've done so far:
I have a chef recipe that sets the locale by doing the following:
template "/etc/profile.d/lang.sh" do
source "lang.sh.erb"
mode "0644"
end
execute "locale-gen" do
command "locale-gen en_US.UTF-8"
end
execute "dpkg-reconfigure-locales" do
command "dpkg-reconfigure locales"
end
where lang.sh.erb looks like:
export LANGUAGE="en_US.UTF-8"
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
This sets up the locale correctly, but unfortunately it doesn't modify the current environment. So I have another recipe that just sets the ENV before including postgresql
ENV["LANGUAGE"] = ENV["LANG"] = ENV["LC_ALL"] = "en_US.UTF-8"
include_recipe "postgresql::server"
This has no effect. The locale is set up correctly:
postgres#precise64:~$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8
But postgres used the "C" locale when it was installed.
postgres#precise64:~$ psql -l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+-------+-----------------------
postgres | postgres | LATIN1 | en_US | en_US |
template0 | postgres | LATIN1 | en_US | en_US | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | LATIN1 | en_US | en_US | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
For attribution's sake, I got all this from http://www.softr.li/blog/2012/05/22/chef-recipe-to-install-a-postgresql-server-on-a-machine-configured-with-en_us-locales.
I found that the solution that worked for me was to either in the bootstrap shell script, or as inline shell, to copy the /etc/default/lang.sh to the box prior to any recipes being run. (So should be first thing done in the Vagrant file after box definitions)
lang file:
export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
From here the database should get setup with the UTF-8 encoding.
Hope this helps as I have spent days searching for solutions to this, and came up with the bits and pieces from various discussions, but realized that the problem was timing of when the values are set...
the environment variable doesn't work for chef resources.
According to the postgresql cookbook, you should set the attribute node['postgresql']['initdb_locale'] to setup the locale when initializing a database. For example, use this section under parent section name: serverin your .kitchen.yml:
attributes:
postgresql:
initdb_locale: "en_US.UTF_8"
You can drop and recreate the postgres template database as UTF-8 after the fact. Not a perfect solution, but it does work in your Chef recipe. See: http://www.pebra.net/blog/2013/06/10/when-struggling-with-postgresql-and-utf8-slash-latin/
include_recipe "postgresql::server"
include_recipe "database::postgresql"
execute "Psql template1 to UTF8" do
user "postgres"
command <<-SQL
echo "
UPDATE pg_database SET datistemplate = FALSE WHERE datname = 'template1';
DROP DATABASE template1;
CREATE DATABASE template1 WITH TEMPLATE = template0 ENCODING = 'UNICODE' LC_CTYPE='en_US.utf8' LC_COLLATE='en_US.utf8';
UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template1';
\\c template1
VACUUM FREEZE;" | psql postgres -t
SQL
# only_if '[ $(echo "select count(*) from pg_database where datname = \'template1\' and datcollate = \'en_US.utf8\'" |psql postgres -t) -eq 0 ]'
end