Postgresql : can't connect with new created users - postgresql

Here's the error :
$ psql -h localhost -U kMbjQ6pR9G -d fzvqFILx0d
Password for user kMbjQ6pR9G:
psql: FATAL: password authentication failed for user "kMbjQ6pR9G"
FATAL: password authentication failed for user "kMbjQ6pR9G"
I'm probably missing a configuration step on using fresh PostgreSQL.
Here's the command lines I tried to create a new user with his own database :
sudo -u postgres bash -c "psql -c \"CREATE USER $USER WITH PASSWORD '$PASSWORD';\"" &&
sudo -u postgres bash -c "psql -c \"CREATE DATABASE $DB;\"" &&
sudo -u postgres bash -c "psql -c \"GRANT ALL PRIVILEGES ON DATABASE $DB to $USER;\"" &&
Here's the PostgreSQL configuration :
$ sudo grep -v ^# /etc/postgresql/9.5/main/pg_hba.conf |grep -v ^$
local all postgres md5
local all all md5
host all all 127.0.0.1/32 md5
host all all ::1/128 md5
Here's all the accound I created to test :
$ sudo -u postgres bash -c 'psql -c "select * from pg_shadow;"'
Password:
usename | usesysid | usecreatedb | usesuper | userepl | usebypassrls | passwd | valuntil | useconfig
------------+----------+-------------+----------+---------+--------------+-------------------------------------+----------+-----------
av6izmbp9l | 16384 | f | f | f | f | md5a49721ef3f5428e269badc03931baf48 | |
rqmejchq7n | 16386 | f | f | f | f | md54edf3a05ca96a435f94152b75495e9dc | |
yyfiknesu8 | 16388 | f | f | f | f | md5b3d4a03913569abbf318bdc490d0f821 | |
qgv2ryqvdw | 16390 | f | f | f | f | md5d0959b4b5e1ed2982e19e4d0af574b11 | |
postgres | 10 | t | t | t | t | md5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | |
pf09joszuj | 16392 | f | f | f | f | md5a920dc31666459e5f0a96e9430d07f02 | |
I think it something simple, but I miss this point.
Thanks for your support,
David.

Did you explicitly specify the database when trying to connect?
psql -h localhost -U myuser -d mydb
This is how I usually set up a user to completely own a given database:
create role myuser with createdb login encrypted password 'mypassword';
create database mydb with owner 'myuser' encoding 'utf8';
pg_hba.conf:
local myuser mydb md5
...

The problem was obvious : username & database names are lowercase !
Here's the script I use to create user and his own database :
#!/bin/bash
if [ -n "$1" ]; then
DB="$1"
else
DB=$(php -r "echo substr(str_shuffle(str_repeat('abcdefghijklmnopqrstuvwxyz', ceil(10/63) )),1,10);")
fi
USER=$(php -r "echo substr(str_shuffle(str_repeat('abcdefghijklmnopqrstuvwxyz', ceil(10/63) )),1,10);")
PASSWORD=$(php -r "echo substr(str_shuffle(str_repeat('0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ', ceil(20/63) )),1,20);")
sudo -u postgres psql -c "CREATE USER $USER WITH PASSWORD '$PASSWORD';"
sudo -u postgres psql -c "CREATE DATABASE $DB;"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE $DB to $USER;"
cat << EndOfMessage
POSTGRESQL_ADDON_DB="$DB"
POSTGRESQL_ADDON_HOST="localhost"
POSTGRESQL_ADDON_PASSWORD="$PASSWORD"
POSTGRESQL_ADDON_PORT="5432"
POSTGRESQL_ADDON_USER="$USER"
EndOfMessage
https://github.com/davidbouche/linux-install/blob/master/pgsql-database-create.sh
Thanks for your contribution.
David

Related

Using sql query in shell script

I have a shell script that uses output from a sql query and based on the value of one column sends out an alert. However i don't think it's capturing the value. although the value is not greater than 0 yet it still sends out an email.
Any idea where i am going wrong? Thanks.
............................................................................
#!/bin/sh
psql -d postgres -U postgres -c "select pid,application_name,pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn) sending_lag,pg_wal_lsn_diff(sent_lsn,flush_lsn) receiving_lag,pg_wal_lsn_diff(flush_lsn, replay_lsn) replaying_lag,pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) total_lag from pg_stat_replication;"| while read total_lag;
do
echo $total_lag
lag1=$(echo $total_lag)
done
if [[ $lag1 -ge 0 ]]
then echo "Current replication lag is $lag1" |mail -s "WARNING!" abcd#mail.com
else
echo "No issue"
fi
............................................................................
this is the output of above query
pid | application_name | sending_lag | receiving_lag | replaying_lag | total_lag
-------+------------------+-------------+---------------+---------------+-----------
27823 | db123 | 0 | 0 | 0 | 0
27824 | db023 | 0 | 0 | 0 | 0

How to extract a value of a Map String

I have this String value which is a Map of user to password
'{"userJohn":"1234","userLinda":"9876"}'
the string has a single quate from each side.
How can I extract the password of say, userLinda
# fishell
~ echo '\'{"userJohn":"1234","userLinda":"9876"}\'' | sed 's/^\'//' | sed 's/\'$//' | jq '.userLinda'
"9876"
# bash zsh
~ echo "'"'{"userJohn":"1234","userLinda":"9876"}'"'" | sed "s/^'//" | sed "s/'$//" | jq '.userLinda'
"9876"

how to use jq command to create a table from a file content

I already used grep sed command to do something (cat service.yaml | grep -Po '(service_[^:]*|version:.+)'| sed -z 's/\nversion//g'), but now I need to create a "table" of content of this file in the output like this:
Here's a variant of #jq170727's solution that only uses:
a yaml-to-json translator; and
jq
To illustrate an alternative to yq for (1), I'll use npm's yaml2json.
yaml2json service.yaml | jq -r '
keys_unsorted as $keys
| (["service", "version", "replica"] | map("\"\(.)\"")), # quote the headers
($keys[] as $k | .[$k] | [$k, .version, .replica])
| #tsv'
Notice also that keys_unsorted is used, mainly to respect the order of the keys in the original YAML file.
Here is a filter which may help
keys[] as $k | .[$k] | [$k, .version, .replica]
Here is an example using it. For reference this is the yq I have from brew
$ yq --help | grep github
See https://github.com/kislyuk/yq for more information.
and/or https://stedolan.github.io/jq
First convert yaml to json
$ yq . service.yaml
{
"service-1": {
"replica": 2,
"version": "0.6.24"
},
"service-2": {
"replica": 3,
"version": "0.21.14"
}
}
Examine top level object
$ yq . service.yaml \
| jq -r 'keys'
[
"service-1",
"service-2"
]
Flatten into a stream of arrays
$ yq . service.yaml \
| jq -c 'keys[] as $k | .[$k] | [$k, .version, .replica]'
["service-1","0.6.24",2]
["service-2","0.21.14",3]
Convert to csv and format as a table
$ yq . service.yaml \
| jq -cr 'keys[] as $k | .[$k] | [$k, .version, .replica] | #csv' \
| column -s, -t
"service-1" "0.6.24" 2
"service-2" "0.21.14" 3
Add titles
$ yq . service.yaml \
| jq -cr 'keys[] as $k | .[$k] | [$k, .version, .replica] | #csv' \
| (echo '"service","version","replica"'; cat -) \
| column -s, -t
"service" "version" "replica"
"service-1" "0.6.24" 2
"service-2" "0.21.14" 3
If you want no quotes around the strings in the final rows this may be better
$ yq . service.yaml \
| jq -cr 'keys[] as $k | .[$k] | "\($k),\(.version),\(.replica)"' \
| (echo '"service","version","replica"'; cat -) \
| column -s, -t
"service" "version" "replica"
service-1 0.6.24 2
service-2 0.21.14 3

How to dynamically pass psql variable value at runtime?

I am trying to create shortcuts for DBAs/admins in psql.
I have created some functions and queries, which give me information such as schema size in bytes, or a list of tables in order of storage used, etc. These will be used whenever an admin or DBA wants to get some info about the database.
I can run these queries naturally with select, or as a function like get_size(), but I want them to be accessible as shortcuts, similar to the native backslash commands (\dx, \dt, etc).
So I have used psql's \set feature to store queries/functions as variables, which I will put in the psqlrc file:
\set size 'select pg_size_pretty(my_size_function(''public''));'
Then when I type :size in psql I would get the size of the "public" schema.
What I want though, is to be able to dynamically pass a schema name, so I could run things like
:size public, :size schema2 etc.
I tried changing the \set to: \set size 'select pg_size_pretty(my_size_function(:schema));', but I can only call that by executing \set schema '''public''' first.
Since the whole point is to use these universally as shortcuts, having to manually run \set commands each time defeats the purpose.
In Oracle the would be colon a bind variable, which would be read at runtime.
How can I do this with psql?
I use these ways.
Postgres Functions (Inside Postgres CLI)
postgres=# CREATE OR REPLACE FUNCTION getSize(tableName varchar) RETURNS varchar LANGUAGE SQL as
postgres-# $$
postgres$# SELECT pg_size_pretty(pg_relation_size(tableName));
postgres$# $$;
CREATE FUNCTION
postgres=# select getSize('test');
getsize
------------
8192 bytes
(1 row)
From Shell (Outside Postgres CLI)
psqlgettablesize(schema, tablename) - Get table size. Pass all for schema argument to get for all schemas.
$ psqlgettablesize customer events
table_name | total_size | table_size | index_size
--------------------------+---------------------+-------------+---------------
test(complete database) | 19039892127 (18 GB) | |
--------- | --------- | --------- | ---------
customer.events | 24576 (24 kB) | 0 (0 bytes) | 24576 (24 kB)
(3 rows)
psqlgettablecount(schema, tablename) - Get count of rows in a table, in a schema
$ psqlgettablecount customer events
count
----------
51850000
(1 row)
psqlgetvacuumdetails(schema, tablename) - Get vacuum details of a table, in a schema
$ psqlgetvacuumdetails customer events
schemaname | relname | n_live_tup | n_dead_tup | last_analyze | analyze_count | last_autoanalyze | autoanalyze_count | last_vacuum | vacuum_count | last_autovacuum | autovacuum_count
------------+-----------+------------+------------+----------------------------+---------------+----------------------------+-------------------+---------------------------+--------------+-----------------+------------------
customer | events | 0 | 0 | 2019-12-02 18:25:04.887653 | 2 | 2019-11-27 18:49:19.002405 | 1 | 2019-11-29 13:11:15.92002 | 1 | | 0
(1 row)
psqltruncatetable(schema, tablename) - Truncate a table, in a schema, after authorization.
$ psqltruncatetable customer events
Are you sure to truncate table 'customer.events' (y/n)? y
Time: 4.944 ms
psqlsettings(category) - Get settings of Postgres
$ psqlsettings Autovacuum
name | setting | unit | category | short_desc | extra_desc | context | vartype | source | min_val | max_val | enumvals | boot_val | reset_val | sourcefile | sourceline | pending_restart
-------------------------------------+-----------+------+------------+-------------------------------------------------------------------------------------------+------------+------------+---------+---------+---------+------------+----------+-----------+-----------+------------+------------+-----------------
autovacuum | on | | Autovacuum | Starts the autovacuum subprocess. | | sighup | bool | default | | | | on | on | | | f
autovacuum_analyze_scale_factor | 0.1 | | Autovacuum | Number of tuple inserts, updates, or deletes prior to analyze as a fraction of reltuples. | | sighup | real | default | 0 | 100 | | 0.1 | 0.1 | | | f
psqlselectrows(schema, tablename) - Get rows from a table, in a schema
$ psqlselectrows customer events
id | name
----+------
1 | Clicked
2 | Page Loaded
(10 rows)
#Colors
B_BLACK='\033[1;30m'
B_RED='\033[1;31m'
B_GREEN='\033[1;32m'
B_YELLOW='\033[1;33m'
B_BLUE='\033[1;34m'
B_PURPLE='\033[1;35m'
B_CYAN='\033[1;36m'
B_WHITE='\033[1;37m'
RESET='\033[0m'
#Postgres Command With Params
psqlcommand="$POSTGRES_BIN/psql -U postgres test -q -c"
function psqlgettablesize()
{
[ -z "$1" ] && echo -e "${B_RED}Argument 1 missing. Schema name needed. ${B_YELLOW}(Pass 'all' to get details from all schema(s)${RESET}" ||
{
criteria="and table_schema = '$1'"
if [ "$1" == "all" ]; then
criteria=""
fi
if [ "$2" != "" ]; then
criteria+=" and table_name = '$2'"
fi
[ -z "$2" ] && echo -e "${B_YELLOW}Table name not given. ${B_GREEN}Showing size of all tables in $1 schema(s)${RESET}"
query="SELECT
concat(current_database(), '(complete database)') AS table_name, concat(pg_database_size(current_database()), ' (', pg_size_pretty(pg_database_size(current_database())), ')') AS total_size, '' AS table_size, '' AS index_size
UNION ALL SELECT '---------','---------','---------','---------'
UNION ALL SELECT
table_name,
concat(total_table_size, ' (', pg_size_pretty(total_table_size), ')'),
concat(table_size, ' (', pg_size_pretty(table_size), ')'),
concat(index_size, ' (', pg_size_pretty(index_size), ')')
FROM (
SELECT
concat(table_schema, '.', table_name) AS table_name,
pg_total_relation_size(concat(table_schema, '.', table_name)) AS total_table_size,
pg_relation_size(concat(table_schema, '.', table_name)) AS table_size,
pg_indexes_size(concat(table_schema, '.', table_name)) AS index_size
FROM information_schema.tables where table_schema !~ '^pg_' AND table_schema <> 'information_schema' $criteria ORDER BY total_table_size) AS sizes";
$psqlcommand "$query"
}
}
function psqlgettablecount()
{
[ -z "$1" ] && echo -e "${B_RED}Argument 1 missing: Need schema name${RESET}"
[ -z "$2" ] && echo -e "${B_RED}Argument 2 missing: Need table name${RESET}" ||
$psqlcommand "select count(*) from $1.$2;"
}
function psqlgetvacuumdetails()
{
[ -z "$1" ] && echo -e "${B_RED}Argument 1 missing: Need schema name${RESET}" ||
[ -z "$2" ] && echo -e "${B_RED}Argument 2 missing: Need table name${RESET}" ||
$psqlcommand "SELECT schemaname, relname, n_live_tup, n_dead_tup, last_analyze::timestamp, analyze_count, last_autoanalyze::timestamp, autoanalyze_count, last_vacuum::timestamp, vacuum_count, last_autovacuum::timestamp, autovacuum_count FROM pg_stat_user_tables where schemaname = '$1' and relname='$2';"
}
function psqltruncatetable()
{
[ -z "$1" ] && echo -e "${B_RED}Argument 1 missing: Need schema name${RESET}" ||
[ -z "$2" ] && echo -e "${B_RED}Argument 2 missing: Need table name${RESET}" ||
{
read -p "$(echo -e ${B_YELLOW}"Are you sure to truncate table '$1.$2' (y/n)? "${RESET})" choice
case "$choice" in
y|Y ) $psqlcommand "TRUNCATE $1.$2;";;
n|N ) echo -e "${B_GREEN}Table '$1.$2' not truncated${RESET}";;
* ) echo -e "${B_RED}Invalid option${RESET}";;
esac
}
}
function psqlsettings()
{
query="select * from pg_settings"
if [ "$1" != "" ]; then
query="$query where category like '%$1%'"
fi
query="$query ;"
$psqlcommand "$query"
if [ -z "$1" ]; then
echo -e "${B_YELLOW}Passing Category as first argument will filter the related settings.${RESET}"
fi
}
function psqlselectrows()
{
[ -z "$1" ] && echo -e "${B_RED}Argument 1 missing: Need schema name${RESET}" ||
[ -z "$2" ] && echo -e "${B_RED}Argument 2 missing: Need table name${RESET}" ||
$psqlcommand "SELECT * from $1.$2"
}
I couldn't figure out how to dynamically pass variables so I settled for a system of aliases and variables that work together to provide what I was looking for. So I can run :load_schema <schemaname> and then run commands interacting with the "loaded" schema. Loading a schema is simply aliasing a \set command. This way people who are unfamiliar with psql can just press : and tab autocomplete will show them all of my shortcuts.
All my code is here.
This isn't exactly what I was looking for so I won't accept it but it's close enough to post in case anyone else wants to do this.

How to preserve new line character while performing psql copy command

I have following content in my csv file(with 3 columns):
141413,"\"'/x=/></script></title><x><x/","Mountain View, CA\"'/x=/></script></title><x><x/"
148443,"CLICK LINK BELOW TO ENTER^^^^^^^^^^^^^^","model\
\
xxx lipsum as it is\
\
100 sometimes unknown\
\
travel evening market\
"
When I import above mentioned csv in mysql using following command, it treats the backslash() as new line; which is the expected behavior.
LOAD DATA INFILE '1.csv' INTO TABLE users FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';
MYSQL Output
But when I try to import to psql using copy command, it treats \ as a normal character.
copy users from '1.csv' WITH (FORMAT csv, DELIMITER ',', ENCODING 'utf8', NULL "\N", QUOTE E'\"', ESCAPE '\');
postgres Output
Try parsing these \ before importing the CSV file, e.g. using perl -pe or sed and the STDIN from psql:
$ cat 1.csv | perl -pe 's/\\\n/\n/g' | psql testdb -c "COPY users FROM STDIN WITH (FORMAT csv, DELIMITER ',', ENCODING 'utf8', NULL "\N", QUOTE E'\"', ESCAPE '\');"
This is how it looks like after the import:
testdb=# select * from users;
id | company | location
--------+-----------------------------------------+-------------------------------------------------
141413 | "'/x=/></script></title><x><x/ | Mountain View, CA"'/x=/></script></title><x><x/
148443 | CLICK LINK BELOW TO ENTER^^^^^^^^^^^^^^ | model +
| | +
| | xxx lipsum as it is +
| | +
| | 100 sometimes unknown +
| | +
| | travel evening market +
| |
(2 Zeilen)