I've added three (3) environment variables which I'm trying to used to setup the db correctly.
These are the variables:
DB_NAME
DB_USER
DB_PW
In the before_script section of .travis.yml I would run something like this:
before_script:
- psql -c "\set pw `echo "$db_pw"`"
- psql -c "\set user `echo "$db_user"`"
- psql -c "\set name `echo "$db_name"`"
- psql -c "create database :'name'"
- psql -c "CREATE :user WITH PASSWORD ':pw';"
However, that doesn't work since the result of \set is lost after each command.
How can I setup the db based on what on my environment variables?
Related
I'm passing env variables from .env file to my bash script and I want to run psql command. Unfortunatelly I'm struggling with a correct string template so my psql always fails.
#!/bin/bash
set -o allexport
source .env
if [[ $SHOULD_UPDATE_CONTENT==1 ]]
then
# 1. Truncate all the tables before inserting data
r="select $DB_SCHEMA.truncate_tables('user','$DB_SCHEMA');"
docker exec $CONTAINER_NAME psql -U dev -d data -v ON_ERROR_STOP=1 -c "${r}"
fi
set +o allexport
.env
DB_HOST=db
DB_PORT=5432
DB_DATABASE=dbTest
DB_SCHEMA=schema
SHOULD_UPDATE_CONTENT=1
I am trying to connect a Postgres RDS serverless instance from a CodeBuild project.
this is where it fails:
psql --host ${PG_HOST} --dbname ${PG_DBNAME} --user ${SECRET_USER} -f /tmp/file
/tmp/file exists, so a file permission/non-existing issue is out of question
What I have tried so far, the username and password have been:
Stored as environment variables in CodeBuild
Stored in AWS Secrets manager
env:
secrets-manager:
# key: secret-id:json-key:version-stage:version-id
SECRET_USER: rds-db-credentials:username
SECRET_PASSWORD: rds-db-credentials:password
Given as parameters in the call directly
psql --host ${PG_HOST} --dbname ${PG_DBNAME} "user=mydbuser password=0fNKJtNv" -f /tmp/file;
Stored in .pgpass file
echo ${PG_HOST}:${PG_PORT}:${PG_DBNAME}:${SECRET_USER}:${SECRET_PASSWORD} > ~/.pgpass
chmod 600 ~/.pgpass
And to be sure, I exported the variable too, PGPASSFILE="~/.pgpass"
When I echo the username and password, I only get *** printed, like:
[Container] 2020/09/27 07:39:34 Running command cat ~/.pgpass
something.eu-central-1.rds.amazonaws.com:5432:spumdb:***:***
Errors:
For psql --host ${PG_HOST} --dbname ${PG_DBNAME} --user ${PG_USER} -f /tmp/file, the error is
psql: warning: extra command-line argument "/tmp/file" ignored
Password for user -f:
psql: fe_sendauth: no password supplied
For psql --host ${PG_HOST} --dbname ${PG_DBNAME} --user ${SECRET_USER} -f /tmp/file, the error is
Password for user ***:
psql: fe_sendauth: no password supplied
For psql --host ${PG_HOST} --dbname ${PG_DBNAME} "user=mydbuser password=0fNKJtNv" -f /tmp/file, the error is:
Password for user user=*** password=***:
psql: fe_sendauth: no password supplied
Just for info,
The RDS security group has the entry to allow all TCP connections from CodeBuild in the AWS region where the project is (35.157.127.248/29)
Both CodeBuild and RDS lie in the same private subnet of an user created VPC
I am able to connect to the RDS instance from an EC2 instance using the same RDS credentials
I am using the latest image for Amazon Linux 2 (aws/codebuild/amazonlinux2-x86_64-standard:3.0)
It looks like a problem at the CodeBuild end, not at the RDS end. For some reason, CodeBuild doesn't get the value of the parameters, that too only username and password, others like hostname, dbname are evaluated correctly!
Does anybody see any problem anywhere? Thank you!
I tried to replicate the issue, but the only thing I found that you should be using --username, not --user. Anyway, here is my buildspec.yml used for the verification:
version: 0.2
env:
variables:
PG_HOST: database-1.cm3c1syrcj06.us-east-1.rds.amazonaws.com
PG_PORT: 5432
PG_DBNAME: mydb
secrets-manager:
SECRET_USER: rds-db-credentials:username
SECRET_PASSWORD: rds-db-credentials:password
phases:
pre_build:
commands:
- echo ${PG_HOST}:${PG_PORT}:${PG_DBNAME}:${SECRET_USER}:${SECRET_PASSWORD} > ~/.pgpass
- chmod 600 ~/.pgpass
build:
commands:
- cat ~/.pgpass | rev
- echo "\dt" > /tmp/file
- psql --host ${PG_HOST} --dbname ${PG_DBNAME} --username ${SECRET_USER} -f /tmp/file
In the above I use a little trick to show the password and username, but in reverse. Otherwise you see only ***.
Everything works as expected, and CB connects the database as evident by the No relations found (this was expected, as my db is empty):
I have a simple dockerfile:
FROM postgres:latest
ENV POSTGRES_PASSWORD password
ENV POSTGRES_USER postgres
ENV POSTGRES_DB evesde
COPY init.sh /docker-entrypoint-initdb.d/
and my init file is chmod' to 777:
#!/bin/bash
psql -U "postgres" -d "evesde" -e "create role yaml with login encrypted password 'password';"
when running a container it will say:
psql: warning: extra command-line argument "create role yaml with
login encrypted password 'password';" ignored
Im not sure why this is happening, and when doing an interactive terminal, this command seemingly worked. I dont see any additional information and wasnt sure what was going wrong.
The postgres docker page is: https://hub.docker.com/_/postgres
When looking at it deeper, I was noticing that running the command itself fails in an interactive Terminal with the same error, but the command runs when I am in postgres: psql -U "postgres" -d "evesde" and run the command, it works.
I think it may be related to passing the command in through the exec command is where it fails. likely related to '.
You want -c instead of -e.
-e turns on "echo queries"
-c runs the command and exits
Have you considered putting just the create role command in a file called create_role.sql and copying that into /docker-entrypoint-initdb.d/?
Based on testing, it looks like an equivalent but simpler solution is to put the SQL command as one line in a file, 00_roles.sql, and copy that into the container instead of the init.sh script.
New to Ansible, Running version 2.1.0. I've written an Ansible playbook that runs a PostgreSQL query against a group of hosts. When I specify the SQL DB password in the shell command it works, but I'm looking to run the playbook against a group of hosts and need a better way to input the passwords as they are all unique. Could anyone suggest a better way to do this?
---
- hosts: Test_Hosts
sudo: yes
sudo_user: root
gather_facts: yes
tasks:
- name: Login to DB and run command
shell: export PGPASSWORD='Password'; psql -U 'user' -d 'db' -c 'select * FROM table';
register: select_all_from_table
- name: Display table contents
debug: msg="{{ select_all_from_table.stdout }}"
I saw another thread on the topic but was not sure how to implement the suggestion: Run a postgresql command with ansible playbook. Postgresql requires password
Ansible allows you to set environment variables for a task using the environment parameter to any task.
So in your case you could just do this:
- name: Login to DB and run command
shell: psql -U 'user' -d 'db' -c 'select * FROM table';
register: select_all_from_table
environment:
PGPASSWORD: '{{ pgpassword }}'
And then set the pgpassword variable at the group or host level.
I just ran into this issue today and this is what worked for me. On Linux you can package all the credentials into a ~/.pgpass hidden file.
Just create it locally (in this case in ./files/pgpass) and then use ansible to copy it onto the host before you run the psql commands.
- name: set passwd file for PSQL
copy:
src: files/pgpass
dest: ~/.pgpass
mode: 0600 ### important: will not work with wrong permissions
- name: PSQL command
shell: "psql -U 'user' -d 'db' -c 'select * FROM table'"
register: select_all_from_table
The file contents must be in the following format:
hostname:port:database:username:password
However, you can use wildcards, so mine looks like this, for example:
*:*:db1:user1:passwd1
*:*:db2:user2:passwd2
See documentation for more details:
https://www.postgresql.org/docs/9.1/static/libpq-pgpass.html
I'm trying to build a setup script to automate the development environments creation, but I'm having trouble both trying to pipe or using the -c modifier for psql.
I've tried:
sudo su postgres psql -c "ALTER ROLE postgres WITH password 'pass'"
and
sudo su postgres psql -c "ALTER ROLE postgres WITH password 'pass';"
Both of which say "ALTER: command not found"
I've also tried pipe, but I'm not able to combine it with su correctly
eg: I tried something like
sudo su postgres echo "ALTER ROLE postgres WITH password 'pass'" | psql
But postgres can't execute "echo"
And:
echo "ALTER ROLE postgres WITH password 'pass'" | sudo su psql
Which just doesn't work.
So, my first question is: how can I execute this simple command from a sh file?
And the second one, less related: how can I use different users in the commands chained with pipe?
What's wrong is the lack of -c or --command for su to indicate that the rest of the line is a command.
But su is not needed anyway, because there's already sudo. Do this instead:
sudo -u postgres psql -c "ALTER ROLE postgres WITH password 'pass'"
If you are just like me, make sure you are not running this from:
bash-4.2$
but:
postgres=#