Run a PostgreSQL script using Ansible - postgresql

I am looking for a way to run a Postgres script using Ansible. While I found a reasonably good example Here, I need to:
Run the script as user postgres
I don't necessarily need to keep a copy of the script on the server so if I need to have a copy, it will only be for temp use.
Can anyone tell me if this is possible and if so an example of running it. Here is what I tried so far using Ansible and it just hung at these points:
- name: Testing DB to make sure it is available
command: psql -U bob image
register: b
- debug: b
- name: Verifying Tables exist in Image
shell: \d image
register: c
- debug: c
- name: Exiting Image DB
shell: \q
register: d
- debug: d
- name: Going to Agent DB
command: psql -U bob agent
register: e
- debug: e
This always hangs at the first part of it when logging into the image DB.

Why it doesn't work
This:
- name: Testing DB to make sure it is available
command: psql -U bob image
register: b
- debug: b
- name: Verifying Tables exist in Image
shell: \d image
register: c
- debug: c
doesn't do what you think it does.
The first command runs psql -U bob image. This starts a psql session. psql waits for input from stdin. Ansible will never send any, it is simply waiting for the command you specified to exit, so it can check the exit code.
So Ansible waits for psql to exit, and psql waits for Ansible to send some input.
Each task in Ansible is independent. The shell or command modules do not change the shell that subsequent commands run in. You simply can't do this the way you expect.
Even if psql exited after the first task (or went to the background), you'd just get an error from the second task like:
bash: d: command not found
So the way you're trying to do this just isn't going to work.
How to do it
You need to run each task as a separate psql command, with a command string:
- name: Testing DB to make sure it is available
command: psql -U bob image -c 'SELECT 1;'
- name: Verifying Tables exist in Image
command: psql -U bob image -c '\d image'
... or with standard input, except that Ansible doesn't seem to support supplying a variable as stdin to a command.
... or with a (possibly templated) SQL script:
- name: Template sql script
template: src="my.sql.j2" dest="{{sometemplocation}}/my.sql"
- name: Execute sql script
shell: "psql {{sometemplocation}}/my.sql"
- name: Delete sql script
file: path="{{sometemplocation}}/my.sql" state=absent
Alternately you can use Ansible's built-in support for querying PostgreSQL to do it, but in that case you cannot use the psql client's backslash commands like \d, you'd have to use only SQL. Query information_schema for table info, etc.
Here's how some of my code looks
Here's an example from an automation module I wrote that does a lot with PostgreSQL.
Really, I should just suck it up and write a psql Ansible task that runs commands via psql, rather than using shell, which is awful and clumsy. For now, though, it works. I use connection strings that're assigned from variables or generated using set_fact to reduce the mess a bit and make connections more flexible.
- name: Wait for the target node to be ready to be joined
shell: "{{postgres_install_dir}}/bin/psql '{{bdr_join_target_dsn}}' -qAtw 'SELECT bdr.bdr_node_join_wait_for_ready();'"
- name: Template pre-BDR-join SQL script
template: src="{{bdr_pre_join_sql_template}}" dest="{{postgres_install_dir}}/bdr_pre_join_{{inventory_hostname}}.sql"
- name: Execute pre-BDR-join SQL script
shell: "{{postgres_install_dir}}/bin/psql '{{bdr_node_dsn}}' -qAtw -f {{postgres_install_dir}}/bdr_pre_join_{{inventory_hostname}}.sql"
- name: Delete pre-BDR-join SQL script
file: path="{{postgres_install_dir}}/bdr_pre_join_{{inventory_hostname}}.sql" state=absent
- name: bdr_group_join
shell: "{{postgres_install_dir}}/bin/psql '{{bdr_node_dsn}}' -qAtw -c \"SELECT bdr.bdr_group_join(local_node_name := '{{inventory_hostname}}', node_external_dsn := '{{bdr_node_dsn}}', join_using_dsn := '{{bdr_join_target_dsn}}');\""
- name: Template post-BDR-join SQL script
template: src="{{bdr_post_join_sql_template}}" dest="{{postgres_install_dir}}/bdr_post_join_{{inventory_hostname}}.sql"
- name: Execute post-BDR-join SQL script
shell: "{{postgres_install_dir}}/bin/psql '{{bdr_node_dsn}}' -qAtw -f {{postgres_install_dir}}/bdr_post_join_{{inventory_hostname}}.sql"
- name: Delete post-BDR-join SQL script
file: path="{{postgres_install_dir}}/bdr_post_join_{{inventory_hostname}}.sql" state=absent

The answer that Craig gives is good, but fails to solve the problem of running the commands as a specific user. That can be done with my additions to his code:
- name: Testing DB to make sure it is available
become: true
become_user: postgres
command: psql -U bob image -c 'SELECT 1;'
- name: Verifying Tables exist in Image
become: true
become_user: postgres
command: psql -U bob image -c '\d image'
Note the "become" and "become_user" parameters. These will tell Ansible to change to the correct user before running the commands.
IMPORTANT: Ansible Version 1.9 and earlier use sudo: yes and sudo_user: postgres instead of become: true and become_user: postgres

Building on the excellent responses above, you can also specify environment variables in your Ansible task as shown below. Note that this assumes you have set up a .pgpass file with the password for the target db.
- name: Execute some sql via psql
command: psql -f /path/to/your/sql
environment:
PGUSER: "{{ db_user }}"
PGDATABASE: "{{ db_name }}"
PGHOST: "{{ db_host }}"
PGPASS: "{{ pgpass_filepath }}"

Related

Docker Compose - Container Bash Forking

I am trying to run netbox based on their standard guide on Docker Hub with a slight difference that I need our existing postgres dump to be restored when the postgres container starts.
I have tried a few approaches like defining a command option in docker-compose file like (and a few more combinations):
sleep 2 && psql -U netbox -f netbox.sql
sleep is required to prevent psql command running before the postgres service is started.
Or defining a bash script that does the database restore but all these approaches cause the container to exit after that command/script is run.
My last resort was to utilize bash forking and this is what the postgres snippet of docker-compose looks like:
postgres:
image: postgres:13-alpine
env_file: env/postgres.env
command:
- sh
- -c
- (sleep 3 && cd /home && psql -U netbox -f netbox.sql) & su -c postgres postgres
volumes:
- ./my_db:/home/
- netbox-postgres-data:/var/lib/postgresql/data
Sadly this throws results in:
postgres: could not access the server configuration file
"/var/lib/postgresql/data/postgresql.conf": No such file or directory
If I omit the command section of docker-compose, the container starts up fine and I can navigate and ls the directory in the error message but it is not what I really need because this container will go on to be part of a much larger jungle of an ecosystem with little to no control over it afterwards.
Could it be my bash forking or the problem lies somewhere else?
Thanks in advance
I was able to find a solution by going through the thread that David Maze shared in the comments.
In my case, placing the *.sql file inside /docker-entrypoint-initdb.d did not work but I wrote a bash script, placed it in /docker-entrypoint-initdb.d directory and it got triggered.
The bash script was a very simple one, it would cd to the directory containing the sql dump and then restore it by running psql:
psql -U netbox -f netbox.sql

docker postgres init script seems will not create db user, command is ignored

I have a simple dockerfile:
FROM postgres:latest
ENV POSTGRES_PASSWORD password
ENV POSTGRES_USER postgres
ENV POSTGRES_DB evesde
COPY init.sh /docker-entrypoint-initdb.d/
and my init file is chmod' to 777:
#!/bin/bash
psql -U "postgres" -d "evesde" -e "create role yaml with login encrypted password 'password';"
when running a container it will say:
psql: warning: extra command-line argument "create role yaml with
login encrypted password 'password';" ignored
Im not sure why this is happening, and when doing an interactive terminal, this command seemingly worked. I dont see any additional information and wasnt sure what was going wrong.
The postgres docker page is: https://hub.docker.com/_/postgres
When looking at it deeper, I was noticing that running the command itself fails in an interactive Terminal with the same error, but the command runs when I am in postgres: psql -U "postgres" -d "evesde" and run the command, it works.
I think it may be related to passing the command in through the exec command is where it fails. likely related to '.
You want -c instead of -e.
-e turns on "echo queries"
-c runs the command and exits
Have you considered putting just the create role command in a file called create_role.sql and copying that into /docker-entrypoint-initdb.d/?
Based on testing, it looks like an equivalent but simpler solution is to put the SQL command as one line in a file, 00_roles.sql, and copy that into the container instead of the init.sh script.

Docker compose: MariaDB got an error with init.sql / entrypoint

have the following service in my docker compose file:
database:
image: mariadb
container_name: database
environment:
MYSQL_ROOT_PASSWORD: 7ctDGg5YUwkCPkCW
entrypoint:
sh -c "echo 'CREATE DATABASE IF NOT EXISTS users;
CREATE DATABASE IF NOT EXISTS data;
CREATE DATABASE IF NOT EXISTS wordpress;
CREATE USER IF NOT EXISTS 'keycloak'#localhost IDENTIFIED BY 'jk2zKvGkJXBsrNMV';
GRANT ALL PRIVILEGES ON 'users'.* TO 'keycloak'#localhost;
CREATE USER IF NOT EXISTS 'wordpress'#localhost IDENTIFIED BY 'QKJFUfZbv7jMB5ba';
GRANT ALL PRIVILEGES ON 'wordpress'.* TO 'wordpress'#localhost;' > /docker-entrypoint-initdb.d/init.sql;
/usr/local/bin/docker-entrypoint.sh --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
"
ports:
- 3306:3306
volumes:
- database_data:/var/lib/mysql
networks:
- backend-network
The mariadb service start ups without problems, but when it comes to the point where it is trying to run the init.sql I got the following error message:
database | 2020-06-13 18:20:03+00:00 [Note] [Entrypoint]: /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
database | ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'jk2zKvGkJXBsrNMV' at line 1
database exited with code 1
Hopefully there is some one out there, who is able to tell me whats wrong.
So far,
Daniel
Take the content of your echo statement and move it to a separate file; for example, init.sql in the current directory. You can bind-mount that specific file into the /docker-entrypoint-initdb.d directory:
database:
image: mariadb
environment:
MYSQL_ROOT_PASSWORD: 7ctDGg5YUwkCPkCW
ports:
- 3306:3306
volumes:
- database_data:/var/lib/mysql
- ./init.sql:/docker-entrypoint-initdb.d/init.sql
What's actually happening with the command you're showing is that you're using single quotes as both a shell quoting delimiter and an SQL quoting delimiter.
echo '...IDENTIFIED BY 'something';'
In this case the single quotes around the password end the shell quoting and just get lost
# same as:
echo ...IDENTIFIED\ BY\ something\;
If you really needed to do it this way you could use the list format for the command to spell out the words specifically (without needing to separately quote them), and then use YAML block scalar syntax to avoid needing to separately quote the command, at which point you can use double quotes to quote the echo argument
command:
- sh
- -c
- >-
echo "... IDENTIFIED BY 'password'; ..."...
But it will be more understandable and less fragile to just split this out into a separate file. Avoid writing complex scripts inline in YAML files.

Specify PostgreSQL passwords in Ansible Playbook

New to Ansible, Running version 2.1.0. I've written an Ansible playbook that runs a PostgreSQL query against a group of hosts. When I specify the SQL DB password in the shell command it works, but I'm looking to run the playbook against a group of hosts and need a better way to input the passwords as they are all unique. Could anyone suggest a better way to do this?
---
- hosts: Test_Hosts
sudo: yes
sudo_user: root
gather_facts: yes
tasks:
- name: Login to DB and run command
shell: export PGPASSWORD='Password'; psql -U 'user' -d 'db' -c 'select * FROM table';
register: select_all_from_table
- name: Display table contents
debug: msg="{{ select_all_from_table.stdout }}"
I saw another thread on the topic but was not sure how to implement the suggestion: Run a postgresql command with ansible playbook. Postgresql requires password
Ansible allows you to set environment variables for a task using the environment parameter to any task.
So in your case you could just do this:
- name: Login to DB and run command
shell: psql -U 'user' -d 'db' -c 'select * FROM table';
register: select_all_from_table
environment:
PGPASSWORD: '{{ pgpassword }}'
And then set the pgpassword variable at the group or host level.
I just ran into this issue today and this is what worked for me. On Linux you can package all the credentials into a ~/.pgpass hidden file.
Just create it locally (in this case in ./files/pgpass) and then use ansible to copy it onto the host before you run the psql commands.
- name: set passwd file for PSQL
copy:
src: files/pgpass
dest: ~/.pgpass
mode: 0600 ### important: will not work with wrong permissions
- name: PSQL command
shell: "psql -U 'user' -d 'db' -c 'select * FROM table'"
register: select_all_from_table
The file contents must be in the following format:
hostname:port:database:username:password
However, you can use wildcards, so mine looks like this, for example:
*:*:db1:user1:passwd1
*:*:db2:user2:passwd2
See documentation for more details:
https://www.postgresql.org/docs/9.1/static/libpq-pgpass.html

Ansible: have sudo but no root

I’d like to use Ansible to manage the configuration of a our Hadoop cluster (running Red Hat).
I have sudo access and can manually ssh into the nodes to execute commands. However, I’m experiencing problems when I try to run Ansible modules to perform the same tasks. Although I have sudo access, I can’t become root. When I try to execute Ansible scripts that require elevated privileges, I get an error like this:
Sorry, user awoolford is not allowed to execute '/bin/bash -c echo
BECOME-SUCCESS- […] /usr/bin/python
/tmp/ansible-tmp-1446662360.01-231435525506280/copy' as awoolford on
[some_hadoop_node].
Looking through the documentation, I thought that the become_allow_same_user property might resolve this, and so I added the following to ansible.cfg:
[privilege_escalation]
become_allow_same_user=yes
Unfortunately, it didn't work.
This post suggests that I need permissions to sudo /bin/sh (or some other shell). Unfortunately, that's not possible for security reasons. Here's a snippet from /etc/sudoers:
root ALL=(ALL) ALL
awoolford ALL=(ALL) ALL, !SU, !SHELLS, !RESTRICT
Can Ansible work in an environment like this? If so, what am I doing wrong?
Well, you simply cannot execute /bin/sh or /bin/bash as your /etc/sudoers shows. What you could do is change ansible's default shell to something else (variable executable in ansible.conf).
Since your sudo policy allows everything by default (does not seem like really secure to me), and I suppose ansible expects an sh-compatible shell, as a really dirty hack you could copy /bin/bash to some other path/name and set the executable variable accordingly (not tested).
In the playbook (some.yml) file, set
runthisplaybook.yml
---
- hosts: label_which_will_work_on_some_servers
sudo: yes
roles:
- some_role_i_want_to_run
Next, in the role//tasks/main.yml for the action which you have to run as sudo.. use something like become_user (where common_user is a variable defined in some role's defaults\main.yml file as common_user: "this_user_can_sudo":
- name: Run chkconfig on init script
command: "sudo -u root /sbin/chkconfig --add tomcat"
# Set execute permission on run_jmeter_test.sh
- name: Set execute permission on run_jmeter_test.sh
command: "chmod -R 755 {{ jmeter_perf_tests_results }}"
become_user: "{{ common_user }}"
# OR Set execute permission on run_jmeter_test.sh
- name: Set execute permission on run_jmeter_test.sh
command: "sudo -u firstuser sudo -u seconduser chmod -R 755 {{ jmeter_perf_tests_results }}"
become_user: "{{ common_user }}"
# OR Set execute permission on run_jmeter_test.sh
- name: Set execute permission on run_jmeter_test.sh
command: "chmod -R 755 {{ jmeter_perf_tests_results }}"
become_user: "{{ common_user }}"
PS: While running ansible-playbook,
ansible-playbook runthisplaybook.yml --sudo-user=this_user_can_sudo -i hosts.yml -u user_which_will_connect_from_source_machine --private-key ${DEPLOYER_KEY_FILE} --extra-vars "target_svr_type=${server_type} deploy_environment=${DEPLOY_ENVIRONMENT} ansible_user=${ANSIBLE_USER}"
After a research over the subject, as of Ansible 2.8 it doesn't seem you have a way to run commands as a different user using become without root permissions.
There's another way to achieve what you were asking without being so, how to put it, 'hacky'.
You can use the shell module with sudo su - <user> -c "COMMAND" to execute a command as a different user, without the need for root access to the original user.
For example,
1 ---
2 - hosts: target_host
3
4 tasks:
5 - shell: 'sudo su EXEC_USER -c "whoami"'
6 register: x
7
8 - debug:
9 msg: "{{ x.stdout_lines }}" # This returns EXEC_USER
However, if your play is complex, you would need to break it down and wrap only the commands that are required to be executed as different user.
This isn't best practice (using sudo + shell instead of become), however that's a solution, and in my opinion a better one than creating dummy shell on every node you manage.
I think now sudo: yes is depricated and replace with become: yes
---
- hosts: servers_on_which_you_want_to_run
become: yes
roles:
- some_role
The smiplist solution is just create a ansible.cfg in your playbook directory with the following content, if it doesn't accept root user:
[defaults]
sudo_user = UsernameToWhichYouWantToUse
Hope, this will solve your problem.