I have a file containing multiple similar sections
...
# PostgreSQL configuration example
#production:
# adapter: postgresql
# database: redmine
# host: localhost
# username: postgres
# password: "postgres"
# SQLite3 configuration example
#production:
# adapter: sqlite3
# database: db/redmine.sqlite3
# SQL Server configuration example
#production:
# adapter: sqlserver
# database: redmine
# host: localhost
# username: jenkins
# password: jenkins
...
I would like to comment/decomment an entire section.
With "SQLite3" I would like to have the following output:
...
# PostgreSQL configuration example
#production:
# adapter: postgresql
# database: redmine
# host: localhost
# username: postgres
# password: "postgres"
# SQLite3 configuration example
production:
adapter: sqlite3
database: db/redmine.sqlite3
# SQL Server configuration example
#production:
# adapter: sqlserver
# database: redmine
# host: localhost
# username: jenkins
# password: jenkins
...
I tried something like:
sed -i -e '/SQLite3/+1,/^\s*$/ s/^#//' FILE
... but it doesn't work because '+1' is not valid.
How can I start a range on the line after the match?
(I tried several variations with braces and 'n', but I didn't find the right spell)
Select the complete range, then apply the substitute command only to a subrange of lines not matching the first pattern:
sed '/SQLite3/,/^$/ { /SQLite3/! s/^#// }' file
Expanded for readability:
sed '/SQLite3/,/^$/ {
/SQLite3/! {
s/^#//
}
}' file
Related
I'd love to run pgadmin4 in our infrastructure in a way, that postgres servers would be preconfigured during docker build/1.st start.
I've tried to modify the internaly used /var/lib/pgadmin/pgadmin4.db sqlite DB on the 1.st start, which however results in an error in the UI (once selecting the particular postgres server:
definition of service "" not found
I've tried following:
Directory structure:
find ./ -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g'
|____
|____dump
| |____servergroup.csv
| |____server.csv
| |____import_db.sh
|____Dockerfile
Where Dockerfile is:
cat Dockerfile
# rebuild:
# docker build -t pgadmin4:3.0-custom .
# run:
# docker run --rm -it -e PGADMIN_DEFAULT_EMAIL=admin -e PGADMIN_DEFAULT_PASSWORD=admin -p8081:80 docker build -t pgadmin4:3.0-custom
FROM dpage/pgadmin4:3.0
COPY dump/ /dump
RUN \
apk add --no-cache sqlite && \
chmod +x /dump/import_db.sh && \
# re rely on the current entrypoint.sh impl
sed -i '/python run_pgadmin.py/a \/dump\/import_db.sh' /entrypoint.sh && \
cat /entrypoint.sh
In fact it just modifies the https://github.com/postgres/pgadmin4/blob/master/pkg/docker/entrypoint.sh to run import_db.sh script on the 1.st start.
Where dump/import_db.sh is:
cat dump/import_db.sh
#!/bin/sh
echo ".tables" | sqlite3 -csv /var/lib/pgadmin/pgadmin4.db
# remove header and `1,1,Servers` entry (would cause duplicates)
cat /dump/servergroup.csv | sed '1d' | grep -v 1,1,Servers > /tmp/servergroup.in.csv
echo "csv servergroup:"
cat /tmp/servergroup.in.csv
echo "DB servergroup:"
sqlite3 -csv -header /var/lib/pgadmin/pgadmin4.db "select * from servergroup;"
echo ".import /tmp/servergroup.in.csv servergroup" | sqlite3 -csv /var/lib/pgadmin/pgadmin4.db
# remove header
cat /dump/server.csv | sed '1d' > /dump/server.in.csv
echo "csv server:"
cat /dump/server.in.csv
echo "DB server:"
sqlite3 -csv -header /var/lib/pgadmin/pgadmin4.db "select * from server;"
echo ".import /dump/server.in.csv server" | sqlite3 -csv /var/lib/pgadmin/pgadmin4.db
Csv files contents:
cat dump/server.csv
id,user_id,servergroup_id,name,host,port,maintenance_db,username,password,role,ssl_mode,comment,discovery_id,hostaddr,db_res,passfile,sslcert,sslkey,sslrootcert,sslcrl,sslcompression,bgcolor,fgcolor,service
1,1,2,servername,localhost,5432,postgres,postgres,"",,prefer,,,"","",,<STORAGE_DIR>/.postgresql/postgresql.crt,<STORAGE_DIR>/.postgresql/postgresql.key,,,0,,,
cat dump/servergroup.csv
id,user_id,name
2,1,my-group
1,1,Servers
Any idea how to fix my error? Or of any other approach that could provide me the pre-configured pgadmin4 docker container?
The current version of image dpage/pgadmin is 4.24. This version has support for external configuration of server definition list (servers.json):
{
"Servers": {
"test": {
"Name": "test",
"Group": "Servers",
"Port": 5432,
"Username": "postgres",
"Host": "postgres",
"SSLMode": "prefer",
"MaintenanceDB": "postgres"
}
}
}
Volume binding can be configured as below:
volumes:
- ./servers.json:/pgadmin4/servers.json
First time container is started server groups and servers will be configured automaticaly.
UPD. JSON format has more fields which are optional. It's important that password can not be imported/exported in such way due to the obvious security reasons.
Looks this change the service column value to an empty string instead of NULL.
Can you try updating the value of service column to NULL
sqlite> UPDATE server SET service = NULL;
commit the changes and Restart pgAdmin4 & try again connecting to that server.
I am trying to connect a Postgres RDS serverless instance from a CodeBuild project.
this is where it fails:
psql --host ${PG_HOST} --dbname ${PG_DBNAME} --user ${SECRET_USER} -f /tmp/file
/tmp/file exists, so a file permission/non-existing issue is out of question
What I have tried so far, the username and password have been:
Stored as environment variables in CodeBuild
Stored in AWS Secrets manager
env:
secrets-manager:
# key: secret-id:json-key:version-stage:version-id
SECRET_USER: rds-db-credentials:username
SECRET_PASSWORD: rds-db-credentials:password
Given as parameters in the call directly
psql --host ${PG_HOST} --dbname ${PG_DBNAME} "user=mydbuser password=0fNKJtNv" -f /tmp/file;
Stored in .pgpass file
echo ${PG_HOST}:${PG_PORT}:${PG_DBNAME}:${SECRET_USER}:${SECRET_PASSWORD} > ~/.pgpass
chmod 600 ~/.pgpass
And to be sure, I exported the variable too, PGPASSFILE="~/.pgpass"
When I echo the username and password, I only get *** printed, like:
[Container] 2020/09/27 07:39:34 Running command cat ~/.pgpass
something.eu-central-1.rds.amazonaws.com:5432:spumdb:***:***
Errors:
For psql --host ${PG_HOST} --dbname ${PG_DBNAME} --user ${PG_USER} -f /tmp/file, the error is
psql: warning: extra command-line argument "/tmp/file" ignored
Password for user -f:
psql: fe_sendauth: no password supplied
For psql --host ${PG_HOST} --dbname ${PG_DBNAME} --user ${SECRET_USER} -f /tmp/file, the error is
Password for user ***:
psql: fe_sendauth: no password supplied
For psql --host ${PG_HOST} --dbname ${PG_DBNAME} "user=mydbuser password=0fNKJtNv" -f /tmp/file, the error is:
Password for user user=*** password=***:
psql: fe_sendauth: no password supplied
Just for info,
The RDS security group has the entry to allow all TCP connections from CodeBuild in the AWS region where the project is (35.157.127.248/29)
Both CodeBuild and RDS lie in the same private subnet of an user created VPC
I am able to connect to the RDS instance from an EC2 instance using the same RDS credentials
I am using the latest image for Amazon Linux 2 (aws/codebuild/amazonlinux2-x86_64-standard:3.0)
It looks like a problem at the CodeBuild end, not at the RDS end. For some reason, CodeBuild doesn't get the value of the parameters, that too only username and password, others like hostname, dbname are evaluated correctly!
Does anybody see any problem anywhere? Thank you!
I tried to replicate the issue, but the only thing I found that you should be using --username, not --user. Anyway, here is my buildspec.yml used for the verification:
version: 0.2
env:
variables:
PG_HOST: database-1.cm3c1syrcj06.us-east-1.rds.amazonaws.com
PG_PORT: 5432
PG_DBNAME: mydb
secrets-manager:
SECRET_USER: rds-db-credentials:username
SECRET_PASSWORD: rds-db-credentials:password
phases:
pre_build:
commands:
- echo ${PG_HOST}:${PG_PORT}:${PG_DBNAME}:${SECRET_USER}:${SECRET_PASSWORD} > ~/.pgpass
- chmod 600 ~/.pgpass
build:
commands:
- cat ~/.pgpass | rev
- echo "\dt" > /tmp/file
- psql --host ${PG_HOST} --dbname ${PG_DBNAME} --username ${SECRET_USER} -f /tmp/file
In the above I use a little trick to show the password and username, but in reverse. Otherwise you see only ***.
Everything works as expected, and CB connects the database as evident by the No relations found (this was expected, as my db is empty):
New to Ansible, Running version 2.1.0. I've written an Ansible playbook that runs a PostgreSQL query against a group of hosts. When I specify the SQL DB password in the shell command it works, but I'm looking to run the playbook against a group of hosts and need a better way to input the passwords as they are all unique. Could anyone suggest a better way to do this?
---
- hosts: Test_Hosts
sudo: yes
sudo_user: root
gather_facts: yes
tasks:
- name: Login to DB and run command
shell: export PGPASSWORD='Password'; psql -U 'user' -d 'db' -c 'select * FROM table';
register: select_all_from_table
- name: Display table contents
debug: msg="{{ select_all_from_table.stdout }}"
I saw another thread on the topic but was not sure how to implement the suggestion: Run a postgresql command with ansible playbook. Postgresql requires password
Ansible allows you to set environment variables for a task using the environment parameter to any task.
So in your case you could just do this:
- name: Login to DB and run command
shell: psql -U 'user' -d 'db' -c 'select * FROM table';
register: select_all_from_table
environment:
PGPASSWORD: '{{ pgpassword }}'
And then set the pgpassword variable at the group or host level.
I just ran into this issue today and this is what worked for me. On Linux you can package all the credentials into a ~/.pgpass hidden file.
Just create it locally (in this case in ./files/pgpass) and then use ansible to copy it onto the host before you run the psql commands.
- name: set passwd file for PSQL
copy:
src: files/pgpass
dest: ~/.pgpass
mode: 0600 ### important: will not work with wrong permissions
- name: PSQL command
shell: "psql -U 'user' -d 'db' -c 'select * FROM table'"
register: select_all_from_table
The file contents must be in the following format:
hostname:port:database:username:password
However, you can use wildcards, so mine looks like this, for example:
*:*:db1:user1:passwd1
*:*:db2:user2:passwd2
See documentation for more details:
https://www.postgresql.org/docs/9.1/static/libpq-pgpass.html
I am looking for a way to run a Postgres script using Ansible. While I found a reasonably good example Here, I need to:
Run the script as user postgres
I don't necessarily need to keep a copy of the script on the server so if I need to have a copy, it will only be for temp use.
Can anyone tell me if this is possible and if so an example of running it. Here is what I tried so far using Ansible and it just hung at these points:
- name: Testing DB to make sure it is available
command: psql -U bob image
register: b
- debug: b
- name: Verifying Tables exist in Image
shell: \d image
register: c
- debug: c
- name: Exiting Image DB
shell: \q
register: d
- debug: d
- name: Going to Agent DB
command: psql -U bob agent
register: e
- debug: e
This always hangs at the first part of it when logging into the image DB.
Why it doesn't work
This:
- name: Testing DB to make sure it is available
command: psql -U bob image
register: b
- debug: b
- name: Verifying Tables exist in Image
shell: \d image
register: c
- debug: c
doesn't do what you think it does.
The first command runs psql -U bob image. This starts a psql session. psql waits for input from stdin. Ansible will never send any, it is simply waiting for the command you specified to exit, so it can check the exit code.
So Ansible waits for psql to exit, and psql waits for Ansible to send some input.
Each task in Ansible is independent. The shell or command modules do not change the shell that subsequent commands run in. You simply can't do this the way you expect.
Even if psql exited after the first task (or went to the background), you'd just get an error from the second task like:
bash: d: command not found
So the way you're trying to do this just isn't going to work.
How to do it
You need to run each task as a separate psql command, with a command string:
- name: Testing DB to make sure it is available
command: psql -U bob image -c 'SELECT 1;'
- name: Verifying Tables exist in Image
command: psql -U bob image -c '\d image'
... or with standard input, except that Ansible doesn't seem to support supplying a variable as stdin to a command.
... or with a (possibly templated) SQL script:
- name: Template sql script
template: src="my.sql.j2" dest="{{sometemplocation}}/my.sql"
- name: Execute sql script
shell: "psql {{sometemplocation}}/my.sql"
- name: Delete sql script
file: path="{{sometemplocation}}/my.sql" state=absent
Alternately you can use Ansible's built-in support for querying PostgreSQL to do it, but in that case you cannot use the psql client's backslash commands like \d, you'd have to use only SQL. Query information_schema for table info, etc.
Here's how some of my code looks
Here's an example from an automation module I wrote that does a lot with PostgreSQL.
Really, I should just suck it up and write a psql Ansible task that runs commands via psql, rather than using shell, which is awful and clumsy. For now, though, it works. I use connection strings that're assigned from variables or generated using set_fact to reduce the mess a bit and make connections more flexible.
- name: Wait for the target node to be ready to be joined
shell: "{{postgres_install_dir}}/bin/psql '{{bdr_join_target_dsn}}' -qAtw 'SELECT bdr.bdr_node_join_wait_for_ready();'"
- name: Template pre-BDR-join SQL script
template: src="{{bdr_pre_join_sql_template}}" dest="{{postgres_install_dir}}/bdr_pre_join_{{inventory_hostname}}.sql"
- name: Execute pre-BDR-join SQL script
shell: "{{postgres_install_dir}}/bin/psql '{{bdr_node_dsn}}' -qAtw -f {{postgres_install_dir}}/bdr_pre_join_{{inventory_hostname}}.sql"
- name: Delete pre-BDR-join SQL script
file: path="{{postgres_install_dir}}/bdr_pre_join_{{inventory_hostname}}.sql" state=absent
- name: bdr_group_join
shell: "{{postgres_install_dir}}/bin/psql '{{bdr_node_dsn}}' -qAtw -c \"SELECT bdr.bdr_group_join(local_node_name := '{{inventory_hostname}}', node_external_dsn := '{{bdr_node_dsn}}', join_using_dsn := '{{bdr_join_target_dsn}}');\""
- name: Template post-BDR-join SQL script
template: src="{{bdr_post_join_sql_template}}" dest="{{postgres_install_dir}}/bdr_post_join_{{inventory_hostname}}.sql"
- name: Execute post-BDR-join SQL script
shell: "{{postgres_install_dir}}/bin/psql '{{bdr_node_dsn}}' -qAtw -f {{postgres_install_dir}}/bdr_post_join_{{inventory_hostname}}.sql"
- name: Delete post-BDR-join SQL script
file: path="{{postgres_install_dir}}/bdr_post_join_{{inventory_hostname}}.sql" state=absent
The answer that Craig gives is good, but fails to solve the problem of running the commands as a specific user. That can be done with my additions to his code:
- name: Testing DB to make sure it is available
become: true
become_user: postgres
command: psql -U bob image -c 'SELECT 1;'
- name: Verifying Tables exist in Image
become: true
become_user: postgres
command: psql -U bob image -c '\d image'
Note the "become" and "become_user" parameters. These will tell Ansible to change to the correct user before running the commands.
IMPORTANT: Ansible Version 1.9 and earlier use sudo: yes and sudo_user: postgres instead of become: true and become_user: postgres
Building on the excellent responses above, you can also specify environment variables in your Ansible task as shown below. Note that this assumes you have set up a .pgpass file with the password for the target db.
- name: Execute some sql via psql
command: psql -f /path/to/your/sql
environment:
PGUSER: "{{ db_user }}"
PGDATABASE: "{{ db_name }}"
PGHOST: "{{ db_host }}"
PGPASS: "{{ pgpass_filepath }}"
How to add rows into table in the Database by Shell Script?
I have following:
Postgres Database Name: dm
Table Name: Error Codes
Password of Database: ******
I want to add rows into table?
I am just starting Shell Scripting...
My code:
#!/bin/sh
DATABASE="dm"
USERNAME="postgres"
HOSTNAME="HBG"
export PGPASSWORD="postgres"
psql -h $HOSTNAME -U $USERNAME $DATABaSE << EOF
select * from Error Codes
EOF
Getting error:
psql: could not connect to server: Connection refused
Is the server running on host "HBG" (192.168.0.241) and accepting
TCP/IP connections on port 5432?
Have a look on how to use the psql command.
You can state any SQL query using this command. For example:
psql -c "SELECT * FROM foo" mydatabase myusername
Of course you can also use INSERT, UPDATE and all other SQL commands, as long as myusername has the rights to do so.
If you want to do this WITHOUT a password (but this is strongly disrecommended and unsecure) you can add this at the start of your script:
set PGPASSWORD=<password>
See here for full documentation: http://www.postgresql.org/docs/7.4/static/app-psql.html