Environment:
We have ~100 databases hosted on multiple machines. All databases have different username and different password on every database.
Issue:
How can I read the username and password and TNS (Service name) from an external file one by one and connect to the each database and run the sql query and pass the output to a logfile.
What is done so far:
Created pass file that contains password, Service_Name, Username and delimiter ":"
cat pass
Loufr#123:PROD:User1
Brinj#6523:TEST:User2
Another file dblist is created with all the dbnames (that matches the service names) in step 1
cat dblist
PROD
TEST
DEV
QA
Quality
Goal:
read name of the database from dblist and find the username and password from pass file for that database.
Use dbname with username and password to connect to database in following string
while read TNS from dblist
sqlplus -s /nolog
connect ${Username}#${TNS}/${Password}
spool output.log
#query.sql
exit
You can parse the file pass this way (replace all the : with spaces and initialize the positional parameters with the resulting tokens):
while read TNS from dblist; do
set -- $(sed -n "/:$TNS:/{s,:, ,gp}" pass)
sqlplus -s /nolog
connect ${1}#${2}/${3}
spool output.log
#query.sql
exit
done
Related
I am trying to copy data from a csv file from my local machine into a remote pgsql table named states, but i am getting an ERROR: Syntax error at or near "FROM". Can someone guide me as to why i am receiving this error?
COPY FROM STDIN states FROM '/Users/Shared/data.csv' DELIMITER AS ',';
The problem is that the path to the file is in the remote server, not the local one.
you need psql and pipe the file to STDIN:
psql -h host -d remoteDB -U myuser -c "copy states from STDIN with delimiter as ',';" < /path/file.csv
alternatively you can also do:
cat /path/file.csv | psql -h host -d remoteDB -U myuser -c "copy states from STDIN with delimiter as ',';"
You can't do it directly with a filename unless the file is on the Postgres server. The docs of COPY state:
COPY with a file name instructs the PostgreSQL server to directly read from or write to a file. The file must be accessible to the server and the name must be specified from the viewpoint of the server. When STDIN or STDOUT is specified, data is transmitted via the connection between the client and the server.
You'll have to pipe the file in via STDIN.
Many Postgres drivers provide a method to make this easier. For example, ruby-pg provides copy_data.
conn.copy_data "COPY states FROM STDIN FORMAT CSV" do
File.foreach('/Users/Shared/data.csv') do |line|
conn.put_copy_data(line)
end
end
Trying to dump the output of a query into a CSV file in an automated job and running into an issue with fields where the column contains my comma delimiter. With the nature of this particular network, I have to jump through a couple of hoops to get things done, and there's a good chance I'm missing something very obvious.
In a nutshell, I kick off my script from a client machine that uses PLINK to run a remote psql command on another box over an SSH connection. That psql command is hitting a Postgres server on a third machine (I can't connect directly from client to DB, hence the extra step in between).
If I manually SSH from client to server 1, connect to the Postgres box, and use \copy... with CSV header, the file that's created is perfect, and any fields that contain a comma are automatically surrounded by double quotes.
However, if I try go issue that \copy (or copy) command in a single command, the output doesn't contain those double quotes, so I end up in that situation where commas in a field are interpreted as a delimiter later one.
In other words, this has the necessary double-quotes:
SSH from client to server1.
psql -Uuser -h server2 database
\copy (select ...) to '~/myfile.csv' with CSV header;
But doesn't:
SSH from client to server1
psql -Uuser -h server2 database -c "\copy (select ...) to '~/myfile.csv' with CSV header;"
Using FORCE_QUOTE
Here is how to do:
psql -U user -h server2 database -c "\copy (select ...) to '~/myfile.csv' WITH (FORMAT CSV, HEADER TRUE, FORCE_QUOTE *);"
COPY command documentation
I run scripts against my database like this...
psql -d myDataBase -a -f myInsertFile.sql
The only problem is I want to be able to specify in this command what schema to run the script against. I could call set search_path='my_schema_01' but the files are supposed to be portable. How can I do this?
You can create one file that contains the set schema ... statement and then include the actual file you want to run:
Create a file run_insert.sql:
set schema 'my_schema_01';
\i myInsertFile.sql
Then call this using:
psql -d myDataBase -a -f run_insert.sql
More universal way is to set search_path (should work in PostgreSQL 7.x and above):
SET search_path TO myschema;
Note that set schema myschema is an alias to above command that is not available in 8.x.
See also: http://www.postgresql.org/docs/9.3/static/ddl-schemas.html
Main Example
The example below will run myfile.sql on database mydatabase using schema myschema.
psql "dbname=mydatabase options=--search_path=myschema" -a -f myfile.sql
The way this works is the first argument to the psql command is the dbname argument. The docs mention a connection string can be provided.
If this parameter contains an = sign or starts with a valid URI prefix
(postgresql:// or postgres://), it is treated as a conninfo string
The dbname keyword specifies the database to connect to and the options keyword lets you specify command-line options to send to the server at connection startup. Those options are detailed in the server configuration chapter. The option we are using to select the schema is search_path.
Another Example
The example below will connect to host myhost on database mydatabase using schema myschema. The = special character must be url escaped with the escape sequence %3D.
psql postgres://myuser#myhost?options=--search_path%3Dmyschema
The PGOPTIONS environment variable may be used to achieve this in a flexible way.
In an Unix shell:
PGOPTIONS="--search_path=my_schema_01" psql -d myDataBase -a -f myInsertFile.sql
If there are several invocations in the script or sub-shells that need the same options, it's simpler to set PGOPTIONS only once and export it.
PGOPTIONS="--search_path=my_schema_01"
export PGOPTIONS
psql -d somebase
psql -d someotherbase
...
or invoke the top-level shell script with PGOPTIONS set from the outside
PGOPTIONS="--search_path=my_schema_01" ./my-upgrade-script.sh
In Windows CMD environment, set PGOPTIONS=value should work the same.
I'm using something like this and works very well:* :-)
(echo "set schema 'acme';" ; \
cat ~/git/soluvas-framework/schedule/src/main/resources/org/soluvas/schedule/tables_postgres.sql) \
| psql -Upostgres -hlocalhost quikdo_app_dev
Note: Linux/Mac/Bash only, though probably there's a way to do that in Windows/PowerShell too.
This works for me:
psql postgresql://myuser:password#myhost/my_db -f myInsertFile.sql
In my case, I wanted to add schema to a file dynamically so that whatever schema name user will provide from the cli, I will run sql file with that provided schema name.
For this, I replaced some text in the sql file. First I added {{schema}} in the file like this
CREATE OR REPLACE FUNCTION {{schema}}.usp_dailygaintablereportdata(
then replace {{schema}} dynamically with user provided schema name with the help of sed command
sed -i "s/{{schema}}/$pgSchemaName/" $filename
result=$(psql -U $user -h $host -p $port -d $dbName -f "$filename" 2>&1)
sed -i "s/$pgSchemaName/{{schema}}/" $filename
First replace is done, then target file is run and then again our replace is reverted back
I was facing similar problems trying to do some dat import on an intermediate schema (that later we move on to the final one). As we rely on things like extensions (for example PostGIS), the "run_insert" sql file did not fully solved the problem.
After a while, we've found that at least with Postgres 9.3 the solution is far easier... just create your SQL script always specifying the schema when refering to the table:
CREATE TABLE "my_schema"."my_table" (...);
COPY "my_schema"."my_table" (...) FROM stdin;
This way using psql -f xxxxx works perfectly, and you don't need to change search_paths nor use intermediate files (and won't hit extension schema problems).
During the installation of my app, I want to create a PostgreSQL-Database and some tables and functions.
For that purpose I use PSQL.EXE that ships with PostgreSQL. I have 2 scripts. The first one creates the database and a corresponding user that has rights to execute scripts on that database. I want to execute the second script as this just created user. Unfortunately I can't find a way to pass the password for that user as a command line argument. Omitting the password leads to a stop of execution and a prompt for the user to enter the password, which I would like to avoid - since this is executed during installtion of my app.
Is there any way to pass the password as argument or is there any other command line tool I could use?
To explain the environment a bit further. I use WiX 3.5 setup as a "MSI-Builder".
You can either use a pgpass file as dbenhur answerd, or you can set the environment variable PGPASSWORD before calling psql:
SET PGPASSWORD=my_very_secret_password
psql somedb someuser
All supported environment variables are documented in the manual: http://www.postgresql.org/docs/current/static/libpq-envars.html
You can't supply password via cmdline arg (and don't want to as that's poor security practice).
You can provide a .pgpass file to support automatic script authentication. Here's the docs.
Better still, if you have access to create the db role then you already have all the access you need without having to carefully log in with a password. Have the second script operate under the same user as the first but include the following line to switch user:
set role my_new_user;
Where my_new_user is the name of the role you want to run it as.
If you only divided the scripts because of the different logins then with this they can go in the same file and just switch role mid way through.
Note:
On the off chance that you are not creating the DB and new role as a super user this may be a little more complex. If this is the case you will need to create the new role with:
create role my_new_role ... ADMIN my_role;
Where my_new_role is the role you're creating and my_role is your current user. Then when you're finished simply:
revoke my_new_role from my_role;
For completion, you can also use URI (doc link)
List dbs
psql "postgresql://username:password#localhost/postgres" -l
I also crafted this command to have only names (please tell me if you know a better way):
psql "postgresql://username:password#localhost/postgres" -l | awk -F '|' '{print $1}'| sed -e '/^\s*$/ d' -e '1,3d'|sed '$d'|awk '{print $1}'
You can also use unix socket to connect:
# ss -x -a |grep postgres|awk '{print $5}'
/var/run/postgresql/.s.PGSQL.5432
Note that the parent directory of the socket is used:
# sudo -u postgres psql -d "postgresql:///postgres?host=/var/run/postgresql/" -l
You can only do this if you have this line in your pg_hba.conf:
local all postgres ident
"ident" uses unix user for authent
dump a db
Here I added a different port number
pg_dump -Fc "postgresql://username:password#localhost:9001/${db}" > "backup_${db}.pgdump"
With dumpall you need a super user or role (with CREATE ROLE ... SUPERUSER). And it must have access to all DB. By default postgres can.
but in my case I couldn't use pg_dumpall with postgres because his password was removed by devs.
So I used:
sudo -u postgres pg_dumpall -d "postgresql:///?host=/var/run/postgresql/" > all.dump
tested version
# cat /opt/postgresql/PG_VERSION
9.6
hth
DB: PostgreSQL 9.0
Client: Windows 7
Server Windows 2008, 64bit
I'm trying to connect remotely to a PostgreSQL instance for purposes of performing a pg_dump to my local machine.
Everything works from my client machine, except that I need to provide a password at the password prompt, and I'd ultimately like to batch this with a script.
I've followed the instructions here:
http://www.postgresql.org/docs/current/static/libpq-pgpass.html
But it's not working.
To recap, I've created a file on the server: C:/Users/postgres/AppData/postgresql/pgpass.conf, where PostgreSQL is the db user.
The file has one line with the following data:
\*:5432:\*postgres:[mypassword]
I've also tried replacing each * with [localhost|myip] and [mydatabasename] respectively.
From my client machine, I connect using:
pg_dump -h <myip> -U postgres -w [mydbname] > [mylocaldumpfile]
I'm presuming that I need to provide the -w switch in order to ignore password prompt, at which point it should look in the AppData directory on the server.
It just comes back with:
connection to database failed: fe_sendauth: no password supplied.
As a hack workaround, if there was a way I could tell the Windows batch file on my client machine to inject the password at the PostgreSQL prompt, that would work as well.
It works for me:
Use command line
cd %appdata%
mkdir postgresql
cd postgresql
notepad pgpass.conf
inside pgpass.conf paste your connection string (*:5432:*postgres:[mypassword]) and save the file.
To connect to postgres use:
psql/pg_dump -U <username> -h <host> -w <other params you want to use>
I have solved similar problem (only in Linux) to use ip address in pgpass and psql.
.pgpass
127.0.0.1:5432:db:dbuser:123
psql params
psql -d db -U dbuser -h 127.0.0.1 -w
pg_hba conf with default settings:
# IPv4 local connections:
84 host all all 127.0.0.1/32 md5
Create pgpass.conf file
Windows > Start > Run
type %APPDATA%
press OK
Create a folder: postgresql
Create a file : pgpass.conf (under postgresql folder)
Open pgpass.conf file
Now you should have below file ready, open it via below (making sure it exists):
Windows > Start > Run
type %APPDATA%\postgresql\pgpass.conf
press OK
Paste pgpass.conf file contents
Paste the below
# serverDomainOrIP:PORT:databaseName:userName:password
# 127.0.0.1:5432:myDbName:postgres:myPassword
# *:5432:*:*:myPassword
You can do one of the below:
- Remove # for the 3rd line, and give your password in the place of "myPassword"
OR
- Remove # for the 2nd line, and give ip (or, yourDomain.com), dbname, username & password
Hope that helps.
I've had a similar problem which I didn't manage to resolve - I couldn't get the script to recognise the pgpass.conf file. I however used a work-around of setting the PGPASSWORD environment variable in the batch file I was using (PostgreSQL 9.6).
Inside a batch file:
SET PGPASSWORD=<<password>>
pg_dump.exe -h <<host>> -p <<port>> -U <<user>> -Fc -b -v -f <<output file path>> <<database>>
I have gotten it to work with the following:
pgpass.conf:
127.0.0.1:5432:*:username:password
However, I have it stored here:
C:\Users\<user>\AppData\Roaming\postgresql
For some reason, on a previous iteration of Postgres, the database had generated the pgpass file and stored it there. I was running into the same issue you were having, moved it to that directory and it worked. I'm not sure why though.
Then, all you'll need to do is:
pg_dump -h myip mydb > mylocaldumpfile
...ensuring that myip and the ip in pgpass.conf are identical. If they are not, it will prompt you for a password.
You could use pgAdmin III to store the password (in the server's properties).
That operation automatically creates a correct pgpass.conf file. You can then schedule a task to run a simple batch file that would read:
"C:\path\to\pg_dump.exe" -U <user> -w <database> > C:\path\to\database.backup
Make sure you are logged in as the user corresponding with the folder where the pgpass.conf file lives.
If you are using UTF-8 encoding, please ensure that you are using without BOM mode.
Otherwise leave the first line as a comment:
# This line may contain hidden BOM bytes
localhost:5432:database:username:password
Also you don't need to escape asterisks \*, just put * to enable wildcard matching.