I have an expect script that loads data from a heroku postgres database into a local .csv file. I need to use expect due to a need for automation and entering a password. So far my script looks like the following:
#!/usr/bin/expect
spawn psql -h <host> -p <port> -U <username> -W <db name> -t -A -F "," -f sql.sql -o output.csv
expect "Password for user <db name>: "
send "<password>\r"
sleep 10
The sql.sql is a sql query, for example select * from my_table.
Notice that I need to add a sleep at the end of my expect script to allow data to be written to .csv file otherwise nothing gets written. However, if the data I am trying to load is too big then I will have to keep adjusting the sleep time every single time. How do I avoid this?
To avoid adjusting sleep for different queries, Just add global timeout in expect script.
set timeout -1
so your script would be something like this :-
#!/usr/bin/expect
set timeout -1
spawn psql -h <host> -p <port> -U <username> -W <db name> -t -A -F "," -f sql.sql -o output.csv
expect "Password for user <db name>: "
send "<password>\r"
#sleep 10
Hope it helps.
Related
i am trying to execute psql queries from the bash command line passing password in following format
set PGPASSWORD=rtttttul psql -U ostgres -h localhost -d postgres -c "select * from logs" -o output.txt
Somehow my queries are not giving any results.i have tried to pass different queries or incorrect credentials but still script execute without any error.
If i don't pass password and try logging in to command prompt,everything works fine.
i want to check what basic thing i am missing above
Below command worked
PGPASSWORD=rtttttul psql -U ostgres -h localhost -d postgres -c "select * from logs" -o output.txt
remove set at start of command fixed it
I'm dealing with psql for the first time and I need a command that executes a .sql file and after executing, it should make a .log file with the script output. I'm looking for something similar to this other command that I use with SQL Server:
sqlcmd -U userid -P password -S serveraddress -i path_to_the_sql_file -o path_where_to_save_log_file
Can you help me, please? :-)
I would spend some time at the psql page.
A quick example:
psql -U userid -h serveraddress -d some_db -f path_to_the_sql_file -L=path_where_to_save_log_file
With Postgres you need to connect to a database with a client. There are other options for inputting commands and capturing output at the link above.
I am migrating an application into Docker. One of the issues that I am bumping into is what is the correct way to load the initial data into PostgreSQL running in Docker? My typical method of restoring a database backup file are not working. I have tried the following ways:
gunzip -c mydbbackup.sql.gz | psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db> -W
That does not work, because PostgreSQL is prompting for a password, and I cannot enter a password because it is reading data from STDOUT. I cannot use the $PGPASSWORD environment variable, because the any environment variable I set in my host is not set in my container.
I also tried a similar command above, except using the -f flag, and specify the path to a sql backup file. This does not work because my file is not on my container. I could copy the file to my container with the ADD statement in my Dockerfile, but this does not seem right.
So, I ask the community. What is the preferred method on loading PostgreSQL database backups into Docker containers?
I cannot use the $PGPASSWORD environment variable, because the any
environment variable I set in my host is not set in my container.
I don't use docker, but your container looks like a remote host in the command shown, with psql running locally. So PGPASSWORD never has to to be set on the remote host, only locally.
If the problems boils down to adding a password to this command:
gunzip -c mydbbackup.sql.gz |
psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db> -W
you may submit it using several methods (in all cases, don't use the -W option to psql)
hardcoded in the invocation:
gunzip -c mydbbackup.sql.gz |
PGPASSWORD=something psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db>
typed on the keyboard
echo -n "Enter password:"
read -s PGPASSWORD
export PGPASSWORD
gunzip -c mydbbackup.sql.gz |
psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db>
Note about the -W or --password option to psql.
The point of this option is to ask for a password to be typed first thing, even if the context makes it unnecessary.
It's frequently misunderstood as the equivalent of the -poption of mysql. This is a mistake: while -p is required on password-protected connections, -W is never required and actually goes in the way when scripting.
-W, --password
Force psql to prompt for a password before connecting to a
database.
This option is never essential, since psql will automatically
prompt for a password if the server demands password
authentication. However, psql will waste a connection attempt
finding out that the server wants a password. In some cases it is
worth typing -W to avoid the extra connection attempt.
I have made required entries in .pgpass file with file permission set to 0600.
I am able to connect to db form shell command line without giving the password.
But when I run a shell script which internally queries postgres, it asks to enter password.
I am not able to figure out what could be wrong.
Here is a sample shell script:
#!/bin/bash
source $1
psql -h $DBHOST -d $DBNAME -U $DBUSER << EOF
select * from students limit 10;
EOF
All values for DBHOST, DBNAME and DBUSER are coming fine.
Nevermind.
I made a stupid mistake. I had edited config file on windows. So it had added crlf at the end of every line. So even though it was not visible it was being used when connecting to postgres.
So in command line
this worked.
psql -h 192.168.1.45 -d somedbname -U $somedbuse
But this did not. (after sourcing config file)
psql -h $DBHOST -d $DBNAME -U $DBUSER
I am creating a PostgreSQL database from the command line (i.e. using psql).
There are some errors in my SQL statements and I want to find out where the errors are occuring (too many objects to fill the screen buffer - so I need to save thios to file)
I have tried just about everything, from using the -o option, the -L option and using tee - I still cant capture the information that scrolls past on the screen.
How do I log this?
This is what I have tried so far:
psql -U -o dbcreate.log -f file.sql
psql -U -L dbcreate.log -f file.sql
psql -U -a -f file.sql | tee dbcreate.log
NONE of which results in the data flashing accross the screen being logged to file - how do I do this?
You need to redirect stderr. On Un*x and Linux:
psql ... 2>error.log
or both stdout and stderr:
psql ... &>error.log
On the other hand if you like to investigate the errors one by one:
psql -v ON_ERROR_STOP=1 ...
A helpful article about executing SQL scripts with psql - here.