I am using Postgres Copy utility to load the data to Postgres table from CSV file. Currently using the below command
psql -h 127.0.0.1 -d target -U postgres -c "\copy TableName FROM 'E:\Dev\XXX_1_0.csv' delimiter '^'" -o E:/Dev/XXX.log
When there is an issue in the data, error information are not getting updated in the log file.
Whereas when there is no error, my log files is updated with loaded row count. fo example (COPY 25)
I tried to execute the above command from command prompt & below error is reported.
Let me know how to get the error information or redirect the errors to log files for the reference.
ERROR: value too long for type character varying(255)
CONTEXT: COPY TableName, line 2, column Name: "NickName..."
I don't know of a way to redirect the error output directly in psql. You can get your shell to do it for you.
This works to combine both stdout and stderr into one file named "log". It works both in bash and in Windows CMD:
psql -c "whatever" > log 2>&1
I am trying to find a way to pass a file to psql while using '\copy'. There is a question posted here use sql file in "\copy" psql command line asking a similar thing, but the accepted solution doesn't actually pass a file to psql, it passes the contents of the file, so this cannot be done when the contents of the sql file exceeds the maximum allowed length of the psql command.
e.g. something like this psql -c data_base "\copy \file_path <file.sql> To './test.csv' With CSV"
Using copy rather than \copy, you can do it like this:
(echo "copy ("; cat file.sql ; echo ") to STDOUT with CSV")| psql -X > ./test.csv
I need to execute a query from a file and write the output of it to a CSV file. I want to keep the delimiter as ;. I have tried below queries.
psql -h localhost -p 5432 -U postgres -d postgres -f \path\to\sqlQuery.sql -o \path\to\result\result.csv
This query puts the result into a file but it is in a postgres result tabular format.
psql -h localhost -p 5432 -U postgres -d postgres -f copy(\path\to\sqlQuery.sql) to \path\to\result.csv csv header;
Above query is giving me a syntax error.
I'm looking for a way to use COPY command and \f or -f together so I'll be able to exceute the query from a file and also write the output to another CSV file with specified delimiter.
Change your SQL script to use COPY:
COPY (/* your query */) TO STDOUT
(FORMAT 'csv', DELIMITER ';');
I am trying to use a psql loop to load a directory of csv files into an AWS RDS Postgres database.
This StackExchange post answers how to do so locally and the following code works with a single file:
PGPASSWORD='MyPassword' psql -h my-rds.instance.us-east-1.rds.amazonaws.com -U user -d mydatabase -c '\COPY dbo.input_table FROM ''MySingleFile.csv'' CSV HEADER'
but I can't "combine" the two features. This code is the closest I can get but I get the following error for each file that is identified by the ls *.csv command:
for x in $(ls *.csv);
do PGPASSWORD='MyPassword' psql -h my-rds.instance.us-east-1.rds.amazonaws.com -U user -d mydatabase -c "\COPY dbo.input_table FROM ''MySingleFile.csv'' CSV HEADER"; done
ERROR: conflicting or redundant options
...
ERROR: conflicting or redundant options
I'm using psql's \dt to list all tables in a database and I need to save the results.
What is the syntax to export the results of a psql command to a file?
From psql's help (\?):
\o [FILE] send all query results to file or |pipe
The sequence of commands will look like this:
[wist#scifres ~]$ psql db
Welcome to psql 8.3.6, the PostgreSQL interactive terminal
db=>\o out.txt
db=>\dt
Then any db operation output will be written to out.txt.
Enter '\o' to revert the output back to console.
db=>\o
The psql \o command was already described by jhwist.
An alternative approach is using the COPY TO command to write directly to a file on the server. This has the advantage that it's dumped in an easy-to-parse format of your choice -- rather than psql's tabulated format. It's also very easy to import to another table/database using COPY FROM.
NB! This requires superuser or pg_write_server_files privileges and will write to a file on the server.
Example: COPY (SELECT foo, bar FROM baz) TO '/tmp/query.csv' (format csv, delimiter ';')
Creates a CSV file with ';' as the field separator.
As always, see the documentation for details
Use o parameter of pgsql command.
-o, --output=FILENAME send query results to file (or |pipe)
psql -d DatabaseName -U UserName -c "SELECT * FROM TABLE" -o /root/Desktop/file.txt
\copy which is a postgres command can work for any user. Don't know if it works for \dt or not, but general syntax is reproduced from the following link Postgres SQL copy syntax
\copy (select * from tempTable limit 100) to 'filenameinquotes' with header delimiter as ','
The above will save the output of the select query in the filename provided as a csv file
EDIT:
For my psql server the following command works this is an older version v8.5
copy (select * from table1) to 'full_path_filename' csv header;
Use the below query to store the result in a CSV file
\copy (your query) to 'file path' csv header;
Example
\copy (select name,date_order from purchase_order) to '/home/ankit/Desktop/result.csv' cvs header;
Hope this helps you.
If you got the following error
ufgtoolspg=> COPY (SELECT foo, bar FROM baz) TO '/tmp/query.csv' (format csv, delimiter ';');
ERROR: must be superuser to COPY to or from a file
HINT: Anyone can COPY to stdout or from stdin. psql's \copy command also works for anyone.
you can run it in this way:
psql somepsqllink_or_credentials -c "COPY (SELECT foo, bar FROM baz) TO STDOUT (format csv, delimiter ';')" > baz.csv
COPY tablename TO '/tmp/output.csv' DELIMITER ',' CSV HEADER;
this command is used to store the entire table as csv
I assume that there exist some internal psql command for this, but you could also run the script command from util-linux-ng package:
DESCRIPTION
Script makes a typescript of everything printed on your terminal.
This approach will work with any psql command from the simplest to the most complex without requiring any changes or adjustments to the original command.
NOTE: For Linux servers.
Save the contents of your command to a file
MODEL
read -r -d '' FILE_CONTENT << 'HEREDOC'
[COMMAND_CONTENT]
HEREDOC
echo -n "$FILE_CONTENT" > sqlcmd
EXAMPLE
read -r -d '' FILE_CONTENT << 'HEREDOC'
DO $f$
declare
curid INT := 0;
vdata BYTEA;
badid VARCHAR;
loc VARCHAR;
begin
FOR badid IN SELECT some_field FROM public.some_base LOOP
begin
select 'ctid - '||ctid||'pagenumber - '||(ctid::text::point) [0]::bigint
into loc
from public.some_base where some_field = badid;
SELECT file||' '
INTO vdata
FROM public.some_base where some_field = badid;
exception
when others then
raise notice 'Block/PageNumber - % ',loc;
raise notice 'Corrupted id - % ', badid;
--return;
end;
end loop;
end;
$f$;
HEREDOC
echo -n "$FILE_CONTENT" > sqlcmd
Run the command
MODEL
sudo -u postgres psql [some_db] -c "$(cat sqlcmd)" >>sqlop 2>&1
EXAMPLE
sudo -u postgres psql some_db -c "$(cat sqlcmd)" >>sqlop 2>&1
View/track your command output
cat sqlop
Done! Thanks! =D
Approach for docker
via psql command
docker exec -i %containerid% psql -U %user% -c '\dt' > tables.txt
or query from sql file
docker exec -i %containerid% psql -U %user% < file.sql > data.txt