I am using Postgres Copy utility to load the data to Postgres table from CSV file. Currently using the below command
psql -h 127.0.0.1 -d target -U postgres -c "\copy TableName FROM 'E:\Dev\XXX_1_0.csv' delimiter '^'" -o E:/Dev/XXX.log
When there is an issue in the data, error information are not getting updated in the log file.
Whereas when there is no error, my log files is updated with loaded row count. fo example (COPY 25)
I tried to execute the above command from command prompt & below error is reported.
Let me know how to get the error information or redirect the errors to log files for the reference.
ERROR: value too long for type character varying(255)
CONTEXT: COPY TableName, line 2, column Name: "NickName..."
I don't know of a way to redirect the error output directly in psql. You can get your shell to do it for you.
This works to combine both stdout and stderr into one file named "log". It works both in bash and in Windows CMD:
psql -c "whatever" > log 2>&1
Related
I am creating a powershell script to read data from a postgres DB
But any lines given after the psql.exe command does not works
after the psql.exe line the console asks for the password
and does nothing it's only when I press Ctrl+C the other lines get executed
I tried using Start-Job but then I am unable to read the output of my select command it only returns the following line
Job started: System.Management.Automation.PSRemotingJob stating that the job has started
I also tried the Invoke-Command but that too didn't help.
Can anyone help me with a simple sample that explains how to enter password for the psql.exe cmd and how to read the output from the select cmd
I am sharing the approach that worked for me
$env:PGPASSWORD='password'
$result=Write-Output "Select * FROM public.table_name" | & $psql -h 127.0.0.1 -p 5432 -U -U postgres -d database_name
Now you can access the output of the select from the result variable.
You can use a for method and iterate over result to read each row.
I am trying to find a way to pass a file to psql while using '\copy'. There is a question posted here use sql file in "\copy" psql command line asking a similar thing, but the accepted solution doesn't actually pass a file to psql, it passes the contents of the file, so this cannot be done when the contents of the sql file exceeds the maximum allowed length of the psql command.
e.g. something like this psql -c data_base "\copy \file_path <file.sql> To './test.csv' With CSV"
Using copy rather than \copy, you can do it like this:
(echo "copy ("; cat file.sql ; echo ") to STDOUT with CSV")| psql -X > ./test.csv
I am using timescaledb.
The doumentation I am following is Using PostgreSQL's COPY to migrate data from a csv file to timescale db. The name of csv file is test.csv.
I created the db named test , the name of table is test1. Table is a hypertable as per the timescaledb documentation.
The table's structure and csv files structure are the same.
While executing the following command in cmd I am not getting a result other than an addition of - symbol in the console command test-#
psql -d test -c "\COPY test1 FROM C:\Users\DEGEJOS\Downloads\test.csv CSV"
If I put ; after the command
psql -d test -c "\COPY test1 FROM C:\Users\DEGEJOS\Downloads\test.csv CSV"; I am getting a syntax error at Line 1.
How can I solve this error and insert data from csv file to db.?
You are trying to run psql with \COPY inside psql session, thus you get an error in the second call, since psql keyword does not exist in SQL. psql is an executable.
To follow the instructions from Timescale, you need to call the command directly in CMD. I.e, call:
psql -d test -c "\COPY test1 FROM C:\Users\DEGEJOS\Downloads\test.csv CSV"
If you are in C:\Users\DEGEJOS as in your screenshoot, it will look like:
C:\Users\DEGEJOS\psql -d test -c "\COPY test1 FROM C:\Users\DEGEJOS\Downloads\test.csv CSV"
I have taken a database backup from the server using:
pg_dump -U postgres -d machine -s --disable-triggers >aftrn.sql
When I am trying to restore that data into my local using:
psql -U postgres -d usernet < aftrn.sql
I am getting the following error of:
invalid command \N
invalid command \N
invalid command \N
out of memory
root#lap-DB3:/home/akhiles#
Does anyone know what's the reason or error related to?
When I've had this issue, it is because it's been loading into the wrong database.
Pipe the output to a file and read the first few lines of the output.
I am trying to copy data from a csv file from my local machine into a remote pgsql table named states, but i am getting an ERROR: Syntax error at or near "FROM". Can someone guide me as to why i am receiving this error?
COPY FROM STDIN states FROM '/Users/Shared/data.csv' DELIMITER AS ',';
The problem is that the path to the file is in the remote server, not the local one.
you need psql and pipe the file to STDIN:
psql -h host -d remoteDB -U myuser -c "copy states from STDIN with delimiter as ',';" < /path/file.csv
alternatively you can also do:
cat /path/file.csv | psql -h host -d remoteDB -U myuser -c "copy states from STDIN with delimiter as ',';"
You can't do it directly with a filename unless the file is on the Postgres server. The docs of COPY state:
COPY with a file name instructs the PostgreSQL server to directly read from or write to a file. The file must be accessible to the server and the name must be specified from the viewpoint of the server. When STDIN or STDOUT is specified, data is transmitted via the connection between the client and the server.
You'll have to pipe the file in via STDIN.
Many Postgres drivers provide a method to make this easier. For example, ruby-pg provides copy_data.
conn.copy_data "COPY states FROM STDIN FORMAT CSV" do
File.foreach('/Users/Shared/data.csv') do |line|
conn.put_copy_data(line)
end
end