I am newbie to postgres and I am finding some difficulties to understand what's going on. I am working with a client's DB in postgres, the job is to extract some tables into CSV and upload it through FTP. The thing is that if the table exports more than few thousands of rows the command stops and throws the error : ERROR: canceling statement due to user request
Here is the command I am running:psql -c "\COPY (select * from <table_name> ) TO 'C:\\Users\<user_name>\<file_name>.csv' DELIMITER ';' CSV HEADER;" -U <user> <db_name>
Is there anyway to find what's causing the error? the feedback psql returns is too vague
Related
I am a beginner in PostgreSQL. I have connected to postgres using
sudo -u postgres psql
Then I connected to 'test' database as 'postgres' user using the following command:
postgres=# \c test
Now when I try to export the results to home directory using the following command;
test=# \copy (select * from person left join car on
person.car_id=car.id) to '/home/navdeep/Downloads/data.csv' delimiter
',' csv header;
I get the following error;
/home/navdeep/Downloads/data.csv: Permission denied
What could be the reason. Please advise. Thanks.
Could you try saving to the temp folder? I don't think you'll have any permission issues there.
I'm trying to import a CSV file into postgresql database.
First of all I tried following query:
copy temporaryData
from '/home/milad/Desktop/myFolder/csvFiles/test.csv'
delimiter ',' csv;
But it gave me below Error:
SQL Error [42501]: ERROR: could not open file "/home/milad/Desktop/myFolder/test.csv" for reading: Permission denied
This error is reasonable as suggested by this thread and also by postgresql tutorial:
Notice that the file must be read directly by the PostgreSQL server, not by the client application. Therefore, it must be accessible by the PostgreSQL server machine. Also, you need to have superuser access in order to execute the COPY statement successfully.
Since I don't like to change my system permissions just to import a file, To solve this issue I decided to move my CSV file to /tmp/ directory which is suggested by this answer.
Then I tried to run below script:
create or replace procedure loadInstructorOfCourses()
language plpgsql
as $$
begin
create table temporaryData(
courseSectionID varchar(10),
instructorID varchar(25),
courseTitle varchar(50)
);
copy temporaryData
from '/tmp/test.csv'
delimiter ',' csv;
drop table temporaryData;
end;$$;
call loadInstructorOfCourses();
And now it gives me below error:
SQL Error [58P01]: ERROR: could not open file "/tmp/test.csv" for reading: No such file or directory
But I'm sure the file has read permission for others and also it's available:
> ls -l -d /tmp
drwxrwxrwt 25 root root 700 Oct 20 09:53 /tmp
> ls -l /tmp/test.csv
-rw-r--r-- 1 milad milad 6700 Oct 20 09:53 /tmp/test.csv
Why does this problem happen and how to resolve it?
INFO:
My OS is Manjaro KDE:
> uname -r
5.4.150-1-MANJARO
And also I use DBeaver to connect to postgresql server and run my scripts.
UPDATES: Using psql I have the same issue too. When I connect to db by psql:
psql -U postgres -d IUTPlanning
and run my procedure:
call loadInstructorOfCourses();
I'll get the same error:
ERROR: could not open file "/tmp/test.csv" for reading: No such file or directory
HINT: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
CONTEXT: SQL statement "copy temporaryData
from '/tmp/test.csv'
delimiter ',' csv"
PL/pgSQL function loadinstructorofcourses() line 9 at SQL statement
Try using psql's \copy command, as the hint in the error message you quoted suggests. COPY FROM tells the server to open the file and process it, while \copy reads the file client side and then passes the output to the server, see https://stackoverflow.com/a/19466558/12760895
Open the psql console with psql -U postgres -d [yourdb]
then run
\copy [target_table] from '/tmp/test.csv' DELIMITER ',' CSV;
I am using timescaledb.
The doumentation I am following is Using PostgreSQL's COPY to migrate data from a csv file to timescale db. The name of csv file is test.csv.
I created the db named test , the name of table is test1. Table is a hypertable as per the timescaledb documentation.
The table's structure and csv files structure are the same.
While executing the following command in cmd I am not getting a result other than an addition of - symbol in the console command test-#
psql -d test -c "\COPY test1 FROM C:\Users\DEGEJOS\Downloads\test.csv CSV"
If I put ; after the command
psql -d test -c "\COPY test1 FROM C:\Users\DEGEJOS\Downloads\test.csv CSV"; I am getting a syntax error at Line 1.
How can I solve this error and insert data from csv file to db.?
You are trying to run psql with \COPY inside psql session, thus you get an error in the second call, since psql keyword does not exist in SQL. psql is an executable.
To follow the instructions from Timescale, you need to call the command directly in CMD. I.e, call:
psql -d test -c "\COPY test1 FROM C:\Users\DEGEJOS\Downloads\test.csv CSV"
If you are in C:\Users\DEGEJOS as in your screenshoot, it will look like:
C:\Users\DEGEJOS\psql -d test -c "\COPY test1 FROM C:\Users\DEGEJOS\Downloads\test.csv CSV"
With PostgreSQL 9.5 on CentOS 7, I have created a database named sample along with several tables. I have .csv data in /home/MyUser/data for each table.
For example, there exists TableName.csv for the table "TableName".
How do I load the csv files into each table?
What I've tried doesn't work and I can't figure out what I'm doing wrong.
Load from within the DB
$ psql sample
sample=# COPY "TableName" FROM '/home/MyUser/data/TableName.csv' WITH CSV;
ERROR: could not open file "/home/MyUser/data/TableName.csv" for reading: Permission denied
This implies a file permission problem. All the files in data/ are -rw-r--r-- and the directory itself is drwxr-xr-x. So file permissions shouldn't be the problem (unless I'm missing something). The internet says that COPY has problems with permissions and to try \copy.
Load from CLI
$ psql \copy sample FROM /home/MyUser/data/TableName.csv WITH CSV
psql: warning: extra command-line argument "FROM" ignored
psql: warning: extra command-line argument "/home/MyUser/data/TableName.csv" ignored
psql: warning: extra command-line argument "WITH" ignored
psql: warning: extra command-line argument "CSV" ignored
psql: FATAL: Peer authentication failed for user "sample"
This appears to be a syntax error, but I'm not finding the documentation particularly helpful (man psql then /\copy). I've also tried the following to the same result.
$ psql \copy sample."TableName" FROM /home/MyUser/data/TableName.csv WITH CSV
$ psql \copy sample FROM /home/MyUser/data/TableName.csv WITH DELIMITER ','
There are several other permutations which yield similar errors.
Web Resources Used
https://www.postgresql.org/docs/9.5/static/app-psql.html
https://www.postgresql.org/docs/9.5/static/sql-copy.html
The correct COPY command to load postgreSQL data from csv file that has single-quoted data?
https://soleil4716.wordpress.com/2010/08/19/using-copy-command-in-postgresql/
Can I use \copy command into a function of postgresql?
https://wiki.postgresql.org/wiki/COPY
About the permissions:
Don't forget that to access a file you need permissions on all directories in the path. So if, for example, the OS user postgres does not have permissions on the /home/MyUser directory, you get the observed error message.
About \copy:
You have to use the -c option to supply a command to psql:
$ psql -c "\copy sample FROM '/home/MyUser/data/TableName.csv' WITH (FORMAT CSV)"
Adding, this sample was helpful from Laurenz, as I got this to work first time today from a windows 10 client.
My target is an enterprise greenplum db (same thing just parallel/scale).
Step 1, Postgres client installed (I didn't install the database itself, but all the other items as part of the windows bundling install, including the shell most importantly for this)
https://www.postgresql.org/download/
Step 2, Create your target table on your workspace/db. I use DBeaver to connect to greenplum db, then run this, you don't have to use DBeaver, you can probably run this in the shell itself.
CREATE TABLE workspacename.scott_test2 (
blah1 varchar(40) NOT NULL,
blah2 varchar(40) NOT NULL
)
DISTRIBUTED BY (blah1);
Step 3, launch the shell command from postgres utilities installed earlier, login as it prompts you, then when you're in, the command I ran in that shell installed above (a command line interface) is simply:
\copy workspacename.scott_test2 FROM 'C:\temp\flatfile.csv' WITH CSV;
Note \copy is not copy. Use \copy
This loaded a half million row table in 2 seconds, fast. file above is comma delimited, the with CSV does that.
Trying to dump the output of a query into a CSV file in an automated job and running into an issue with fields where the column contains my comma delimiter. With the nature of this particular network, I have to jump through a couple of hoops to get things done, and there's a good chance I'm missing something very obvious.
In a nutshell, I kick off my script from a client machine that uses PLINK to run a remote psql command on another box over an SSH connection. That psql command is hitting a Postgres server on a third machine (I can't connect directly from client to DB, hence the extra step in between).
If I manually SSH from client to server 1, connect to the Postgres box, and use \copy... with CSV header, the file that's created is perfect, and any fields that contain a comma are automatically surrounded by double quotes.
However, if I try go issue that \copy (or copy) command in a single command, the output doesn't contain those double quotes, so I end up in that situation where commas in a field are interpreted as a delimiter later one.
In other words, this has the necessary double-quotes:
SSH from client to server1.
psql -Uuser -h server2 database
\copy (select ...) to '~/myfile.csv' with CSV header;
But doesn't:
SSH from client to server1
psql -Uuser -h server2 database -c "\copy (select ...) to '~/myfile.csv' with CSV header;"
Using FORCE_QUOTE
Here is how to do:
psql -U user -h server2 database -c "\copy (select ...) to '~/myfile.csv' WITH (FORMAT CSV, HEADER TRUE, FORCE_QUOTE *);"
COPY command documentation