Create query that copies from a CSV file on my computer to the DB located on another computer in Postgres - postgresql

I am trying to create a query that will copy data from a CSV file that is located on my computer to a Postgres DB that is on a different computer.
Our Postgres DB is located on another computer, and I work on my own to import and query data. I have successfully copied data from the CSV file on MY computer TO the DB in PSQL Console using the following:
\COPY table_name FROM 'c:\path\to\file.csv' CSV DELIMITER E'\t' HEADER;
But when writing a query using the SQL Editor, I use the same code above without the '\' in the beginning. I get the following error:
ERROR: could not open file "c:\pgres\dmi_vehinventory.csv" for reading: No such file or directory
********** Error **********
ERROR: could not open file "c:\pgres\dmi_vehinventory.csv" for reading: No such file or directory
SQL state: 58P01
I assume the query is actually trying to find the file on the DB's computer rather than my own.
How do I write a query that tells Postgres to look for the file on MY computer rather than the DB's computer?
Any help will be much appreciated !

\COPY is a correct way if you want to upload file from local computer (computer where you've stared psql)
COPY is correct when you want to upload on remote host from remote directory
here is an example, i've connected with psql to remote server:
test=# COPY test(i, i1, i3)
FROM './test.csv' WITH DELIMITER ',';
ERROR: could not open file "./test.csv" for reading: No such file
test=# \COPY test(i, i1, i3)
FROM './test.csv' WITH DELIMITER ',';
test=# select * from test;
i | i1 | i3
---+----+----
1 | 2 | 3
(1 row)

There are several common misconceptions when dealing with PostgreSQL's COPY command.
Even though psql's \COPY FROM '/path/to/file/on/client' command has identical syntax (other than the backslash) to the backend's COPY FROM '/path/to/file/on/server' command, they are totally different. When you include a backslash, psql actually rewrites it to a COPY FROM STDIN command instead, and then reads the file itself and transfers it over the connection.
Executing a COPY FROM 'file' command tells the backend to itself open the given path and load it into a given table. As such, the file must be mapped in the server's filesystem and the backend process must have the correct permissions to read it. However, the upside of this variant is that it is supported by any postgresql client that supports raw sql.
Successfully executing a COPY FROM STDIN places the connection into a special COPY_IN state during which an entirely different (and much simpler) sub-protocol is spoken between the client and server, which allows for data (which may or may not come from a file) to be transferred from the client to the server. As such, this command is not well supported outside of libpq, the official client library for C. If you aren't using libpq, you may or may not be able to use this command, but you'll have to do your own research.
COPY FROM STDIN/COPY TO STDOUT doesn't really have anything to do with standard input or standard output; rather the client needs to speak the sub-protocol on the database connection. In the COPY IN case, libpq provides two commands, one to send data to the backend, and another to either commit or roll back the operation. In the COPY OUT case, libpq provides one function that receives either a row of data or an end of data marker.
I don't know anything about SQL Editor, but it's likely that issuing a COPY FROM STDIN command will leave the connection in an unusable state from its point of view, especially if it's connecting via an ODBC driver. As far as I know, ODBC drivers for PostgreSQL do not support COPY IN.

Related

Postgresql Copy Function from the Server Computer to the Client Computer

I would like to import a table from the server computer into a Client computer using the copy command. I know this is a recurring issue for users, but I have not been able to get an answer to this particular one and it's also a different scenario, and I believe this to be common.
I used a copy command to copy a Table from the server to the client computer using the code below:
COPY (Select * from Table_Name) TO 'C:\somedirectory\file.csv' DELIMITER ',' CSV HEADER;
However, I got the following
ERROR: relative path not allowed for COPY to file
My question is: How do I use the correct COPY command to copy from the server computer to the client computer in Postgres.
Thank you in anticipation
Please check if your user has read/write access to the destination folder.
This is one thread I found, see if it helps
https://dba.stackexchange.com/questions/158466/relative-path-for-psql-copy-file
https://postgrespro.com/list/thread-id/1116997
Try with network through access using client public IP.
How do I use the correct COPY command to copy from the server computer to the client computer in Postgres
You simply can't.
Which is clearly stated in the manual
COPY with a file name instructs the PostgreSQL server to directly read from or write to a file. The file must be accessible by the PostgreSQL user (the user ID the server runs as) and the name must be specified from the viewpoint of the server
(emphasis mine)
You need to use psql's \copy command or any other export tool that works on the client side.

SQL DATABASE(postgresql)

ERROR: could not open file "C:\Users\lenovo\Downloads\Owners.csv" for reading: Permission denied
HINT: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
SQL state: 42501
I am trying to import a csv file into postgresql. But this error pops up. I search everywhere. But i Couldn't get the answer of it PLEASE HELP ME.
THANKS IN ADVANCE!!
COPY mytable FROM /path/thefile.csv WITH CSV,HEADER; is executed by the DBMS server, the .csv-file is read by the server. The server (typically) runs as user postgres, which cannot access arbitrary users's files. (Also: the client and server don't have to be running on the same machine) There are two possible solutions to this:
copy the csv-file to a place where the server can access it, in /tmp/, or somewhere under its home-directory.
use psql's \copy mytable(col1,col2,...) FROM '/path/file.csv'... (slightly different syntax)

Export Postgres table to csv

I am trying to export my Postgres table to a csv on my desktop and I get this error:
ERROR: could not open file "C:\Users\blah\Desktop\countyreport.csv" for writing: Permission denied
SQL state: 42501
This is my query which I believe is the correct syntax
COPY countyreport TO 'C:\\Users\\blah\\Desktop\\countyreport.csv' DELIMITER ',' CSV HEADER;
According to the user manual:
Files named in a COPY command are read or written directly by the
server, not by the client application.
https://www.postgresql.org/docs/current/static/sql-copy.html
The common mistake is to believe that the filesystem access will be that of the (client) user, but it's not. It's normal to run the postgresql server as its own user. Therefore action carried out by the server will be done as a different OS user to the client. The server is usually run as an OS user postgres.
Assuming that you are running the server on your local machine then the simplest way to fix it would be to give postgres access to your home directory or desktop. This can be done by changing the windows security settings on your home directory.
Before you do this.... Stop and think. Is this what you are looking for? If the server is in development then will it always run on the user's machine. If not then you may need to use COPY to write to the stdout. See the manual for information on this.

How do you copy data from file to table in SQL? [duplicate]

This question already has answers here:
Postgres ERROR: could not open file for reading: Permission denied
(17 answers)
Closed 9 years ago.
How do you copy data from a file to a table in SQL? I'm using pgAdmin3 on a Macbook.
The table name is tutor, and the name of the file is tutor.rtf.
I use the following query:
COPY tutor
FROM /Users/.../tutor.rtf
WITH DELIMITER ',';
but got the error "permission denied'.
The file is not locked. So how do you solve this problem? Or is there any other quicker way to copy data from file to table except for INSERT INTO ... VALUE(); ?
COPY opens the file using the PostgreSQL server backend, so it requires that the user postgresql runs as have read permission (for COPY FROM) for the file in question. It also requires the same SQL-level access rights to the table as INSERT, but I suspect it's file permissions that're getting you here.
Most likely the postgres or postgres_ (depending on how you installed PostgreSQL) user doesn't have read access to /Users/somepath/tutor.rtf or some parent directory of that file.
The easiest solution is to use psql's \copy command, which reads the file using the client permissions, rather than those of the server, and uses a path relative to the client's current working directory. This command is not available in PgAdmin-III.
Newer PgAdmin-III versions have the Import command in the table context menu. See importing tables from file in the PgAdmin-III docs. This does the equivalent of psql's \copy command, reading the file with the access rights of the PgAdmin-III application.
Alternately you can use the server-side COPY command by making sure every directory from /Users up somepath has world-execute rights - meaning users can traverse it, cd into it, etc, but can't list its contents without r rights too. Then either set the file to group postgres and make sure it has group read rights, or make it world-readable.

Can PostgreSQL COPY read CSV from a remote location?

I've been using JDBC with a local Postgres DB to copy data from CSV files into the database with the Postgres COPY command. I use Java to parse the existing CSV file into a CSV format matches the tables in the DB. I then save this parsed CSV to my local disk. I then have JDBC execute a COPY command using the parsed CSV to my local DB. Everything works as expected.
Now I'm trying to perform the same process on a Postgres DB on a remote server using JDBC. However, when JDBC tries to execute the COPY I get
org.postgresql.util.PSQLException: ERROR: could not open file "C:\data\datafile.csv" for reading: No such file or directory
Am I correct in understanding that the COPY command tells the DB to look locally for this file? I.E. the remote server is looking on its C: drive (doesn't exist).
If this is the case, is there anyway to indicate to the copy command to look on my computer rather than "locally" on the remote machine? Reading through the copy documentation I didn't find anything that indicated this functionality.
If the functionality doesn't exist, I'm thinking of just populating the whole database locally then copying to database to the remote server but just wanted to check that I wasn't missing anything.
Thanks for your help.
Create your sql file as follows on your client machine
COPY testtable (column1, c2, c3) FROM STDIN WITH CSV;
1,2,3
4,5,6
\.
Then execute, on your client
psql -U postgres -f /mylocaldrive/copy.sql -h remoteserver.example.com
If you use JDBC, the best solution for you is to use the PostgreSQL COPY API
http://jdbc.postgresql.org/documentation/publicapi/org/postgresql/copy/CopyManager.html
Otherwise (as already noted by others) you can use \copy from psql which allows accessing the local files on the client machine
To my knowledge, the COPY command can only be used to read locally (either from stdin or from file) from the machine where the database is running.
You could make a shell script where you run you the java conversion, then use psql to do a \copy command, which reads from a file on the client machine.