I'm trying to pass a sql file to psql. After reading the docs, tried:
psql_args=(
"password='$INPUT_PASSWORD'"
dbname=analytics
"host='$INPUT_HOST'"
user=analytics
port=32648
file=query.sql
)
psql "${psql_args[*]}"
psql: error: invalid connection option "file"
root#380773cb4e26:/#
If I remove the file=query.sql arg this results in a connection to psql. I just don't know how to pass it a query file?
On the docs, two arguments look like ones of interest here:
-f filename
--file=filename
Read commands from the file filename, rather than standard input
and also:
-c command
--command=command
Specifies that psql is to execute the given command string, command
I tried the file=query.sql one but that failed with the error message above.The command one wants a string whereas I want to pass a .sql file. I tried anyway:
psql_args=(
"password='$INPUT_PASSWORD'"
dbname=analytics
"host='$INPUT_HOST'"
user=analytics
port=32648
command=query.sql
)
psql "${psql_args[*]}"
psql: error: invalid connection option "command"
Is there a way that I can pass query.sql to psql in order to run a query?
You seem to be packaging options up into a connection string. But --file must be given directly as an option to psql, not as part of a connection string.
psql "${psql_args[*]}" --file=query.sql
Since other answer seem to overlook this.
Here is how to store dynamic options into an array, and pass it as arguments to the command:
#!/usr/bin/env bash
psql_args=(
"--dbname=analytics"
"--host=$INPUT_HOST"
"--user=analytics"
"--port=32648"
"--file=query.sql"
)
psql "${psql_args[#]}"
According with the documentation https://www.postgresql.org/docs/current/sql-copy.html , PSQL command COPY cannot create a table from a tsv or csv file. You need to create a table and its columns before you can COPY to it.
Is there any workaround for this issue?
There's some ways to workaround this issue.
You can find some scripts around the internet which will do what you're looking for, but the best way I know is using this Data Mover Project.
Besides, it's already published to docker as techindicium/spark-datamover:v0.1.
You can call it from the command line:
docker run --network host techindicium/spark-datamover:v0.1 -s /home/path/your_file.csv --filetype csv --destination "jdbc:postgresql://localhost:PORT/DATABASE?user=USERNAME&password=PASSWD" --destination-table MY_DEST_TABLE
I'm using my Windows PC, and I'm trying to import a "dump.sql" into the database "TEST" created with Postgres, using command prompt. I do it in two steps:
Step 1:
cd C:\Program Files\PostgreSQL\12\bin
Step 2:
psql -U username -d TEST < C:\Users\Username\Desktop\University\Politechnic\III year\INFORMATIC TECHNOLOGIES FOR THE WEB\PDF SL\SL\Materials\TIW_IOL_ServletJSP\db\dump.sql`
Long path, I know. But the result is: "Impossible to find specified file".
What can I do?
Not sure how security is where you are at, but can you attempt to write to a file with a simpler destination? Such that you take out any possibility of spaces and/or length being the issue? Then you will at least be able to remove those variables or narrow down to them depending on the outcome. Note that the max path length is 260 characters
(From comment on question)
I am trying to log a complete session in psql into a .txt file. The command given to me was initially this:
psql db_name| tee file_name.txt
However, my SSH client does nothing until I quit it. That means, it does not recognize any command. More like a document, no action happens no matter what I write. So far, only '\q' is recognised which lets me get out of it. Any ideas what is happening? How am I to write the query if shell will not read anything. Also, I tried the following (this is before connecting to database) :
script filename.txt
It does show the message : script started, file is filename.txt, but I dont know where this file is stored and how to retrieve it.
Any help with the above will be welcome and really appreciated! Thanks a lot :)
There is option to psql for log query and results:
-L filename
--log-file filename
Write all query output into file filename, in addition to the normal output destination.
Try this:
psql db_name -L file_name.txt
Computer: Mac OS X, version 10.8
Database: Postgres
Trying to import csv file into postgres.
pg> copy items_ordered from '/users/darchcruise/desktop/items_ordered.csv' with CSV;
ERROR: could not open file "/users/darchcruise/desktop/items_ordered.csv" for reading: Permission denied
Then I tried
$> chown postgres /users/darchcruise/desktop/items_ordered.csv
chown: /users/darchcruise/desktop/items_ordered.csv: Operation not permitted
Lastly, I tried
$> ls -l
-rw-r--r-- 1 darchcruise staff 1016 Oct 18 21:04 items_ordered.csv
Any help is much appreciated!
Assuming the psql command-line tool, you may use \copy instead of copy.
\copy opens the file and feeds the contents to the server, whereas copy tells the server the open the file itself and read it, which may be problematic permission-wise, or even impossible if client and server run on different machines with no file sharing in-between.
Under the hood, \copy is implemented as COPY FROM stdin and accepts the same options than the server-side COPY.
Copy the CSV file to /tmp
For me this solved the issue.
chmod a+rX /users/darchcruise/ /users/darchcruise/desktop /users/darchcruise/desktop/items_ordered.csv
This will change access rights for your folder. Note that everyone will be able to read your file.
You can't use chown being a user without administrative rights.
Also consider learning umask to ease creation of shared files.
Copy your CSV file into the /tmp folder
Files named in a COPY command are read or written directly by the server, not by the client application. Therefore, they must reside on or be accessible to the database server machine, not the client. They must be accessible to and readable or writable by the PostgreSQL user (the user ID the server runs as), not the client. COPY naming a file is only allowed to database superusers, since it allows reading or writing any file that the server has privileges to access.
I had the issue when I was trying to export data from a remote server into the local disk. I hadn't realised that SQL copy actually is executed on the server and that it tries to write to a server folder. Instead the correct thing to do was to use \copy which is the psql command and it writes to the local file system as I expected. http://www.postgresql.org/message-id/CAFjNrYsE4Za_KWzmfgN1_-MG7GTw_vpMRxPk=OEjAiLqLskxdA#mail.gmail.com
Perhaps that might be useful to someone else too.
Another way to do this, if you have pgAdmin and are comfortable using the GUI is to go the table in the schema and right click on the table you wish to import the file to and select "Import" browse your computer for the file, select the type your file is, the columns you want the data to be imputed into, and then select import.
That was done using pgAdmin III and the 9.4 version of PostgreSQL
I resolved the same issue with a recursive chown on the parent folder:
sudo chown -R postgres:postgres /home/my_user/export_folder
(my export being in /home/my_user/export_folder/export_1.csv)
for macbook first i opened terminal then type
open /tmp
or in finder directory you directly enter command+shift+g then type /tmp in go to the folder.
it opens temp folder in finder. then i paste copied csv file into this folder.then again i go to postgres terminal and typed below command and then it is copied my csv data into db table
\copy recharge_operator FROM '/private/tmp/operator.csv' DELIMITER ',' CSV;
COPY your table (Name, Latitude, Longitude) FROM 'C:\Temp\your file.csv' DELIMITERS ',' CSV HEADER;
Use c:\Temp\"Your File"\.
For me it worked to simply to add sudo (or run as root) for the chown command:
sudo chown postgres /users/darchcruise/desktop/items_ordered.csv
You must grant the pg_read_server_files permission to the user if you are not using postgres superuser.
Example:
GRANT pg_read_server_files TO my_user WITH ADMIN OPTION;
just in case you're facing this problem under windows 10 , add the group of users "youcomputer\Users" on the security Tab and grant it full control , that solved my issue
I had the same error message but was using psycopg2 to communicate with PostgreSQL. I fixed the permission issues by using the functions copy_from and copy_expert that will open the file on the client side as the user running the python script and feed the data to the database over STDIN.
Refer to this link for further information.
This answer is only for Linux Beginners.
Assuming initially the DB user didn't have file/folder(directory) permission on the client side.
Let's constrain ourselves to the following:
User: postgres
Purpose: You wanted to (write to / read from) a specific folder
Tool: psql
Connected to a specific database: YES
FILE_PATH: /home/user/training/sql/csv_example.csv
Query: \copy (SELECT * FROM table_name TO FILE_PATH, DELIMITER ',' CSV HEADER;
Actual Results: After running the query you got an error : Permission Denied
Expected Results: COPY COUNT_OF_ROWS_COPIED
Here are the steps I'd follow to try and resolve it.
Confirm the FILE_PATH permissions on your File system.
Inside a terminal to view the permissions for a file/folder you need to long list them by entering the command ls -l.
The output has a section that shows sth like this -> drwxrwxr-x
Which is interpreted in the following way:
TYPE | OWNER RIGHTS | GROUP RIGHTS | USER RIGHTS
rwx (r: Read, W: Write, X: Execute)
TYPE (1 Char) = d: directory, -: file
OWNER RIGHTS (3 Chars after TYPE)
GROUP RIGHTS (3 Chars after OWNER)
USER RIGHTS (3 Chars after GROUP)
If permissions are not enough (Ensure that a user can at least enter all folders in the path you wanted path) - x.
This means for FILE_PATH, All the directories (home , user, training, sql) should have at least an x in the USER RIGHTS.
Change permissions for all parent folders that you need to enter to have a x. You can use chmod rights_you_want parent_folder
Assuming /training/ didn't have an execute permission.
I'd go the user folder and enter chmod a+x training
Change the destination folder/directory to have a w if you want to write to it. or at least a r if you want to read from it
Assuming /sql didn't have a write permission.
I would now chmod a+w sql
Restart the postgresql server sudo systemctl restart postgresql
Try again.
This would most probably help you now get a successful expected result.
On Linux you can fix this by giving the postgres user read/write/execute permissions on the target directory. Eg:
setfacl -m u:postgres:rwx /home/hi
I just copied the source csv file to another folder in which you have more permissions (C:/temp), and it worked fine.
May be You are using pgadmin by connecting remote host then U are trying to update there from your system but it searches for that file in remote system's file system... its the error wat I faced May be its also for u check it