ERROR: could not open file "C:\Program Files\PostgreSQL\10\data\Data_copy\student.csv" for reading: No such file or directory
HINT: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
SQL state: 58P01
The error message says it all.
If you use SQL COPY command the file to be loaded must be accessible on database server side.
If this is not possible you can use psql CLI \copy command from the client side because it can access a file on client side.
Example:
$ cat t.csv
1,ONE
2,TWO
3,THREE
In psql:
# \d t
Table "public.t"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+---------
x | integer | | |
y | text | | |
# \copy t from 't.csv' delimiter ',';
COPY 3
# select * from t;
x | y
---+-------
1 | ONE
2 | TWO
3 | THREE
(3 rows)
Using psql run the following command
\copy students from 'C:\Program Files\PostgreSQL\10\data\Data_copy\student.csv' csv;
I have a PostGIS enabled PostgreSQL database that is no longer needed in production. I would like to back it up, but the geometry columns of PostGIS should be transformed to a simple, long-term stable text format like WKT.
I'm aware of the ST_AsText function.
SELECT road_id, ST_AsText(road_geom) AS geom, road_name FROM roads;
But how do I apply this to backup the complete database with several tables with geometries and many without?
That's the backup strategy I finally applied:
1. Normal dump of the PostGIS-Postgres database with pg_dump. Plain-text format to improve readability.
2. CSV backup of all tables
For this I created a copy of my initial database ("la_as_text") and transformed all the geometry columns to text with the following script (thanks #Jim Jones). It's very specific for my case, but I decided to post it like this anyway. Just in case somebody runs in the came issues as I did with views that depend on geometry columns and gist indexes. The views won't work if you change the datatype to text and text columns are also not valid for gist indexing.
#!/bin/bash
# has to be run by a user who has owner permissions for the database
rm delete_views.txt delete_views_mod.txt alter_script.txt alter_script_mod.txt
# delete all views - not necessary in long-term-backup and cause problems when altering geometry columns
echo "select 'drop view ' ||table_name|| ';' from information_schema.views where table_schema not in ('pg_catalog', 'information_schema') and table_name "'!'"~ '^pg_';" | psql la_as_text >> delete_views.txt
cat delete_views.txt | tail -n +3 | head -n -2 >> delete_views_mod.txt
cat delete_views_mod.txt | psql la_as_text
# delete one particular gist index that depends on a geometry column -- text can't be indexed with gist
echo "drop index idx_combined_points_the_geom;" | psql la_as_text
# change data type geometry (EWKB) to human readable text (WKT)
echo "select 'alter table '||table_schema||'.'||table_name||' alter column '||column_name||' type text USING ST_AsText('||column_name||');' from information_schema.columns where table_schema = 'public' and udt_name = 'geometry';" | psql la_as_text >> alter_script.txt
cat alter_script.txt | tail -n +3 | head -n -2 >> alter_script_mod.txt
cat alter_script_mod.txt | psql la_as_text
The resulting database lacks a lot of its initial functionality due to the missing geometry data type, but it's human readable.
Instead of a normal dump I exported all the individual tables as ';'-separated text files with the following script:
#!/bin/bash
SCHEMA="public"
DB="la_as_text"
psql -Atc "select tablename from pg_tables where schemaname='$SCHEMA'" $DB |\
while read TBL; do
psql -c "copy $SCHEMA.$TBL to stdout with csv delimiter ';'" $DB > $TBL.csv
done
I'm pretty confident that this backup can be reconstructed in the future -- if necessary.
I have a batch script that I want to be able to run on both my laptop and desktop. Depending on which machine I'm running it on, I need to pass 1 of 2 file paths to a postgresql script.
if "%computername%"=="LAPTOP" (
set init_path=C:\path1
) else (
set init_path=C:\path2
)
psql -U postgres -d dbname -v full_path=%init_path%\csvfile.csv -qf sql/sqlfile.sql
My postgresql script looks like this:
begin;
drop table if exists schm.table_name;
create table schm.table_name (
var1 date,
var2 float,
var3 text,
var4 float
);
truncate table schm.table_name;
\set quoted_path '\'' :full_path '\''
\copy schm.table_name from :quoted_path with delimiter as ',' csv quote as '"';
commit;
When I run it, I get the error:
:quoted_path: No such file or directory
Is this the correct way to quote a variable? I was basing my syntax on this thread.
I have an sql script defining a function in create_database_function.sql file:
CREATE OR REPLACE FUNCTION increment_display_count(in_host_id int, in_host_owner_id int, in_date_ date) RETURNS void as $$
BEGIN
UPDATE display_count set display_count = display_count + 1
WHERE host_id = in_host_id AND host_owner_id = in_host_owner_id AND date_ = in_date_;
IF FOUND THEN
RETURN;
END IF;
BEGIN
INSERT INTO display_count (host_id, host_owner_id, date_, display_count) VALUES (in_host_id, in_host_owner_id, in_date_, 1);
EXCEPTION WHEN OTHERS THEN
UPDATE display_count set display_count = display_count + 1
WHERE host_id = in_host_id AND host_owner_id = in_host_owner_id AND date_ = in_date_;
END;
RETURN;
END;
$$
LANGUAGE plpgsql;
I want to execute in on Windows. To do so, I run a usual command:
psql -h localhost -U postgres -d database -f create_database_function.sql
But this script gave me a huge number of syntax errors. An hour-long googling didn't bear fruit. But I went on tinkering with this script and eventually found the problem.
The solution was to prepend all semicolon ; signs with backslash \.
Though this solves the problem, it introduces another one. I work on the project with another guy. But he works on Linux. In his case the script should be without \ before semicolons.
So, why do I need to prepend ; with \ on Windows? Can this be somehow avoided or done another way?
I googled for it a lot and haven't found any similar problem.
Update
The output when I use \; instead of ;:
C:\Xubuntu_shared\pixel\pixel\src\main\scripts>psql -h localhost -U postgres -d
pixel_test -f create_database_function.sql
Password:
CREATE FUNCTION
The output with errors when I execute the script without backslash escaping:
C:\Xubuntu_shared\pixel\pixel\src\main\scripts>psql -h localhost -U postgres -d
pixel_test -f create_database_function.sql
Password:
psql:create_database_function.sql:4: ERROR: unterminated dollar-quoted string a
t or near "$$
BEGIN
UPDATE display_count set display_count = display_count + 1
WHERE host_id = in_host_id AND host_owner_id = in_host_owner_id AND date_ =
in_date_;" at character 121
psql:create_database_function.sql:6: ERROR: syntax error at or near "IF" at cha
racter 5
psql:create_database_function.sql:7: ERROR: syntax error at or near "IF" at cha
racter 9
psql:create_database_function.sql:9: ERROR: syntax error at or near "INSERT" at
character 19
psql:create_database_function.sql:12: ERROR: syntax error at or near "EXCEPTION
" at character 5
psql:create_database_function.sql:13: WARNING: there is no transaction in progr
ess
COMMIT
psql:create_database_function.sql:14: ERROR: syntax error at or near "RETURN" a
t character 5
psql:create_database_function.sql:15: WARNING: there is no transaction in progr
ess
COMMIT
psql:create_database_function.sql:17: ERROR: unterminated dollar-quoted string
at or near "$$
LANGUAGE plpgsql;" at character 1
Other information that may be important:
create_database_function.sql encoding is UTF-8, without BOM. Line ending is in Windows format.
Update 2
Version
pixel=> SELECT version();
version
-------------------------------------------------------------
PostgreSQL 9.2.3, compiled by Visual C++ build 1600, 32-bit
(1 row)
pixel=>
Update 3
Output from output of select name, setting, source from pg_settings where source <> 'default'; command:
Oleg#OLEG-PC /C/Xubuntu_shared/pixel/pixel/src/main/scripts (pixel-dev2)
$ psql -U postgres
Password:
Welcome to psql.exe 7.4.6, the PostgreSQL interactive terminal.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help on internal slash commands
\g or terminate with semicolon to execute query
\q to quit
Warning: Console codepage (866) differs from windows codepage (1251)
8-bit characters will not work correctly. See PostgreSQL
documentation "Installation on Windows" for details.
postgres=# select name,setting,source from pg_settings where source <> 'default'
;
name | setting
| source
----------------------------+---------------------------------------------------
---+----------------------
config_file | C:/Program Files/PostgreSQL/9.2/data/postgresql.co
nf | override
data_directory | C:/Program Files/PostgreSQL/9.2/data
| override
DateStyle | ISO, MDY
| configuration file
default_text_search_config | pg_catalog.english
| configuration file
hba_file | C:/Program Files/PostgreSQL/9.2/data/pg_hba.conf
| override
ident_file | C:/Program Files/PostgreSQL/9.2/data/pg_ident.conf
| override
lc_collate | C
| override
lc_ctype | C
| override
lc_messages | C
| configuration file
lc_monetary | C
| configuration file
lc_numeric | C
| configuration file
lc_time | C
| configuration file
listen_addresses | *
| configuration file
log_destination | stderr
| configuration file
log_line_prefix | %t
| configuration file
log_timezone | Europe/Moscow
| configuration file
logging_collector | on
| configuration file
max_connections | 100
| configuration file
max_stack_depth | 2048
| environment variable
port | 5432
| configuration file
server_encoding | UTF8
| override
shared_buffers | 4096
| configuration file
TimeZone | Europe/Moscow
| configuration file
transaction_deferrable | off
| override
transaction_isolation | read committed
| override
transaction_read_only | off
| override
wal_buffers | 128
| override
(27 rows)
For those who may encounter this rare problem.
It is related to Postgres server and client versions mismatch. I were using Server of version 9.2.3 and Client of version 7.4.6.
Is important to mention that Postgres installer for Windows already includes psql client. So there is no need to install another one.
I don't remember exactly why I installed a separate client, but I guess it was because psql didn't start from console. I think it could be cured by rebooting Windows after PosrgreSQL isntallation (as a lot of problems are solved with this OS this way) or manually adding path to psql.exe to environment path variable.
So, if you face the same problem, check the version of Server and Client. And if they don't match, set the path environment variable to the correct client from original PosgreSQL shipping.
I have a somewhat detailed query in a script that uses ? placeholders. I wanted to test this same query directly from the psql command line (outside the script). I want to avoid going in and replacing all the ? with actual values, instead I'd like to pass the arguments after the query.
Example:
SELECT *
FROM foobar
WHERE foo = ?
AND bar = ?
OR baz = ? ;
Looking for something like:
%> {select * from foobar where foo=? and bar=? or baz=? , 'foo','bar','baz' };
You can use the -v option e.g:
$ psql -v v1=12 -v v2="'Hello World'" -v v3="'2010-11-12'"
and then refer to the variables in SQL as :v1, :v2 etc:
select * from table_1 where id = :v1;
Please pay attention to how we pass string/date values using two quotes " '...' " But this way of interpolation is prone to SQL injections, because it's you who's responsible for quoting. E.g. need to include a single quote? -v v2="'don''t do this'".
A better/safer way is to let PostgreSQL handle it:
$ psql -c 'create table t (a int, b varchar, c date)'
$ echo "insert into t (a, b, c) values (:'v1', :'v2', :'v3')" \
| psql -v v1=1 -v v2="don't do this" -v v3=2022-01-01
Found out in PostgreSQL, you can PREPARE statements just like you can in a scripting language. Unfortunately, you still can't use ?, but you can use $n notation.
Using the above example:
PREPARE foo(text,text,text) AS
SELECT *
FROM foobar
WHERE foo = $1
AND bar = $2
OR baz = $3 ;
EXECUTE foo('foo','bar','baz');
DEALLOCATE foo;
In psql there is a mechanism via the
\set name val
command, which is supposed to be tied to the -v name=val command-line option. Quoting is painful, In most cases it is easier to put the whole query meat inside a shell here-document.
Edit
oops, I should have said -v instead of -P (which is for formatting options) previous reply got it right.
You can also pass-in the parameters at the psql command-line, or from a batch file. The first statements gather necessary details for connecting to your database.
The final prompt asks for the constraint values, which will be used in the WHERE column IN() clause. Remember to single-quote if strings, and separate by comma:
#echo off
echo "Test for Passing Params to PGSQL"
SET server=localhost
SET /P server="Server [%server%]: "
SET database=amedatamodel
SET /P database="Database [%database%]: "
SET port=5432
SET /P port="Port [%port%]: "
SET username=postgres
SET /P username="Username [%username%]: "
SET /P bunos="Enter multiple constraint values for IN clause [%constraints%]: "
ECHO you typed %constraints%
PAUSE
REM pause
"C:\Program Files\PostgreSQL\9.0\bin\psql.exe" -h %server% -U %username% -d %database% -p %port% -e -v v1=%constraints% -f test.sql
Now in your SQL code file, add the v1 token within your WHERE clause, or anywhere else in the SQL. Note that the tokens can also be used in an open SQL statement, not just in a file. Save this as test.sql:
SELECT * FROM myTable
WHERE NOT someColumn IN (:v1);
In Windows, save the whole file as a DOS BATch file (.bat), save the test.sql in the same directory, and launch the batch file.
Thanks for Dave Page, of EnterpriseDB, for the original prompted script.
I would like to offer another answer inspired by #malcook's comment (using bash).
This option may work for you if you need to use shell variables within your query when using the -c flag. Specifically, I wanted to get the count of a table, whose name was a shell variable (which you can't pass directly when using -c).
Assume you have your shell variable
$TABLE_NAME='users'
Then you can get the results of that by using
psql -q -A -t -d databasename -c <<< echo "select count(*) from $TABLE_NAME;"
(the -q -A -t is just to print out the resulting number without additional formatting)
I will note that the echo in the here-string (the <<< operator) may not be necessary, I originally thought the quotes by themselves would be fine, maybe someone can clarify the reason for this.
It would appear that what you ask can't be done directly from the command line. You'll either have to use a user-defined function in plpgsql or call the query from a scripting language (and the latter approach makes it a bit easier to avoid SQL injection).
I've ended up using a better version of #vol7ron answer:
DO $$
BEGIN
IF NOT EXISTS(SELECT 1 FROM pg_prepared_statements WHERE name = 'foo') THEN
PREPARE foo(text,text,text) AS
SELECT *
FROM foobar
WHERE foo = $1
AND bar = $2
OR baz = $3;
END IF;
END$$;
EXECUTE foo('foo','bar','baz');
This way you can always execute it in this order (the query prepared only if it does not prepared yet), repeat the execution and get the result from the last query.