I have an sql script which copies data from a file:
--myscript.sql
\set file :dir 'data.csv'
copy data from :'file' csv;
and execute it with psql providing the dir variable
psql -v dir="D:\data\" -f myscript.sql
now I would like the copy command executed only if some other variable is let, e.g. -v doit=
Are there any script control structures available for this? Looking for something like
$if :{?doit}
copy data from :'file' csv;
$endif;
Have tried to wrap it by an anonymous block
do '
begin
if ':{?doit}' then
copy data from :'file' csv;
end if;
end';
But it gives the error
Error: syntax error (approximate position "TRUE")
LINE 3: if 'TRUE' then
^
Answering my own question.
There were two issues there
psql variables cannot be directly substituted in do statements
This can be solved by putting a psql variable into a server setting as suggested here
the copy command does not accept any functions for the file path name, so the first solution won't work here.
I ended up formating the do block like this
select format('
begin
if ''%s''=''t'' then
copy rates from ''%s'' csv header delimiter E''\t'';
end if;
end;',:{?doit},:'file') as do \gset
do :'do';
The format function lets us use a multiline string. The resulting string is then assigned to a new varialbe with the help of \gset and feed it to do.
Note: The format function formats 'TRUE' as 't', so we have to treat it as such.
Thanks to #Mark for the directions.
Related
I have got 100 state codes. I need to use them in a query like this:
select *
from public.clients
where state = '01'
and get 100 different .csv files. I haven't got superuser rights, so if I got it right I can only do it with \copy in psql command line. But I can't understand how to use it properly.
I tried to make it like this:
DO
$do$
declare
i text;
arr text[] := array(
select distinct state
from public.clients c
where state is not null
order by state);
BEGIN
FOREACH i IN ARRAY arr
LOOP
EXECUTE format($x$\COPY (
SELECT *
FROM public.clients c
WHERE c."state" = %1$s)
TO 'G:\Desktop\states\.%1$s.out.csv' WITH DELIMITER ',' CSV HEADER$x$, i);
END LOOP;
END
$do$
But I got ERROR: syntax error at or near "\" and I think it's not the last error because I'm not sure in this WHERE c."state" = %1$s and this TO 'G:\Desktop\states\.%1$s.out.csv' WITH DELIMITER ',' CSV HEADER$x$, i);
DO sends it program to the server to run. Since \copy is a psql metacommand, the server won't know how to run it. So you can't do it this way.
You also can't use the \gexec metacommand, as that can only be used to run SQL commands, not other metacommands.
So you can write a script (say, perl or python, or bash) to print out a series of \copy... and then pipe that into psql, or send it to a file and execute psql -f file.sql
I'm trying to invoke a SQL script within another SQL script using pgsql.
I already saw that I can use
\i 'path/to/file.sql'
where path/to/files.sql is between single quotes.
I was trying to replace 'path/to/file.sql' with a variable instead, like
DO $$
DECLARE
ls_id INT := 271195;
tokens VARCHAR(20);
BEGIN
tokens := CONCAT(ls_id, '_tokens.sql');
\i tokens
END $$;
Is this possible some way or another?
It's not something you can do directly in sql because \i is a psql command and not actually SQL at all.
But the following should demonstrate how you can go about it.
SELECT ('something' || '_and_' || 'something_else.sql') as filename \gset
\i :filename
The gset will create a psql variable for you based on the result of the query. Again the \gset is actually just for psql rather than being sent to the backend.
Then you can reference the psql variable prefixed by a colon.
Note that this is just a macro-style text substitution. You will need to deal with quoting spaces and backslashes and so on yourself.
I'm working on a prototype that uses Postgres as its backend. I don't do a lot of SQL, so I'm feeling my way through it. I made a .pgsql file I run with psql that executes each of many files that set up my database, and I use a variable to define the schema that will be used so I can test features without mucking up my "good" instance:
\set schema_name 'example_schema'
\echo 'The Schema name is' :schema_name
\ir sql/file1.pgsql
\ir sql/file2.pgsql
This has been working well. I've defined several functions that expand :schema_name properly:
CREATE OR REPLACE FUNCTION :schema_name.get_things_by_category(...
For reasons I can't figure out, this isn't working in my newest function:
CREATE OR REPLACE FUNCTION :schema_name.update_thing_details(_id uuid, _details text)
RETURNS text
LANGUAGE 'plpgsql'
AS $BODY$
BEGIN
UPDATE :schema_name.things
...
The syntax error indicates it's interpreting :schema_name literally after UPDATE instead of expanding it. How do I get it to use the variable value instead of the literal value here? I get that maybe within the BEGIN..END is a different context, but surely there's a way to script this schema name in all places?
I can think of three approaches, since psql cannot do this directly.
Shell script
Use a bash script to perform the variable substitution and pipe the results into psql, like.
#!/bin/bash
$schemaName = $1
$contents = `cat script.sql | sed -e 's/#SCHEMA_NAME#/$schemaName'`
echo $contents | psql
This would probably be a lot of boiler plate if you have a lot of .sql scripts.
Staging Schema
Keep the approach you have now with a hard-coded schema of something like staging and then have a bash script go and rename staging to whatever you want the actual schema to be.
Customize the search path
Your entry point could be an inline script within bash that is piped into psql, does an up-front update of the default connection schema, then uses \ir to include all of your .sql files, which should not specify a schema.
#!/bin/bash
$schemaName = $1
psql <<SCRIPT
SET search_path TO $schemaName;
\ir sql/file1.pgsql
\ir sql/file2.pgsql
SCRIPT
Some details: How to select a schema in postgres when using psql?
Personally I am leaning towards the latter approach as it seems the simplest and most scalable.
The documentation says:
Variable interpolation will not be performed within quoted SQL literals and identifiers. Therefore, a construction such as ':foo' doesn't work to produce a quoted literal from a variable's value (and it would be unsafe if it did work, since it wouldn't correctly handle quotes embedded in the value).
Now the function body is a “dollar-quoted%rdquo; string literal ($BODY$...$BODY$), so the variable will not be replaced there.
I can't think of a way to do this with psql variables.
I have two scripts master.sql category.sql and I am setting value for substitute variable in master.sql and use it in category value to export data in csv file - the file name is stored in substitute variable. Here is my code:
master.sql
set feedback off;
set serveroutput on;
SET SQLFORMAT csv;
var CatCode char(5) ;
COL fileName NEW_VALUE csvFile noprint
exec :CatCode := '18';
select ('c:\temp\CatCodes_' || trim(:CatCode) || '.csv') fileName from dual;
#category.sql;
exec :CatCode := '19';
select ('c:\temp\CatCodes_' || trim(:CatCode) || '.csv') fileName from dual;
category.sql
set termout off
spool &csvFile
SELECT Catgry_ID, Category_Name FROM Categories WHERE Catgry_ID = :CatCode;
spool off;
set termout off
When I run script master.sql (F5) on one machine it just works fine and creates two different csv files in c:\temp folder but when I run same script on different machine it prompts for csvFile! I am sure it should be some setting issue but I can't find it. I checked DEFINE setting on machine where it does not prompt, show define "&" (hex 26) and on another one it shows define "&". Is there anything else do I need to set to ignore prompt?
Write this in your script
set define off
If you are using substitution variables, you can set define to some other character(ensure that character is not used elsewhere)
set define <<Character>>
Example:
set define #
I ended up removing SQL Developer and reinstalling it and it worked :)
This PostgreSQL COPY command works:
copy tablename from E'c:\\abc\\a.txt';
but I want the tablename to be dynamically generated. How can I do this?
You need to build a string, concatenating in the dynamic table name, and then use execute. Note that you escape the ' by ''. This also includes a dynamic name to save the file too. You need to replace savedir with the actual directory you are using.
CREATE OR REPLACE FUNCTION dynamicCopy(tablename text, outname text) RETURNS VOID AS $$
DECLARE STATEMENT TEXT;
BEGIN
STATEMENT := 'COPY (select * from ' || quote_ident(tablename) || ') to ''savedir' || outname ||'.txt''';
EXECUTE STATEMENT;
END;
$$ LANGUAGE 'plpgsql';
EDIT:
Since I first wrote this, I have discovered the format function, which I think is generally easier to read than SQL generated with the concatenation operator || and more flexible.
CREATE OR REPLACE FUNCTION dynamicCopy(tablename text, outname text) RETURNS VOID AS
$BODY$
BEGIN
EXECUTE FORMAT('COPY (SELECT * FROM %s) TO ''savedir%s.csv''',
tablename,
outname);
END
$BODY$
LANGUAGE plpgsql;
See the official docs for a full discussion: https://www.postgresql.org/docs/current/static/plpgsql-statements.html#PLPGSQL-STATEMENTS-EXECUTING-DYN
You can copy from multi csv files to an table via shell script.
Make an script file: vim csvtotable
write csvtotable script. Here is my example:
#!/bin/sh DBNAME=postgres files=$1 for file in ${files}; do psql -d ${DBNAME} -c "\copy parent_tree(parent_id, some_text) FROM '${file}' delimiters ',' csv header" done
Execute the script.
./csv2table "$(ls *.out.csv)"
Obviously, Local CSV file should be match with table. Then It will import from csv to database tables if csv file name ending with .out.csv.
I am not sure csv name match is global or just at present directory match.
Reference link:
-c command reference https://www.postgresql.org/docs/current/app-psql.html#APP-PSQL-PATTERNS
chomd command: https://linuxize.com/post/chmod-command-in-linux/
bash shanebang: https://linuxize.com/post/bash-shebang/