I'm working on a prototype that uses Postgres as its backend. I don't do a lot of SQL, so I'm feeling my way through it. I made a .pgsql file I run with psql that executes each of many files that set up my database, and I use a variable to define the schema that will be used so I can test features without mucking up my "good" instance:
\set schema_name 'example_schema'
\echo 'The Schema name is' :schema_name
\ir sql/file1.pgsql
\ir sql/file2.pgsql
This has been working well. I've defined several functions that expand :schema_name properly:
CREATE OR REPLACE FUNCTION :schema_name.get_things_by_category(...
For reasons I can't figure out, this isn't working in my newest function:
CREATE OR REPLACE FUNCTION :schema_name.update_thing_details(_id uuid, _details text)
RETURNS text
LANGUAGE 'plpgsql'
AS $BODY$
BEGIN
UPDATE :schema_name.things
...
The syntax error indicates it's interpreting :schema_name literally after UPDATE instead of expanding it. How do I get it to use the variable value instead of the literal value here? I get that maybe within the BEGIN..END is a different context, but surely there's a way to script this schema name in all places?
I can think of three approaches, since psql cannot do this directly.
Shell script
Use a bash script to perform the variable substitution and pipe the results into psql, like.
#!/bin/bash
$schemaName = $1
$contents = `cat script.sql | sed -e 's/#SCHEMA_NAME#/$schemaName'`
echo $contents | psql
This would probably be a lot of boiler plate if you have a lot of .sql scripts.
Staging Schema
Keep the approach you have now with a hard-coded schema of something like staging and then have a bash script go and rename staging to whatever you want the actual schema to be.
Customize the search path
Your entry point could be an inline script within bash that is piped into psql, does an up-front update of the default connection schema, then uses \ir to include all of your .sql files, which should not specify a schema.
#!/bin/bash
$schemaName = $1
psql <<SCRIPT
SET search_path TO $schemaName;
\ir sql/file1.pgsql
\ir sql/file2.pgsql
SCRIPT
Some details: How to select a schema in postgres when using psql?
Personally I am leaning towards the latter approach as it seems the simplest and most scalable.
The documentation says:
Variable interpolation will not be performed within quoted SQL literals and identifiers. Therefore, a construction such as ':foo' doesn't work to produce a quoted literal from a variable's value (and it would be unsafe if it did work, since it wouldn't correctly handle quotes embedded in the value).
Now the function body is a “dollar-quoted%rdquo; string literal ($BODY$...$BODY$), so the variable will not be replaced there.
I can't think of a way to do this with psql variables.
Related
I'm trying to invoke a SQL script within another SQL script using pgsql.
I already saw that I can use
\i 'path/to/file.sql'
where path/to/files.sql is between single quotes.
I was trying to replace 'path/to/file.sql' with a variable instead, like
DO $$
DECLARE
ls_id INT := 271195;
tokens VARCHAR(20);
BEGIN
tokens := CONCAT(ls_id, '_tokens.sql');
\i tokens
END $$;
Is this possible some way or another?
It's not something you can do directly in sql because \i is a psql command and not actually SQL at all.
But the following should demonstrate how you can go about it.
SELECT ('something' || '_and_' || 'something_else.sql') as filename \gset
\i :filename
The gset will create a psql variable for you based on the result of the query. Again the \gset is actually just for psql rather than being sent to the backend.
Then you can reference the psql variable prefixed by a colon.
Note that this is just a macro-style text substitution. You will need to deal with quoting spaces and backslashes and so on yourself.
I've very complex data that I'm inserting into postgresql and am using double dollar ($$) to escape. However I've one row which ends with dollar sign and is causing error.
The original row is like 'dd^d\w=dd$' and when escaped '$$dd^d\w=dd$$$'.
How can I escape this specific row?
Use any string inside the double dollar to differentiate it:
select $anything$abc$$anything$;
?column?
----------
abc$
The insert is similar:
insert into t (a, b) values
($anything$abc$$anything$, $xyz$abc$$xyz$);
INSERT 0 1
select * from t;
a | b
------+------
abc$ | abc$
I found this question troubleshooting problem with executing query with double dollar in literal from within linux shell. For example select '$abc$' in psql gives correct result $abc$ while psql -U me -c "select '$abc$'" called from linux shell produces incorrect result $ (provided there's no system variable abc).
In that case, wrapping into another delimiter ($wrapper$$abc$$wrapper$) won't help since the primary problem is interpreting dollars in shell context. Possible solution is escaping dollars (psql -U me -c "select '\$abc\$'") however this produces backslashes literally when called in psql. To produce same query usable in both psql and linux shell, psql -U me -c "select concat(chr(36),'abc',chr(36))" is universal solution.
While Clodoaldo is quite right I think there's another aspect of this you need to look at:
Why are you doing the quoting yourself? You should generally be using parameterised ("prepared") statements via your programming language's client driver. See bobby tables for some examples in common languages. Using parameterised statements means you don't have to care about doing any quoting at all anymore, it's taken care of for you by the database client driver.
I'd give you an example but you haven't mentioned your client language/tools.
I have been searching but so far I only found how to insert date into tables based on a csv files.
I have the following scenario:
Directory name = ticketID
Inside this directory I have a couple of files, like:
Description.txt
Summary.txt - Contains ticket header and has been imported succefully.
Progress_#.txt - this is everytime a ticket gets udpdated. I get a new file.
Solution.txt
Importing the Issue.txt was easy since this was actually a CSV.
Now my problem is with Description and Progress files.
I need to update the existing rows with the data from this files. Something on the line of
update table_ticket set table_ticket.description = Description.txt where ticket_number = directoryname
I'm using PostgreSQL and the COPY command is valid for new data and it would still fail due to the ',;/ special chars.
I wanted to do this using bash script, but it seem that it is it won't be possible:
for i in `find . -type d`
do
update table_ticket
set table_ticket.description = $i/Description.txt
where ticket_number = $i
done
Of course the above code would take into consideration connection to the database.
Anyone has a idea on how I could achieve this using shell script. Or would it be better to just make something in Java and read and update the record, although I would like to avoid this approach.
Thanks
Alex
Thanks for the answer, but I came across this:
psql -U dbuser -h dbhost db
\set content = `cat PATH/Description.txt`
update table_ticket set description = :'content' where ticketnr = TICKETNR;
Putting this into a simple script I created the following:
#!/bin/bash
for i in `find . -type d|grep ^./CS`
do
p=`echo $i|cut -b3-12 -`
echo $p
sed s/PATH/${p}/g cmd.sql > cmd.tmp.sql
ticketnr=`echo $p|cut -b5-10 -`
sed -i s/TICKETNR/${ticketnr}/g cmd.tmp.sql
cat cmd.tmp.sql
psql -U supportAdmin -h localhost supportdb -f cmd.tmp.sql
done
The downside is that it will create always a new connection, later I'll change to create a single file
But it does exactly what I was looking for, putting the contents inside a single column.
psql can't read the file in for you directly unless you intend to store it as a large object in which case you can use lo_import. See the psql command \lo_import.
Update: #AlexandreAlves points out that you can actually slurp file content in using
\set myvar = `cat somefile`
then reference it as a psql variable with :'myvar'. Handy.
While it's possible to read the file in using the shell and feed it to psql it's going to be awkward at best as the shell offers neither a native PostgreSQL database driver with parameterised query support nor any text escaping functions. You'd have to roll your own string escaping.
Even then, you need to know that the text encoding of the input file is valid for your client_encoding otherwise you'll insert garbage and/or get errors. It quickly lands up being easier to do it in a langage with proper integration with PostgreSQL like Python, Perl, Ruby or Java.
There is a way to do what you want in bash if you really must, though: use Pg's delimited dollar quoting with a randomized delimiter to help prevent SQL injection attacks. It's not perfect but it's pretty darn close. I'm writing an example now.
Given problematic file:
$ cat > difficult.txt <__END__
Shell metacharacters like: $!(){}*?"'
SQL-significant characters like "'()
__END__
and sample table:
psql -c 'CREATE TABLE testfile(filecontent text not null);'
You can:
#!/bin/bash
filetoread=$1
sep=$(printf '%04x%04x\n' $RANDOM $RANDOM)
psql <<__END__
INSERT INTO testfile(filecontent) VALUES (
\$x${sep}\$$(cat ${filetoread})\$x${sep}\$
);
__END__
This could be a little hard to read and the random string generation is bash specific, though I'm sure there are probably portable approaches.
A random tag string consisting of alphanumeric characters (I used hex for convenience) is generated and stored in seq.
psql is then invoked with a here-document tag that isn't quoted. The lack of quoting is important, as <<'__END__' would tell bash not to interpret shell metacharacters within the string, wheras plain <<__END__ allows the shell to interpret them. We need the shell to interpret metacharacters as we need to substitute sep into the here document and also need to use $(...) (equivalent to backticks) to insert the file text. The x before each substitution of seq is there because here-document tags must be valid PostgreSQL identifiers so they must start with a letter not a number. There's an escaped dollar sign at the start and end of each tag because PostgreSQL dollar quotes are of the form $taghere$quoted text$taghere$.
So when the script is invoked as bash testscript.sh difficult.txt the here document lands up expanding into something like:
INSERT INTO testfile(filecontent) VALUES (
$x0a305c82$Shell metacharacters like: $!(){}*?"'
SQL-significant characters like "'()$x0a305c82$
);
where the tags vary each time, making SQL injection exploits that rely on prematurely ending the quoting difficult.
I still advise you to use a real scripting language, but this shows that it is indeed possible.
The best thing to do is to create a temporary table, COPY those from the files in question, and then run your updates.
Your secondary option would be to create a function in a language like pl/perlu and do this in the stored procedure, but you will lose a lot of performance optimizations that you can do when you update from a temp table.
Background: I need to port a ksh script from SunOS 5.10 to RHEL 5.8. It makes a call to isql to retrieve some data and, quite contrary to the intended application of final endpoint client utilities such as isql, it parses its out to be used by a variable in the shell script. Please note that I just inherited this and by no means did design such a hack myself. I certainly never would be parsing isql out to assign value to a var in shell -- if the script needed that info, I would use Perl with some API like DBD::DBI that is designed to marshall data between the application and the data store. But I have what I have and must work within the parameters.
What is happening is that the following piped input does return data on SunOS but not in RHEL:
echo "SELECT some_field FROM some_table WHERE some_crtra = 'X' \ngo" | isql -U$USER -P$PASS -D$DB -S$SERVER
That output on Solaris being:
some_field
------
Y
(1 row affected)
From that point, the script uses awk to extract just the field value from the above stream but let's ignore that because that's not the problem.
Also please note that I am able to get the data executing the piped commands separately, i.e. by going manually into isql and running the SQL. So the SQL or the connection string are not the problem -- it is either how the piping streams data OR isql itself works differently on the different platforms.
Can anybody see why there is disparate response to the same input on the two systems? Any idea how I can change the piping to make it work?
Thanks
echo "SELECT some_field FROM some_table WHERE some_crtra = 'X' \ngo"
is non portable.
I would suggest instead:
printf "SELECT some_field FROM some_table WHERE some_crtra = 'X' \ngo\n"
From the ksh93 manual page:
When the first arg does not begin with a -, and none of the
arguments contain a \, then echo prints each of its arguments
separated by a space and terminated by a new-line. Otherwise,
the behavior of echo is system dependent and print or printf
described below should be used. See echo(1) for usage and
description.
psql has a construct for passing named arguments:
psql -v name='value'
which can then be referenced inside a script:
SELECT :name;
which will give the result
?column?
----------
value
(1 row)
During development, I need to drop and recreate copies of the database fairly frequently, so I'm trying to automate the process. So I need to run a query that forcibly disconnects all users and then drops the database. But the database this operates on will vary, so the database name needs to be an argument.
The problem is that the query to disconnect the users requires a string (WHERE pg_stat_activity.datname = 'dbname') and the query that drops requires an unquoted token (DROP DATABASE IF EXISTS dbname). (Sorry. Not sure what to call that kind of token.)
I can use the named argument fine without quotes in the DROP query, but quoting the named argument in the disconnect query causes the argument to not be expanded. I.e., I would get the string ':name' instead of the string 'value'.
Is there any way to turn the unquoted value into a string or turn a string into an unquoted token for the DROP query? I can work around it by putting the disconnect and DROP queries in separate scripts and passing the argument in with quotes to the disconnect and without quotes to the DROP, but I'd prefer they were in the same script since they're really two steps in a single process.
Use:
... WHERE pg_stat_activity.datname = :'name'
Note the placement of the colon before the single quote.
The manual:
If an unquoted colon (:) followed by a psql variable name appears
within an argument, it is replaced by the variable's value, as
described in SQL Interpolation below. The forms
:'variable_name' and :"variable_name" described there work as well.
And:
To quote the value of a variable as an SQL literal, write a colon
followed by the variable name in single quotes.