I need to do an UPDATE script using the Replace() function of Postgres but I don't know the exact string that I have to replace and I'd like to know if there is some way that I can do this similary the LIKEoperator, using Wildcards.
My problem is that I got a table that contains some scripts and at the end of each one there is a tag <signature> like this:
'SELECT SCRIPT WHATEVER.... < signature>782798e2a92c72b270t920b< signature>'
What I need to do is:
UPDATE table SET script = REPLACE(script,'<signature>%<signature>','<signature>1234ABCDEF567890<signature>')
Whatever the signature is, I need to replace with a new one defined by me. I know using the '%' doesn't work, it was just to ilustrate the effect i want to perform. Is there any way to do this in Postgres 9.5?
with expr
as
(select 'Hello <signature>thisismysig</signature>'::text as
full_text, '<signature>'::text as open,
'</signature>'::text as close
)
select
substring(full_text from
position(open in full_text)+char_length(open)
for
position(close in full_text)- char_length(open)-position(open in full_text)
)
note: with part added for ease of understanding (hopefully).
Use POSIX regex to do the same thing as other answer (but shorter)
select
substring('a bunch of other stuff <signature>mysig</signature>'
from '<signature>(.*?)</signature>')
Related
If I run the following query in Postgres or Snowflake, it will remove test from the end of the input string, even though the trimming text is best:
SELECT rtrim('rtrimtest', 'best');
See:
https://www.db-fiddle.com/f/kKYwe5tNLpVoacM2q1nJY7/0
However, I need to rtrim to only remove if the trimming text is an exact match. How do I do that?
The order of the characters doesn't matter for rtrim().
rtrim('rtrimtest', 'best') is the same as rtrim('rtrimtest', 'stbe')
On Postgres you can use regex_replace() for what you want to do:
regexp_replace('rtrimtest', 'best$', '');
Online example
I'm working on a prototype that uses Postgres as its backend. I don't do a lot of SQL, so I'm feeling my way through it. I made a .pgsql file I run with psql that executes each of many files that set up my database, and I use a variable to define the schema that will be used so I can test features without mucking up my "good" instance:
\set schema_name 'example_schema'
\echo 'The Schema name is' :schema_name
\ir sql/file1.pgsql
\ir sql/file2.pgsql
This has been working well. I've defined several functions that expand :schema_name properly:
CREATE OR REPLACE FUNCTION :schema_name.get_things_by_category(...
For reasons I can't figure out, this isn't working in my newest function:
CREATE OR REPLACE FUNCTION :schema_name.update_thing_details(_id uuid, _details text)
RETURNS text
LANGUAGE 'plpgsql'
AS $BODY$
BEGIN
UPDATE :schema_name.things
...
The syntax error indicates it's interpreting :schema_name literally after UPDATE instead of expanding it. How do I get it to use the variable value instead of the literal value here? I get that maybe within the BEGIN..END is a different context, but surely there's a way to script this schema name in all places?
I can think of three approaches, since psql cannot do this directly.
Shell script
Use a bash script to perform the variable substitution and pipe the results into psql, like.
#!/bin/bash
$schemaName = $1
$contents = `cat script.sql | sed -e 's/#SCHEMA_NAME#/$schemaName'`
echo $contents | psql
This would probably be a lot of boiler plate if you have a lot of .sql scripts.
Staging Schema
Keep the approach you have now with a hard-coded schema of something like staging and then have a bash script go and rename staging to whatever you want the actual schema to be.
Customize the search path
Your entry point could be an inline script within bash that is piped into psql, does an up-front update of the default connection schema, then uses \ir to include all of your .sql files, which should not specify a schema.
#!/bin/bash
$schemaName = $1
psql <<SCRIPT
SET search_path TO $schemaName;
\ir sql/file1.pgsql
\ir sql/file2.pgsql
SCRIPT
Some details: How to select a schema in postgres when using psql?
Personally I am leaning towards the latter approach as it seems the simplest and most scalable.
The documentation says:
Variable interpolation will not be performed within quoted SQL literals and identifiers. Therefore, a construction such as ':foo' doesn't work to produce a quoted literal from a variable's value (and it would be unsafe if it did work, since it wouldn't correctly handle quotes embedded in the value).
Now the function body is a “dollar-quoted%rdquo; string literal ($BODY$...$BODY$), so the variable will not be replaced there.
I can't think of a way to do this with psql variables.
I have been searching but so far I only found how to insert date into tables based on a csv files.
I have the following scenario:
Directory name = ticketID
Inside this directory I have a couple of files, like:
Description.txt
Summary.txt - Contains ticket header and has been imported succefully.
Progress_#.txt - this is everytime a ticket gets udpdated. I get a new file.
Solution.txt
Importing the Issue.txt was easy since this was actually a CSV.
Now my problem is with Description and Progress files.
I need to update the existing rows with the data from this files. Something on the line of
update table_ticket set table_ticket.description = Description.txt where ticket_number = directoryname
I'm using PostgreSQL and the COPY command is valid for new data and it would still fail due to the ',;/ special chars.
I wanted to do this using bash script, but it seem that it is it won't be possible:
for i in `find . -type d`
do
update table_ticket
set table_ticket.description = $i/Description.txt
where ticket_number = $i
done
Of course the above code would take into consideration connection to the database.
Anyone has a idea on how I could achieve this using shell script. Or would it be better to just make something in Java and read and update the record, although I would like to avoid this approach.
Thanks
Alex
Thanks for the answer, but I came across this:
psql -U dbuser -h dbhost db
\set content = `cat PATH/Description.txt`
update table_ticket set description = :'content' where ticketnr = TICKETNR;
Putting this into a simple script I created the following:
#!/bin/bash
for i in `find . -type d|grep ^./CS`
do
p=`echo $i|cut -b3-12 -`
echo $p
sed s/PATH/${p}/g cmd.sql > cmd.tmp.sql
ticketnr=`echo $p|cut -b5-10 -`
sed -i s/TICKETNR/${ticketnr}/g cmd.tmp.sql
cat cmd.tmp.sql
psql -U supportAdmin -h localhost supportdb -f cmd.tmp.sql
done
The downside is that it will create always a new connection, later I'll change to create a single file
But it does exactly what I was looking for, putting the contents inside a single column.
psql can't read the file in for you directly unless you intend to store it as a large object in which case you can use lo_import. See the psql command \lo_import.
Update: #AlexandreAlves points out that you can actually slurp file content in using
\set myvar = `cat somefile`
then reference it as a psql variable with :'myvar'. Handy.
While it's possible to read the file in using the shell and feed it to psql it's going to be awkward at best as the shell offers neither a native PostgreSQL database driver with parameterised query support nor any text escaping functions. You'd have to roll your own string escaping.
Even then, you need to know that the text encoding of the input file is valid for your client_encoding otherwise you'll insert garbage and/or get errors. It quickly lands up being easier to do it in a langage with proper integration with PostgreSQL like Python, Perl, Ruby or Java.
There is a way to do what you want in bash if you really must, though: use Pg's delimited dollar quoting with a randomized delimiter to help prevent SQL injection attacks. It's not perfect but it's pretty darn close. I'm writing an example now.
Given problematic file:
$ cat > difficult.txt <__END__
Shell metacharacters like: $!(){}*?"'
SQL-significant characters like "'()
__END__
and sample table:
psql -c 'CREATE TABLE testfile(filecontent text not null);'
You can:
#!/bin/bash
filetoread=$1
sep=$(printf '%04x%04x\n' $RANDOM $RANDOM)
psql <<__END__
INSERT INTO testfile(filecontent) VALUES (
\$x${sep}\$$(cat ${filetoread})\$x${sep}\$
);
__END__
This could be a little hard to read and the random string generation is bash specific, though I'm sure there are probably portable approaches.
A random tag string consisting of alphanumeric characters (I used hex for convenience) is generated and stored in seq.
psql is then invoked with a here-document tag that isn't quoted. The lack of quoting is important, as <<'__END__' would tell bash not to interpret shell metacharacters within the string, wheras plain <<__END__ allows the shell to interpret them. We need the shell to interpret metacharacters as we need to substitute sep into the here document and also need to use $(...) (equivalent to backticks) to insert the file text. The x before each substitution of seq is there because here-document tags must be valid PostgreSQL identifiers so they must start with a letter not a number. There's an escaped dollar sign at the start and end of each tag because PostgreSQL dollar quotes are of the form $taghere$quoted text$taghere$.
So when the script is invoked as bash testscript.sh difficult.txt the here document lands up expanding into something like:
INSERT INTO testfile(filecontent) VALUES (
$x0a305c82$Shell metacharacters like: $!(){}*?"'
SQL-significant characters like "'()$x0a305c82$
);
where the tags vary each time, making SQL injection exploits that rely on prematurely ending the quoting difficult.
I still advise you to use a real scripting language, but this shows that it is indeed possible.
The best thing to do is to create a temporary table, COPY those from the files in question, and then run your updates.
Your secondary option would be to create a function in a language like pl/perlu and do this in the stored procedure, but you will lose a lot of performance optimizations that you can do when you update from a temp table.
I have a postgres function which runs the following loop
while x<=colnum LOOP
EXECUTE
'Update attendrpt set
slot'||x||' = pres
from (SELECT branch, semester, attend_date , div, array_to_string(ARRAY_AGG(first_name||':'||alias_name||':'||lect_type||':'||
to_char(present,'99')),';')
As pres
from attend1 where lecture_slot_no ='||x||'
group by branch, semester, attend_date , div ) j
where attendrpt.branch=j.branch and attendrpt.semester=j.semester
and attendrpt.attenddate=j.attend_date and attendrpt.div=j.div;';
`x:=x+1;
END LOOP;`
The problem here is it is conflicting the single quotes closing in query and the execute command.Is there anyway to solve this.
Thanks in advance.
Quote your function definition with dollar-quoting (like $BODY$ or just $$) as per the manual.
Use execute ... using instead of string substitution. For substituting identifiers use the %I format specifier from the format function.
If you absolutely must use || string concatenation, say if you're on some ancient version of PostgreSQL, you need to use the quote_literal and quote_ident functions to avoid issues with quoting and potential security problems.
Beyond that, it looks like the whole approach is completely unnecessary; you're doing something that looks like it can be done in simple SQL.
What is wrong with this statement that it is still giving me spaces after the field. This makes me think that the syntax combining the WHEN statements is off. My boss wants them combined in one statement. What am I doing wrong?
Case WHEN LTRIM(RTRIM(cSHortName))= '' Then NULL
WHEN cShortname is NOT NULL THEN
REPLACE (cShortName,SUBSTRING,(cShortName,PATINDEX('%A-Za-z0-9""},1,) ''_
end AS SHORT_NAME
Judging from the code, it seems that you may be trying to strip spaces and non-alphanumeric characters from the beginning and ending of the string.
If so, would this work for you?
I think it provides the substring from the first alphanumeric occurrence to the last.
SELECT
SUBSTRING(
cShortName,
PATINDEX('%A-Za-z0-9',cShortName),
( LEN(cShortName)
-PATINDEX('%A-Za-z0-9',REVERSE(cShortName))
-PATINDEX('%A-Za-z0-9',cShortName)
)
) AS SHORTNAME
Replace TRIM with LTRIM.
You can also test LEN(cShortName) = 0
Ummm there seems to be some problems in this script, but try this.
Case
WHEN LTRIM(RTRIM(cSHortName))= '' Then NULL
WHEN cShortname is NOT NULL THEN REPLACE(cShortName, SUBSTRING(cShortName, PATINDEX('%A-Za-z0-9', 1) , ''), '')
end AS SHORT_NAME
Why do you think it is supposed not to give you spaces after the field?
Edit:
As far as I understand you are trying to remove any characters from the string that do not match this regular expression range [a-zA-Z0-9] (add any other characters that you want to preserve).
I see no clean way to do that in Microsoft SQL Server (you are using Microsoft SQL Server it seems) using the built-in functions. There are some examples on the web that use a temporary table and a while loop, but this is ugly. I would either return the strings as is and process them on the caller side, or write a function that does that using the CLR and invoke it from the select statement.
I hope this helps.