I have this redshift SQL query. I extracted a number with decimal from the comment using the "REGEXP_SUBSTR" function. I also need to convert it from string to number/decimal. Then, I need to subtract that number from the total.
This is my query
SELECT sc.comment,
sm.subtotal,
to_number(REGEXP_SUBSTR(sc.comment, '[0.00-9]+..[0.00-9]+', 1),'9999999D99')
FROM "sales_memo_comments" sc INNER JOIN "sales_memo" sm ON sc.foreign_id = sm.parent_id
I tried using the "to_number" function on Redshift SQL, but its giving me the following: ERROR: invalid input syntax for type numeric: " "
This is the current output Before extracting the number refund amount from the comment column:
comment
"SAR719.00 Refund transaction executed successfully, Refund Request ID:504081288877953603 \n , Authorization Code:095542 "
"AUD52.07 Refund transaction executed successfully, Refund Request ID:6J45695858A90833"
Canceled by : ron.johnd#company.co.us
refund amount is [MYR197.41]
"Please Ignore Order refunded by Refund Request ID:5002758809696048 , Authorization Code:2587759"
OMR37.83($98.23) Refund transaction executed successfully
This is it after using the above SQL query with REGEXP. I still get some anomalies.
comment
719
52.07
.co.
197.41
5.0027621
37.83($98.23
Two questions
How do I edit the REGEXP to take account for the anomalies seen above
How do I convert my string REGEXP to a numeric value to do a subtraction with another numeric column?
Any help would be appreciated.
Here is a way - you need to be able to test whether a string is numeric and for that you need a UDF - so just run this once to define that function
create or replace function isnumeric (aval VARCHAR(20000))
returns bool
IMMUTABLE
as $$
try:
x = int(aval);
except:
return (1==2);
else:
return (1==1);
$$ language plpythonu;
Then, you could change your code as follows
SELECT sc.comment,
sm.subtotal,
to_number(
case when isnumeric(REGEXP_SUBSTR(sc.comment, '[0.00-9]+..[0.00-9]+', 1))
then REGEXP_SUBSTR(sc.comment, '[0.00-9]+..[0.00-9]+', 1)
else 0 end
,'9999999D99')
FROM "sales_memo_comments" sc INNER JOIN "sales_memo" sm ON sc.foriegn_id = sm.parent_id
My approach would be to first add two columns: one for string length and the other counting allowed characters. From this table, you could filter for only rows where the two matched (ie no non-allowed characters) and then just cast the remaining values to floats or decimals or whatever.
with temp as (
SELECT '719' as comment
UNION SELECT '52.07'
UNION SELECT '.co.'
UNION SELECT '197.41'
UNION SELECT '5.0027621'
UNION SELECT '37.83($98.23'
),
temp2 as (
SELECT *
,regexp_count(comment, '[0-9.]') as good_char_length
,len(comment) as str_length
FROM
temp
)
SELECT *
,comment::float as comment_as_float
FROM
temp2
WHERE
good_char_length = str_length
I have seen some answers already but my query is a little bit different:
Here is an original query:
cmd.CommandText = "select count(Table1.UserID) from Table1 INNER JOIN
Table2 ON Table1.ID = Table2.ID where Table1.Userid = " + UserID + " and
Table1.Number != '" + Number +"' and Table2.ID < 4";
Here is a modified query for SQL Injections:
cmd.CommandText = "select count(Table1.UserID) from Table1 INNER JOIN
Table2 ON Table1.ID = Table2.ID where Table1.Userid = #userId and
Table1.ID != #Number and Table2.ID < 4";
If you can notice, the first query has UserId surrounded by double quotes: ..." + UserID +"... and Number us surrounded by single and double quotes: ...'" + Number + "'...
Here is how I'm setting parameters:
cmd.CommandType = CommandType.Text;
cmd.Parameters.AddWithValue("#Number", Number);
cmd.Parameters.AddWithValue("#userId",UserID);
where UserID is an integer and Number is a string.
So, my question is, if the modified query formatted the right way? Is there any difference how to put #UserId and #Number parameters into a query considering the different ways they are specified in the original query?
I have been working on .net Mvc for a long time, and I can ensure you the parameters are correctly fixed by yourself in the second case, and you do not need to worry. By the way you can still debug and test if you can inject yourself. Briefly, your code looks great and invulnerable.
This is how i do it, which is similar and also as safe as yours:
string Query = #"select a1, a2, a3, a4 from table1 where a1 in
(select b1 from table2 where b2 = #start or b2 = #end)";
using (SqlCommand Comm = new SqlCommand(Query, Conn))
{
Comm.Parameters.Add("#start", SqlDbType.NVarChar).Value = start;
Comm.Parameters.Add("#end", SqlDbType.Int).Value = end;
}
In your initial query, the double quotes belonged to the actual text of the query, not the parameter. The single quotes you would add when appending a string into the sql query. I do not know why you would put single quotes around something called Number. If in fact that is a numeric type variable, it can go into the query without the single quotes. But if it has single quotes, the only thing that happens is that Sql sees it as a string, and then converts it to a number if it is going to use it as one. For example, if Table1.Number is numeric.
But, as you have noted, building your query string by appending your parameters into your query string is terrible practice as it opens the door, wide open, for sql injection attacks. So, you go with parameterized queries, as you have.
In parameterized queries, you do not worry about quotes. For parameters that are string values, the environment will worry about encasing them in quotes as it builds the command to pass to your sql db. For parameters that are numeric, quotes are not needed, and again, that is taken care of for you.
I think your 2nd version of the query is much better and from the looks of it, it should work just fine.
Adding parameters instead of concatenating your values is much safer against sql injection. And in this example, and I can't see any way to do a sql injection.
EDIT
When using parametrised queries, you dont need to add any quotes, just like when you declare a variable and use it in a query - you dont need to use quotes.
DECLARE #x CHAR(10) = 'abc'
SELECT #x
When using concatenation of values inside a query, if the value you're trying to add into the query is a CHAR, you need to wrap it between single quotes. If it's an INT, it shouldn't be wrapped between single quotes.
SELECT 'abc', 1
The double quotes you have in your first query dont have anything to do with the sql statement, they are used in your c# code to build the sql statement string you're trying to assign to CommandText.
string abcVar = "abc";
int intVar = 1;
string sqlCommand = "SELECT '" + abcVar + "', " + intVar;
I've been given a table with a few fields that hold comma-separated values (either blank or Y/N) like so (and the field name where this data is stored is People_Notified):
Y,,N,
,Y,,N
,,N,Y
Each 'slot' relates to a particular field value and I need to now include that particular field name in the string as well (in this case Parent, Admin, Police and Medical) but inserting a "N" if the current value is blank but leaving the existing Y's and N's in place. So for the above example, where there are four known slots, I would want a tsql statement to end up with:
Parent=Y,Admin=N,Police=N,Medical=N
Parent=N,Admin=Y,Police=N,Medical=N
Parent=N,Admin=N,Police=N,Medical=Y
I tried to use a combination of CHARINDEX and CASE but haven't figured a way to make this work.
js
Although a bit messy, in theory can be done in one statement:
select
'Parent=' +stuff((stuff((stuff(
substring((replace(
(','+(replace((replace(#People_Notified,',,,',',N,N,')),',,',',N,'))+','),',,',',N,')),2,7),7,0,
'Medical=')),5,0,'Police=')),3,0,'Admin=')
broken down is easier to follow:
declare #People_Notified varchar(100)=',,Y,Y' -- test variable
-- Insert Ns
set #People_Notified= (select replace(#People_Notified,',,,',',N,N,')) -- case two consecutive missing
set #People_Notified= (select replace(#People_Notified,',,',',N,')) -- case one missing
set #People_Notified= (select replace((','+#People_Notified+','),',,',',N,')) -- case start or end missing
set #People_Notified= substring(#People_Notified,2,7) -- remove extra commas added previously
-- Stuff the labels
select 'Parent=' +stuff((stuff((stuff(#People_Notified,7,0,'Medical=')),5,0,'Police=')),3,0,'Admin=')
If you're able to use XQuery in SQL Server, I don't think you need to get too complex. You could do something like this:
SELECT CONVERT(XML, REPLACE('<pn>' + REPLACE(People_Notified, ',', '</pn><pn>') + '</pn>', '<pn></pn>', '<pn>N</pn>')).query('
concat("Parent=", data(/pn[1])[1], ",Admin=", data(/pn[2])[1], ",Police=", data(/pn[3])[1], ",Medical=", data(/pn[4])[1])
')
FROM ...
Explanation: Construct an XML-like string out of the original delimited string by replacing commas with closing and opening tags. Add an opening tag to the start and a closing tag to the end. Replace each empty element with one containing "N". Convert the XML-like string into actual XML data so that you can use XQuery. Then just concatenate what you need using concat() and the right indexes for the elements' data.
Here's one way to do it:
;WITH cteXML (Id, Notified)
AS
(
SELECT Id,
CONVERT(XML,'<Notified><YN>'
+ REPLACE([notified],',', '</YN><YN>')
+ '</YN></Notified>') AS Notified
FROM People_Notified
)
select id,
'Parent=' + case Notified.value('/Notified[1]/YN[1]','varchar(1)') when '' then 'N' else Notified.value('/Notified[1]/YN[1]','varchar(1)') end + ',' +
'Admin=' + case Notified.value('/Notified[1]/YN[2]','varchar(1)') when '' then 'N' else Notified.value('/Notified[1]/YN[2]','varchar(1)') end + ',' +
'Police=' + case Notified.value('/Notified[1]/YN[3]','varchar(1)') when '' then 'N' else Notified.value('/Notified[1]/YN[3]','varchar(1)') end + ',' +
'Medical=' + case Notified.value('/Notified[1]/YN[4]','varchar(1)') when '' then 'N' else Notified.value('/Notified[1]/YN[4]','varchar(1)') end Notified
from cteXML
SQL Fiddle
Check this page out for an explanation of what the XML stuff is doing.
This page has a pretty thorough look at the various ways you can split a delimited string into rows.
Whenever I replace placeholders in the SQL query using on it surrounds the replacement with '', is there a way to prevent this?
It means I can't do things like
SQL("SELECT * FROM {table} blah").on("table" -> tabletouse)
because it wraps the table name with '' which causes an SQL syntax error.
you could certainly combine both approaches, using the format function for data you don't want to be escaped
SQL(
"""
select %s from %s
where
name = {name} and
date between {start} and {end}
order by %s
""".format(fields, table, order)
).on(
'name -> name,
'start -> startDate,
'end -> endDate
)
Just take into account that the data you are sending using the format function should NOT come from user input, otherwise it should be properly sanitized
You cannot do what you are trying. Anorm's replacement is based on PreparedStatements. Meaning all data will automatically be escaped, meaning you cannot use replacement for :
table names,
column names,
whatever operand, SQL keyword, etc.
The best you can do here is a String concatenation (and what is really a bad way in my opinion) :
SQL("SELECT * FROM " + tabletouse + " blah").as(whatever *)
PS : Checkout this question about table names in PreparedStatements.
I am trying to execute a native query and pass run-time parameters and get the result as a List. When I try to process the Object [], one of the columns fetched is a String. But it comes out as java.lang.Character instead of String. Here is the query below:
SELECT CASE
WHEN (TRUNC(abm.credit_card_expiration_date) BETWEEN trunc(SYSDATE) AND
trunc(last_day(SYSDATE))) THEN
'Expires'
ELSE
'Expired'
END EXP_STATUS,
TO_CHAR(abm.credit_card_expiration_date, 'MM/YY') EXP_DATE
FROM account_billing_methods abm
WHERE abm.account_id = 201103
AND abm.billing_type_id = 1
AND TRUNC(abm.credit_card_expiration_date) <= TRUNC(LAST_DAY(SYSDATE))
The EXP_STATUS column could not be typecasted into String as it is of type Character. Any ideas of why it does not work?
Regards,
-Anand
I had the same problem and changed the select clause of my query to:
EXP_STATUS || '' as EXP_STATUS
Then it is a VARCHAR instead of a CHAR and JPA will return it as a String instead of a Character.
If someone knows a better/more elegant solution, I would appreciate if you could share it.