Related
Im trying to convert a script from PLSQL to TSQL and am stuff with a couple of lines
table(cast(multiset(select level from dual connect by level <= len (regexp_replace(t.image, '[^**]+'))/2) as sys.OdciNumberList)) levels
where substr(REGEXP_SUBSTR (t.image, '[^**]+',1, levels.column_value),1,instr( REGEXP_SUBSTR (t.image, '[^**]+',1, levels.column_value),'=',1) -1)
IMAGE
Any help would be great.
Chris
For a better answer it would be good to include some sample input and desired results. Especially when addressing a different version of SQL. Perhaps including a PL/SQL tag would help find someone who understands PL/SQL and T-SQL. It would also be helpful to include DDL, specifically the datatype for "Level". Again, I say this not to be critical but rather guide you towards getting better answers here.
All That said, you can accomplish what you are trying to do in T-SQL leveraging a tally table, an N-Grams function and a couple other functions which I are included at the end of this post.
regexp_replace
To replace or remove characters that match a pattern in t-SQL you can use patreplace8k. Here's an example of how to use it to replace numbers with *'s:
SELECT pr.NewString
FROM samd.patReplace8K('My phone number is 555-2211','[0-9]','*') AS pr;
Returns: My phone number is -*
regexp_subsr
Here's an example of how to extract all phone numbers from a string:
DECLARE
#string VARCHAR(8000) = 'Call me later at 222-3333 or tomorrow at 312.555.2222,
(313)555-6789, or at 1+800-555-4444 before noon. Thanks!',
#pattern VARCHAR(50) = '%[^0-9()+.-]%';
-- EXTRACTOR
SELECT ItemNumber = ROW_NUMBER() OVER (ORDER BY f.position),
ItemIndex = f.position,
ItemLength = itemLen.l,
Item = SUBSTRING(f.token, 1, itemLen.l)
FROM
(
SELECT ng.position, SUBSTRING(#string,ng.position,DATALENGTH(#string))
FROM samd.NGrams8k(#string, 1) AS ng
WHERE PATINDEX(#pattern, ng.token) < --<< this token does NOT match the pattern
ABS(SIGN(ng.position-1)-1) + --<< are you the first row? OR
PATINDEX(#pattern,SUBSTRING(#string,ng.position-1,1)) --<< always 0 for 1st row
) AS f(position, token)
CROSS APPLY (VALUES(ISNULL(NULLIF(PATINDEX(#pattern,f.token),0), --CROSS APPLY (VALUES(ISNULL(NULLIF(PATINDEX('%'+#pattern+'%',f.token),0),
DATALENGTH(#string)+2-f.position)-1)) AS itemLen(l)
WHERE itemLen.L > 6 -- this filter is more harmful to the extractor than the splitter
ORDER BY ItemNumber;
T-SQL INSTR Function
I included a T-SQL version of Oracles INSTR function at the end of this post. Note these examples:
DECLARE
#string VARCHAR(8000) = 'AABBCC-AA123-AAXYZPDQ-AA-54321',
#search VARCHAR(8000) = '-AA',
#position INT = 1,
#occurance INT = 2;
-- 1.1. Get me the 2nd #occurance "-AA" in #string beginning at #position 1
SELECT f.* FROM samd.instr8k(#string,#search,#position,#occurance) AS f;
-- 1.2. Retreive everything *BEFORE* the second instance of "-AA"
SELECT
ItemIndex = f.ItemIndex,
Item = SUBSTRING(#string,1,f.itemindex-1)
FROM samd.instr8k(#string,#search,#position,#occurance) AS f;
-- 1.3. Retreive everything *AFTER* the second instance of "-AA"
SELECT
ItemIndex = MAX(f.ItemIndex),
Item = MAX(SUBSTRING(#string,f.itemindex+f.itemLength,8000))
FROM samd.instr8k(#string,#search,#position,#occurance) AS f;
regexp_replace (ADVANCED)
Here's a more complex example, leveraging ngrams8k to replace phone numbers with the text "REMOVED"
DECLARE
#string VARCHAR(8000) = 'Call me later at 222-3333 or tomorrow at 312.555.2222, (313)555-6789, or at 1+800-555-4444 before noon. Thanks!',
#pattern VARCHAR(50) = '%[0-9()+.-]%';
SELECT NewString = (
SELECT IIF(IsMatch=1 AND patSplit.item LIKE '%[0-9][0-9][0-9]%','<REMOVED>', patSplit.item)
FROM
(
SELECT 1, i.Idx, SUBSTRING(#string,1,i.Idx), CAST(0 AS BIT)
FROM (VALUES(PATINDEX(#pattern,#string)-1)) AS i(Idx) --FROM (VALUES(PATINDEX('%'+#pattern+'%',#string)-1)) AS i(Idx)
WHERE SUBSTRING(#string,1,1) NOT LIKE #pattern
UNION ALL
SELECT r.RN,
itemLength = LEAD(r.RN,1,DATALENGTH(#string)+1) OVER (ORDER BY r.RN)-r.RN,
item = SUBSTRING(#string,r.RN,
LEAD(r.RN,1,DATALENGTH(#string)+1) OVER (ORDER BY r.RN)-r.RN),
isMatch = ABS(t.p-2+1)
FROM core.rangeAB(1,DATALENGTH(#string),1,1) AS r
CROSS APPLY (VALUES (
CAST(PATINDEX(#pattern,SUBSTRING(#string,r.RN,1)) AS BIT),
CAST(PATINDEX(#pattern,SUBSTRING(#string,r.RN-1,1)) AS BIT),
SUBSTRING(#string,r.RN,r.Op+1))) AS t(c,p,s)
WHERE t.c^t.p = 1
) AS patSplit(ItemIndex, ItemLength, Item, IsMatch)
FOR XML PATH(''), TYPE).value('.','varchar(8000)');
Returns:
Call me later at or tomorrow at , , or at before noon. Thanks!
CREATE FUNCTION core.rangeAB
(
#Low BIGINT, -- (start) Lowest number in the set
#High BIGINT, -- (stop) Highest number in the set
#Gap BIGINT, -- (step) Difference between each number in the set
#Row1 BIT -- Base: 0 or 1; should RN begin with 0 or 1?
)
/****************************************************************************************
[Purpose]:
Creates a lazy, in-memory, forward-ordered sequence of up to 531,441,000,000 integers
starting with #Low and ending with #High (inclusive). RangeAB is a pure, 100% set-based
alternative to solving SQL problems using iterative methods such as loops, cursors and
recursive CTEs. RangeAB is based on Itzik Ben-Gan's getnums function for producing a
sequence of integers and uses logic from Jeff Moden's fnTally function which includes a
parameter for determining if the "row-number" (RN) should begin with 0 or 1.
I wanted to use the name "Range" because it functions and performs almost identically to
the Range function built into Python and Clojure. RANGE is a reserved SQL keyword so I
went with "RangeAB". Functions/Algorithms developed using rangeAB can be easilty ported
over to Python, Clojure or any other programming language that leverages a lazy sequence.
The two major differences between RangeAB and the Python/Clojure versions are:
1. RangeAB is *Inclusive* where the other two are *Exclusive". range(0,3) in Python and
Clojure return [0 1 2], core.rangeAB(0,3) returns [0 1 2 3].
2. RangeAB has a fourth Parameter (#Row1) to determine if RN should begin with 0 or 1.
[Author]:
Alan Burstein
[Compatibility]:
SQL Server 2008+
[Syntax]:
SELECT r.RN, r.OP, r.N1, r.N2
FROM core.rangeAB(#Low,#High,#Gap,#Row1) AS r;
[Parameters]:
#Low = BIGINT; represents the lowest value for N1.
#High = BIGINT; represents the highest value for N1.
#Gap = BIGINT; represents how much N1 and N2 will increase each row. #Gap is also the
difference between N1 and N2.
#Row1 = BIT; represents the base (first) value of RN. When #Row1 = 0, RN begins with 0,
when #row = 1 then RN begins with 1.
[Returns]:
Inline Table Valued Function returns:
RN = BIGINT; a row number that works just like T-SQL ROW_NUMBER() except that it can
start at 0 or 1 which is dictated by #Row1. If you need the numbers:
(0 or 1) through #High, then use RN as your "N" value, ((#Row1=0 for 0, #Row1=1),
otherwise use N1.
OP = BIGINT; returns the "finite opposite" of RN. When RN begins with 0 the first number
in the set will be 0 for RN, the last number in will be 0 for OP. When returning the
numbers 1 to 10, 1 to 10 is retrurned in ascending order for RN and in descending
order for OP.
Given the Numbers 1 to 3, 3 is the opposite of 1, 2 the opposite of 2, and 1 is the
opposite of 3. Given the numbers -1 to 2, the opposite of -1 is 2, the opposite of 0
is 1, and the opposite of 1 is 0.
The best practie is to only use OP when #Gap > 1; use core.O instead. Doing so will
improve performance by 1-2% (not huge but every little bit counts)
N1 = BIGINT; This is the "N" in your tally table/numbers function. this is your *Lazy*
sequence of numbers starting at #Low and incrementing by #Gap until the next number
in the sequence is greater than #High.
N2 = BIGINT; a lazy sequence of numbers starting #Low+#Gap and incrementing by #Gap. N2
will always be greater than N1 by #Gap. N2 can also be thought of as:
LEAD(N1,1,N1+#Gap) OVER (ORDER BY RN)
[Dependencies]:
N/A
[Developer Notes]:
1. core.rangeAB returns one billion rows in exactly 90 seconds on my laptop:
4X 2.7GHz CPU's, 32 GB - multiple versions of SQL Server (2005-2019)
2. The lowest and highest possible numbers returned are whatever is allowable by a
bigint. The function, however, returns no more than 531,441,000,000 rows (8100^3).
3. #Gap does not affect RN, RN will begin at #Row1 and increase by 1 until the last row
unless its used in a subquery where a filter is applied to RN.
4. #Gap must be greater than 0 or the function will not return any rows.
5. Keep in mind that when #Row1 is 0 then the highest RN value (ROWNUMBER) will be the
number of rows returned minus 1
6. If you only need is a sequential set beginning at 0 or 1 then, for best performance
use the RN column. Use N1 and/or N2 when you need to begin your sequence at any
number other than 0 or 1 or if you need a gap between your sequence of numbers.
7. Although #Gap is a bigint it must be a positive integer or the function will
not return any rows.
8. The function will not return any rows when one of the following conditions are true:
* any of the input parameters are NULL
* #High is less than #Low
* #Gap is not greater than 0
To force the function to return all NULLs instead of not returning anything you can
add the following code to the end of the query:
UNION ALL
SELECT NULL, NULL, NULL, NULL
WHERE NOT (#High&#Low&#Gap&#Row1 IS NOT NULL AND #High >= #Low AND #Gap > 0)
This code was excluded as it adds a ~5% performance penalty.
9. There is no performance penalty for sorting by RN ASC; there is a large performance
penalty, however for sorting in descending order. If you need a descending sort the
use OP in place of RN then sort by rn ASC.
10. When setting the #Row1 to 0 and sorting by RN you will see that the 0 is added via
MERGE JOIN concatination. Under the hood the function is essentially concatinating
but, because it's using a MERGE JOIN operator instead of concatination the cost
estimations are needlessly high. You can circumvent this problem by changing:
ORDER BY core.rangeAB.RN to: ORDER BY ROW_NUMBER() OVER (ORDER BY (SELECT NULL))
[Examples]:
-----------------------------------------------------------------------------------------
[Revision History]:
Rev 00 - 20140518 - Initial Development - AJB
Rev 05 - 20191122 - Developed this "core" version for open source distribution;
updated notes and did some final code clean-up
*****************************************************************************************/
RETURNS TABLE WITH SCHEMABINDING AS RETURN
WITH
L1(N) AS
(
SELECT 1
FROM (VALUES
($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),
($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),
($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),
($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),
($),($)) T(N) -- 90 values
),
L2(N) AS (SELECT 1 FROM L1 a CROSS JOIN L1 b CROSS JOIN L1 c),
iTally(RN) AS (SELECT ROW_NUMBER() OVER (ORDER BY (SELECT 1)) FROM L2 a CROSS JOIN L2 b)
SELECT r.RN, r.OP, r.N1, r.N2
FROM
(
SELECT
RN = 0,
OP = (#High-#Low)/#Gap,
N1 = #Low,
N2 = #Gap+#Low
WHERE #Row1 = 0
UNION ALL -- (#High-#Low)/#Gap+1:
SELECT TOP (ABS((ISNULL(#High,0)-ISNULL(#Low,0))/ISNULL(#Gap,0)+ISNULL(#Row1,1)))
RN = i.RN,
OP = (#High-#Low)/#Gap+(2*#Row1)-i.RN,
N1 = (i.rn-#Row1)*#Gap+#Low,
N2 = (i.rn-(#Row1-1))*#Gap+#Low
FROM iTally AS i
ORDER BY i.RN
) AS r
WHERE #High&#Low&#Gap&#Row1 IS NOT NULL AND #High >= #Low
AND #Gap > 0;
GO
CREATE FUNCTION samd.ngrams8k
(
#String VARCHAR(8000), -- Input string
#N INT -- requested token size
)
/*****************************************************************************************
[Purpose]:
A character-level N-Grams function that outputs a contiguous stream of #N-sized tokens
based on an input string (#String). Accepts strings up to 8000 varchar characters long.
For more information about N-Grams see: http://en.wikipedia.org/wiki/N-gram.
[Author]:
Alan Burstein
[Compatibility]:
SQL Server 2008+, Azure SQL Database
[Syntax]:
--===== Autonomous
SELECT ng.Position, ng.Token
FROM samd.ngrams8k(#String,#N) AS ng;
--===== Against a table using APPLY
SELECT s.SomeID, ng.Position, ng.Token
FROM dbo.SomeTable AS s
CROSS APPLY samd.ngrams8k(s.SomeValue,#N) AS ng;
[Parameters]:
#String = The input string to split into tokens.
#N = The size of each token returned.
[Returns]:
Position = BIGINT; the position of the token in the input string
token = VARCHAR(8000); a #N-sized character-level N-Gram token
[Dependencies]:
1. core.rangeAB (iTVF)
[Developer Notes]:
1. ngrams8k is not case sensitive;
2. Many functions that use ngrams8k will see a huge performance gain when the optimizer
creates a parallel execution plan. One way to get a parallel query plan (if the
optimizer does not choose one) is to use make_parallel by Adam Machanic which can be
found here:
sqlblog.com/blogs/adam_machanic/archive/2013/07/11/next-level-parallel-plan-porcing.aspx
3. When #N is less than 1 or greater than the datalength of the input string then no
tokens (rows) are returned. If either #String or #N are NULL no rows are returned.
This is a debatable topic but the thinking behind this decision is that: because you
can't split 'xxx' into 4-grams, you can't split a NULL value into unigrams and you
can't turn anything into NULL-grams, no rows should be returned.
For people who would prefer that a NULL input forces the function to return a single
NULL output you could add this code to the end of the function:
UNION ALL
SELECT 1, NULL
WHERE NOT(#N > 0 AND #N <= DATALENGTH(#String)) OR (#N IS NULL OR #String IS NULL)
4. ngrams8k is deterministic. For more about deterministic functions see:
https://msdn.microsoft.com/en-us/library/ms178091.aspx
[Examples]:
--===== 1. Split the string, "abcd" into unigrams, bigrams and trigrams
SELECT ng.Position, ng.Token FROM samd.ngrams8k('abcd',1) AS ng; -- unigrams (#N=1)
SELECT ng.Position, ng.Token FROM samd.ngrams8k('abcd',2) AS ng; -- bigrams (#N=2)
SELECT ng.Position, ng.Token FROM samd.ngrams8k('abcd',3) AS ng; -- trigrams (#N=3)
[Revision History]:
------------------------------------------------------------------------------------------
Rev 00 - 20140310 - Initial Development - Alan Burstein
Rev 01 - 20150522 - Removed DQS N-Grams functionality, improved iTally logic. Also Added
conversion to bigint in the TOP logic to remove implicit conversion
to bigint - Alan Burstein
Rev 05 - 20171228 - Small simplification; changed:
(ABS(CONVERT(BIGINT,(DATALENGTH(ISNULL(#String,''))-(ISNULL(#N,1)-1)),0)))
to:
(ABS(CONVERT(BIGINT,(DATALENGTH(ISNULL(#String,''))+1-ISNULL(#N,1)),0)))
Rev 06 - 20180612 - Using CHECKSUM(N) in the to convert N in the token output instead of
using (CAST N as int). CHECKSUM removes the need to convert to int.
Rev 07 - 20180612 - re-designed to: Use core.rangeAB - Alan Burstein
*****************************************************************************************/
RETURNS TABLE WITH SCHEMABINDING AS RETURN
SELECT
Position = r.RN,
Token = SUBSTRING(#String,CHECKSUM(r.RN),#N)
FROM core.rangeAB(1,LEN(#String)+1-#N,1,1) AS r
WHERE #N > 0 AND #N <= LEN(#String);
GO
CREATE FUNCTION samd.patReplace8K
(
#string VARCHAR(8000),
#pattern VARCHAR(50),
#replace VARCHAR(20)
)
/*****************************************************************************************
[Purpose]:
Given a string (#string), a pattern (#pattern), and a replacement character (#replace)
patReplace8K will replace any character in #string that matches the #Pattern parameter
with the character, #replace.
[Author]:
Alan Burstein
[Compatibility]:
SQL Server 2008+
[Syntax]:
--===== Basic Syntax Example
SELECT pr.NewString
FROM samd.patReplace8K(#String,#Pattern,#Replace) AS pr;
[Developer Notes]:
1. Required SQL Server 2008+
2. #Pattern IS case sensitive but can be easily modified to make it case insensitive
3. There is no need to include the "%" before and/or after your pattern since since we
are evaluating each character individually
4. Certain special characters, such as "$" and "%" need to be escaped with a "/"
like so: [/$/%]
[Examples]:
--===== 1. Replace numeric characters with a "*"
SELECT pr.NewString
FROM samd.patReplace8K('My phone number is 555-2211','[0-9]','*') AS pr;
[Revision History]:
Rev 00 - 10/27/2014 Initial Development - Alan Burstein
Rev 01 - 10/29/2014 Mar 2007 - Alan Burstein
- Redesigned based on the dbo.STRIP_NUM_EE by Eirikur Eiriksson
(see: http://www.sqlservercentral.com/Forums/Topic1585850-391-2.aspx)
- change how the cte tally table is created
- put the include/exclude logic in a CASE statement instead of a WHERE clause
- Added Latin1_General_BIN Colation
- Add code to use the pattern as a parameter.
Rev 02 - 20141106
- Added final performane enhancement (more cudo's to Eirikur Eiriksson)
- Put 0 = PATINDEX filter logic into the WHERE clause
Rev 03 - 20150516
- Updated to deal with special XML characters
Rev 04 - 20170320
- changed #replace from char(1) to varchar(1) to address how spaces are handled
Rev 05 - Re-write using samd.NGrams
*****************************************************************************************/
RETURNS TABLE WITH SCHEMABINDING AS RETURN
SELECT newString =
(
SELECT CASE WHEN #string = CAST('' AS VARCHAR(8000)) THEN CAST('' AS VARCHAR(8000))
WHEN #pattern+#replace+#string IS NOT NULL THEN
CASE WHEN PATINDEX(#pattern,token COLLATE Latin1_General_BIN)=0
THEN ng.token ELSE #replace END END
FROM samd.NGrams8K(#string, 1) AS ng
ORDER BY ng.position
FOR XML PATH(''),TYPE
).value('text()[1]', 'VARCHAR(8000)');
GO
CREATE FUNCTION samd.Instr8k
(
#string VARCHAR(8000),
#search VARCHAR(8000),
#position INT,
#occurance INT
)
/*****************************************************************************************
[Purpose]:
Returns the position (ItemIndex) of the Nth(#occurance) occurrence of one string(#search) within
another(#string). Similar to Oracle's PL/SQL INSTR funtion.
https://www.techonthenet.com/oracle/functions/instr.php
[Author]:
Alan Burstein
[Compatibility]:
SQL Server 2008+
[Syntax]:
--===== Autonomous
SELECT ins.ItemIndex, ins.ItemLength, ins.ItemCount
FROM samd.Instr8k(#string,#search,#position,#occurance) AS ins;
--===== Against a table using APPLY
SELECT s.SomeID, ins.ItemIndex, ins.ItemLength, ins.ItemCount
FROM dbo.SomeTable AS s
CROSS APPLY samd.Instr8k(s.string,#search,#position,#occurance) AS ins
[Parameters]:
#string = VARCHAR(8000); Input sting to evaluate
#search = VARCHAR(8000); Token to search for inside of #string
#position = INT; Where to begin searching for #search; identical to the third
parameter in SQL Server CHARINDEX [, start_location]
#occurance = INT; Represents the Nth instance of the search string (#search)
[Returns]:
ItemIndex = Position of the Nth (#occurance) instance of #search inside #string
ItemLength = Length of #search (in case you need it, no need to re-evaluate the string)
ItemCount = Number of times #search appears inside #string
[Dependencies]:
1. samd.ngrams8k
1.1. dbo.rangeAB (iTVF)
2. samd.substringCount8K_lazy
[Developer Notes]:
1. samd.Instr8k does not treat the input strings (#string and #search) as case sensitive.
2. Don't use instr8k for "SubstringBetween" functionality; for better performance use
samd.SubstringBetween8k instead.
3. The #position parameter is the key benefit of this function when dealing with long
strings where the search item is towards the back of the string. For example, take a
5000 character string where, what you are looking for is always *at least* 3000
characters deep. Setting #position to 3000 will dramatically improve performance.
4. Unlike Oracle's PL/SQL INSTR function, Instr8k does not accept numbers less than 1.
[Examples]:
[Revision History]:
------------------------------------------------------------------------------------------
Rev 00 - 20191112 - Initial Development - Alan Burstein
*****************************************************************************************/
RETURNS TABLE WITH SCHEMABINDING AS RETURN
SELECT
ItemIndex = ISNULL(MAX(ISNULL(instr.Position,1)+(a.Pos-1)),0),
ItemLength = ISNULL(MAX(LEN(#search)),LEN(#search)),
ItemCount = ISNULL(MAX(items.SubstringCount),0)
FROM (VALUES(ISNULL(#position,1),LEN(#search))) AS a(Pos,SrchLn)
CROSS APPLY (VALUES(SUBSTRING(#string,a.Pos,8000))) AS f(String)
CROSS APPLY samd.substringCount8K_lazy(f.string,#search) AS items
CROSS APPLY
(
SELECT TOP (#occurance) RN = ROW_NUMBER() OVER (ORDER BY ng.position), ng.position
FROM samd.ngrams8k(f.string,a.SrchLn) AS ng
WHERE ng.token = #search
ORDER BY RN
) AS instr
WHERE a.Pos > 0
AND #occurance <= items.SubstringCount
AND instr.RN = #occurance;
GO
CREATE FUNCTION samd.substringCount8K_lazy
(
#string varchar(8000),
#searchstring varchar(1000)
)
/*****************************************************************************************
[Purpose]:
Scans the input string (#string) and counts how many times the search character
(#searchChar) appears. This function is Based on Itzik Ben-Gans cte numbers table logic
[Compatibility]:
SQL Server 2008+
Uses TABLE VALUES constructor (not available pre-2008)
[Author]: Alan Burstein
[Syntax]:
--===== Autonomous
SELECT f.substringCount
FROM samd.substringCount8K_lazy(#string,#searchString) AS f;
--===== Against a table using APPLY
SELECT f.substringCount
FROM dbo.someTable AS t
CROSS APPLY samd.substringCount8K_lazy(t.col, #searchString) AS f;
Parameters:
#string = VARCHAR(8000); input string to analyze
#searchString = VARCHAR(1000); substring to search for
[Returns]:
Inline table valued function returns -
substringCount = int; Number of times that #searchChar appears in #string
[Developer Notes]:
1. substringCount8K_lazy does NOT take overlapping values into consideration. For
example, this query will return a 1 but the correct result is 2:
SELECT substringCount FROM samd.substringCount8K_lazy('xxx','xx')
When overlapping values are a possibility or concern then use substringCountAdvanced8k
2. substringCount8K_lazy is what is referred to as an "inline" scalar UDF." Technically
it's aninline table valued function (iTVF) but performs the same task as a scalar
valued user defined function (UDF); the difference is that it requires the APPLY table
operator to accept column values as a parameter. For more about "inline" scalar UDFs
see thisarticle by SQL MVP Jeff Moden:
http://www.sqlservercentral.com/articles/T-SQL/91724/
and for more about how to use APPLY see the this article by SQL MVP Paul White:
http://www.sqlservercentral.com/articles/APPLY/69953/.
Note the above syntax example and usage examples below to better understand how to
use the function. Although the function is slightly more complicated to use than a
scalar UDF it will yield notably better performance for many reasons. For example,
unlike a scalar UDFs or multi-line table valued functions, the inline scalar UDF does
not restrict the query optimizer's ability generate a parallel query execution plan.
3. substringCount8K_lazy returns NULL when either input parameter is NULL and returns 0
when either input parameter is blank.
4. substringCount8K_lazy does not treat parameters as cases senstitive
5. substringCount8K_lazy is deterministic. For more deterministic functions see:
https://msdn.microsoft.com/en-us/library/ms178091.aspx
[Examples]:
--===== 1. How many times does the substring "abc" appear?
SELECT f.* FROM samd.substringCount8k_lazy('abc123xxxabc','abc') AS f;
--===== 2. Return records from a table where the substring "ab" appears more than once
DECLARE #table TABLE (string varchar(8000));
DECLARE #searchString varchar(1000) = 'ab';
INSERT #table VALUES ('abcabc'),('abcd'),('bababab'),('baba'),(NULL);
SELECT searchString = #searchString, t.string, f.substringCount
FROM #table AS t
CROSS APPLY samd.substringCount8k_lazy(string,'ab') AS f
WHERE f.substringCount > 1;
-----------------------------------------------------------------------------------------
[Revision History]:
Rev 00 - 20180625 - Initial Development - Alan Burstein
Rev 01 - 20190102 - Added logic to better handle #searchstring = char(32) - Alan Burstein
*****************************************************************************************/
RETURNS TABLE WITH SCHEMABINDING AS RETURN
SELECT substringCount = (LEN(v.s)-LEN(REPLACE(v.s,v.st,'')))/d.l
FROM (VALUES(DATALENGTH(#searchstring))) AS d(l)
CROSS APPLY (VALUES(#string,CASE WHEN d.l>0 THEN #searchstring END)) AS v(s,st);
GO
i have a starting table where there are some meteo data stored every 15 minutes, one field stores leaf wet at 1 minute sampling in a numeric array form, thus i have a 15 values array each row.
Now i want to create a 1 hour aggregation of this table, crating an array of 60 values for this field.
I tried array_cat at first place, but says
array_cat(numeric[]) not existing
the function obviuously exists, so i tought the format was not the one expected, i tried first unnesting and then aggregating, not working again.
Finally i was able to aggregate trough string conversion, but it's not what i wanted (i might in the future apply some numeric elaboration oh that 60-values array)
I paste the query for further investigations
SELECT dati1_v.id_stazione,
to_char(dati1_v.data_ora, 'YYYY-MM-DD HH24:00:00'::text) AS date_hour,
round(avg(dati1_v.temp1_media), 2) AS t_avg,
round(avg(dati1_v.ur1_media), 2) AS hum_avg,
sum(dati1_v.pioggia) AS rain_tot,
max(dati1_v.pioggia) AS rain_max,
round((avg((SELECT avg(lw.lw) AS avg FROM unnest(dati1_v.lw_top_array) lw(lw))) - lws.top_min) /
(lws.top_max - lws.top_min) * 100::numeric, 2) AS lw_top_avg,
array_agg((SELECT round((avg(lw.lw) - lws.top_min) / (lws.top_max - lws.top_min) * 100::numeric, 2) AS round
FROM unnest(dati1_v.lw_top_array) lw(lw))) AS lw_top_array,
array_cat(dati1_v.lw_top_array) AS lw_top_array_tot,
-- array_agg((select lw_top_array from unnest(dati1_v.lw_top_array))) AS lw_top_array_tot,
-- array_agg(array_to_string(dati1_v.lw_top_array, ',')) AS lw_top_array_tot,
round((avg((SELECT avg(lw.lw) AS avg FROM unnest(dati1_v.lw_bottom_array) lw(lw))) - lws.bottom_min) /
(lws.bottom_max - lws.bottom_min) * 100::numeric, 2) AS lw_bottom_avg,
array_agg((SELECT round((avg(lw.lw) - lws.bottom_min) / (lws.bottom_max - lws.bottom_min) * 100::numeric,
2) AS round
FROM unnest(dati1_v.lw_bottom_array) lw(lw))) AS lw_bottom_array
FROM dati1_v,
lw_settings lws
WHERE lws.id = 1
GROUP BY dati1_v.id_stazione, to_char(dati1_v.data_ora, 'YYYY-MM-DD HH24:00:00'::text), lws.top_min, lws.top_max,
lws.bottom_min, lws.bottom_max
ORDER BY dati1_v.id_stazione, to_char(dati1_v.data_ora, 'YYYY-MM-DD HH24:00:00'::text)
in particular, my tries were related to this specific block:
array_cat(dati1_v.lw_top_array) AS lw_top_array_tot,
-- array_agg((select lw_top_array from unnest(dati1_v.lw_top_array))) AS lw_top_array_tot,
-- array_agg(array_to_string(dati1_v.lw_top_array, ',')) AS lw_top_array_tot
Thanks
For me in similar case helped UNNEST in subquery and ARRAY_AGG of unnnested
SELECT
ARRAY_AGG(
DISTINCT lw_top
) as lw_top_array
FROM (
SELECT
UNNEST(lw_top_array) AS lw_top
FROM
dati1_v
) as tmp;
for me helped next query
SELECT
my_table.key,
array_agg(_unnested.item) as array_coll
from my_table
left join LATERAL (SELECT unnest(my_table.array_coll) as item) _unnested ON TRUE
GROUP by my_table.key
In PostgreSQL, the Group_concat function is not available but you can get similar result as string_agg and array_to_string.
string_agg(array_to_string(file_ids, ','), ',') filter ( where file_ids notnull ) AS file_ids_str
array_to_string and array_to_string works in next way
array_to_string([1, 2, 456], ',') => '1,2,456'
string_agg(['a', 'ab'], ',') => 'a,ab'
the only problem is that result is string with ',' as separator
I get this error with my query, but I can't figure out what is wrong with it:
Query-specified return tuple has 135 columns but crosstab returns 295.
And here's my query: (The part highlighted in bold returns same number of rows when run separately in pgAdmin.)
SELECT X.*, pi.productcode, pi.productitemdesc, pi.retailsalesprice, cat.productcategorydesc FROM (
SELECT * FROM crosstab (
'SELECT a.productitem AS productitemid, l.locationcode, (CASE WHEN SUM(a.netamount) IS NOT NULL THEN SUM(a.netamount) ELSE 0 END) || ''#'' || (CASE WHEN SUM(a.quantity) IS NOT NULL THEN SUM(a.quantity) ELSE 0 END) AS sales_qty FROM invoiceitem a INNER JOIN invoiceinfo b ON a.invoice = b.invoiceid INNER JOIN locationinfo l ON b.location = l.locationid WHERE b.status !=2 AND l.locationtype = 1 AND l.status = 1 AND TO_CHAR (b.invoicedate, ''YYYY-MM-DD'')>=''2018-03-01'' AND TO_CHAR (b.invoicedate, ''YYYY-MM-DD'')<=''2018-03-03'' GROUP BY a.productitem, l.locationcode ORDER BY a.productitem',
'SELECT l.locationcode FROM locationinfo l INNER JOIN invoiceinfo b ON b.location = l.locationid
WHERE b.status !=2 AND l.locationtype = 1 AND l.status = 1 AND TO_CHAR (b.invoicedate, ''YYYY-MM-DD'')>=''2018-03-01'' AND TO_CHAR (b.invoicedate, ''YYYY-MM-DD'')<=''2018-03-03'' GROUP BY l.locationcode order by l.locationcode')
AS (productitemid int, "0007" text,"BE101" text,"BE1013" text,"BE1014" text,"BE102" text,"BE103" text,"BE1034" text,"BE104" text,"BE1040" text,"BE1043" text,"BE1044" text,"BE1045" text,"BE1046" text,"BE105" text,"BE106" text,"BE107" text,"BE108" text,"BE109" text,"BE110" text,"BE111" text,"BE112" text,"BE123" text,"BE1265" text,"BE1266" text,"BE1271" text,"BE1272" text,"BE1273" text,"BE1274" text,"BE1279" text,"BE1280" text,"BE1281" text,"BE1282" text,"BE1351" text,"BE1400" text,"BE1401" text,"BE1404" text,"BE141" text,"BE142" text,"BE193" text,"BE194" text,"BE2125" text,"BE2126" text,"BE2127" text,"BE2128" text,"BE3001" text,"BE3002" text,"BE3005" text,"BE3006" text,"BE3009" text,"BE3010" text,"BE3031" text,"BE3032" text,"BE3121" text,"BE3122" text,"BE3123" text,"BE3124" text,"BE3127" text,"BE3128" text,"BE3131" text,"BE3132" text,"BE3203" text,"BE3204" text,"BE325" text,"BE3253" text,"BE3254" text,"BE326" text,"BE332" text,"BE3503" text,"BE3504" text,"BE355" text,"BE356" text,"BE365" text,"BE366" text,"BE381" text,"BE382" text,"BE383" text,"BE384" text,"BE400" text,"BE401" text,"BE402" text,"BE403" text,"BE405" text,"BE406" text,"BE408" text,"BE409" text,"BE411" text,"BE412" text,"BE4311" text,"BE4316" text,"BE4401" text,"BE4402" text,"BE4521" text,"BE4522" text,"BE4551" text,"BE4552" text,"BE470" text,"BE473" text,"BE475" text,"BE481" text,"BE482" text,"BE601" text,"BE604" text,"BE609" text,"BE610" text,"BE7040" text,"BE7043" text,"BE7045" text,"BE7046" text,"BE7048" text,"BE7049" text,"BE708" text,"BE7111" text,"BE7112" text,"BE7127" text,"BE7128" text,"BE7217" text,"BE7218" text,"BE7307" text,"BE7308" text,"BE7351" text,"BE7352" text,"BE801" text,"BE802" text,"BE803" text,"BE804" text,"BE831" text,"BE832" text,"BE860" text,"BE861" text,"BE862" text,"BE863" text,"BE865" text,"BE981" text,"BE982" text
)) X
LEFT JOIN productitem pi ON X.productitemid = pi.productitemid
LEFT JOIN productcategory cat ON pi.productcategory = cat.productcategoryid
The bold part of
productitemid int, "0007" text,"BE101" text,"BE1013" text,"BE1014" text,"BE102" text,"BE103" text,"BE1034" text,"BE104" text,"BE1040" text,"BE1043" text,"BE1044" text,"BE1045" text,"BE1046" text,"BE105" text,"BE106" text,"BE107" text,"BE108" text,"BE109" text,"BE110" text,"BE111" text,"BE112" text,"BE123" text,"BE1265" text,"BE1266" text,"BE1271" text,"BE1272" text,"BE1273" text,"BE1274" text,"BE1279" text,"BE1280" text,"BE1281" text,"BE1282" text,"BE1351" text,"BE1400" text,"BE1401" text,"BE1404" text,"BE141" text,"BE142" text,"BE193" text,"BE194" text,"BE2125" text,"BE2126" text,"BE2127" text,"BE2128" text,"BE3001" text,"BE3002" text,"BE3005" text,"BE3006" text,"BE3009" text,"BE3010" text,"BE3031" text,"BE3032" text,"BE3121" text,"BE3122" text,"BE3123" text,"BE3124" text,"BE3127" text,"BE3128" text,"BE3131" text,"BE3132" text,"BE3203" text,"BE3204" text,"BE325" text,"BE3253" text,"BE3254" text,"BE326" text,"BE332" text,"BE3503" text,"BE3504" text,"BE355" text,"BE356" text,"BE365" text,"BE366" text,"BE381" text,"BE382" text,"BE383" text,"BE384" text,"BE400" text,"BE401" text,"BE402" text,"BE403" text,"BE405" text,"BE406" text,"BE408" text,"BE409" text,"BE411" text,"BE412" text,"BE4311" text,"BE4316" text,"BE4401" text,"BE4402" text,"BE4521" text,"BE4522" text,"BE4551" text,"BE4552" text,"BE470" text,"BE473" text,"BE475" text,"BE481" text,"BE482" text,"BE601" text,"BE604" text,"BE609" text,"BE610" text,"BE7040" text,"BE7043" text,"BE7045" text,"BE7046" text,"BE7048" text,"BE7049" text,"BE708" text,"BE7111" text,"BE7112" text,"BE7127" text,"BE7128" text,"BE7217" text,"BE7218" text,"BE7307" text,"BE7308" text,"BE7351" text,"BE7352" text,"BE801" text,"BE802" text,"BE803" text,"BE804" text,"BE831" text,"BE832" text,"BE860" text,"BE861" text,"BE862" text,"BE863" text,"BE865" text,"BE981" text,"BE982" text
AND
SELECT l.locationcode FROM locationinfo l INNER JOIN invoiceinfo b ON b.location = l.locationid
WHERE b.status !=2 AND l.locationtype = 1 AND l.status = 1 AND TO_CHAR (b.invoicedate, ''YYYY-MM-DD'')>=''2018-03-01'' AND TO_CHAR (b.invoicedate, ''YYYY-MM-DD'')<=''2018-03-03'' GROUP BY l.locationcode order by l.locationcode
When run seperately, I get 295 results which is correct. However putting it together in the whole query gets the error.
The manual:
The remaining output columns must have the type of the last column of the source_sql query's result, and there must be exactly as many of them as there are rows in the category_sql query's result.
crosstab(text, text) with a dynamic (!) SELECT query as 2nd parameter is very error prone, since the output column list is static. You should also generate the output column list dynamically. Example:
Dynamic alternative to pivot with CASE and GROUP BY
That aside, the different number of rows returned from the same query, may very well be due to a different timezone setting in the two different sessions.
You have the expression TO_CHAR(b.invoicedate, 'YYYY-MM-DD') in your predicates. If b.invoicedate is type timestamptz the result depends on the timezone setting. Consider:
SET timezone = '+10';
SELECT TO_CHAR(timestamptz '2018-04-07 23:30+0', 'YYYY-MM-DD');
to_char
----------
2018-04-08
SET timezone = '-10';
SELECT TO_CHAR(timestamptz '2018-04-07 23:30+0', 'YYYY-MM-DD');
to_char
----------
2018-04-07
Solution
To remove the dependency on the timezone setting, use absolute values.
to_char() is expensive complication in this place to begin with. Drop that and adjust the predicate.
Provide timestamptz values to match your column. This also allows indexes to be used (if applicable).
Return Table Type from A function in PostgreSQL
Use time zone names (not offsets or abbreviations) to avoid more misunderstandings.
Ignoring timezones altogether in Rails and PostgreSQL
Use dollar-quoting to clean up the quote mess.
Insert text with single quotes in PostgreSQL
I suggest this as 2nd parameter:
$$
SELECT l.locationcode
FROM locationinfo l
JOIN invoiceinfo b ON b.location = l.locationid
WHERE b.status <> 2 AND l.locationtype = 1 AND l.status = 1
AND b.invoicedate >= timestamptz '2018-03-01 Europe/Vienna'
AND b.invoicedate < timestamptz '2018-03-04 Europe/Vienna'
GROUP BY 1
ORDER BY 1
$$
Replace Europe/Vienna in my example with the time zone name defining your dates.
Note 2018-03-04 to include all of 2018-03-03 like you intended.
i am in the need of sum result from array_to_string function which has only integer value in array.
SELECT
state_name,
"ST_CODE" ,
dist_name ,
"DIST_CODE",
subdist_name,
"SUBDIST_CODE",
array_to_string(array_agg(distinct jan_sale::numeric), ',') as super
FROM village_details_new , aw
WHERE
"ST_CODE"::integer=3
AND village_details_new.aw_code::integer = aw.aw_code::integer
group by
"SUBDIST_CODE",
state_name,
"ST_CODE" ,
dist_name ,
"DIST_CODE",
subdist_name
order by "SUBDIST_CODE"
i need sum in
sum(array_to_string(array_agg(distinct jan_sale::numeric), ',') as super)
but its not working please help.........
"Punjab";"03";"Gurdaspur";"0301";"Dhar Kalan";"0301001";"10.1620708"
"Punjab";"03";"Gurdaspur";"0301";"Pathankot";"0301002";"8.7883687"
"Punjab";"03";"Gurdaspur";"0301";"Gurdaspur";"0301003";"0"
"Punjab";"03";"Gurdaspur";"0301";"Batala";"0301004";"0"
"Punjab";"03";"Gurdaspur";"0301";"Dera Baba Nanak";"0301005";"5.6477731"
"Punjab";"03";"Kapurthala";"0302";"Bhulath";"0302001";"6.6241262"
"Punjab";"03";"Kapurthala";"0302";"Kapurthala";"0302002";"12.6775959"
"Punjab";"03";"Kapurthala";"0302";"Sultanpur Lodhi";"0302003";"3.1599889"
"Punjab";"03";"Kapurthala";"0302";"Phagwara";"0302004";"3.3310623,8.7898848"
"Punjab";"03";"Jalandhar";"0303";"Shahkot";"0303001";"14.2610899"
"Punjab";"03";"Jalandhar";"0303";"Nakodar";"0303002";"14.2610899"
sum(distinct jan_sale)::numeric as super
I wish to write a Query for SAP B1 (t-sql) that will list all Income and Expenses Items by total and month by month.
I have successfully written a Query using PIVOT, but I do not want the column headings to be hardcoded like: Jan-11, Feb-11, Mar-11 ... Dec-11.
Rather I want the column headings to be parametrically generated, so that if I input:
--------------------------------------
Query - Selection Criteria
--------------------------------------
Posting Date greater or equal 01.09.10
Posting Date smaller or equal 31.08.11
[OK] [Cancel]
the Query will generate the following columns:
Sep-10, Oct-10, Nov-10, ..... Aug-11
I guess DYNAMIC PIVOT can do the trick.
So, I modified one SQL obtained from another forum to suit my purpose, but it does not work. The error message I get is Incorrect Syntax near 20100901.
Could anybody help me locate my error?
Note: In SAP B1, '[%1]' is an input variable
Here's my query:
/*Section 1*/
DECLARE #listCol VARCHAR(2000)
DECLARE #query VARCHAR(4000)
-------------------------------------
/*Section 2*/
SELECT #listCol =
STUFF(
( SELECT DISTINCT '],[' + CONVERT(VARCHAR, MONTH(T0.RefDate), 102)
FROM JDT1
FOR XML PATH(''))
, 1, 2, '') + ']'
------------------------------------
/*Section 3*/
SET #query = '
SELECT * FROM
(
SELECT
T0.Account,
T1.GroupMask,
T1.AcctName,
MONTH(T0.RefDate) as [Month],
(T0.Debit - T0.Credit) as [Amount]
FROM dbo.JDT1 T0
JOIN dbo.OACT T1 ON T0.Account = T1.AcctCode
WHERE
T1.GroupMask IN (4,5,6,7) AND
T0.[Refdate] >= '[%1]' AND
T0.[Refdate] <= '[%2]'
) S
PIVOT
(
Sum(Amount)
FOR [Month] IN ('+#listCol+')
) AS pvt
'
--------------------------------------------
/*Section 4*/
EXECUTE (#query)
I don't know SAP, but a couple of things spring to mind:
It looks like you want #listCol to contain a collection of numbers within square brackets, for example [07],[08],[09].... However, your code appears not to put a [ at the start of this string.
Try replacing the lines
T0.[Refdate] >= '[%1]' AND
T0.[Refdate] <= '[%2]'
with
T0.[Refdate] >= ''[%1]'' AND
T0.[Refdate] <= ''[%2]''
(I also added a space before the AND in the first of these two lines while I was editing your question.)