TSQL GROUP BY part of string - tsql

I have a simple query. Something like this:
SELECT l.list_name, COUNT(order_id)
FROM orders o JOIN lists l ON l.order_id=o.order_id
WHERE l.list_name LIKE 'orders_1%' or l.list_name LIKE 'orders_2%'
GROUP BY l.list_name
The situation looks like this: overight a stored procedure is updating the lists table, but it chops lists in parts if there are more than 1000 orders.
If I have 1200 orders with criteria or list 'orders_1', then my procedure will create two lists: 'orders_1_1' 'and orders_1_2', the first having 1,000 and second 200 orders.
So when I run my query to count those orders I will get results like so:
list_name count
orders_1 100
orders_1_more_than_100_1 1000
orders_1_more_than_100_2 200
orders_2 400
orders_3_1 1000
orders_3_2 1000
orders_3_3 420
orders_3_more_than_100_1 1000
orders_3_more_than_100_2 900
orders_3_more_than_200_1 1000
orders_3_more_than_200_2 1000
orders_3_more_than_200_3 100
orders_4 200
orders_4_more_than_300 200
The result I would like to get should look like this:
list_name count
orders_1 100
orders_1_more_than_100 1200
orders_2 400
orders_3 2420
orders_3_more_than_100 1900
orders_3_more_than_200 2100
orders_4 200
orders_4_more_than_300 200
So that it will sum all lists that start the same.
Any ideas on that? :)
These are the exact values that I have in my list_names column:
WYS_AUT_PISMO_NR_6
WYS_AUT_PISMO_NR_5_POWYZEJ_240
WYS_AUT_PISMO_NR_5_DO_240
WYS_AUT_PISMO_NR_4_POWYZEJ_240_5
WYS_AUT_PISMO_NR_4_POWYZEJ_240_4
WYS_AUT_PISMO_NR_4_POWYZEJ_240_3
WYS_AUT_PISMO_NR_4_POWYZEJ_240_2
WYS_AUT_PISMO_NR_4_POWYZEJ_240_1
WYS_AUT_PISMO_NR_4_DO_240
WYS_AUT_PISMO_NR_3_POWYZEJ_240
WYS_AUT_PISMO_NR_3_DO_240
WYS_AUT_PISMO_NR_2_POWYZEJ_240
WYS_AUT_PISMO_NR_2_DO_240
WYS_AUT_PISMO_NR_1
What I want is to group them like so:
WYS_AUT_PISMO_NR_6
WYS_AUT_PISMO_NR_5_POWYZEJ_240
WYS_AUT_PISMO_NR_5_DO_240
WYS_AUT_PISMO_NR_4_POWYZEJ_240 /*here I must group those 5 lists*/
WYS_AUT_PISMO_NR_4_DO_240
WYS_AUT_PISMO_NR_3_POWYZEJ_240
WYS_AUT_PISMO_NR_3_DO_240
WYS_AUT_PISMO_NR_2_POWYZEJ_240
WYS_AUT_PISMO_NR_2_DO_240
WYS_AUT_PISMO_NR_1

This monstruous expression will isolate string starting from beginning of parameter to last _ if there are more than one underscores:
select case when len (l.list_name) - len (replace (l.list_name, '_', '')) > 1
then left(l.list_name,
len (l.list_name) - charindex('_', reverse(l.list_name)))
else l.list_name
end
Alternatively you might strip 'orders_' from string, replace underscore with dot and convert it to float, then to int to remove decimals, and then back to string using this monstrosity:
select 'orders_' + cast (cast (cast (
replace (substring (#str, 8, 100), '_', '.')
as float) as int) as varchar(100))
To avoid repeating this blobs, use derived table instead of lists:
SELECT l.TrimmedListName, COUNT(order_id)
FROM orders o
JOIN
(
select lists.*,
-- Remove optional list continuation number
case when len (list_name) - len (replace (list_name, '_', '')) > 1
then left(list_name,
len (list_name) - charindex('_', reverse(list_name)))
else list_name
end AS TrimmedListName
from lists
) l ON l.order_id=o.order_id
WHERE (l.list_name LIKE 'orders_1%' or l.list_name LIKE 'orders_2%')
GROUP BY l.TrimmedListName

try something like
select substring(l.list_name, 0, 8), count(order_Id)
FROM orders o JOIN lists l ON l.order_id=o.order_id
WHERE l.list_name LIKE 'orders_1%' or l.list_name LIKE 'orders_2%'
group by substring(l.list_name, 0, 8)
Updated answer for the updated question:
select newColName, COUNT(order_id)
from
(
select case when GetSubstringCount(l.list_name, '_', '') > 1 then
SUBSTRING(l.list_name, 0, len(l.list_name) - 2)
else l.list_name end as NewColName
, order_Id
FROM orders o JOIN lists l ON l.order_id=o.order_id
WHERE l.list_name LIKE 'orders_1%' or l.list_name LIKE 'orders_2%'
) mySubTable
group by newColName
You'll need something like This to create the GetSubstringCount method

Related

PLSQL to TSQL - REGEXP

Im trying to convert a script from PLSQL to TSQL and am stuff with a couple of lines
table(cast(multiset(select level from dual connect by level <= len (regexp_replace(t.image, '[^**]+'))/2) as sys.OdciNumberList)) levels
where substr(REGEXP_SUBSTR (t.image, '[^**]+',1, levels.column_value),1,instr( REGEXP_SUBSTR (t.image, '[^**]+',1, levels.column_value),'=',1) -1)
IMAGE
Any help would be great.
Chris
For a better answer it would be good to include some sample input and desired results. Especially when addressing a different version of SQL. Perhaps including a PL/SQL tag would help find someone who understands PL/SQL and T-SQL. It would also be helpful to include DDL, specifically the datatype for "Level". Again, I say this not to be critical but rather guide you towards getting better answers here.
All That said, you can accomplish what you are trying to do in T-SQL leveraging a tally table, an N-Grams function and a couple other functions which I are included at the end of this post.
regexp_replace
To replace or remove characters that match a pattern in t-SQL you can use patreplace8k. Here's an example of how to use it to replace numbers with *'s:
SELECT pr.NewString
FROM samd.patReplace8K('My phone number is 555-2211','[0-9]','*') AS pr;
Returns: My phone number is -*
regexp_subsr
Here's an example of how to extract all phone numbers from a string:
DECLARE
#string VARCHAR(8000) = 'Call me later at 222-3333 or tomorrow at 312.555.2222,
(313)555-6789, or at 1+800-555-4444 before noon. Thanks!',
#pattern VARCHAR(50) = '%[^0-9()+.-]%';
-- EXTRACTOR
SELECT ItemNumber = ROW_NUMBER() OVER (ORDER BY f.position),
ItemIndex = f.position,
ItemLength = itemLen.l,
Item = SUBSTRING(f.token, 1, itemLen.l)
FROM
(
SELECT ng.position, SUBSTRING(#string,ng.position,DATALENGTH(#string))
FROM samd.NGrams8k(#string, 1) AS ng
WHERE PATINDEX(#pattern, ng.token) < --<< this token does NOT match the pattern
ABS(SIGN(ng.position-1)-1) + --<< are you the first row? OR
PATINDEX(#pattern,SUBSTRING(#string,ng.position-1,1)) --<< always 0 for 1st row
) AS f(position, token)
CROSS APPLY (VALUES(ISNULL(NULLIF(PATINDEX(#pattern,f.token),0), --CROSS APPLY (VALUES(ISNULL(NULLIF(PATINDEX('%'+#pattern+'%',f.token),0),
DATALENGTH(#string)+2-f.position)-1)) AS itemLen(l)
WHERE itemLen.L > 6 -- this filter is more harmful to the extractor than the splitter
ORDER BY ItemNumber;
T-SQL INSTR Function
I included a T-SQL version of Oracles INSTR function at the end of this post. Note these examples:
DECLARE
#string VARCHAR(8000) = 'AABBCC-AA123-AAXYZPDQ-AA-54321',
#search VARCHAR(8000) = '-AA',
#position INT = 1,
#occurance INT = 2;
-- 1.1. Get me the 2nd #occurance "-AA" in #string beginning at #position 1
SELECT f.* FROM samd.instr8k(#string,#search,#position,#occurance) AS f;
-- 1.2. Retreive everything *BEFORE* the second instance of "-AA"
SELECT
ItemIndex = f.ItemIndex,
Item = SUBSTRING(#string,1,f.itemindex-1)
FROM samd.instr8k(#string,#search,#position,#occurance) AS f;
-- 1.3. Retreive everything *AFTER* the second instance of "-AA"
SELECT
ItemIndex = MAX(f.ItemIndex),
Item = MAX(SUBSTRING(#string,f.itemindex+f.itemLength,8000))
FROM samd.instr8k(#string,#search,#position,#occurance) AS f;
regexp_replace (ADVANCED)
Here's a more complex example, leveraging ngrams8k to replace phone numbers with the text "REMOVED"
DECLARE
#string VARCHAR(8000) = 'Call me later at 222-3333 or tomorrow at 312.555.2222, (313)555-6789, or at 1+800-555-4444 before noon. Thanks!',
#pattern VARCHAR(50) = '%[0-9()+.-]%';
SELECT NewString = (
SELECT IIF(IsMatch=1 AND patSplit.item LIKE '%[0-9][0-9][0-9]%','<REMOVED>', patSplit.item)
FROM
(
SELECT 1, i.Idx, SUBSTRING(#string,1,i.Idx), CAST(0 AS BIT)
FROM (VALUES(PATINDEX(#pattern,#string)-1)) AS i(Idx) --FROM (VALUES(PATINDEX('%'+#pattern+'%',#string)-1)) AS i(Idx)
WHERE SUBSTRING(#string,1,1) NOT LIKE #pattern
UNION ALL
SELECT r.RN,
itemLength = LEAD(r.RN,1,DATALENGTH(#string)+1) OVER (ORDER BY r.RN)-r.RN,
item = SUBSTRING(#string,r.RN,
LEAD(r.RN,1,DATALENGTH(#string)+1) OVER (ORDER BY r.RN)-r.RN),
isMatch = ABS(t.p-2+1)
FROM core.rangeAB(1,DATALENGTH(#string),1,1) AS r
CROSS APPLY (VALUES (
CAST(PATINDEX(#pattern,SUBSTRING(#string,r.RN,1)) AS BIT),
CAST(PATINDEX(#pattern,SUBSTRING(#string,r.RN-1,1)) AS BIT),
SUBSTRING(#string,r.RN,r.Op+1))) AS t(c,p,s)
WHERE t.c^t.p = 1
) AS patSplit(ItemIndex, ItemLength, Item, IsMatch)
FOR XML PATH(''), TYPE).value('.','varchar(8000)');
Returns:
Call me later at or tomorrow at , , or at before noon. Thanks!
CREATE FUNCTION core.rangeAB
(
#Low BIGINT, -- (start) Lowest number in the set
#High BIGINT, -- (stop) Highest number in the set
#Gap BIGINT, -- (step) Difference between each number in the set
#Row1 BIT -- Base: 0 or 1; should RN begin with 0 or 1?
)
/****************************************************************************************
[Purpose]:
Creates a lazy, in-memory, forward-ordered sequence of up to 531,441,000,000 integers
starting with #Low and ending with #High (inclusive). RangeAB is a pure, 100% set-based
alternative to solving SQL problems using iterative methods such as loops, cursors and
recursive CTEs. RangeAB is based on Itzik Ben-Gan's getnums function for producing a
sequence of integers and uses logic from Jeff Moden's fnTally function which includes a
parameter for determining if the "row-number" (RN) should begin with 0 or 1.
I wanted to use the name "Range" because it functions and performs almost identically to
the Range function built into Python and Clojure. RANGE is a reserved SQL keyword so I
went with "RangeAB". Functions/Algorithms developed using rangeAB can be easilty ported
over to Python, Clojure or any other programming language that leverages a lazy sequence.
The two major differences between RangeAB and the Python/Clojure versions are:
1. RangeAB is *Inclusive* where the other two are *Exclusive". range(0,3) in Python and
Clojure return [0 1 2], core.rangeAB(0,3) returns [0 1 2 3].
2. RangeAB has a fourth Parameter (#Row1) to determine if RN should begin with 0 or 1.
[Author]:
Alan Burstein
[Compatibility]:
SQL Server 2008+
[Syntax]:
SELECT r.RN, r.OP, r.N1, r.N2
FROM core.rangeAB(#Low,#High,#Gap,#Row1) AS r;
[Parameters]:
#Low = BIGINT; represents the lowest value for N1.
#High = BIGINT; represents the highest value for N1.
#Gap = BIGINT; represents how much N1 and N2 will increase each row. #Gap is also the
difference between N1 and N2.
#Row1 = BIT; represents the base (first) value of RN. When #Row1 = 0, RN begins with 0,
when #row = 1 then RN begins with 1.
[Returns]:
Inline Table Valued Function returns:
RN = BIGINT; a row number that works just like T-SQL ROW_NUMBER() except that it can
start at 0 or 1 which is dictated by #Row1. If you need the numbers:
(0 or 1) through #High, then use RN as your "N" value, ((#Row1=0 for 0, #Row1=1),
otherwise use N1.
OP = BIGINT; returns the "finite opposite" of RN. When RN begins with 0 the first number
in the set will be 0 for RN, the last number in will be 0 for OP. When returning the
numbers 1 to 10, 1 to 10 is retrurned in ascending order for RN and in descending
order for OP.
Given the Numbers 1 to 3, 3 is the opposite of 1, 2 the opposite of 2, and 1 is the
opposite of 3. Given the numbers -1 to 2, the opposite of -1 is 2, the opposite of 0
is 1, and the opposite of 1 is 0.
The best practie is to only use OP when #Gap > 1; use core.O instead. Doing so will
improve performance by 1-2% (not huge but every little bit counts)
N1 = BIGINT; This is the "N" in your tally table/numbers function. this is your *Lazy*
sequence of numbers starting at #Low and incrementing by #Gap until the next number
in the sequence is greater than #High.
N2 = BIGINT; a lazy sequence of numbers starting #Low+#Gap and incrementing by #Gap. N2
will always be greater than N1 by #Gap. N2 can also be thought of as:
LEAD(N1,1,N1+#Gap) OVER (ORDER BY RN)
[Dependencies]:
N/A
[Developer Notes]:
1. core.rangeAB returns one billion rows in exactly 90 seconds on my laptop:
4X 2.7GHz CPU's, 32 GB - multiple versions of SQL Server (2005-2019)
2. The lowest and highest possible numbers returned are whatever is allowable by a
bigint. The function, however, returns no more than 531,441,000,000 rows (8100^3).
3. #Gap does not affect RN, RN will begin at #Row1 and increase by 1 until the last row
unless its used in a subquery where a filter is applied to RN.
4. #Gap must be greater than 0 or the function will not return any rows.
5. Keep in mind that when #Row1 is 0 then the highest RN value (ROWNUMBER) will be the
number of rows returned minus 1
6. If you only need is a sequential set beginning at 0 or 1 then, for best performance
use the RN column. Use N1 and/or N2 when you need to begin your sequence at any
number other than 0 or 1 or if you need a gap between your sequence of numbers.
7. Although #Gap is a bigint it must be a positive integer or the function will
not return any rows.
8. The function will not return any rows when one of the following conditions are true:
* any of the input parameters are NULL
* #High is less than #Low
* #Gap is not greater than 0
To force the function to return all NULLs instead of not returning anything you can
add the following code to the end of the query:
UNION ALL
SELECT NULL, NULL, NULL, NULL
WHERE NOT (#High&#Low&#Gap&#Row1 IS NOT NULL AND #High >= #Low AND #Gap > 0)
This code was excluded as it adds a ~5% performance penalty.
9. There is no performance penalty for sorting by RN ASC; there is a large performance
penalty, however for sorting in descending order. If you need a descending sort the
use OP in place of RN then sort by rn ASC.
10. When setting the #Row1 to 0 and sorting by RN you will see that the 0 is added via
MERGE JOIN concatination. Under the hood the function is essentially concatinating
but, because it's using a MERGE JOIN operator instead of concatination the cost
estimations are needlessly high. You can circumvent this problem by changing:
ORDER BY core.rangeAB.RN to: ORDER BY ROW_NUMBER() OVER (ORDER BY (SELECT NULL))
[Examples]:
-----------------------------------------------------------------------------------------
[Revision History]:
Rev 00 - 20140518 - Initial Development - AJB
Rev 05 - 20191122 - Developed this "core" version for open source distribution;
updated notes and did some final code clean-up
*****************************************************************************************/
RETURNS TABLE WITH SCHEMABINDING AS RETURN
WITH
L1(N) AS
(
SELECT 1
FROM (VALUES
($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),
($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),
($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),
($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),($),
($),($)) T(N) -- 90 values
),
L2(N) AS (SELECT 1 FROM L1 a CROSS JOIN L1 b CROSS JOIN L1 c),
iTally(RN) AS (SELECT ROW_NUMBER() OVER (ORDER BY (SELECT 1)) FROM L2 a CROSS JOIN L2 b)
SELECT r.RN, r.OP, r.N1, r.N2
FROM
(
SELECT
RN = 0,
OP = (#High-#Low)/#Gap,
N1 = #Low,
N2 = #Gap+#Low
WHERE #Row1 = 0
UNION ALL -- (#High-#Low)/#Gap+1:
SELECT TOP (ABS((ISNULL(#High,0)-ISNULL(#Low,0))/ISNULL(#Gap,0)+ISNULL(#Row1,1)))
RN = i.RN,
OP = (#High-#Low)/#Gap+(2*#Row1)-i.RN,
N1 = (i.rn-#Row1)*#Gap+#Low,
N2 = (i.rn-(#Row1-1))*#Gap+#Low
FROM iTally AS i
ORDER BY i.RN
) AS r
WHERE #High&#Low&#Gap&#Row1 IS NOT NULL AND #High >= #Low
AND #Gap > 0;
GO
CREATE FUNCTION samd.ngrams8k
(
#String VARCHAR(8000), -- Input string
#N INT -- requested token size
)
/*****************************************************************************************
[Purpose]:
A character-level N-Grams function that outputs a contiguous stream of #N-sized tokens
based on an input string (#String). Accepts strings up to 8000 varchar characters long.
For more information about N-Grams see: http://en.wikipedia.org/wiki/N-gram.
[Author]:
Alan Burstein
[Compatibility]:
SQL Server 2008+, Azure SQL Database
[Syntax]:
--===== Autonomous
SELECT ng.Position, ng.Token
FROM samd.ngrams8k(#String,#N) AS ng;
--===== Against a table using APPLY
SELECT s.SomeID, ng.Position, ng.Token
FROM dbo.SomeTable AS s
CROSS APPLY samd.ngrams8k(s.SomeValue,#N) AS ng;
[Parameters]:
#String = The input string to split into tokens.
#N = The size of each token returned.
[Returns]:
Position = BIGINT; the position of the token in the input string
token = VARCHAR(8000); a #N-sized character-level N-Gram token
[Dependencies]:
1. core.rangeAB (iTVF)
[Developer Notes]:
1. ngrams8k is not case sensitive;
2. Many functions that use ngrams8k will see a huge performance gain when the optimizer
creates a parallel execution plan. One way to get a parallel query plan (if the
optimizer does not choose one) is to use make_parallel by Adam Machanic which can be
found here:
sqlblog.com/blogs/adam_machanic/archive/2013/07/11/next-level-parallel-plan-porcing.aspx
3. When #N is less than 1 or greater than the datalength of the input string then no
tokens (rows) are returned. If either #String or #N are NULL no rows are returned.
This is a debatable topic but the thinking behind this decision is that: because you
can't split 'xxx' into 4-grams, you can't split a NULL value into unigrams and you
can't turn anything into NULL-grams, no rows should be returned.
For people who would prefer that a NULL input forces the function to return a single
NULL output you could add this code to the end of the function:
UNION ALL
SELECT 1, NULL
WHERE NOT(#N > 0 AND #N <= DATALENGTH(#String)) OR (#N IS NULL OR #String IS NULL)
4. ngrams8k is deterministic. For more about deterministic functions see:
https://msdn.microsoft.com/en-us/library/ms178091.aspx
[Examples]:
--===== 1. Split the string, "abcd" into unigrams, bigrams and trigrams
SELECT ng.Position, ng.Token FROM samd.ngrams8k('abcd',1) AS ng; -- unigrams (#N=1)
SELECT ng.Position, ng.Token FROM samd.ngrams8k('abcd',2) AS ng; -- bigrams (#N=2)
SELECT ng.Position, ng.Token FROM samd.ngrams8k('abcd',3) AS ng; -- trigrams (#N=3)
[Revision History]:
------------------------------------------------------------------------------------------
Rev 00 - 20140310 - Initial Development - Alan Burstein
Rev 01 - 20150522 - Removed DQS N-Grams functionality, improved iTally logic. Also Added
conversion to bigint in the TOP logic to remove implicit conversion
to bigint - Alan Burstein
Rev 05 - 20171228 - Small simplification; changed:
(ABS(CONVERT(BIGINT,(DATALENGTH(ISNULL(#String,''))-(ISNULL(#N,1)-1)),0)))
to:
(ABS(CONVERT(BIGINT,(DATALENGTH(ISNULL(#String,''))+1-ISNULL(#N,1)),0)))
Rev 06 - 20180612 - Using CHECKSUM(N) in the to convert N in the token output instead of
using (CAST N as int). CHECKSUM removes the need to convert to int.
Rev 07 - 20180612 - re-designed to: Use core.rangeAB - Alan Burstein
*****************************************************************************************/
RETURNS TABLE WITH SCHEMABINDING AS RETURN
SELECT
Position = r.RN,
Token = SUBSTRING(#String,CHECKSUM(r.RN),#N)
FROM core.rangeAB(1,LEN(#String)+1-#N,1,1) AS r
WHERE #N > 0 AND #N <= LEN(#String);
GO
CREATE FUNCTION samd.patReplace8K
(
#string VARCHAR(8000),
#pattern VARCHAR(50),
#replace VARCHAR(20)
)
/*****************************************************************************************
[Purpose]:
Given a string (#string), a pattern (#pattern), and a replacement character (#replace)
patReplace8K will replace any character in #string that matches the #Pattern parameter
with the character, #replace.
[Author]:
Alan Burstein
[Compatibility]:
SQL Server 2008+
[Syntax]:
--===== Basic Syntax Example
SELECT pr.NewString
FROM samd.patReplace8K(#String,#Pattern,#Replace) AS pr;
[Developer Notes]:
1. Required SQL Server 2008+
2. #Pattern IS case sensitive but can be easily modified to make it case insensitive
3. There is no need to include the "%" before and/or after your pattern since since we
are evaluating each character individually
4. Certain special characters, such as "$" and "%" need to be escaped with a "/"
like so: [/$/%]
[Examples]:
--===== 1. Replace numeric characters with a "*"
SELECT pr.NewString
FROM samd.patReplace8K('My phone number is 555-2211','[0-9]','*') AS pr;
[Revision History]:
Rev 00 - 10/27/2014 Initial Development - Alan Burstein
Rev 01 - 10/29/2014 Mar 2007 - Alan Burstein
- Redesigned based on the dbo.STRIP_NUM_EE by Eirikur Eiriksson
(see: http://www.sqlservercentral.com/Forums/Topic1585850-391-2.aspx)
- change how the cte tally table is created
- put the include/exclude logic in a CASE statement instead of a WHERE clause
- Added Latin1_General_BIN Colation
- Add code to use the pattern as a parameter.
Rev 02 - 20141106
- Added final performane enhancement (more cudo's to Eirikur Eiriksson)
- Put 0 = PATINDEX filter logic into the WHERE clause
Rev 03 - 20150516
- Updated to deal with special XML characters
Rev 04 - 20170320
- changed #replace from char(1) to varchar(1) to address how spaces are handled
Rev 05 - Re-write using samd.NGrams
*****************************************************************************************/
RETURNS TABLE WITH SCHEMABINDING AS RETURN
SELECT newString =
(
SELECT CASE WHEN #string = CAST('' AS VARCHAR(8000)) THEN CAST('' AS VARCHAR(8000))
WHEN #pattern+#replace+#string IS NOT NULL THEN
CASE WHEN PATINDEX(#pattern,token COLLATE Latin1_General_BIN)=0
THEN ng.token ELSE #replace END END
FROM samd.NGrams8K(#string, 1) AS ng
ORDER BY ng.position
FOR XML PATH(''),TYPE
).value('text()[1]', 'VARCHAR(8000)');
GO
CREATE FUNCTION samd.Instr8k
(
#string VARCHAR(8000),
#search VARCHAR(8000),
#position INT,
#occurance INT
)
/*****************************************************************************************
[Purpose]:
Returns the position (ItemIndex) of the Nth(#occurance) occurrence of one string(#search) within
another(#string). Similar to Oracle's PL/SQL INSTR funtion.
https://www.techonthenet.com/oracle/functions/instr.php
[Author]:
Alan Burstein
[Compatibility]:
SQL Server 2008+
[Syntax]:
--===== Autonomous
SELECT ins.ItemIndex, ins.ItemLength, ins.ItemCount
FROM samd.Instr8k(#string,#search,#position,#occurance) AS ins;
--===== Against a table using APPLY
SELECT s.SomeID, ins.ItemIndex, ins.ItemLength, ins.ItemCount
FROM dbo.SomeTable AS s
CROSS APPLY samd.Instr8k(s.string,#search,#position,#occurance) AS ins
[Parameters]:
#string = VARCHAR(8000); Input sting to evaluate
#search = VARCHAR(8000); Token to search for inside of #string
#position = INT; Where to begin searching for #search; identical to the third
parameter in SQL Server CHARINDEX [, start_location]
#occurance = INT; Represents the Nth instance of the search string (#search)
[Returns]:
ItemIndex = Position of the Nth (#occurance) instance of #search inside #string
ItemLength = Length of #search (in case you need it, no need to re-evaluate the string)
ItemCount = Number of times #search appears inside #string
[Dependencies]:
1. samd.ngrams8k
1.1. dbo.rangeAB (iTVF)
2. samd.substringCount8K_lazy
[Developer Notes]:
1. samd.Instr8k does not treat the input strings (#string and #search) as case sensitive.
2. Don't use instr8k for "SubstringBetween" functionality; for better performance use
samd.SubstringBetween8k instead.
3. The #position parameter is the key benefit of this function when dealing with long
strings where the search item is towards the back of the string. For example, take a
5000 character string where, what you are looking for is always *at least* 3000
characters deep. Setting #position to 3000 will dramatically improve performance.
4. Unlike Oracle's PL/SQL INSTR function, Instr8k does not accept numbers less than 1.
[Examples]:
[Revision History]:
------------------------------------------------------------------------------------------
Rev 00 - 20191112 - Initial Development - Alan Burstein
*****************************************************************************************/
RETURNS TABLE WITH SCHEMABINDING AS RETURN
SELECT
ItemIndex = ISNULL(MAX(ISNULL(instr.Position,1)+(a.Pos-1)),0),
ItemLength = ISNULL(MAX(LEN(#search)),LEN(#search)),
ItemCount = ISNULL(MAX(items.SubstringCount),0)
FROM (VALUES(ISNULL(#position,1),LEN(#search))) AS a(Pos,SrchLn)
CROSS APPLY (VALUES(SUBSTRING(#string,a.Pos,8000))) AS f(String)
CROSS APPLY samd.substringCount8K_lazy(f.string,#search) AS items
CROSS APPLY
(
SELECT TOP (#occurance) RN = ROW_NUMBER() OVER (ORDER BY ng.position), ng.position
FROM samd.ngrams8k(f.string,a.SrchLn) AS ng
WHERE ng.token = #search
ORDER BY RN
) AS instr
WHERE a.Pos > 0
AND #occurance <= items.SubstringCount
AND instr.RN = #occurance;
GO
CREATE FUNCTION samd.substringCount8K_lazy
(
#string varchar(8000),
#searchstring varchar(1000)
)
/*****************************************************************************************
[Purpose]:
Scans the input string (#string) and counts how many times the search character
(#searchChar) appears. This function is Based on Itzik Ben-Gans cte numbers table logic
[Compatibility]:
SQL Server 2008+
Uses TABLE VALUES constructor (not available pre-2008)
[Author]: Alan Burstein
[Syntax]:
--===== Autonomous
SELECT f.substringCount
FROM samd.substringCount8K_lazy(#string,#searchString) AS f;
--===== Against a table using APPLY
SELECT f.substringCount
FROM dbo.someTable AS t
CROSS APPLY samd.substringCount8K_lazy(t.col, #searchString) AS f;
Parameters:
#string = VARCHAR(8000); input string to analyze
#searchString = VARCHAR(1000); substring to search for
[Returns]:
Inline table valued function returns -
substringCount = int; Number of times that #searchChar appears in #string
[Developer Notes]:
1. substringCount8K_lazy does NOT take overlapping values into consideration. For
example, this query will return a 1 but the correct result is 2:
SELECT substringCount FROM samd.substringCount8K_lazy('xxx','xx')
When overlapping values are a possibility or concern then use substringCountAdvanced8k
2. substringCount8K_lazy is what is referred to as an "inline" scalar UDF." Technically
it's aninline table valued function (iTVF) but performs the same task as a scalar
valued user defined function (UDF); the difference is that it requires the APPLY table
operator to accept column values as a parameter. For more about "inline" scalar UDFs
see thisarticle by SQL MVP Jeff Moden:
http://www.sqlservercentral.com/articles/T-SQL/91724/
and for more about how to use APPLY see the this article by SQL MVP Paul White:
http://www.sqlservercentral.com/articles/APPLY/69953/.
Note the above syntax example and usage examples below to better understand how to
use the function. Although the function is slightly more complicated to use than a
scalar UDF it will yield notably better performance for many reasons. For example,
unlike a scalar UDFs or multi-line table valued functions, the inline scalar UDF does
not restrict the query optimizer's ability generate a parallel query execution plan.
3. substringCount8K_lazy returns NULL when either input parameter is NULL and returns 0
when either input parameter is blank.
4. substringCount8K_lazy does not treat parameters as cases senstitive
5. substringCount8K_lazy is deterministic. For more deterministic functions see:
https://msdn.microsoft.com/en-us/library/ms178091.aspx
[Examples]:
--===== 1. How many times does the substring "abc" appear?
SELECT f.* FROM samd.substringCount8k_lazy('abc123xxxabc','abc') AS f;
--===== 2. Return records from a table where the substring "ab" appears more than once
DECLARE #table TABLE (string varchar(8000));
DECLARE #searchString varchar(1000) = 'ab';
INSERT #table VALUES ('abcabc'),('abcd'),('bababab'),('baba'),(NULL);
SELECT searchString = #searchString, t.string, f.substringCount
FROM #table AS t
CROSS APPLY samd.substringCount8k_lazy(string,'ab') AS f
WHERE f.substringCount > 1;
-----------------------------------------------------------------------------------------
[Revision History]:
Rev 00 - 20180625 - Initial Development - Alan Burstein
Rev 01 - 20190102 - Added logic to better handle #searchstring = char(32) - Alan Burstein
*****************************************************************************************/
RETURNS TABLE WITH SCHEMABINDING AS RETURN
SELECT substringCount = (LEN(v.s)-LEN(REPLACE(v.s,v.st,'')))/d.l
FROM (VALUES(DATALENGTH(#searchstring))) AS d(l)
CROSS APPLY (VALUES(#string,CASE WHEN d.l>0 THEN #searchstring END)) AS v(s,st);
GO

concatenate arrays in aggregate query

i have a starting table where there are some meteo data stored every 15 minutes, one field stores leaf wet at 1 minute sampling in a numeric array form, thus i have a 15 values array each row.
Now i want to create a 1 hour aggregation of this table, crating an array of 60 values for this field.
I tried array_cat at first place, but says
array_cat(numeric[]) not existing
the function obviuously exists, so i tought the format was not the one expected, i tried first unnesting and then aggregating, not working again.
Finally i was able to aggregate trough string conversion, but it's not what i wanted (i might in the future apply some numeric elaboration oh that 60-values array)
I paste the query for further investigations
SELECT dati1_v.id_stazione,
to_char(dati1_v.data_ora, 'YYYY-MM-DD HH24:00:00'::text) AS date_hour,
round(avg(dati1_v.temp1_media), 2) AS t_avg,
round(avg(dati1_v.ur1_media), 2) AS hum_avg,
sum(dati1_v.pioggia) AS rain_tot,
max(dati1_v.pioggia) AS rain_max,
round((avg((SELECT avg(lw.lw) AS avg FROM unnest(dati1_v.lw_top_array) lw(lw))) - lws.top_min) /
(lws.top_max - lws.top_min) * 100::numeric, 2) AS lw_top_avg,
array_agg((SELECT round((avg(lw.lw) - lws.top_min) / (lws.top_max - lws.top_min) * 100::numeric, 2) AS round
FROM unnest(dati1_v.lw_top_array) lw(lw))) AS lw_top_array,
array_cat(dati1_v.lw_top_array) AS lw_top_array_tot,
-- array_agg((select lw_top_array from unnest(dati1_v.lw_top_array))) AS lw_top_array_tot,
-- array_agg(array_to_string(dati1_v.lw_top_array, ',')) AS lw_top_array_tot,
round((avg((SELECT avg(lw.lw) AS avg FROM unnest(dati1_v.lw_bottom_array) lw(lw))) - lws.bottom_min) /
(lws.bottom_max - lws.bottom_min) * 100::numeric, 2) AS lw_bottom_avg,
array_agg((SELECT round((avg(lw.lw) - lws.bottom_min) / (lws.bottom_max - lws.bottom_min) * 100::numeric,
2) AS round
FROM unnest(dati1_v.lw_bottom_array) lw(lw))) AS lw_bottom_array
FROM dati1_v,
lw_settings lws
WHERE lws.id = 1
GROUP BY dati1_v.id_stazione, to_char(dati1_v.data_ora, 'YYYY-MM-DD HH24:00:00'::text), lws.top_min, lws.top_max,
lws.bottom_min, lws.bottom_max
ORDER BY dati1_v.id_stazione, to_char(dati1_v.data_ora, 'YYYY-MM-DD HH24:00:00'::text)
in particular, my tries were related to this specific block:
array_cat(dati1_v.lw_top_array) AS lw_top_array_tot,
-- array_agg((select lw_top_array from unnest(dati1_v.lw_top_array))) AS lw_top_array_tot,
-- array_agg(array_to_string(dati1_v.lw_top_array, ',')) AS lw_top_array_tot
Thanks
For me in similar case helped UNNEST in subquery and ARRAY_AGG of unnnested
SELECT
ARRAY_AGG(
DISTINCT lw_top
) as lw_top_array
FROM (
SELECT
UNNEST(lw_top_array) AS lw_top
FROM
dati1_v
) as tmp;
for me helped next query
SELECT
my_table.key,
array_agg(_unnested.item) as array_coll
from my_table
left join LATERAL (SELECT unnest(my_table.array_coll) as item) _unnested ON TRUE
GROUP by my_table.key
In PostgreSQL, the Group_concat function is not available but you can get similar result as string_agg and array_to_string.
string_agg(array_to_string(file_ids, ','), ',') filter ( where file_ids notnull ) AS file_ids_str
array_to_string and array_to_string works in next way
array_to_string([1, 2, 456], ',') => '1,2,456'
string_agg(['a', 'ab'], ',') => 'a,ab'
the only problem is that result is string with ',' as separator

How to match starting date to the closest ending date in order to get the time difference

SELECT
OUT.EMP_ID,
OUT.DT_TM "DateTimeOut",
IN.DT_TM "DateTimeIn",
cast(timestampdiff( 4, char(timestamp(IN.DT_TM) - timestamp(OUT.DT_TM))) as decimal(30,1))/60 "Duration Out"
FROM (
select
e1.EMP_ID,
e1.DT_TM
from
hr.timeout e1
WHERE
month(e1.DT_TM)=09
and year(e1.DT_TM)=2016
AND e1.CD='OUT'
) OUT
LEFT JOIN (
select
e2.EMP_ID,
e2.DT_TM
from
hr.timeout e2
WHERE
month(e2.dt_tm)=09
and year(e2.dt_tm)=2016
AND e2.CD='IN'
) IN
on out.EMP_ID=in.EMP_ID
Trying to get the closest DateTimeIn match with the DateTimeOut.
Currently it repeats the same DateTimeOut and DateTimeIn multiple times.
I think its normal because your table dont have constraint unique on emp_id, dt_tm and cd. But if you want unique result try this :
With Period as (
select e1.EMP_ID, e1.DT_TM from hr.timeout e1
WHERE month(e1.DT_TM)=09 and year(e1.DT_TM)=2016
)
Select distinct OUT.EMP_ID, IN.DT_TM as DateTimeIn, OUT.DT_TM as DateTimeOut,
TIMESTAMPDIFF(2 , CAST(timestamp(IN.DT_TM) - timestamp(ifnull(OUT.DT_TM, current date)) AS CHAR(22)) ) as DurationSecond
from Period in left outer join Period out
on out.EMP_ID=in.EMP_ID and out.CD='OUT' and in.CD='IN'
order by 1, 2, 3
Like you can see, i use timestampdiff with '2' like first parameter for second (you divided by 60), i use ifnull because you do a left outer join (out.DT_TM can be null) and i do a distinct for unique result

Generate series for various mixes of numbers and letteres

I am using this syntax:
generate_series(1, COALESCE((string_to_array(table.id_number, '-')) [2] :: INT, 1)) AS n (numbers)
To generate IDs in elements that have got ID like 32.22.1-4 to get 4 rows with IDs 32.22.1, 32.22.2, 32.22.3 and 32.22.4. How to change it to accept also letters?
So for 32.22.a-c there would be:
32.22.a, 32.22.b, 32.22.c
And for 32.22.d1-d4 there would be
32.22.d1, 32.22.d2, 32.22.d3, 32.22.d4
EDIT:
The whole code looks like:
INSERT INTO ...
(
SELECT
...
FROM table
CROSS JOIN LATERAL
generate_series(1, COALESCE((string_to_array(table.id_number, '-')) [2] :: INT, 1)) AS n (numbers)
WHERE table.id_number LIKE ...
);
WITH t(id_number) AS ( VALUES
('32.33.a1-a5'::TEXT),
('32.34.a-c'::TEXT),
('32.35.b-e'::TEXT)
), stats AS (
SELECT
chars,
chars[1] pattern, -- pattern use
CASE
WHEN (ascii(chars[3]) - ascii(chars[2])) = 0
THEN FALSE
ELSE TRUE
END char_pattern, -- check if series of chars
CASE
WHEN (ascii(chars[3]) - ascii(chars [2])) = 0
THEN right(chars[3],1)::INTEGER
ELSE (ascii(chars[3]) + 1 - ascii(chars[2]))::INTEGER
END i -- number of series
FROM t,
regexp_matches(t.id_number, '(.*\.)(\w*)-(\w*)$') chars
)
SELECT
CASE WHEN char_pattern
THEN pattern || chr(ascii(chars[2]) - 1 + s)
ELSE pattern || left(chars[2],1) || s::TEXT
END output
FROM stats,generate_series(1,stats.i) s;
Result:
output
---------
32.33.a1
32.33.a2
32.33.a3
32.33.a4
32.33.a5
32.34.a
32.34.b
32.34.c
32.35.b
32.35.c
32.35.d
32.35.e
(12 rows)

Dynamic pivot - how to obtain column titles parametrically?

I wish to write a Query for SAP B1 (t-sql) that will list all Income and Expenses Items by total and month by month.
I have successfully written a Query using PIVOT, but I do not want the column headings to be hardcoded like: Jan-11, Feb-11, Mar-11 ... Dec-11.
Rather I want the column headings to be parametrically generated, so that if I input:
--------------------------------------
Query - Selection Criteria
--------------------------------------
Posting Date greater or equal 01.09.10
Posting Date smaller or equal 31.08.11
[OK] [Cancel]
the Query will generate the following columns:
Sep-10, Oct-10, Nov-10, ..... Aug-11
I guess DYNAMIC PIVOT can do the trick.
So, I modified one SQL obtained from another forum to suit my purpose, but it does not work. The error message I get is Incorrect Syntax near 20100901.
Could anybody help me locate my error?
Note: In SAP B1, '[%1]' is an input variable
Here's my query:
/*Section 1*/
DECLARE #listCol VARCHAR(2000)
DECLARE #query VARCHAR(4000)
-------------------------------------
/*Section 2*/
SELECT #listCol =
STUFF(
( SELECT DISTINCT '],[' + CONVERT(VARCHAR, MONTH(T0.RefDate), 102)
FROM JDT1
FOR XML PATH(''))
, 1, 2, '') + ']'
------------------------------------
/*Section 3*/
SET #query = '
SELECT * FROM
(
SELECT
T0.Account,
T1.GroupMask,
T1.AcctName,
MONTH(T0.RefDate) as [Month],
(T0.Debit - T0.Credit) as [Amount]
FROM dbo.JDT1 T0
JOIN dbo.OACT T1 ON T0.Account = T1.AcctCode
WHERE
T1.GroupMask IN (4,5,6,7) AND
T0.[Refdate] >= '[%1]' AND
T0.[Refdate] <= '[%2]'
) S
PIVOT
(
Sum(Amount)
FOR [Month] IN ('+#listCol+')
) AS pvt
'
--------------------------------------------
/*Section 4*/
EXECUTE (#query)
I don't know SAP, but a couple of things spring to mind:
It looks like you want #listCol to contain a collection of numbers within square brackets, for example [07],[08],[09].... However, your code appears not to put a [ at the start of this string.
Try replacing the lines
T0.[Refdate] >= '[%1]' AND
T0.[Refdate] <= '[%2]'
with
T0.[Refdate] >= ''[%1]'' AND
T0.[Refdate] <= ''[%2]''
(I also added a space before the AND in the first of these two lines while I was editing your question.)