SQL Server discarding SPACE during GROUP BY - tsql

Looks like SQL Server (tried on 2008 R2) is doing an RTRIM on columns in GROUP BY clause. Did anyone notice this? Am I missing something here?
The two selects are returning the same result set in the query below, which should not be the case I believe.
declare #t table(Name varchar(100), Age int)
insert into #t values ('A', 20)
insert into #t values ('B', 30)
insert into #t values ('C', 40)
insert into #t values ('D', 25)
insert into #t values (' A', 21)
insert into #t values ('A ', 32)
insert into #t values (' A ', 28)
select
Name,
count(*) Count
from #t
group by Name
select
rtrim(Name) RtrimmedName,
count(*) Count
from #t
group by rtrim(Name)
Please let me know your thoughts...


It's actually doing the opposite, but the observable effects are the same.
When comparing two strings of unequal length, one of the rules of SQL (the standard, not just SQL Server) is that the shorter string is padded with spaces until it's the same length, and then the comparison is performed.
If you want to avoid being surprised, you'll need to add a non-space character at the end of each string.
In fact, checking the standard text, it appears that there are two options:
4.6 Type conversions and mixing of data types
...
When values of unequal length
are compared, if the collating sequence for the comparison has
the NO PAD attribute and the shorter value is equal to a prefix of
the longer value, then the shorter value is considered less than
the longer value. If the collating sequence for the comparison has
the PAD SPACE attribute, for the purposes of the comparison, the
shorter value is effectively extended to the length of the longer
by concatenation of <space>s on the right.
But all of the SQL Server collations I'm aware of are PAD SPACE.


This is easier to see:
declare #t table (Name varchar(100), Age int)
insert #t values('A', 20),('B', 30),('C', 40),('D ', 25)
,(' A', 21),('A ', 32),(' A ', 28),('D ',10);
select Name, Replace(Name,' ','-'),
count(*) Count
from #t
group by Name
--
NAME COLUMN_1 COUNT
A -A 2
A A- 2
B B 1
C C 1
D D-- 2
Notice the space between A and dot. It chose the 1-space version over the 0-space.
Notice also that the D group chooses the one with 2 trailing spaces over the 4.
So, no it's not performing an RTRIM. It's somewhat of a soft bug however, because it's arbitrarily choosing one of the two columns (the one it came across first) as the result of the GROUP BY which could possibly throw you off if spaces mattered.


I know this is an old thread but this might help someone else. I've used the following for the group by to properly identify differences in trailing spaces:
declare #t table(Name varchar(100), Age int)
insert into #t values ('A', 20)
insert into #t values ('B', 30)
insert into #t values ('C', 40)
insert into #t values ('D', 25)
insert into #t values (' A', 21)
insert into #t values ('A ', 32)
insert into #t values (' A ', 28)
select
Name,
count(*) Count
from #t
group by Name, DATALENGTH(Name)
By adding the additional DATALENGTH on the group by it will also identify the difference in length of the items.

Related

T-sql Concatenate variables for use in an Insert Into while loop

I'm trying to create a SQL statement that allows me to save time when creating a series of CTEs or temp tables that are all the same and only increment in certain places. I could do this with VBA, but I can't figure out if this is possible with SQL
I'm trying to run the following code
DECLARE #N as INT
DECLARE #POINTS as TABLE(ID int Not Null,n varchar(3) Not Null)
DECLARE #TABLENAME varchar(6)
Set #INC = 1
Set #N = 5
--WHILE #INC <= #N
--BEGIN
Declare #N1 INT
Declare #N2 INT
Declare #N3 INT
Declare #N4 INT
Declare #N5 INT
SET #N1=25
SET #N2=50
SET #N3=100
SET #N4=250
SET #N5=500
--END
WHILE #INC <= #N
BEGIN
INSERT INTO #POINTS(ID, n)
VALUES (#INC, CONCAT('#N',#INC))
SET #INC = #INC + 1
END
Select * from #POINTS
These are the results of the code after it runs:
ID n
1 #N1
2 #N2
3 #N3
4 #N4
5 #N5
Is it possible to have the values I set for each #Nx variable be inserted to the n column instead of the concatenated character values? If so, how? I haven't been able to find any results on someone trying to do this.
I can get around this problem by using outside tables, but I want to know if this is possible.

If you only have a small number of values values, you might as well write a single insert statement that will insert all 5 rows to the table:
INSERT INTO #POINTS(ID, n) VALUES
(1, '5'),
(2, '25'),
(3, '50'),
(4, '100'),
(5, '250'),
(6, '500');
If you have too many values to write like this, you can use a cte to generate an inline tally table, and use insert...select from that tally table:
DECLARE #Points as TABLE
(
ID int Not Null,
n varchar(3) Not Null
);
WITH E10(N) AS
(
SELECT 1
FROM (VALUES(0), (1), (2), (3), (4), (5), (6), (7), (8), (9))V(N)
), Tally(N) AS
(
-- Since all the numbers are multiples of 5,
-- there's no point of populating the tally
-- with numbers that aren't multiples of 5....
SELECT ROW_NUMBER() OVER(ORDER BY ##SPID) * 5
FROM E10 As Ten
CROSS JOIN E10 As Hundred
--Need more? Add more cross joins
-- each will multiply the number of rows by 10:
--CROSS JOIN E10 As Thousand
--CROSS JOIN E10 As [Ten Thousand] -- and so on
)
INSERT INTO #Points(ID, n)
SELECT ROW_NUMBER() OVER(ORDER BY N), CAST(N as varchar(3))
FROM Tally
WHERE N IN(5, 25, 50, 100, 250, 500);
SELECT *
FROM #Points;
Results:
ID n
1 5
2 25
3 50
4 100
5 250
6 500
Of course, if you already have a tally table, you don't need the cte...

Removing all the Alphabets from a string using a single SQL Query [duplicate]

I'm currently doing a data conversion project and need to strip all alphabetical characters from a string. Unfortunately I can't create or use a function as we don't own the source machine making the methods I've found from searching for previous posts unusable.
What would be the best way to do this in a select statement? Speed isn't too much of an issue as this will only be running over 30,000 records or so and is a once off statement.

You can do this in a single statement. You're not really creating a statement with 200+ REPLACEs are you?!
update tbl
set S = U.clean
from tbl
cross apply
(
select Substring(tbl.S,v.number,1)
-- this table will cater for strings up to length 2047
from master..spt_values v
where v.type='P' and v.number between 1 and len(tbl.S)
and Substring(tbl.S,v.number,1) like '[0-9]'
order by v.number
for xml path ('')
) U(clean)
Working SQL Fiddle showing this query with sample data
Replicated below for posterity:
create table tbl (ID int identity, S varchar(500))
insert tbl select 'asdlfj;390312hr9fasd9uhf012 3or h239ur ' + char(13) + 'asdfasf'
insert tbl select '123'
insert tbl select ''
insert tbl select null
insert tbl select '123 a 124'
Results
ID S
1 390312990123239
2 123
3 (null)
4 (null)
5 123124

CTE comes for HELP here.
;WITH CTE AS
(
SELECT
[ProductNumber] AS OrigProductNumber
,CAST([ProductNumber] AS VARCHAR(100)) AS [ProductNumber]
FROM [AdventureWorks].[Production].[Product]
UNION ALL
SELECT OrigProductNumber
,CAST(STUFF([ProductNumber], PATINDEX('%[^0-9]%', [ProductNumber]), 1, '') AS VARCHAR(100) ) AS [ProductNumber]
FROM CTE WHERE PATINDEX('%[^0-9]%', [ProductNumber]) > 0
)
SELECT * FROM CTE
WHERE PATINDEX('%[^0-9]%', [ProductNumber]) = 0
OPTION (MAXRECURSION 0)
output:
OrigProductNumber ProductNumber
WB-H098 098
VE-C304-S 304
VE-C304-M 304
VE-C304-L 304
TT-T092 092

RichardTheKiwi's script in a function for use in selects without cross apply,
also added dot because in my case I use it for double and money values within a varchar field
CREATE FUNCTION dbo.ReplaceNonNumericChars (#string VARCHAR(5000))
RETURNS VARCHAR(1000)
AS
BEGIN
SET #string = REPLACE(#string, ',', '.')
SET #string = (SELECT SUBSTRING(#string, v.number, 1)
FROM master..spt_values v
WHERE v.type = 'P'
AND v.number BETWEEN 1 AND LEN(#string)
AND (SUBSTRING(#string, v.number, 1) LIKE '[0-9]'
OR SUBSTRING(#string, v.number, 1) LIKE '[.]')
ORDER BY v.number
FOR
XML PATH('')
)
RETURN #string
END
GO
Thanks RichardTheKiwi +1

Well if you really can't use a function, I suppose you could do something like this:
SELECT REPLACE(REPLACE(REPLACE(LOWER(col),'a',''),'b',''),'c','')
FROM dbo.table...
Obviously it would be a lot uglier than that, since I only handled the first three letters, but it should give the idea.

Select query to remove non-numeric characters

I've got dirty data in a column with variable alpha length. I just want to strip out anything that is not 0-9.
I do not want to run a function or proc. I have a script that is similar that just grabs the numeric value after text, it looks like this:
Update TableName
set ColumntoUpdate=cast(replace(Columnofdirtydata,'Alpha #','') as int)
where Columnofdirtydata like 'Alpha #%'
And ColumntoUpdate is Null
I thought it would work pretty good until I found that some of the data fields I thought would just be in the format Alpha # 12345789 are not.
Examples of data that needs to be stripped
AB ABCDE # 123
ABCDE# 123
AB: ABC# 123
I just want the 123. It is true that all data fields do have the # prior to the number.
I tried substring and PatIndex, but I'm not quite getting the syntax correct or something. Anyone have any advice on the best way to address this?

See this blog post on extracting numbers from strings in SQL Server. Below is a sample using a string in your example:
DECLARE #textval NVARCHAR(30)
SET #textval = 'AB ABCDE # 123'
SELECT LEFT(SUBSTRING(#textval, PATINDEX('%[0-9.-]%', #textval), 8000),
PATINDEX('%[^0-9.-]%', SUBSTRING(#textval, PATINDEX('%[0-9.-]%', #textval), 8000) + 'X') -1)

Here is an elegant solution if your server supports the TRANSLATE function (on sql server it's available on sql server 2017+ and also sql azure).
First, it replaces any non numeric characters with a # character.
Then, it removes all # characters.
You may need to add additional characters that you know may be present in the second parameter of the TRANSLATE call.
select REPLACE(TRANSLATE([Col], 'abcdefghijklmnopqrstuvwxyz+()- ,#+', '##################################'), '#', '')

You can use stuff and patindex.
stuff(Col, 1, patindex('%[0-9]%', Col)-1, '')
SQL Fiddle

This works well for me:
CREATE FUNCTION [dbo].[StripNonNumerics]
(
#Temp varchar(255)
)
RETURNS varchar(255)
AS
Begin
Declare #KeepValues as varchar(50)
Set #KeepValues = '%[^0-9]%'
While PatIndex(#KeepValues, #Temp) > 0
Set #Temp = Stuff(#Temp, PatIndex(#KeepValues, #Temp), 1, '')
Return #Temp
End
Then call the function like so to see the original something next to the sanitized something:
SELECT Something, dbo.StripNonNumerics(Something) FROM TableA

In case if there are some characters possible between digits (e.g. thousands separators), you may try following:
declare #table table (DirtyCol varchar(100))
insert into #table values
('AB ABCDE # 123')
,('ABCDE# 123')
,('AB: ABC# 123')
,('AB#')
,('AB # 1 000 000')
,('AB # 1`234`567')
,('AB # (9)(876)(543)')
;with tally as (select top (100) N=row_number() over (order by ##spid) from sys.all_columns),
data as (
select DirtyCol, Col
from #table
cross apply (
select (select C + ''
from (select N, substring(DirtyCol, N, 1) C from tally where N<=datalength(DirtyCol)) [1]
where C between '0' and '9'
order by N
for xml path(''))
) p (Col)
where p.Col is not NULL
)
select DirtyCol, cast(Col as int) IntCol
from data
Output is:
DirtyCol IntCol
--------------------- -------
AB ABCDE # 123 123
ABCDE# 123 123
AB: ABC# 123 123
AB # 1 000 000 1000000
AB # 1`234`567 1234567
AB # (9)(876)(543) 9876543
For update, add ColToUpdate to select list of the data cte:
;with num as (...),
data as (
select ColToUpdate, /*DirtyCol, */Col
from ...
)
update data
set ColToUpdate = cast(Col as int)

CREATE FUNCTION FN_RemoveNonNumeric (#Input NVARCHAR(512))
RETURNS NVARCHAR(512)
AS
BEGIN
DECLARE #Trimmed NVARCHAR(512)
SELECT #Trimmed = #Input
WHILE PATINDEX('%[^0-9]%', #Trimmed) > 0
SELECT #Trimmed = REPLACE(#Trimmed, SUBSTRING(#Trimmed, PATINDEX('%[^0-9]%', #Trimmed), 1), '')
RETURN #Trimmed
END
GO
SELECT dbo.FN_RemoveNonNumeric('ABCDE# 123')

Pretty late to the party, I found the following which I though worked brilliantialy.. if anyone is still looking
SELECT
(SELECT CAST(CAST((
SELECT SUBSTRING(FieldToStrip, Number, 1)
FROM master..spt_values
WHERE Type='p' AND Number <= LEN(FieldToStrip) AND
SUBSTRING(FieldToStrip, Number, 1) LIKE '[0-9]' FOR XML Path(''))
AS xml) AS varchar(MAX)))
FROM
SourceTable

Here's a version which pulls all digits from a string; i.e. given I'm 35 years old; I was born in 1982. The average family has 2.4 children. this would return 35198224. i.e. it's good where you've got numeric data which may have been formatted as a code (e.g. #123,456,789 / 123-00005), but isn't appropriate if you're looking to pull out specific numbers (i.e. as opposed to digits / just the numeric characters) from the text. Also it only handles digits; so won't return negative signs (-) or periods .).
declare #table table (id bigint not null identity (1,1), data nvarchar(max))
insert #table (data)
values ('hello 123 its 45613 then') --outputs: 12345613
,('1 some other string 98 example 4') --outputs: 1984
,('AB ABCDE # 123') --outputs: 123
,('ABCDE# 123') --outputs: 123
,('AB: ABC# 123') --outputs: 123
; with NonNumerics as (
select id
, data original
--the below line replaces all digits with blanks
, replace(replace(replace(replace(replace(replace(replace(replace(replace(replace(data,'0',''),'1',''),'2',''),'3',''),'4',''),'5',''),'6',''),'7',''),'8',''),'9','') nonNumeric
from #table
)
--each iteration of the below CTE removes another non-numeric character from the original string, putting the result into the numerics column
, Numerics as (
select id
, replace(original, substring(nonNumeric,1,1), '') numerics
, replace(nonNumeric, substring(nonNumeric,1,1), '') charsToreplace
, len(replace(nonNumeric, substring(nonNumeric,1,1), '')) charsRemaining
from NonNumerics
union all
select id
, replace(numerics, substring(charsToreplace,1,1), '') numerics
, replace(charsToreplace, substring(charsToreplace,1,1), '') charsToreplace
, len(replace(charsToreplace, substring(charsToreplace,1,1), '')) charsRemaining
from Numerics
where charsRemaining > 0
)
--we select only those strings with `charsRemaining=0`; i.e. the rows for which all non-numeric characters have been removed; there should be 1 row returned for every 1 row in the original data set.
select * from Numerics where charsRemaining = 0
This code works by removing all the digits (i.e. the characters we want) from a the given strings by replacing them with blanks. Then it goes through the original string (which includes the digits) removing all of the characters that were left (i.e. the non-numeric characters), thus leaving only the digits.
The reason we do this in 2 steps, rather than just removing all non-numeric characters in the first place is there are only 10 digits, whilst there are a huge number of possible characters; so replacing that small list is relatively fast; then gives us a list of those non-numeric characters which actually exist in the string, so we can then replace that small set.
The method makes use of recursive SQL, using common table expressions (CTEs).

To add on to Ken's answer, this handles commas and spaces and parentheses
--Handles parentheses, commas, spaces, hyphens..
declare #table table (c varchar(256))
insert into #table
values
('This is a test 111-222-3344'),
('Some Sample Text (111)-222-3344'),
('Hello there 111222 3344 / How are you?'),
('Hello there 111 222 3344 ? How are you?'),
('Hello there 111 222 3344. How are you?')
select
replace(LEFT(SUBSTRING(replace(replace(replace(replace(replace(c,'(',''),')',''),'-',''),' ',''),',',''), PATINDEX('%[0-9.-]%', replace(replace(replace(replace(replace(c,'(',''),')',''),'-',''),' ',''),',','')), 8000),
PATINDEX('%[^0-9.-]%', SUBSTRING(replace(replace(replace(replace(replace(c,'(',''),')',''),'-',''),' ',''),',',''), PATINDEX('%[0-9.-]%', replace(replace(replace(replace(replace(c,'(',''),')',''),'-',''),' ',''),',','')), 8000) + 'X') -1),'.','')
from #table

Create function fn_GetNumbersOnly(#pn varchar(100))
Returns varchar(max)
AS
BEGIN
Declare #r varchar(max) ='', #len int ,#c char(1), #x int = 0
Select #len = len(#pn)
while #x <= #len
begin
Select #c = SUBSTRING(#pn,#x,1)
if ISNUMERIC(#c) = 1 and #c <> '-'
Select #r = #r + #c
Select #x = #x +1
end
return #r
End

In your case It seems like the # will always be after teh # symbol so using CHARINDEX() with LTRIM() and RTRIM() would probably perform the best. But here is an interesting method of getting rid of ANY non digit. It utilizes a tally table and table of digits to limit which characters are accepted then XML technique to concatenate back to a single string without the non-numeric characters. The neat thing about this technique is it could be expanded to included ANY Allowed characters and strip out anything that is not allowed.
DECLARE #ExampleData AS TABLE (Col VARCHAR(100))
INSERT INTO #ExampleData (Col) VALUES ('AB ABCDE # 123'),('ABCDE# 123'),('AB: ABC# 123')
DECLARE #Digits AS TABLE (D CHAR(1))
INSERT INTO #Digits (D) VALUES ('0'),('1'),('2'),('3'),('4'),('5'),('6'),('7'),('8'),('9')
;WITH cteTally AS (
SELECT
I = ROW_NUMBER() OVER (ORDER BY (SELECT NULL))
FROM
#Digits d10
CROSS APPLY #Digits d100
--add more cross applies to cover longer fields this handles 100
)
SELECT *
FROM
#ExampleData e
OUTER APPLY (
SELECT CleansedPhone = CAST((
SELECT TOP 100
SUBSTRING(e.Col,t.I,1)
FROM
cteTally t
INNER JOIN #Digits d
ON SUBSTRING(e.Col,t.I,1) = d.D
WHERE
I <= LEN(e.Col)
ORDER BY
t.I
FOR XML PATH('')) AS VARCHAR(100))) o

Declare #MainTable table(id int identity(1,1),TextField varchar(100))
INSERT INTO #MainTable (TextField)
VALUES
('6B32E')
declare #i int=1
Declare #originalWord varchar(100)=''
WHile #i<=(Select count(*) from #MainTable)
BEGIN
Select #originalWord=TextField from #MainTable where id=#i
Declare #r varchar(max) ='', #len int ,#c char(1), #x int = 0
Select #len = len(#originalWord)
declare #pn varchar(100)=#originalWord
while #x <= #len
begin
Select #c = SUBSTRING(#pn,#x,1)
if(#c!='')
BEGIN
if ISNUMERIC(#c) = 0 and #c <> '-'
BEGIN
Select #r = cast(#r as varchar) + cast(replace((SELECT ASCII(#c)-64),'-','') as varchar)
end
ELSE
BEGIN
Select #r = #r + #c
END
END
Select #x = #x +1
END
Select #r
Set #i=#i+1
END

I have created a function for this
Create FUNCTION RemoveCharacters (#text varchar(30))
RETURNS VARCHAR(30)
AS
BEGIN
declare #index as int
declare #newtexval as varchar(30)
set #index = (select PATINDEX('%[A-Z.-/?]%', #text))
if (#index =0)
begin
return #text
end
else
begin
set #newtexval = (select STUFF ( #text , #index , 1 , '' ))
return dbo.RemoveCharacters(#newtexval)
end
return 0
END
GO

Here is the answer:
DECLARE #t TABLE (tVal VARCHAR(100))
INSERT INTO #t VALUES('123')
INSERT INTO #t VALUES('123S')
INSERT INTO #t VALUES('A123,123')
INSERT INTO #t VALUES('a123..A123')
;WITH cte (original, tVal, n)
AS
(
SELECT t.tVal AS original,
LOWER(t.tVal) AS tVal,
65 AS n
FROM #t AS t
UNION ALL
SELECT tVal AS original,
CAST(REPLACE(LOWER(tVal), LOWER(CHAR(n)), '') AS VARCHAR(100)),
n + 1
FROM cte
WHERE n <= 90
)
SELECT t1.tVal AS OldVal,
t.tval AS NewVal
FROM (
SELECT original,
tVal,
ROW_NUMBER() OVER(PARTITION BY tVal + original ORDER BY original) AS Sl
FROM cte
WHERE PATINDEX('%[a-z]%', tVal) = 0
) t
INNER JOIN #t t1
ON t.original = t1.tVal
WHERE t.sl = 1

You can create SQL CLR scalar function in order to be able to use regular expressions like replace patterns.
Here you can find example of how to create such function.
Having such function will solve the issue with just the following lines:
SELECT [dbo].[fn_Utils_RegexReplace] ('AB ABCDE # 123', '[^0-9]', '');
SELECT [dbo].[fn_Utils_RegexReplace] ('ABCDE# 123', '[^0-9]', '');
SELECT [dbo].[fn_Utils_RegexReplace] ('AB: ABC# 123', '[^0-9]', '');
More important, you will be able to solve more complex issues as the regular expressions will bring a whole new world of options directly in your T-SQL statements.

Use this:
REPLACE(TRANSLATE(SomeString, REPLACE(TRANSLATE(SomeString, '0123456789', '##########'), '#', ''), REPLICATE('#', LEN(REPLACE(TRANSLATE(SomeString, '0123456789', '##########'), '#', '') + 'x') - 1)), '#', '')
Demo:
DROP TABLE IF EXISTS #MyTempTable;
CREATE TABLE #MyTempTable (SomeString VARCHAR(255));
INSERT INTO #MyTempTable
VALUES ('ssss123ssg99d362sdg')
, ('hey 62q&*^(n43')
, (NULL)
, ('')
, ('hi')
, ('123');
SELECT SomeString
, REPLACE(TRANSLATE(SomeString, REPLACE(TRANSLATE(SomeString, '0123456789', '##########'), '#', ''), REPLICATE('#', LEN(REPLACE(TRANSLATE(SomeString, '0123456789', '##########'), '#', '') + 'x') - 1)), '#', '')
FROM #MyTempTable;
DROP TABLE IF EXISTS #MyTempTable;
Results:
SomeString
(No column name)
ssss123ssg99d362sdg
12399362
hey62q&*^(n43
6243
NULL
NULL
hi
123
123

While the OP wanted to "strip out anything that is not 0-9", the post is also tagged with "substring" and "patindex", and the OP mentioned the concern "not quite getting the syntax correct or something". To address that the requirements note that "all data fields do have the # prior to the number" and to provide an answer that addresses the challenges with substring/patindex, consider the following:
/* A sample select */
;WITH SampleValues AS
( SELECT 'AB ABCDE # 123' [Columnofdirtydata]
UNION ALL SELECT 'AB2: ABC# 123')
SELECT
s.Columnofdirtydata,
f1.pos1,
'['+ f2.substr +']' [InspectOutput]
FROM
SampleValues s
CROSS APPLY (SELECT PATINDEX('%# %',s.Columnofdirtydata) [pos1]) f1
CROSS APPLY (SELECT SUBSTRING(s.Columnofdirtydata, f1.pos1 + LEN('#-'),LEN(s.Columnofdirtydata)) [substr]) f2
/* Using update scenario from OP */
UPDATE t1
SET t1.Columntoupdate = CAST(f2.substr AS INT)
FROM
TableName t1
CROSS APPLY (SELECT PATINDEX('%# %',t1.Columnofdirtydata) [pos1]) f1
CROSS APPLY (SELECT SUBSTRING(t1.Columnofdirtydata, f1.pos1 + LEN('#-'),LEN(t1.Columnofdirtydata)) [substr]) f2
Note that my syntax advice for patindex/substring, is to:
consider using APPLY as a way to temporarily alias results from one function for use as parameters in the next. It's not uncommon to (in ETL, for example) need to parse out parameter/position-based substrings in an updatable column of a staging table. If you need to "debug" and potentially fix some parsing logic, this style will help.
consider using LEN('PatternSample') in your substring logic, to account for reusing this pattern or adjusting it when your source data changes (instead of "+ 1"
SUBSTRING() requires a length parameter, but it can be greater than the length of the string. Therefore, if you are getting "the rest of the string" after the pattern, you can just use "The source length"

DECLARE #STR VARCHAR(400)
DECLARE #specialchars VARCHAR(50) = '%[~,#,#,$,%,&,*,(,),!^?:]%'
SET #STR = '1, 45 4,3 68.00-'
WHILE PATINDEX( #specialchars, #STR ) > 0
---Remove special characters using Replace function
SET #STR = Replace(Replace(REPLACE( #STR, SUBSTRING( #STR, PATINDEX( #specialchars, #STR ), 1 ),''),'-',''), ' ','')
SELECT #STR

SELECT REGEXP_REPLACE( col, '[^[:digit:]]', '' ) AS new_col FROM my_table

TSQL: Remove duplicates based on max(date)

I am searching for a query to select the maximum date (a datetime column) and keep its id and row_id. The desire is to DELETE the rows in the source table.
Source Data
id date row_id(unique)
1 11/11/2009 1
1 12/11/2009 2
1 13/11/2009 3
2 1/11/2009 4
Expected Survivors
1 13/11/2009 3
2 1/11/2009 4
What query would I need to achieve the results I am looking for?

Tested on PostgreSQL:
delete from table where (id, date) not in (select id, max(date) from table group by id);

There are various ways of doing this, but the basic idea is the same:
- Indentify the rows you want to keep
- Compare each row in your table to the ones you want to keep
- Delete any that don't match
DELETE
[source]
FROM
yourTable AS [source]
LEFT JOIN
yourTable AS [keep]
ON [keep].id = [source].id
AND [keep].date = (SELECT MAX(date) FROM yourTable WHERE id = [keep].id)
WHERE
[keep].id IS NULL
DELETE
[yourTable]
FROM
[yourTable]
LEFT JOIN
(
SELECT id, MAX(date) AS date FROM yourTable GROUP BY id
)
AS [keep]
ON [keep].id = [yourTable].id
AND [keep].date = [yourTable].date
WHERE
[keep].id IS NULL
DELETE
[source]
FROM
yourTable AS [source]
WHERE
[source].row_id != (SELECT TOP 1 row_id FROM yourTable WHERE id = [source].id ORDER BY date DESC)
DELETE
[source]
FROM
yourTable AS [source]
WHERE
NOT EXISTS (SELECT id FROM yourTable GROUP BY id HAVING id = [source].id AND MAX(date) != [source].date)

Because you are using SQL Server 2000, you'er not able to use the Row Over technique of setting up a sequence and to identify the top row for each unique id.
So, your proposed technique is to use a datetime column to get the top 1 row to remove duplicates. That might work, but there is a possibility that you might still get duplicates having the same datetime value. But that's easy enough to check for.
First check the assumption that all rows are unique based on the id and date columns:
CREATE TABLE #TestTable (rowid INT IDENTITY(1,1), thisid INT, thisdate DATETIME)
INSERT INTO #TestTable (thisid,thisdate) VALUES (1, '11/11/2009')
INSERT INTO #TestTable (thisid,thisdate) VALUES (1, '12/11/2009')
INSERT INTO #TestTable (thisid,thisdate) VALUES (1, '12/12/2009')
INSERT INTO #TestTable (thisid,thisdate) VALUES (2, '1/11/2009')
INSERT INTO #TestTable (thisid,thisdate) VALUES (2, '1/11/2009')
SELECT COUNT(*) AS thiscount
FROM #TestTable
GROUP BY thisid, thisdate
HAVING COUNT(*) > 1
This example returns a value of 2 - indicating that you will still end up with duplicates even after using the date column to remove duplicates. If you return 0, then you have proven that your proposed technique will work.
When de-duping production data, I think one should take some precautions and test before and after. You should create a table to hold the rows you plan to remove so you can recover them easily if you need to after the delete statement has been executed.
Also, it's a good idea to know beforehand how many rows you plan to remove so you can verify the count before and after - and you can gauge the magnitude of the delete operation. Based on how many rows will be affected, you can plan when to run the operation.
To test before the de-duping process, find the occurrences.
-- Get occurrences of duplicates
SELECT COUNT(*) AS thiscount
FROM
#TestTable
GROUP BY thisid
HAVING COUNT(*) > 1
ORDER BY thisid
That gives you the rows with more than one row with the same id. Capture the rows from this query into a temporary table and then run a query using the SUM to get the total number of rows that are not unique based on your key.
To get the number of rows you plan to delete, you need the count of rows that are duplicate based on your unique key, and the number of distinct rows based on your unique key. You subtract the distinct rows from the count of occurrences. All that is pretty straightforward - so I'll leave you to it.

Try this
declare #t table (id int, dt DATETIME,rowid INT IDENTITY(1,1))
INSERT INTO #t (id,dt) VALUES (1, '11/11/2009')
INSERT INTO #t (id,dt) VALUES (1, '11/12/2009')
INSERT INTO #t (id,dt) VALUES (1, '11/13/2009')
INSERT INTO #t (id,dt) VALUES (2, '11/01/2009')
Query:
delete from #t where rowid not in(
select t.rowid from #t t
inner join(
select MAX(dt)maxdate
from #t
group by id) X
on t.dt = X.maxdate )
select * from #t
Output:
id dt rowid
1 2009-11-13 00:00:00.000 3
2 2009-11-01 00:00:00.000 4

delete from temp where row_id not in (
select t.row_id from temp t
right join
(select id,MAX(dt) as dt from temp group by id) d
on t.dt = d.dt and t.id = d.id)
I have tested this answer..

INSERT INTO #t (id,dt) VALUES (1, '11/11/2009')
INSERT INTO #t (id,dt) VALUES (1, '11/12/2009')
INSERT INTO #t (id,dt) VALUES (1, '11/13/2009')
INSERT INTO #t (id,dt) VALUES (2, '11/01/2009')
select * from #t
;WITH T AS(
select dense_rank() over(partition by id order by dt desc)NO,DT,ID,rowid from #t )
DELETE T WHERE NO>1

T-SQL: Pivot but for semicolon-separated values instead of columns

I've got semicolon-separated values in a column Values in my table:
Values
1;2;3;4;5
I would like to transform it in a procedure to have there values as rows:
Values
1
2
3
4
5
How could I do it in T-SQL?

Solution 1(using xml):
declare #str varchar(20)
declare #xml as xml
set #str= '1;2;3;4;5'
SET #xml = cast(('<x>'+replace(#str,';' ,'</x><x>')+'</x>') as xml)
SELECT col.value('.', 'varchar(10)') as value FROM #xml.nodes('x') as tbl(col)
Solution 2(using recursive cte)
declare #str as varchar(100)
declare #delimiter as char(1)
set #delimiter = ';'
set #str = '1;2;3;4;5' -- original data
set #str = #delimiter + #str + #delimiter
;with num_cte as
(
select 1 as rn
union all
select rn +1 as rn
from num_cte
where rn <= len(#str)
)
, get_delimiter_pos_cte as
(
select
ROW_NUMBER() OVER (ORDER BY rn) as rowid,
rn as delimiterpos
from num_cte
cross apply( select substring(#str,rn,1) AS chars) splittedchars
where chars = #delimiter
)
select substring(#str,a.delimiterpos+1 ,c2.delimiterpos - a.delimiterpos - 1) as Countries
from get_delimiter_pos_cte a
inner join get_delimiter_pos_cte c2 on c2.rowid = a.rowid+1
option(maxrecursion 0)

The thing that struck me as possibly leaving room for an additional answer, or additional improvement was that most of the answers/links given were how to split values like this for a single scalar value as opposed to how to apply that kind of splitting logic for a column of values in a table.
I include both a numbers table solution and an XML solution. The XML solution was inspired by the earlier post priyanka.sarkar. I think that a numbers table solution, using an actual numbers table instead of the CTE as in the below solution is probably the fastest, but the XML approach deserves to be developed upon because it's really nice looking.
So, here goes my attempt.
CREATE PROCEDURE PARSE_DELIMITED_VALUES
AS
WITH FIRST_NUMBERS (N) AS (
SELECT 1 UNION ALL SELECT 1
), SECOND_NUMBERS (N) AS (
SELECT E1.N
FROM FIRST_NUMBERS E1
CROSS JOIN FIRST_NUMBERS E2
), THIRD_NUMBERS (N) AS (
SELECT E1.N
FROM SECOND_NUMBERS E1
CROSS JOIN SECOND_NUMBERS E2
), FOURTH_NUMBERS (N) AS (
SELECT E1.N
FROM THIRD_NUMBERS E1
CROSS JOIN THIRD_NUMBERS E2
), FIFTH_NUMBERS (N) AS (
SELECT E1.N
FROM FOURTH_NUMBERS E1
CROSS JOIN FOURTH_NUMBERS E2
), NUMBERS (N) AS (
SELECT N
FROM NUMBERS
WHERE N <= 8000 /*adjust these as needed to come up with a max number equal to the max character length allowed in the Values column*/
/*or better yet, if you can, just remove this first...numbers... header stuff so long as you create a temp or permanent table that contains the same numbers to work with*/
)
SELECT SUBSTRING(
MYTABLE.Values,
CASE
WHEN NUMBERS.NUMBER = 1 THEN 1
ELSE NUMBERS.NUMBER + 1
END,
CASE CHARINDEX(';', MYTABLE.Values, NUMBERS.NUMBER + 1)
WHEN 0 THEN LEN('^' + MYTABLE.Values + '^') - 2 + 1
ELSE CHARINDEX(';', MYTABLE.Values, NUMBERS.NUMBER + 1)
END
- CASE
WHEN NUMBERS.NUMBER = 1 THEN 1
ELSE NUMBERS.NUMBER + 1
END
) AS PARSED_VALUE
FROM MYTABLE
INNER JOIN NUMBERS
ON NUMBERS.NUMBER <= LEN('^' + MYTABLE.Values + '^') - 2
AND (
NUMBERS.NUMBER = 1
OR SUBSTRING(MYTABLE.Values, NUMBERS.NUMBER, 1) = ';'
)
GO
-- if your values column can contain NULL values I would change the join at the end as follows:
--from INNER JOIN NUMBERS
--to LEFT OUTER JOIN NUMBERS
The above would probably be most performant if the WITH NUMBERS ... CTEs were replaced by a temporary or permanent table containing the same numeric values.
On the other hand the CTE does the job and keeps it more in one place.
CREATE PROCEDURE PARSE_DELIMITED_VALUES
AS
SELECT E.x.value('.', 'VARCHAR(MAX)') AS PARSED_VALUE
FROM (
SELECT CAST('<x>' + REPLACE(Values, ';', '</x><x>') + '</x>' AS XML) my_x
FROM MYTABLE
) TT
CROSS APPLY my_x.nodes('/x') AS E(x)
GO
-- if your values column can contain NULL values I would change the join at the end as follows:
from `CROSS APPLY`
to `OUTER APPLY`

It's not the most elegant approach, but this might be worth a try. It creates a Sql Command as a string, and at the end executes it.
DECLARE #Values VARCHAR(8000)
-- Flatten all values lists into one string
SET #Values = REPLACE(REPLACE((SELECT [Value] FROM [dbo.MyTable] FOR XML PATH('')), '<Value>', ''), '</Value>', ';')
SET #Values = SUBSTRING(#Values, 0, LEN(#Values))
DECLARE #SeparatorIndex INT
SET #SeparatorIndex = (SELECT TOP 1 PATINDEX('%[;]%', #Values))
DECLARE #InsertClause VARCHAR(50)
SET #InsertClause = 'INSERT INTO [dbo.MyTable] VALUES ('
DECLARE #SQL VARCHAR(500)
SET #SQL = #InsertClause + SUBSTRING(#Values, 0, #SeparatorIndex) + '); '
SET #Values = RIGHT(#Values, LEN(#Values) - (#SeparatorIndex - 1))
SET #SQL = REPLACE(#SQL + (SELECT (REPLACE(#Values, ';', '); ' + #InsertClause))) + ')', '; )', '')
EXEC (#SQL)
The command ends up (in Sql Server 2005) as:
INSERT INTO [dbo.MyTable] VALUES (1); INSERT INTO [dbo.MyTable] VALUES (2); INSERT INTO [dbo.MyTable] VALUES (3); INSERT INTO [dbo.MyTable] VALUES (4); INSERT INTO [dbo.MyTable] VALUES (5) ...'

Do you actually mean, "rows," as in, "tuples," (so you can insert the data into another table, one element per row) or do you mean you want the data displayed vertically?
I'd think a string Replace (look up T-SQL's String Functions) would do the trick, no? Depending on the output target, you'd replace ; with CRLF or . You could even use Replace to create dynamic SQL Insert statements that could be executed by the SP to do row inserts (if that was your intent).
For presentation purposes, this is bad practice.
If it is purely for presentation and you are permitted, I'd output everything as XML then XSLT it any way you want. Honestly, I don't remember the last time I operated directly on a recordset. I always output to XML.