Related
I'm currently doing a data conversion project and need to strip all alphabetical characters from a string. Unfortunately I can't create or use a function as we don't own the source machine making the methods I've found from searching for previous posts unusable.
What would be the best way to do this in a select statement? Speed isn't too much of an issue as this will only be running over 30,000 records or so and is a once off statement.
You can do this in a single statement. You're not really creating a statement with 200+ REPLACEs are you?!
update tbl
set S = U.clean
from tbl
cross apply
(
select Substring(tbl.S,v.number,1)
-- this table will cater for strings up to length 2047
from master..spt_values v
where v.type='P' and v.number between 1 and len(tbl.S)
and Substring(tbl.S,v.number,1) like '[0-9]'
order by v.number
for xml path ('')
) U(clean)
Working SQL Fiddle showing this query with sample data
Replicated below for posterity:
create table tbl (ID int identity, S varchar(500))
insert tbl select 'asdlfj;390312hr9fasd9uhf012 3or h239ur ' + char(13) + 'asdfasf'
insert tbl select '123'
insert tbl select ''
insert tbl select null
insert tbl select '123 a 124'
Results
ID S
1 390312990123239
2 123
3 (null)
4 (null)
5 123124
CTE comes for HELP here.
;WITH CTE AS
(
SELECT
[ProductNumber] AS OrigProductNumber
,CAST([ProductNumber] AS VARCHAR(100)) AS [ProductNumber]
FROM [AdventureWorks].[Production].[Product]
UNION ALL
SELECT OrigProductNumber
,CAST(STUFF([ProductNumber], PATINDEX('%[^0-9]%', [ProductNumber]), 1, '') AS VARCHAR(100) ) AS [ProductNumber]
FROM CTE WHERE PATINDEX('%[^0-9]%', [ProductNumber]) > 0
)
SELECT * FROM CTE
WHERE PATINDEX('%[^0-9]%', [ProductNumber]) = 0
OPTION (MAXRECURSION 0)
output:
OrigProductNumber ProductNumber
WB-H098 098
VE-C304-S 304
VE-C304-M 304
VE-C304-L 304
TT-T092 092
RichardTheKiwi's script in a function for use in selects without cross apply,
also added dot because in my case I use it for double and money values within a varchar field
CREATE FUNCTION dbo.ReplaceNonNumericChars (#string VARCHAR(5000))
RETURNS VARCHAR(1000)
AS
BEGIN
SET #string = REPLACE(#string, ',', '.')
SET #string = (SELECT SUBSTRING(#string, v.number, 1)
FROM master..spt_values v
WHERE v.type = 'P'
AND v.number BETWEEN 1 AND LEN(#string)
AND (SUBSTRING(#string, v.number, 1) LIKE '[0-9]'
OR SUBSTRING(#string, v.number, 1) LIKE '[.]')
ORDER BY v.number
FOR
XML PATH('')
)
RETURN #string
END
GO
Thanks RichardTheKiwi +1
Well if you really can't use a function, I suppose you could do something like this:
SELECT REPLACE(REPLACE(REPLACE(LOWER(col),'a',''),'b',''),'c','')
FROM dbo.table...
Obviously it would be a lot uglier than that, since I only handled the first three letters, but it should give the idea.
I have a database with several tables and I need to search every varchar column across the database, for columns that simultaneously contain lower and upper case characters.
To clarify:
If one column contains helLo the name of the column should be returned by the query, but if the column values only contain either hello or HELLO then the name of the column is not returned.
Let's exclude all UPPER and all LOWER, the rest will be MIXED.
SELECT someColumn
FROM someTable
WHERE someColumn <> UPPER(someColumn) AND someColumn <> LOWER(someColumn)
EDIT:
As suggested in comments and described in detail here I need to specify a case-sensitive collation.
SELECT someColumn
FROM someTable
WHERE someColumn <> UPPER(someColumn) AND
someColumn <> LOWER(someColumn)
Collate SQL_Latin1_General_CP1_CS_AS
It sounds like you are after a case sensitive search, so you'd need to use a case sensitive collation for there WHERE clause.
e.g. if your collation is currently SQL_Latin1_General_CP1_CI_AS which is case insensitive, you can write a case sensitive query using:
SELECT SomeColumn
FROM dbo.SomeTable
WHERE SomeField LIKE '%helLo%' COLLATE SQL_Latin1_General_CP1_CS_AS
Here, COLLATE SQL_Latin1_General_CP1_CS_AS tells it to use a case sensitive collation to perform the filtering.
I think I understand that you want to find any varchar column with mixed case data within it?
If so, you can achieve this with a cursor looking at your column types, which then executes some dynamic SQL on the varchar columns it finds to check the data for mixed case values.
I thoroughly recommend doing this on a non-production server using a copy of your database, not least because you need to create a table to deposit your findings into:
create table VarcharColumns (TableName nvarchar(max), ColumnName nvarchar(max))
declare #sql nvarchar(max)
declare my_cursor cursor local static read_only forward_only
for
select 'insert into VarcharColumns select t,c from(select ''' + s.name + '.' + tb.name + ''' t, ''' + c.name + ''' c from ' + s.name + '.' + tb.name + ' where ' + c.name + ' like ''%[abcdefghijklmnopqrstuvwxyz]%'' COLLATE SQL_Latin1_General_CP1_CS_AS and ' + c.name + ' like ''%[ABCDEFGHIJKLMNOPQRSTUVWXYZ]%'' COLLATE SQL_Latin1_General_CP1_CS_AS having count(1) > 0) a' as s
from sys.columns c
inner join sys.types t
on(c.system_type_id = t.system_type_id
and t.name = 'varchar'
)
inner join sys.tables tb
on(c.object_id = tb.object_id)
inner join sys.schemas s
on(tb.schema_id = s.schema_id)
open my_cursor
fetch next from my_cursor into #sql
while ##fetch_status = 0
begin
print #sql
exec(#sql)
fetch next from my_cursor into #sql
end
close my_cursor
deallocate my_cursor
select * from VarcharColumns
You can check the hash compared to its upper and lower values... here is a simple test:
declare #test varchar(256)
set #test = 'MIX' -- Try changing this to a mix case, and then all lower case
select case
when hashbytes('SHA1',#test) <> hashbytes('SHA1',upper(#test)) and hashbytes('SHA1',#test) <> hashbytes('SHA1',lower(#test))
then 'MixedCase'
else 'Not Mixed Case'
end
So using this in a table... you can do something like this
create table #tempT (SomeColumn varchar(256))
insert into #tempT (SomeColumn) values ('some thing lower'),('SOME THING UPPER'),('Some Thing Mixed')
SELECT SomeColumn
FROM #tempT
WHERE 1 = case
when hashbytes('SHA1',SomeColumn) <> hashbytes('SHA1',upper(SomeColumn)) and hashbytes('SHA1',SomeColumn) <> hashbytes('SHA1',lower(SomeColumn)) then 1
else 0
end
I can someone help me turn this sql into a stored proc..
select * from voilets
where cfrw = 'F16'
UNION
(select *
from voilets
where cfrw in ('B05','B12','R02','F01','F16','F17','U11','U03','U04','U21'))
ORDER BY DSCA
Where 'F16 is a variable called #default
and
'B05','B12','R02','F01','F16','F17','U11','U03','U04','U21' is an array of #voilets
This is not working for me:
#sCarrierSelect varchar(max)
AS
BEGIN
declare #SQL nvarchar(4000)
set #SQL = '
select * from voilets
where t_cfrw = ' + #default + '
UNION
(select *
from carriers
where t_cfrw in (' + #voilets+'))
ORDER BY T_DSCA
'
print #SQL
exec sp_executesql #SQL
END
IF you SQL Server IS >=2008 then:
USE tempdb;
GO
CREATE TABLE voilets
(cfrw char(3), DSCA int)
go
INSERT INTO voilets VALUES ('R02', 2)
INSERT INTO voilets VALUES ('F16', 5)
INSERT INTO voilets VALUES ('F16', 4)
INSERT INTO voilets VALUES ('X77', 9)
go
CREATE TYPE myType AS TABLE (id CHAR(3));
GO
CREATE PROCEDURE usp_myProc
#default char(3),
#voiletsTVP myType READONLY
AS
select * from voilets
where cfrw = #default
UNION
(select *
from voilets
where cfrw in (SELECT * FROM #voiletsTVP))
ORDER BY DSCA
GO
-------------------------
DECLARE #default char(3)
SET #default='F16'
DECLARE #voiletsTVP AS myType;
INSERT INTO #voiletsTVP SELECT * FROM (VALUES ('B05'),('B12'),('R02'),('F01'),('F16'),('F17'),('U11'),('U03'),('U04'),('U21')) q(x)
EXEC usp_myProc #default,#voiletsTVP
GO
Result-set:
cfrw DSCA
R02 2
F16 4
F16 5
Performing that safely in a sproc is actually quite tricky; there are a few common approaches:
use a udf to split a string on a token - google for "split udf" (there will be many), and join on the results
use a table valued parameter
Personally, I rarely use sprocs these days; I'd use dapper:
List<string> foo = ...
var items = conn.Query<SomeType>(
"select * from [table] where colName in #foo", new { foo }).ToList();
Most LINQ providers and ORMs will have options here too, involving Contains etc.
You can learn about Passing Arrays in SQL Parameters using XML Data Type in SQL Server 2005
See sample:
/* for this xml:
<list>
<item>42</item>
<item>73</item>
<item>2007</item>
</list>
*/
CREATE FUNCTION [lm].[SplitList]
(
#list AS XML
)
RETURNS TABLE
AS
RETURN
(
SELECT tempTable.item.value('.', 'VARCHAR(MAX)') AS Item
FROM #list.nodes('list/item') tempTable(item)
);
Why not use a sql CLR function to split your values, passing those into your procedure. Here is a very good and fast split string implementation: CLR Split String. If you can't use sql clr, then look online for 'sql split string'. Whichever you use you put the result of that work into a temporary table and join that to your main table.
-- Given a CSV string like this:
declare #roles varchar(800)
select #roles = 'Pub,RegUser,ServiceAdmin'
-- Question: How to get roles into a table view like this:
select 'Pub'
union
select 'RegUser'
union
select 'ServiceAdmin'
After posting this, I started playing with some dynamic SQL. This seems to work, but seems like there might be some security risks by using dynamic SQL - thoughts on this?
declare #rolesSql varchar(800)
select #rolesSql = 'select ''' + replace(#roles, ',', ''' union select ''') + ''''
exec(#rolesSql)
If you're working with SQL Server compatibility level 130 then the STRING_SPLIT function is now the most succinct method available.
Reference link: https://msdn.microsoft.com/en-gb/library/mt684588.aspx
Usage:
SELECT * FROM string_split('Pub,RegUser,ServiceAdmin',',')
RESULT:
value
-----------
Pub
RegUser
ServiceAdmin
See my answer from here
But basically you would:
Create this function in your DB:
CREATE FUNCTION dbo.Split(#origString varchar(max), #Delimiter char(1))
returns #temptable TABLE (items varchar(max))
as
begin
declare #idx int
declare #split varchar(max)
select #idx = 1
if len(#origString )<1 or #origString is null return
while #idx!= 0
begin
set #idx = charindex(#Delimiter,#origString)
if #idx!=0
set #split= left(#origString,#idx - 1)
else
set #split= #origString
if(len(#split)>0)
insert into #temptable(Items) values(#split)
set #origString= right(#origString,len(#origString) - #idx)
if len(#origString) = 0 break
end
return
end
and then call the function and pass in the string you want to split.
Select * From dbo.Split(#roles, ',')
Here's a thorough discussion of your options:
Arrays and Lists in SQL Server
What i do in this case is just using some string replace to convert it to json and open the json like a table. May not be suitable for every use case but it is very simple to get running and works with strings and files. With files you just need to watch your line break character, mostly i find it to be "Char(13)+Char(10)"
declare #myCSV nvarchar(MAX)= N'"Id";"Duration";"PosX";"PosY"
"•P001";223;-30;35
"•P002";248;-28;35
"•P003";235;-26;35'
--CSV to JSON
--convert to json by replacing some stuff
declare #myJson nvarchar(MAX)= '[['+ replace(#myCSV, Char(13)+Char(10), '],[' ) +']]'
set #myJson = replace(#myJson, ';',',') -- Optional: ensure coma delimiters for json if the current delimiter differs
-- set #myJson = replace(#myJson, ',,',',null,') -- Optional: empty in between
-- set #myJson = replace(#myJson, ',]',',null]') -- Optional: empty before linebreak
SELECT
ROW_NUMBER() OVER (ORDER BY (SELECT 0))-1 AS LineNumber, *
FROM OPENJSON( #myJson )
with (
col0 varchar(255) '$[0]'
,col1 varchar(255) '$[1]'
,col2 varchar(255) '$[2]'
,col3 varchar(255) '$[3]'
,col4 varchar(255) '$[4]'
,col5 varchar(255) '$[5]'
,col6 varchar(255) '$[6]'
,col7 varchar(255) '$[7]'
,col8 varchar(255) '$[8]'
,col9 varchar(255) '$[9]'
--any name column count is possible
) csv
order by (SELECT 0) OFFSET 1 ROWS --hide header row
Using SQL Server's built in XML parsing is also an option. Of course, this glosses over all the nuances of an RFC-4180 compliant CSV.
-- Given a CSV string like this:
declare #roles varchar(800)
select #roles = 'Pub,RegUser,ServiceAdmin'
-- Here's the XML way
select split.csv.value('.', 'varchar(100)') as value
from (
select cast('<x>' + replace(#roles, ',', '</x><x>') + '</x>' as xml) as data
) as csv
cross apply data.nodes('/x') as split(csv)
If you are using SQL 2016+, using string_split is better, but this is a common way to do this prior to SQL 2016.
Using BULK INSERT you can import a csv file into your sql table -
http://blog.sqlauthority.com/2008/02/06/sql-server-import-csv-file-into-sql-server-using-bulk-insert-load-comma-delimited-file-into-sql-server/
Even the accepted answer is working fine. but I got this function much faster even for thousands of record. create below function and use.
IF EXISTS (
SELECT 1
FROM Information_schema.Routines
WHERE Specific_schema = 'dbo'
AND specific_name = 'FN_CSVToStringListTable'
AND Routine_Type = 'FUNCTION'
)
BEGIN
DROP FUNCTION [dbo].[FN_CSVToStringListTable]
END
GO
CREATE FUNCTION [dbo].[FN_CSVToStringListTable] (#InStr VARCHAR(MAX))
RETURNS #TempTab TABLE (Id NVARCHAR(max) NOT NULL)
AS
BEGIN
;-- Ensure input ends with comma
SET #InStr = REPLACE(#InStr + ',', ',,', ',')
DECLARE #SP INT
DECLARE #VALUE VARCHAR(1000)
WHILE PATINDEX('%,%', #INSTR) <> 0
BEGIN
SELECT #SP = PATINDEX('%,%', #INSTR)
SELECT #VALUE = LEFT(#INSTR, #SP - 1)
SELECT #INSTR = STUFF(#INSTR, 1, #SP, '')
INSERT INTO #TempTab (Id)
VALUES (#VALUE)
END
RETURN
END
GO
---Test like this.
declare #v as NVARCHAR(max) = N'asdf,,as34df,234df,fs,,34v,5fghwer,56gfg,';
SELECT Id FROM dbo.FN_CSVToStringListTable(#v)
I was about you use the solution mentioned in the accepted answer, but doing more research led me to use Table Value Types:
These are far more efficient and you don't need a TVF (Table valued function) just to create a table from csv. You can use it directly in your scripts or pass that to a stored procedure as a Table Value Parameter. The Type can be created as :
CREATE TYPE [UniqueIdentifiers] AS TABLE(
[Id] [varchar](20) NOT NULL
)
This question already has answers here:
How to concatenate text from multiple rows into a single text string in SQL Server
(47 answers)
Closed 7 years ago.
If I issue SELECT username FROM Users I get this result:
username
--------
Paul
John
Mary
but what I really need is one row with all the values separated by comma, like this:
Paul, John, Mary
How do I do this?
select
distinct
stuff((
select ',' + u.username
from users u
where u.username = username
order by u.username
for xml path('')
),1,1,'') as userlist
from users
group by username
had a typo before, the above works
This should work for you. Tested all the way back to SQL 2000.
create table #user (username varchar(25))
insert into #user (username) values ('Paul')
insert into #user (username) values ('John')
insert into #user (username) values ('Mary')
declare #tmp varchar(250)
SET #tmp = ''
select #tmp = #tmp + username + ', ' from #user
select SUBSTRING(#tmp, 0, LEN(#tmp))
good review of several approaches:
http://blogs.msmvps.com/robfarley/2007/04/07/coalesce-is-not-the-answer-to-string-concatentation-in-t-sql/
Article copy -
Coalesce is not the answer to string concatentation in T-SQL I've seen many posts over the years about using the COALESCE function to get string concatenation working in T-SQL. This is one of the examples here (borrowed from Readifarian Marc Ridey).
DECLARE #categories varchar(200)
SET #categories = NULL
SELECT #categories = COALESCE(#categories + ',','') + Name
FROM Production.ProductCategory
SELECT #categories
This query can be quite effective, but care needs to be taken, and the use of COALESCE should be properly understood. COALESCE is the version of ISNULL which can take more than two parameters. It returns the first thing in the list of parameters which is not null. So really it has nothing to do with concatenation, and the following piece of code is exactly the same - without using COALESCE:
DECLARE #categories varchar(200)
SET #categories = ''
SELECT #categories = #categories + ',' + Name
FROM Production.ProductCategory
SELECT #categories
But the unordered nature of databases makes this unreliable. The whole reason why T-SQL doesn't (yet) have a concatenate function is that this is an aggregate for which the order of elements is important. Using this variable-assignment method of string concatenation, you may actually find that the answer that gets returned doesn't have all the values in it, particularly if you want the substrings put in a particular order. Consider the following, which on my machine only returns ',Accessories', when I wanted it to return ',Bikes,Clothing,Components,Accessories':
DECLARE #categories varchar(200)
SET #categories = NULL
SELECT #categories = COALESCE(#categories + ',','') + Name
FROM Production.ProductCategory
ORDER BY LEN(Name)
SELECT #categories
Far better is to use a method which does take order into consideration, and which has been included in SQL2005 specifically for the purpose of string concatenation - FOR XML PATH('')
SELECT ',' + Name
FROM Production.ProductCategory
ORDER BY LEN(Name)
FOR XML PATH('')
In the post I made recently comparing GROUP BY and DISTINCT when using subqueries, I demonstrated the use of FOR XML PATH(''). Have a look at this and you'll see how it works in a subquery. The 'STUFF' function is only there to remove the leading comma.
USE tempdb;
GO
CREATE TABLE t1 (id INT, NAME VARCHAR(MAX));
INSERT t1 values (1,'Jamie');
INSERT t1 values (1,'Joe');
INSERT t1 values (1,'John');
INSERT t1 values (2,'Sai');
INSERT t1 values (2,'Sam');
GO
select
id,
stuff((
select ',' + t.[name]
from t1 t
where t.id = t1.id
order by t.[name]
for xml path('')
),1,1,'') as name_csv
from t1
group by id
;
FOR XML PATH is one of the only situations in which you can use ORDER BY in a subquery. The other is TOP. And when you use an unnamed column and FOR XML PATH(''), you will get a straight concatenation, with no XML tags. This does mean that the strings will be HTML Encoded, so if you're concatenating strings which may have the < character (etc), then you should maybe fix that up afterwards, but either way, this is still the best way of concatenating strings in SQL Server 2005.
building on mwigdahls answer. if you also need to do grouping here is how to get it to look like
group, csv
'group1', 'paul, john'
'group2', 'mary'
--drop table #user
create table #user (groupName varchar(25), username varchar(25))
insert into #user (groupname, username) values ('apostles', 'Paul')
insert into #user (groupname, username) values ('apostles', 'John')
insert into #user (groupname, username) values ('family','Mary')
select
g1.groupname
, stuff((
select ', ' + g.username
from #user g
where g.groupName = g1.groupname
order by g.username
for xml path('')
),1,2,'') as name_csv
from #user g1
group by g1.groupname
You can use this query to do the above task:
DECLARE #test NVARCHAR(max)
SELECT #test = COALESCE(#test + ',', '') + field2 FROM #test
SELECT field2 = #test
For detail and step by step explanation visit the following link
http://oops-solution.blogspot.com/2011/11/sql-server-convert-table-column-data.html
DECLARE #EmployeeList varchar(100)
SELECT #EmployeeList = COALESCE(#EmployeeList + ', ', '') +
CAST(Emp_UniqueID AS varchar(5))
FROM SalesCallsEmployees
WHERE SalCal_UniqueID = 1
SELECT #EmployeeList
source:
http://www.sqlteam.com/article/using-coalesce-to-build-comma-delimited-string
In SQLite this is simpler. I think there are similar implementations for MySQL, MSSql and Orable
CREATE TABLE Beatles (id integer, name string );
INSERT INTO Beatles VALUES (1, "Paul");
INSERT INTO Beatles VALUES (2, "John");
INSERT INTO Beatles VALUES (3, "Ringo");
INSERT INTO Beatles VALUES (4, "George");
SELECT GROUP_CONCAT(name, ',') FROM Beatles;
you can use stuff() to convert rows as comma separated values
select
EmployeeID,
stuff((
SELECT ',' + FPProjectMaster.GroupName
FROM FPProjectInfo AS t INNER JOIN
FPProjectMaster ON t.ProjectID = FPProjectMaster.ProjectID
WHERE (t.EmployeeID = FPProjectInfo.EmployeeID)
And t.STatusID = 1
ORDER BY t.ProjectID
for xml path('')
),1,1,'') as name_csv
from FPProjectInfo
group by EmployeeID;
Thanks #AlexKuznetsov for the reference to get this answer.
A clean and flexible solution in MS SQL Server 2005/2008 is to create a CLR Agregate function.
You'll find quite a few articles (with code) on google.
It looks like this article walks you through the whole process using C#.
If you're executing this through PHP, what about this?
$hQuery = mysql_query("SELECT * FROM users");
while($hRow = mysql_fetch_array($hQuery)) {
$hOut .= $hRow['username'] . ", ";
}
$hOut = substr($hOut, 0, strlen($hOut) - 1);
echo $hOut;