I'm trying to edit an existing process we have that is called from SQL Server 2008r2 via xp_cmdshell. It replaces/removes certain characters in a csv file before it is then imported. I've managed to add in removing fullstops but am having trouble getting it to remove backticks. Every way I have tried to escape it I get the error sed: -e expression #1, char 97: Unterminated `s' command.
In SQL:
DECLARE
#FileLocation VARCHAR(255) = '\\FILELOCATION\'
,#ArchiveLocation VARCHAR(255) = '\\FILELOCATION\Archive\'
,#FileName VARCHAR(50) = 'DATE_FILENAME.csv'
,#UtilitiesLocation VARCHAR(100) = '\\NETWORKLOCATION\UTILITIES\'
,#Command VARCHAR(2000)
DECLARE #SedCommand VARCHAR(100) =
's/£//g;' -- remove pound signs
+ 's/\.//g;' -- remove full stop
+ 's/\`//g;' -- remove grave accent (backtick)
+ 's/\"\([A-Za-z- '']*\),\([A-Za-z- '']*\)\"/\1|\2/g;' -- "Last, First" replaced by Last|First
+ 's/\"\([0-9.-]*\),\([0-9.-]*\)\"/\1\2/g;' -- remove thousands separators in numbers
SET #Command =
'copy '
+ #FileLocation + #FileName + ' '
+ #ArchiveLocation + 'QN' + #FileName
PRINT #Command -- copy \\FILELOCATION\DATE_FILENAME.csv \\FILELOCATION\Archive\QNDATE_FILENAME.csv
EXEC xp_cmdshell #Command
SET #Command =
#UtilitiesLocation + 'sed --text -e '
+ '"' + #SedCommand + '" '
+ #ArchiveLocation + 'QN' + #FileName
+ ' > '
+ #FileLocation + #FileName
PRINT #Command -- \\NETWORKLOCATION\UTILITIES\sed --text -e "s/£//g;s/\.//g;s/\`//g;s/\"\([A-Za-z- ']*\),\([A-Za-z- ']*\)\"/\1|\2/g;s/\"\([0-9.-]*\),\([0-9.-]*\)" \\FILELOCATION\Archive\QNDATE_FILENAME.csv > \\FILELOCATION\DATE_FILENAME.csv
EXEC xp_cmdshell #Command
Is anyone able to help please? I've tried using 3 backslashes, ^ as the escape character, putting the backtick in double quotes but can't get anything to work.
Related
Currently in an SP, we have columns created in a temp table that are being set to 0 by default using the first SET statement below but I am now needing the columns to be set to an empty string value and not null:
SET #columns = #columns + ',' + CHAR(13) + CHAR(10) + ' [' + cast((#i+1) as nvarchar) + '] int Default 0' SET #i = #i + 1;
I tried using this solution:
SET #columns = #columns + ',' + CHAR(13) + CHAR(10) + ' [' + cast((#i+1) as nvarchar) + '] nvarchar Default []' SET #i = #i+1;
but when I create the ALTER statement:
Alter TABLE #Survivaltmp Add ' + STUFF(#columns, 1, 1, '') + ''
that adds the column with stuffing this variable I am getting this error:
An object or column name is missing. For Select Into statements verify each column has a name.
My question is, is there any way to add an empty string value to a column? Any help/direction would be appreciated. Thanks.
Strings are enclosed in single quotes, not brackets, they're for identifiers.
Try:
... + '] nvarchar Default ''''' ...
I am using postgres to create several INSERT scripts to insert data from one database to another.
The statement is giving me an error for invalid syntax for integer.
Here is the script I am trying to run
select 'INSERT INTO public."HoldingMasters"(
"AccountID", "AssetID", "CreatedDate", "DateAcquired", "GainsLongTerm", "LotNumber", "Managed", "ModifiedDate", "Sweep", "UID", "Units", "UnitsPledged", "IsModified", "ModifiedCount")
SELECT ' + CAST("AccountID" as integer) + ',' + CAST("AssetID" as integer) + ',' + "CreatedDate" + ',' + "DateAcquired" + ',' + "GainsLongTerm" + ',' + CAST("LotNumber" as integer) + ',' + "Managed" + ',' + "ModifedDate" + ',' + "Sweep" + ',' + CAST("UID" as integer) + ',' + "Units" + ',' + ',' + "UnitsPledged" + ',' + "IsModifed" + ',' + CAST("ModifiedCount" as integer) +
'WHERE NOT EXISTS(SELECT "UID" from "HoldingMasters" where "UID" = ' + CAST("UID" as integer) +')'
as script from "HoldingMasters";
And this gives me the error:
ERROR: invalid input syntax for integer: "INSERT INTO public."HoldingMasters"(
"AccountID", "AssetID", "CreatedDate", "DateAcquired", "GainsLongTerm", "LotNumber", "Managed", "ModifiedDate", "Sweep", "UID", "Units", "UnitsPledged", "IsModified", "ModifiedCount")
SELECT "
LINE 1: select 'INSERT INTO public."HoldingMasters"(
^
SQL state: 22P02
Character: 8
Several of these columns are integers and I am not sure what syntax to use to get them to cast properly in the INSERT statement within my scripts.
I have tried:
SUM(NULLIF("AccountID", '')::integer)
"AccountID"::integer
to_number - does not seem to work on my machine
I have run this INSERT statement by itself without any issue, it just seems to have troubles when within the "as script" generator. I have not seen anyone try this syntax within a script generator.
Any help would be appreciated.
I am attempting to semi automate creation of my databases
As part of this I want to add extended properties of column descriptions.
When I try to run sp_sqlexec in my script ( or even just Exec(#mystring) I get an error. However, if while debugging, I copy the dynamic sql string from the watch window and then run sp_sqlexec on the copied string in a seperate window I get no errors and the extended properties are added correctly.
The following script demonstrates the problem:
--Create a table to apply column descriptions to
Create table dbo.table1 (id int, name nvarchar(20));
--Create the table that contains our column descriptions
Create table dbo.column_descs_table (schemaname nvarchar(20), tablename nvarchar(20), columnname nvarchar(20), column_description nvarchar(20))
Insert into column_descs_table (schemaname, tablename, columnname, column_description)
values ('dbo', 'table1', 'id', 'the id column'), ('dbo' , 'table1', 'name', 'the name column');
--Dynamic sql string varaible to hold the commands
Declare #dyn_sql nvarchar(max);
Set #dyn_sql = 'N'''; --Set to opening quote
--now create the string containing commands to add column escriptions
SELECT #dyn_sql = #dyn_sql + N' EXEC sp_addextendedproperty ''''Col Desc'''', ''''' + column_description + N''''', ''''SCHEMA'''', ' + schemaname + N', ''''TABLE'''', ' + tablename + N', ''''COLUMN'''', ' + columnname + N' ;'
FROM dbo.column_descs_table
Set #dyn_sql = #dyn_sql + ''''; --add the closing quote
Print #dyn_sql --If I copy the contents of #dyn_sql here and run seperately it works OK
Exec sp_sqlexec #dyn_sql -- this line causes error
The error I get is
Msg 102, Level 15, State 1, Line 1
Incorrect syntax near ' EXEC sp_addextendedproperty 'Col Desc', 'the id column', 'SCHEMA', dbo, 'TABLE', table1, 'COLUMN', id ; EXEC sp_addextendedprope'.
Yet if I step through the code and copy the contents of #dyn_sql then paste this as follows:
Exec sp_sqlexec N' EXEC sp_addextendedproperty ''Col Desc'', ''the id column'', ''SCHEMA'', dbo, ''TABLE'', table1, ''COLUMN'', id ; EXEC sp_addextendedproperty ''Col Desc'', ''the name column'', ''SCHEMA'', dbo, ''TABLE'', table1, ''COLUMN'', name ;'
Then the above works fine and the column descriptions are added as expected.
Any help on this specific copying problem is much appreciated. I do understand the security issues with dynamic sql ( this script will be removed from the database once my setup is complete)
Thanks in advance
Jude
It looks like it's because your leading N is included within the string to execute; you don't need it at all. In other words, you are ending up with something like this:
exec sp_execsql 'N'' exec sp_addextendedproperty /* etc. */ '''
But it should be like this:
exec sp_execsql N'exec sp_addextendedproperty /* etc. */ '
But why are you even using dynamic SQL here? All values passed to sp_addextendedproperty can be passed as parameters so there is no obvious reason to use dynamic SQL, unless you've simplified something for the question.
Finally, you should be using sp_executesql, it's the preferred way to execute dynamic SQL.
I believe that I have resolved my string copying problem. SQL was detecting double quotes in by concatenated string as empty strings and removing them. A simple example showing the problem and my solution is below:
--Example to Select 'simple string' and then 'concat string' into results sets
DECLARE
#Simplestring nvarchar( max ) = '' ,
#Concatstring nvarchar( max ) = '' ,
#Stringvar nvarchar( 10 ) = 'string';
--The double quotes in next line are the quotemark we want plus a quotemark acting
--as an escape character
--#simplestring will be set to 'Select 'simple string' '
SET #Simplestring = 'Select ''simple string'' ';
--Similarly we need #concatstring to be set to 'Select 'Concat string' '
SET #Concatstring = 'Select '' concat' + #Stringvar + ''; -- this wont work the last
--double quote will be removed
--Add a character that cannot appear in any othe part of the concatenation - I've used *
SET #Concatstring = 'Select '' Concat ' + #Stringvar + '*';
--Now replace the * with a quote mark
SET #Concatstring = REPLACE( #Concatstring , '*' , '''' ); -- This will work
EXEC sp_executesql #Simplestring;
EXEC sp_executesql #Concatstring;
There may be a simpler solution than mine.
Many thanks for the advice on using sp_executesql. I am working on changing my code to use this ( with variables passed in as parametrs).
Jude
What I am trying to do is export a Tsql query to a csv file. Simple enough, however I need to be able to specify which fields are wrapped in quotes "". I can get my query to export with all the feilds wrapped.
"SCHEN001","Joe Bloggs Inc","1","1","1","1","1","1","13","6","Mr John Smith"
What I would like to export is
"SCHEN001","Joe Bloggs Inc",1,1,1,1,1,1,13,6,"Mr John Smith"
Is this possible using Tsql?
Any ideas would be greatly appreciated
Take a look at the bcp.exe utility. It is ment for bulk operations and you can specify templates for exports etc.
A link that seems reasonable: http://www.simple-talk.com/sql/database-administration/creating-csv-files-using-bcp-and-stored-procedures/
Another approach is to use SQL Server Integration Services (if you have MS SQL Server Standard or Enterprise edition)
or, alternatively, you can copy grid results into Excel, and export CSV from there :-)
Try to use this script.
Set variable #TblName to the name of your table.
The script uses information_schema.columns
to get the datatypes for every column in selected table.
DECLARE #TblName varchar(128)
DECLARE #WhereClause varchar(255)
DECLARE #cmd varchar(7000)
SET #TblName = '<YOUR TABLENAME>' --TABLENAME
SET #cmd = ''
create table #tableDef (id int identity (1,1), ColType int, ColName varchar(128))
--Fetch table information
insert #tableDef (ColType, ColName)
select case when DATA_TYPE like '%char%' then 1
when DATA_TYPE like '%datetime%' then 2
else 0 end ,
COLUMN_NAME
from information_schema.columns
where TABLE_NAME = #TblName
order by ORDINAL_POSITION
SELECT #cmd = #cmd
+ ' CASE WHEN ' + ColName + ' IS NULL '
+ ' THEN ''NULL'' '
+ ' ELSE '
+ case ColType
when 1 then ''''''''' + ' + ColName + ' + '''''''''
when 2 then ''''''''' + ' + 'CONVERT(VARCHAR(20),' + ColName + ')' + ' + '''''''''
else 'CONVERT(VARCHAR(20),' + ColName + ')' end
+ ' END + '','' + '
from #tableDef
order by id
select #cmd = 'SELECT ' + left(#cmd,len(#cmd)-8) + '+'''' FROM ' + #tblName
exec (#cmd)
drop table #tableDef
I'm using SELECT ... FOR XML to generate XML and it creates exactly what I want to see - in SSMS. But...
The problem I have is, the exported XML file has the entire XML body on a single line. It's valid XML, but how can I format the output? (one element per line, terminated with \r\n and preferably indented) The file should be human readable in notepad.
Also, is there a better way to tack on the XML header?
Of course I could write a CLR stored proc to do the export, seems like overkill though. (Why is this even hard?)
ALTER PROCEDURE [dbo].[ExportConfigTablesToXML]
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
--EXEC sp_configure 'show advanced options', 1
--RECONFIGURE
--EXEC sp_configure 'xp_cmdshell', 1
--RECONFIGURE
--EXEC sp_configure 'show advanced options', 0
--RECONFIGURE
DECLARE #FileName VARCHAR(50)
DECLARE #SQLCmd VARCHAR(500)
DECLARE #CreateXmlHeader varchar(500)
DECLARE #AppendXmlBody varchar(500)
DECLARE #DeleteTempFile varchar(500)
SELECT #FileName = 'C:\Temp\SampleXMLOutput.xml'
SELECT #SQLCmd = 'bcp ' +
'"SELECT Name AS [#Name], ServiceName, MachineName,' +
' CASE PollInterval WHEN 10 THEN NULL ELSE PollInterval END AS PollInterval,' +
' CASE Alerts WHEN 1 THEN NULL ELSE ''false'' END AS Alerts, ' +
' CASE Pages WHEN 1 THEN NULL ELSE ''false'' END AS Pages' +
' FROM Watchdog.dbo.[Config.Services] [Service]' +
' FOR XML PATH(''Service''), ROOT(''Services'')"' +
' queryout ' +
#FileName + '.tmp' +
' -S' + ##SERVERNAME +
' -T -c -r -t'
SET #CreateXmlHeader = 'ECHO ^<?xml version="1.0" ?^> > ' + #FileName
SET #AppendXmlBody = 'TYPE ' + #FileName + '.tmp >> ' + #FileName
SET #DeleteTempFile = 'DEL ' + #FileName + '.tmp'
EXECUTE master..xp_cmdshell #SQLCmd
EXECUTE master..xp_cmdshell #CreateXmlHeader
EXECUTE master..xp_cmdshell #AppendXmlBody
EXECUTE master..xp_cmdshell #DeleteTempFile
END
You could include the addition of the header by putting the XML extract into a sub-query. So your BCP command becomes:
SELECT #SQLCmd = 'bcp ' +
'"SELECT ''<?xml version="1.0"?>'' + ' +
'(SELECT Name AS [#Name], ServiceName, MachineName,' +
' CASE PollInterval WHEN 10 THEN NULL ELSE PollInterval END AS PollInterval,' +
' CASE Alerts WHEN 1 THEN NULL ELSE ''false'' END AS Alerts, ' +
' CASE Pages WHEN 1 THEN NULL ELSE ''false'' END AS Pages' +
' FROM Watchdog.dbo.[Config.Services] [Service]' +
' FOR XML PATH(''Service''), ROOT(''Services''))"' +
' queryout ' +
#FileName + '.tmp' +
' -S' + ##SERVERNAME +
' -T -c -r -t'
To close the loop, I did find a solution that writes the XML one tag per line with no indenting. It's a disgusting hack but it's acceptable for my purposes, let's see if it passes code review. If someone wants to look at the file with indenting they can open it in IE (by double clicking it).
Of course the real solution would be a CLR stored proc, but for now I prefer it this way to keep the deployment simple, i.e. just run a single SQL script to deploy the DB.
ALTER PROCEDURE [dbo].[ExportConfigTablesToXML]
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON;
--EXEC sp_configure 'show advanced options', 1
--RECONFIGURE
--EXEC sp_configure 'xp_cmdshell', 1
--RECONFIGURE
--EXEC sp_configure 'show advanced options', 0
--RECONFIGURE
DECLARE #FileName VARCHAR(50)
DECLARE #SQLCmd VARCHAR(1000)
SELECT #FileName = 'C:\Temp\SampleXMLOutput.xml'
SELECT #SQLCmd = 'bcp ' +
'"DECLARE #xml xml; ' +
'DECLARE #text varchar(MAX); ' +
'SET #xml = (SELECT Name AS [#Name], ServiceName, MachineName, ' +
' CASE PollInterval WHEN 10 THEN NULL ELSE PollInterval END AS PollInterval, ' +
' CASE Alerts WHEN 1 THEN NULL ELSE ''false'' END AS Alerts, ' +
' CASE Pages WHEN 1 THEN NULL ELSE ''false'' END AS Pages ' +
'FROM Watchdog.dbo.[Config.Services] [Service] ' +
'FOR XML PATH(''Service''), ROOT(''Services''), TYPE); ' +
'SET #text = ''<?xml version=""1.0"" ?>'' + CHAR(13) + CHAR(10); ' +
'SET #text = #text + REPLACE(CAST(#xml AS varchar(MAX)), ''><'', ''>'' + CHAR(13) + CHAR(10) + ''<''); ' +
'SELECT #text" ' +
' queryout ' +
#FileName +
' -S' + ##SERVERNAME +
' -T -c -r -t'
EXECUTE master..xp_cmdshell #SQLCmd
END
Since you're running commands anyway, you could perhaps:
xmllint --format
From Free XML Formatting tool