I am using SQL Server 2017 version, and I want to import multiple .csv files into multiple tables in SQL server.
I found the following script in the net,
--BULK INSERT MULTIPLE FILES From a Folder
--a table to loop thru filenames drop table ALLFILENAMES
CREATE TABLE ALLFILENAMES(WHICHPATH VARCHAR(255),WHICHFILE varchar(255))
--some variables
declare #filename varchar(255),
#path varchar(255),
#sql varchar(8000),
#cmd varchar(1000)
--get the list of files to process:
SET #path = 'C:\Dump\'
SET #cmd = 'dir ' + #path + '*.csv /b'
INSERT INTO ALLFILENAMES(WHICHFILE)
EXEC Master..xp_cmdShell #cmd
UPDATE ALLFILENAMES SET WHICHPATH = #path where WHICHPATH is null
--cursor loop
declare c1 cursor for SELECT WHICHPATH,WHICHFILE FROM ALLFILENAMES where WHICHFILE like '%.csv%'
open c1
fetch next from c1 into #path,#filename
While ##fetch_status <> -1
begin
--bulk insert won't take a variable name, so make a sql and execute it instead:
set #sql = 'BULK INSERT Temp FROM ''' + #path + #filename + ''' '
+ ' WITH (
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''\n'',
FIRSTROW = 2
) '
print #sql
exec (#sql)
fetch next from c1 into #path,#filename
end
close c1
deallocate c1
But the problem is I cannot use the command 'EXEC Master..xp_cmdShell' cause it was disabled by DBA's due to some security reasons, and they are not permitting me to use it. Is there any alternative command that I can use instead of 'xp_cmdShell' in the same script.
In this script near bulk insert command (set #sql = 'BULK INSERT Temp FROM ''' + #path + #filename + ''' '
+ ') I see only one table name 'Test', and how can I mention multiple table names in the Bulk insert command?
Any help please.
It's been a long time since I have had to do this, but this is how I used to do these kinds of things.
DECLARE #intFlag INT
SET #intFlag = 1
WHILE (#intFlag <=100)
BEGIN
PRINT #intFlag
declare #fullpath1 varchar(1000)
select #fullpath1 = '''\\FTP\' + convert(varchar, getdate()- #intFlag , 112) + '_your_file.csv'''
declare #cmd1 nvarchar(1000)
select #cmd1 = 'bulk insert [dbo].[your_table] from ' + #fullpath1 + ' with (FIELDTERMINATOR = ''\t'', FIRSTROW = 5, ROWTERMINATOR=''0x0a'')'
exec (#cmd1)
SET #intFlag = #intFlag + 1
END
GO
As you can tell, this is looping through a bunch of files with dates as file names. The first part of each file name was in this date format: convert(varchar, getdate()- #intFlag , 112)
I'm guessing your files have names that match some specific pattern.
SQl Server has a tool that does this for you. Goto to your SQL Server folder
Open SQL Server Import and Export Wizard.
Choose a Data Source Microsoft Excel
Select the Excel File. And following the steps
Related
trying to get an ETL package going which requires bulk inserting a bunch of .csv files off of a drive. Instead of manually having to scroll through the code and change the file path name for each insert (it increments with each month), would just like to declare the variable at the top.
Eg.
DECLARE #Month NVARCHAR(255)
SET #Month = 'November'
DECLARE #Year NVARCHAR(255)
SET #Year = '2019'
BULK INSERT #MonthlyMetrics
FROM '\\SomeServer\SomeFolder\Metrics_#Month_#Year.csv'
WITH
(
FIELDTERMINATOR = ','
, ROWTERMINATOR = '0x0a'
, FIRSTROW = 3
)
GO
There's a dozen of inserts like this where the only part of the file name that changes is the month. i.e. Metrics_November_2019.csv is that actual file name. Since the FROM is a string, not sure how to get the declared variable in there.
You can try using a dynamic SQL to load the data, only you will need to enable xp_cmdshell like this:
This turns on advanced options and is needed to configure xp_cmdshell
sp_configure 'show advanced options', '1';
RECONFIGURE
This enables xp_cmdshell
sp_configure 'xp_cmdshell', '1'
RECONFIGURE
After that you can load the data in your temp table:
DECLARE #dir varchar(300)
DECLARE #files table (FileName varchar(800));
DECLARE #filesCmd varchar(500);
DECLARE #filename varchar(100);
DECLARE #cmd_BulkInsert nvarchar(max);
SET #dir = '\\SomeServer\SomeFolder';
SET #filesCmd = 'DIR /b ' + #dir + ' | findstr csv';
INSERT INTO #files EXECUTE xp_cmdshell #filesCmd;
BEGIN TRY
DECLARE C_TABLE CURSOR LOCAL FORWARD_ONLY READ_ONLY STATIC FOR
SELECT FileName FROM #files WHERE NOT FileName IS NULL;
OPEN C_TABLE;
FETCH NEXT FROM C_TABLE INTO #filename;
WHILE ##FETCH_STATUS = 0
BEGIN
SET #cmd_BulkInsert = N'
BULK INSERT #MonthlyMetrics
FROM '''+#dir+'\'+#filename+'''
WITH
(
FIELDTERMINATOR = '',''
,ROWTERMINATOR = ''0x0a''
,FIRSTROW = 3
)'
EXECUTE sp_executesql #cmd_BulkInsert;
FETCH NEXT FROM C_TABLE INTO #filename;
END
END TRY
BEGIN CATCH
SELECT
GETDATE()
,ERROR_LINE()
,ERROR_MESSAGE()
,ERROR_NUMBER()
,ERROR_STATE()
,ERROR_SEVERITY()
,#cmd_BulkInsert
END CATCH
Using SQL Server 2008 for Transnational replication.
Some times my team folks altering the table during development before removing the table from subscription in the replication. So it leads to replication failure in the environment. To avoid that, planned to write a DDL trigger which will remove the table from subscription when it is getting altered.
Below is the trigger i have written to achieve the same.
CREATE TRIGGER RemoveArticleFromRepliction ON DATABASE
FOR DROP_TABLE
, ALTER_TABLE AS
BEGIN
SET NOCOUNT ON;
DECLARE #EventData XML = EVENTDATA();
DECLARE #TableName VARCHAR(255) = #EventData.value('(/EVENT_INSTANCE/ObjectName)[1]', 'NVARCHAR(255)')
DECLARE #CMD VARCHAR(255) = 'IF EXISTS (SELECT 1 FROM SYS.TABLES WHERE NAME = ''' + #TableName + ''') EXEC sp_mck_DropSubscriptionArticle ''' + #TableName + ''''
EXEC #CMD
END
The trigger applied successfully in SQL server 2008.
Note: sp_mck_DropSubscriptionArticle is a custom procedure. the logic to remove the article from subscription is added in it.
When i execute the alter statement, i am getting the below error
Could not find stored procedure 'IF EXISTS (SELECT 1 FROM SYS.TABLES WHERE NAME = 'NOTE') EXEC sp_mck_DropSubscriptionArticle 'NOTE''.
Please help me resolve this issue or suggest me any other approach to fix this problem
Looks like you are missing the brackets after EXEC. Please try the below changed code
CREATE TRIGGER RemoveArticleFromRepliction ON DATABASE
FOR DROP_TABLE
, ALTER_TABLE AS
BEGIN
SET NOCOUNT ON;
DECLARE #EventData XML = EVENTDATA();
DECLARE #TableName VARCHAR(255) = #EventData.value('(/EVENT_INSTANCE/ObjectName)[1]', 'NVARCHAR(255)')
DECLARE #CMD VARCHAR(255) = 'IF EXISTS (SELECT 1 FROM SYS.TABLES WHERE NAME = ''' + #TableName + ''') EXEC sp_mck_DropSubscriptionArticle ''' + #TableName + ''''
EXEC (#CMD)
END
I have a database with all tables needed, on which is perfectly usable. But for test purposes, I need to make copies of the database for, lets say 100 times. (My application will loop on each database to execute some scripts).
The databases generated should bear different names of course. To use Backup/Restore or even Detach/Copy/Attach a 100 times is not do-able. So I would like to know if there's a script which can loop to copy/restore a database several times on different names?
Thanks
Ok found something that's working for me, by simple WHILE LOOP;
DECLARE #index int
DECLARE #dbName varchar(25)
declare #HRNET varchar(200)
declare #HRNET_LOG varchar(200)
declare #sql varchar(2000)
SET #index = 5
WHILE (#index < 200)
BEGIN
-- Construct db name and corresponding files name
SET #dbName = 'BDName' + Right('0000' + CONVERT(NVARCHAR, #index), 4)
set #MDF = '''C:\Program Files\Microsoft SQL Server\MSSQL10_50.SQL2008\MSSQL\DATA\' + #dbName + '.mdf'''
SET #LDF = '''C:\Program Files\Microsoft SQL Server\MSSQL10_50.SQL2008\MSSQL\DATA\' + #dbName + '_1.ldf'''
-- Restore db from backup bak file
SELECT #sql = 'RESTORE DATABASE ' + #dbName + '
FROM DISK = ''C:\DB Backup\DBName1919.bak''
WITH FILE = 1,
MOVE ''WEEKLY_UK_CO_E_REPORTING_Data'' TO ' + #MDF + ',
MOVE ''WEEKLY_UK_CO_E_REPORTING_Log'' TO ' + #LDF +
', NOUNLOAD, STATS = 10'
exec(#sql)
SET #index = #index + 1
END
GO
To retrieve the backup path location (MDF & LDF), just run the following;
RESTORE FILELISTONLY
FROM DISK = N'C:\DB Backup\DBName1919.bak'
Our application does not delete data as we retain it for a period of time, instead we have a column "deleted" (bit) in most tables of the database that store data which we mark 1 when deleted, otherwise the default is 0.
I'd like to create a stored procedure that iterates all tables in the database, checks for the existence of a column named "deleted" and if it exists, I run a check against the LastUpdatedUtc column (datetime2) and if the date is over 6 months old and deleted = 1 then we delete the row.
This application is under continuous development so tables could be added which is why I want to create a script that iterates tables instead of having to add a line for each table and remember to add them as new tables are added.
Any help in a SQL Server 2008 R2 stored procedure to this would be a great help.
Thank you.
EDIT (thank you Omaer) here is what I've come up with so far. Anyone that knows a better way let me know.
IF OBJECT_ID('tempdb..#tmpTables') IS NOT NULL DROP TABLE #tmpTables
GO
CREATE TABLE #tmpTables
(
ID INT,
TableName NVARCHAR(100) NOT NULL
)
GO
SET NOCOUNT ON
GO
INSERT #tmpTables
SELECT [object_id], [name] FROM sys.all_objects WHERE type_desc = 'USER_TABLE' ORDER BY [name]
DECLARE #TN NVARCHAR(100)
DECLARE #SQL NVARCHAR(max)
DECLARE #PurgeDate VARCHAR(50)
SET #PurgeDate = DATEADD(MONTH, -6, GETUTCDATE())
WHILE (SELECT COUNT(*) FROM #tmpTables) > 0
BEGIN
SELECT TOP 1 #TN = TableName FROM #tmpTables
IF EXISTS(SELECT * FROM sys.columns WHERE name = 'deleted' AND OBJECT_ID = OBJECT_ID(#TN))
BEGIN
IF EXISTS(SELECT * FROM sys.columns WHERE name = 'LastUpdatedUtc' AND OBJECT_ID = OBJECT_ID(#TN))
BEGIN
SET #SQL = 'SELECT Count(*) As Counter FROM ' + #TN + ' WHERE [deleted] = 1 AND [LastUpdatedUtc] < ''' + #PurgeDate + '''' -- this will be the delete line when the code is final, just outputting results for now
EXEC(#SQL)
END
END
DELETE #tmpTables WHERE TableName=#TN
END
DROP TABLE #tmpTables
This is my first attempt, not tested it so there might be some typos/syntax errors but this should get you started:
declare #date6MonthsBack varchar(50)
select #date6MonthsBack = dateadd(month, -6, getdate());
declare c cursor for
select 'delete from ' + quotename(name) + ' where [deleted] = 1 and [LastUpdatedUtc] <= ''' + #date6MonthsBack + '''' from sys.tables
where object_id in (select object_id from sys.columns where name = 'deleted')
and object_id in (select object_id from sys.columns where name = 'LastUpdatedUtc')
declare #sql varchar(max)
open c; fetch next from c into #sql
while (##fetch_status = 0) begin
print(#sql)
--exec(#sql) --uncomment this line to do the actual deleting once you have verified the commands.
fetch next from c into #sql; end
close c; deallocate c
You could use undocummented sp_MSforeactable procedure instead of loop or cursor. Something like code below. I created procedure that runs your code and is executed with sp_MSforeachtable. The disadvantage is - the procedure is undocumented and may not be supported in next SQL Server releases
IF OBJECT_ID('dbo.usp_cleanup') IS NULL
EXEC ('CREATE PROCEDURE dbo.usp_cleanup AS SELECT 1')
GO
ALTER PROCEDURE dbo.usp_cleanup
#sTblName VARCHAR(200)
AS
BEGIN
-- your variables
DECLARE #PurgeDate VARCHAR(50)
DECLARE #SQL VARCHAR(MAX)
SET #PurgeDate = DATEADD(MONTH, -6, GETUTCDATE())
-- we can check columns existence in one condition
IF
EXISTS(SELECT * FROM sys.columns WHERE name = 'deleted' AND OBJECT_ID = OBJECT_ID(#sTblName))
AND EXISTS(SELECT * FROM sys.columns WHERE name = 'LastUpdatedUtc' AND OBJECT_ID = OBJECT_ID(#sTblName))
BEGIN
SET #SQL = 'SQL CODE GOES HERE' -- this will be the delete line when the code is final, just outputting results for now
PRINT #SQL
--EXEC(#SQL) -- uncomment for execution
END
ELSE
-- for debugging
BEGIN
PRINT #sTblName + ' has no [delete] and [LastUpdatedUtc] columns'
END
END
EXEC sp_MSforeachtable 'exec usp_cleanup ''?'''
GO
When I try to run the following SQL snippet inside a cursor loop,
set #cmd = N'exec sp_rename ' + #test + N',' +
RIGHT(#test,LEN(#test)-3) + '_Pct' + N',''COLUMN'''
I get the following message,
Msg 15248, Level 11, State 1, Procedure sp_rename, Line 213
Either the parameter #objname is ambiguous or the claimed #objtype (COLUMN) is wrong.
What is wrong and how do I fix it ? I tried wrapping the column name in brackets [], and double quotes "" like some of the search results suggested.
Edit 1 -
Here is the entire script. How do I pass the table name to the rename sp ? I'm not sure how to do that since the column names are in one of many tables.
BEGIN TRANSACTION
declare #cnt int
declare #test nvarchar(128)
declare #cmd nvarchar(500)
declare Tests cursor for
SELECT COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE COLUMN_NAME LIKE 'pct%' AND TABLE_NAME LIKE 'TestData%'
open Tests
fetch next from Tests into #test
while ##fetch_status = 0
BEGIN
set #cmd = N'exec sp_rename ' + #test + N',' + RIGHT(#test,LEN(#test)-3) + '_Pct' + N', column'
print #cmd
EXEC sp_executeSQL #cmd
fetch next from Tests into #test
END
close Tests
deallocate Tests
ROLLBACK TRANSACTION
--COMMIT TRANSACTION
Edit 2 -
The script is designed to rename columns whose names match a pattern, in this case with a "pct" prefix. The columns occur in a variety of tables within the database. All table names are prefixed with "TestData".
Here is slightly modified version. Changes are noted as code commentary.
BEGIN TRANSACTION
declare #cnt int
declare #test nvarchar(128)
-- variable to hold table name
declare #tableName nvarchar(255)
declare #cmd nvarchar(500)
-- local means the cursor name is private to this code
-- fast_forward enables some speed optimizations
declare Tests cursor local fast_forward for
SELECT COLUMN_NAME, TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE COLUMN_NAME LIKE 'pct%'
AND TABLE_NAME LIKE 'TestData%'
open Tests
-- Instead of fetching twice, I rather set up no-exit loop
while 1 = 1
BEGIN
-- And then fetch
fetch next from Tests into #test, #tableName
-- And then, if no row is fetched, exit the loop
if ##fetch_status <> 0
begin
break
end
-- Quotename is needed if you ever use special characters
-- in table/column names. Spaces, reserved words etc.
-- Other changes add apostrophes at right places.
set #cmd = N'exec sp_rename '''
+ quotename(#tableName)
+ '.'
+ quotename(#test)
+ N''','''
+ RIGHT(#test,LEN(#test)-3)
+ '_Pct'''
+ N', ''column'''
print #cmd
EXEC sp_executeSQL #cmd
END
close Tests
deallocate Tests
ROLLBACK TRANSACTION
--COMMIT TRANSACTION