T-SQL stored procedure to get data from any table on the server for CSV export (SQL Server 2016) - tsql

Question

Answered / Solved.
Long story short, I need a stored procedure that would get the data from a few different views and put it into a .CSV file. Easy enough, but me being me, I decided to write something that could get the data from any table one could potentially desire. I decided to go with 2 procedures in the end:
Loop through a table with all the parameters (catalog, schema, table name, export path/file name etc. etc.) and feed it to 2nd stored procedure (in theory it should make it easier to manage in future, if/when different data needs to be exported). This one is fairly straightforward and doesn't cause any issues.
Pick up the column names (which was surprisingly easy - below in case it helps anyone)
select #SQL = 'insert into Temp_Export_Headers ' +
'select COLUMN_NAME ' +
'from [' + #loc_Source_Database + '].information_schema.columns ' +
'where table_name = ''' + #loc_Source_Table + ''''
and
select #Headers = coalesce(#Headers + ',', '') + convert(varchar, Column_Name)
from Temp_Export_Headers
After that, I want to dump all the data from "actual" table into temp one, which in itself is easy enough, but that's where things start to go downhill for me.
select #SQL =
'drop table if exists TempData ' +
'select * ' +
'into TempData ' +
'from [' + #loc_Source_Database + '].' + #loc_Source_Schema + '.' + #loc_Source_Table + ' with (nolock) '
Select * is just temporary, will probably replace it with a variable later on, for now it can live in this state on dev.
Now I want to loop through TempData and insert things I want (everything at the moment, will add some finesse and where clauses in near future) and put it into yet another temp table that holds all the stuff for actual CSV export.
Is there any way to add a self incrementing column to my TempData without having to look for and get rid of the original PK / Identity? (Different tables will have different values / names for those, making it a bit of a nightmare for someone with my knowledge / experience to loop through in a sensible manner, so I'd just like a simple column starting with 1 and ending with whatever last row number is)
#ShubhamPandey 's answer was exactly what I was after, code below is a product of my tired mind on the verge of madness (It does, however, work)
select #SQL =
'alter table TempData ' +
'add Uni_Count int'
select #SQL2 =
'declare #UniCount int ' +
'select #UniCount = 0 ' +
'update tempdata with (rowlock) ' +
'set #UniCount = Uni_Count = #UniCount + 1'
Both versions execute quicker than select * into without any other manipulation. Something I cannot yet comprehend.
Is there a better / more sensible way of doing this? (My reasoning with the loop - there will potentially be a lot of data for some of the tables / views, with most of them executed daily, plan was to export everything on Sat/Sun when system isn't that busy, and have daily "updates" going from last highest unique id to current.)
Looping was a horrible idea. To illustrate just how bad it was:
Looping through 10k rows meant execution time of 1m 21s.
Not looping through 500k rows resulted in execution time of 56s.

Answer 1

Since you are doing a table creation while insertion, you can always go forward with a statement like:
select #SQL =
'drop table if exists TempData ' +
'select ROW_NUMBER() OVER (<some column name>) AS [Id], * ' +
'into TempData ' +
'from [' + #loc_Source_Database + '].' + #loc_Source_Schema + '.' + #loc_Source_Table + ' with (nolock) '
This would create an auto-incrementing index for you in the TempData table