I'm using SSIS 2008 and trying to work on a package for importing a specified file into a table created for its layout. It will take in the destination table & source file as package variables.
The main problem I'm running into is that the file layouts are subject to change, they're not consistent. The table I'd be importing into will match the file though. I had initial success, but soon after changing the source file/destination it throws the vs_needsnewmetadata error.
Are there any workarounds discovered that could potentially be used here for files not fitting the layout the package was designed with?
Edit: These are .txt files, tab-delimited.
Edit2: Tried fiddling with OPENROWSET as well, hit a security error on our server.
I am assuming here that said file is a CSV file.
I have just been faced with the exact same problem a couple of weeks ago. You need to use dynamic SQL to achieve this.
Create a stored procedure on your database with the code below (change the 2 "C:\Folder\" locations to the location of your file):
CREATE PROCEDURE [dbo].[CreateAndImportCSVs] (#FILENAME NVARCHAR(200))
AS
BEGIN
SET NOCOUNT ON;
DECLARE #PATH NVARCHAR(4000) = N'C:\Folder\' + #FILENAME + ''
DECLARE #TABLE NVARCHAR(50) = SUBSTRING(#FILENAME,0,CHARINDEX('.',#FILENAME))
DECLARE #SQL NVARCHAR(4000) = N'IF OBJECT_ID(''dbo.' + #TABLE + ''' , ''U'') IS NOT NULL DROP TABLE dbo.[' + #TABLE + ']
SELECT * INTO [' + #TABLE + ']
FROM OPENROWSET(''MSDASQL''
,''Driver={Microsoft Access Text Driver (*.txt, *.csv)};DefaultDir=C:\Folder;''
,''SELECT * FROM ' + #FILENAME + ''')'
EXEC(#SQL)
END
You might need to download the Microsoft Access Database Engine from:
https://www.microsoft.com/en-gb/download/details.aspx?id=13255
and install on your machine/server for the Microsoft Access Text Driver to work.
Then create an Execute SQL Task in SSIS with the relevant connection details to your SQL server database. Then pass the file name to the stored procedure you created:
EXEC dbo.CreateAndImportCSVs 'filename.csv'
It will then create the table based on the structure and data contained within the CSV, it also names the table the same as the csv file name.
*This stored procedure can also be used to run through a list of files.
Hope this helps!
Related
I'm trying to create 1000+ Procs in MS SQL from supplied physical files as part of legacy migration located on Network . For now I plan to use sp with dynamic SQL to loop over all of them like in segment below, I had problem with BULK ROWTERMINATOR, so I just dummied it with bunch of ZZZZ, is there any other correct way to set it to NONE, so all string will be loaded into single row for run. I also use Nvarchar(Max) for my field.
DROP TABLE IF EXISTS #imp;
CREATE TABLE #imp (Col varchar(max))
BULK INSERT #imp
FROM '//TFSNetwork/log/Install/sp_Test02.sql'
WITH (ROWTERMINATOR = '\nzzzzzzzzzZZZ') ---<< ?????
select top 1 #Sql = Col from #imp
EXEC (#sql);
----------------------------------------------------sp_Test02.sql
CREATE PROCEDURE [dbo].[sp_Test]
AS
BEGIN
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
SET NOCOUNT ON;
SELECT GETDATE() AS TS
END
-----------------------------------------------------------------
Load whole file into single row/column
ROWTERMINATOR = '\n' is what used by default ,that's why you get it once omitted at all. Don't think we can or will want to change this behavior rather use your Z combo).
Same thing can be done with another BULK , in this case no need any ROWTERM options.
declare #myFile varchar(max)
select #myFile = BulkColumn
from openrowset(BULK '//Network/Path/Test02.sql', single_blob) x;
SELECT #myFile
I am attempting to build a view to be used in crystal reports that allows us to look up GL codes. Unfortunately, our ERP creates a new SQL table each year and appends it the last 2 digits onto the table name.
Unless I can find a way to change which table it looks at based off the date I will need to manually change the view every year for each of the views I am creating. Any advice?
This Year: select * from GL000016
Next Year: select * from GL000017
Here is the MSSQL version:
DECLARE #SQLQuery AS NVARCHAR(500)
DECLARE #TableName AS NVARCHAR(100)
SET #TableName = 'GL0000' + RIGHT(CONVERT(CHAR(4), GETDATE(), 120),2)
SET #SQLQuery = 'SELECT * FROM ' + #TableName
EXECUTE sp_executesql #SQLQuery
You could also use a stored procedure depending on the environment. #Tablename will hold the table name if that is all you need (i.e. SELECT #Tablename).
You can use the T-SQL Year function.
Returns an integer that represents the year of the specified date.
https://msdn.microsoft.com/en-us/library/ms186313.aspx
So for this year, the following will return 17.
select (YEAR(GETDATE()) % 100) + 1
Not exactly possible to switch tables dynamically for Views
If you want to switch the table you are selecting from, you'll need the to use IF statements or do Dynamic sql. Considering that you want to do this in a view, both of those are not available to you. So from my perspective, your options are:
Switch to use a stored procedure and use dynamic sql or if statements
Switch to use a function that returns a table (again, dynamic sql or if statements)
A Sql job that periodically runs a stored procedure that uses dynamic sql to re-create the view with the correct GL Account table name.
If you have to use a view, then 3 is probably your option, but it comes with a maintenance and handover overhead. Next person working on this project might be wondering why their view changes keeps getting overwritten.
Create yourself a temporary table that match the common structure of your GL0000XX table.
You then have to use dynamic SQL to query your tables.
CREATE TABLE #GL ....;
DECLARE #year char(2) = YEAR(GETDATE()) % 100;
INSERT INTO #GL
EXEC('SELECT * FROM GL0000' + #year);
Haven't found an answer via Google. I need to execute this code from SQL Server stored proc.
I have a folder with 100+ access dbs with a table called tblReports. Some of the access db's have an extra column in tblReports called AdminReport.
I need to capture the extra column if it exists, thus... I need to test how many columns are in tblReports so that I can use an if/else statement in the sp to generate the correct sql based on the column count.
I'd love to read your thoughts, here's the relevant snippet.
set #sql = 'Insert into CustomerServiceIntranet.dbo.ReportCriteria
(UserInfo,RptNbr,RptType,RptDesc,GroupCDBrk,ClientCDBrk,CategoryCDBrk,
UserIDBrk,UnitCDBrk,WrkTypeBrk,StatCDBrk,StatDatBrk,
ExperBrk,GroupList,ClientList,CategoryList,UserIDList,BusAreaList,
WrkTypList,StatusList,QueueList,ReviewDay,ReviewDayNA,
ErrorImpact,DateRange,DataSource,RptPathFile)'
+ 'Select '''+ #userfilename + ''', ors.* '
+ 'from (select * From Openrowset(''Microsoft.ACE.OLEDB.12.0'','''
+ #CurrentName
+ ''';''Admin'';,''select * from tblReports'')) ors'
The standard approach would be to link to tblReports by calling DoCmd.TransferDatabase. You would then be able to count number of the fields in the table, before embarking on any SQL. At the end of the look you would delete the link by calling DoCmd.DeleteObject.
It certainly looks neater than what you are trying to do.
In my T-SQL script, I refer to same long table name several times. I use this query on different tables.
Is there a way to refer a table name by variable? If so, I can simple declare one variable at the top which script will use and just by setting value, I can run it on various tables without making changes in the script.
A couple of options.
Within a single SQL statement you can alias table names like so:
SELECT *
FROM MySuperLongTableName T
WHERE T.SomeField=1
If you need to do this over lots of statements across several scripts a synonym might be a better option:
CREATE SYNONYM SuperT FOR dbo.MySuperLongTableName
You could create a synonym for that table but obviously you'd need to make sure that nobody changed the definition of the synonym whilst the script was running (and no parallel invocations of the script)
Are you running these in SSMS? If so you could set SQL CMD mode (on the "Query" menu) and use
:setvar tablename "spt_values"
use master
select * from $(tablename)
You could do this as such:
Declare #TableName As nvarchar(max)
Declare #SQL AS nvarchar(max)
#TableName = 'longtablename'
#SQL = 'Select * From ' + #TableName
EXEC(#SQL)
We've got a system (MS SQL 2008 R2-based) that has a number of "input" database and a one "output" database. I'd like to write a query that will read from the output DB, and JOIN it to data in one of the source DB. However, the source table may be one or more individual tables :( The name of the source DB is included in the output DB; ideally, I'd like to do something like the following (pseudo-SQL ahoy)
select o.[UID]
,o.[description]
,i.[data]
from [output].dbo.[description] as o
left join (select [UID]
,[data]
from
[output.sourcedb].dbo.datatable
) as i
on i.[UID] = o.[UID];
Is there any way to do something like the above - "dynamically" specify the database and table to be joined on for each row in the query?
Try using the exec function, then specify the select as a string, adding variables for database names and tables where appropriate. Simple example:
DECLARE #dbName VARCHAR(255), #tableName VARCHAR(255), #colName VARCHAR(255)
...
EXEC('SELECT * FROM ' + #dbName + '.dbo.' + #tableName + ' WHERE ' + #colName + ' = 1')
No, the table must be known at the time you prepare the query. Otherwise how would the query optimizer know what indexes it might be able to use? Or if the table you reference even has an UID column?
You'll have to do this in stages:
Fetch the sourcedb value from your output database in one query.
Build an SQL query string, interpolating the value you fetched in the first query into the FROM clause of the second query.
Be careful to check that this value contains a legitimate database name. For instance, filter out non-alpha characters or apply a regular expression or look it up in a whitelist. Otherwise you're exposing yourself to a SQL Injection risk.
Execute the new SQL string you built with exec() as #user353852 suggests.