I have an ancient system that uses an Access 97 database to store information. I want to copy the data from the 90-some tables to a SQL Server 2008 database on a daily basis. I already have the tables defined in SS2008.
There is an equally ancient DTS job that has a separate box-line-box for each table. I'd rather use an easier to maintain method that was written in code, not lines and boxes. (yes, I know that SSIS lines and boxes are translated into XML, but that's kind of hard for me read and write.)
I can't use Linked Server or OPENROWSET because my SS2008 server runs as a 64-bit process, so the OLEDB Jet driver is not available. The OLEDB MSOffice ACE 12.0 driver is 64-bit, but it isn't supposed to be used with database servers because it is not threadsafe (according to Microsoft). Also, I can't get it to work ("Could not find installable ISAM") within SS2008 despite extensive research. I can read the Access table with OLEDB Jet in a 32-bit program such as SSIS.
So, I'm looking for a modern, non-box-and-line, elegant 32-bit solution to copy the tables from the Access mdb/mdw file to SS2008.
Can I do this with:
a single T-SQL script
some C# thing that does introspection to determine table structure and then executes SQL for each table
some magic "copy every table from this OLEDB to that SQL Server" package
There are several close dups of this question (Copy access database to SQL server periodically, Migrating Access Tables to SQL Server - Beginner), but none that deal with the 32-bit limitation that makes OPENROWSET/Linked Server a non-option.
You could probably do it from within Access itself using VBA like the following:
Public Function CopyTableDataToSqlServer()
Dim tbd As DAO.TableDef, qdf As DAO.QueryDef, connStr As String
connStr = _
"ODBC;" & _
"Driver={SQL Server};" & _
"Server=.\SQLEXPRESS;" & _
"Database=cloneDB;" & _
"Trusted_Connection=yes;"
For Each tbd In CurrentDb.TableDefs
If Not ((tbd.Name Like "MSys*") Or (tbd.Name Like "~*")) Then
Debug.Print tbd.Name
Set qdf = CurrentDb.CreateQueryDef("")
qdf.Connect = connStr
qdf.SQL = "DELETE FROM [" & tbd.Name & "]"
qdf.ReturnsRecords = False
qdf.Execute
Set qdf = Nothing
CurrentDb.Execute _
"INSERT INTO [" & connStr & "].[" & tbd.Name & "] " & _
"SELECT * " & _
"FROM [" & tbd.Name & "] ", _
dbFailOnError
End If
Next
Set tbd = Nothing
Debug.Print "Done."
End Function
I cast my vote for some C# thing. You may need to watch out for memory usage if you have large tables.
The basic idea goes like this:
foreach(tableName in access)
get the table from access
optionally clear the target table
sqlbulkcopy it to target database
A more complicated solution would be to grab both tables and only update the changed rows.
Related
MigrateDatabaseToLatestVersion is used. The database that is stored within SQL Server Express is updated.
When opening a local stored .sdf file (SQL Server CE database) with a valid path and file name, this file is not updated.
Database.SetInitializer(new MigrateDatabaseToLatestVersion<DTDataContext, Configuration>());
var connection = DTDataContext.GetConnectionSqlServerCE40(fullPathName);
dataBaseContext = new DTDataContext(connection, true);
dataBaseContext.Database.Initialize(true);
The MigrationHistory entries will be made in SQL Server Express and not in the local SQL Server CE database file.
What would be the easiest way to update a local SQL Server CE database file?
After a few experiments, an adequate solution was found (which fits for my purpose).
The question was focused about the old sdf(s) that were previously written but with an older model in contrast to the code.
I decided not to migrate old files (which are applied as a kind of backups).
Only reading will be made within those files. Obviously, it is possible that newer sdf(s) will be read once in the future but that's not a big deal.
Before reading stuff of an entity that could maybe not exist (in a sdf), it will be checked via SqlQuery and count(*).
[System.Diagnostics.CodeAnalysis.SuppressMessage( "Microsoft.Design", "CA1031:DoNotCatchGeneralExceptionTypes" )]
private bool TestIfTableExists( string tableName, DTDataContext dataContext )
{
try
{
int cnt = dataContext.Database.SqlQuery<int>( "select count(*) from " + tableName ).First();
return cnt > 0;
}
catch( Exception ex ) { /*available SqlCeException assembly does not fit --- table does not exist*/ return false; }
}
btw When using SqlCeException (v3.5), which could be provided as a reference via the assembly search, the above situation would fail (=unhandled exception!). Have not tested it with v4 because I wanna avoid a 'manual' reference because it must be checked in (no need for any path problems with other workstations).
Concerning writing a sdf:
When writing a new sdf with the current model, this is not a problem at all.
Database.CreateIfNotExists() was applied.
In my case, updating a sdf was not necessary --- and a quick solution for that was not found.
I've been trying to update and insert data into an existing database table. I am getting no errors, but the actual table data is not being updated.
This is my code for updating one of the rows:
using (SqlConnection con = new SqlConnection(connection))
{
con.Open();
SqlTransaction t = con.BeginTransaction();
SqlCommand cmd = new SqlCommand();
cmd.Connection = con;
cmd.Transaction = t;
cmd.CommandText = "Update tblName set PersonName = 'Wes' where PersonID = 2";
int i = cmd.ExecuteNonQuery();
t.Commit();
Console.WriteLine("change: "+i);
}
The output is
change: 1.
The change is happening when it runs but the actual table data is still the same. Any ideas?
The whole User Instance and AttachDbFileName= approach is flawed - at best! Visual Studio will be copying around the .mdf file (from App_Data into the output directory .\bin\debug) and most likely, your INSERT works just fine against the .mdf in .\bin\debug - but you're just looking at the wrong .mdf file (in App_Data) in the end!
If you want to stick with this approach, then try putting a breakpoint on the myConnection.Close() call - and then inspect the .mdf file with SQL Server Mgmt Studio Express - I'm almost certain your data is there.
The real solution in my opinion would be to
install SQL Server Express (and you've already done that anyway)
install SQL Server Management Studio Express
create your database in SSMS Express, give it a logical name (e.g. Testing)
connect to it using its logical database name (given when you create it on the server) - and don't mess around with physical database files and user instances. In that case, your connection string would be something like:
Data Source=.\\SQLEXPRESS;Database=Testing;Integrated Security=True
and everything else is exactly the same as before...
I need some help with a SSIS Script Task (SQL 2008 R2) that dynamically creates a package. I am refining a package that copies data from a Sage Timberline (Now rebranded to Sage 300) Pervasive SQL environment to a SQL server data warehouse. I can create a package that opens the connection to Timberline and copies the data to a table in SQL Server. The problem is, for each company in timberline and each table in SQL, I need to create a separate data flow task. Given the three Timberline company folders and the number of tables in each folder, this would take a lot of time to create and be cumbersome to maintain and troubleshoot.
I am trying to create a package that uses a Foreach Loop to create a package that creates a ADO/ODBC source (Timberline), a OLE destination (SQL) and dynamically handles the column mapping. I found code here that almost does what I need.
I tested this code and it works great using OLE SQL source and destinations. What makes this script work is that it dynamically handles the column mapping. So, it you placed it into a Foreach Loop of the 100 or so tables, with each loop it could dynamically create the data flow and map the columns, then execute the new package.
My problem is that I can only connect to Timberline using ODBC. So, I need to modify the script to create the source connection with ADO NET (ODBC) instead of OLE. I’m having a lot of trouble trying to figure this out. Could someone please help me out with this?
Here the other couple of things I tried first, other than this approach:
Solution: Setup a Linked server to Timberline Pervasive SQL
Problem: SQL server is 64-bit and the Timberline driver is 32-bit. Using a linked server returns a architecture mismatch error. I called Sage and they said they have no plans to release a 64-bit drive.
Solution: Use one of the SQL Transfer tasks
Problem: Only works with SQL databases. This source is a Pervasive SQL database
Solution: Use a “INSERT … INTO …” type script
Problem: This requires a linked server. See the problem above
Here’s the section of the original VB .NET code I need help with:
'To Create a package named [Sample Package]
Dim package As New Package()
package.Name = "Sample Package"
package.PackageType = DTSPackageType.DTSDesigner100
package.VersionBuild = 1
'To add Connection Manager to the package
'For source database (OLTP)
Dim OLTP As ConnectionManager = package.Connections.Add("OLEDB")
OLTP.ConnectionString = "Data Source=.;Initial Catalog=OLTP;Provider=SQLNCLI10;Integrated Security=SSPI;Auto Translate=False;"
OLTP.Name = "LocalHost.OLTP"
'To add Load Employee Dim to the package [Data Flow Task]
Dim dataFlowTaskHost As TaskHost = DirectCast(package.Executables.Add("SSIS.Pipeline.2"), TaskHost)
dataFlowTaskHost.Name = "Load Employee Dim"
dataFlowTaskHost.FailPackageOnFailure = True
dataFlowTaskHost.FailParentOnFailure = True
dataFlowTaskHost.DelayValidation = False
dataFlowTaskHost.Description = "Data Flow Task"
'-----------Data Flow Inner component starts----------------
Dim dataFlowTask As MainPipe = TryCast(dataFlowTaskHost.InnerObject, MainPipe)
' Source OLE DB connection manager to the package.
Dim SconMgr As ConnectionManager = package.Connections("LocalHost.OLTP")
' Create and configure an OLE DB source component.
Dim source As IDTSComponentMetaData100 = dataFlowTask.ComponentMetaDataCollection.[New]()
source.ComponentClassID = "DTSAdapter.OLEDBSource.2"
' Create the design-time instance of the source.
Dim srcDesignTime As CManagedComponentWrapper = source.Instantiate()
' The ProvideComponentProperties method creates a default output.
srcDesignTime.ProvideComponentProperties()
source.Name = "Employee Dim from OLTP"
' Assign the connection manager.
source.RuntimeConnectionCollection(0).ConnectionManagerID = SconMgr.ID
source.RuntimeConnectionCollection(0).ConnectionManager = DtsConvert.GetExtendedInterface(SconMgr)
' Set the custom properties of the source.
srcDesignTime.SetComponentProperty("AccessMode", 0)
' Mode 0 : OpenRowset / Table - View
srcDesignTime.SetComponentProperty("OpenRowset", "[dbo].[Employee_Dim]")
' Connect to the data source, and then update the metadata for the source.
srcDesignTime.AcquireConnections(Nothing)
srcDesignTime.ReinitializeMetaData()
srcDesignTime.ReleaseConnections()
Thanks in advance!
The C# code here is what you need if you need a Derived Column transform between the Source and Destination...
http://bifuture.blogspot.com/2011/01/ssis-adding-derived-column-to-ssis.html
To get the Source & Destination connections working, there is some secret sauce here to get things working between COM and .Net...
http://blogs.msdn.com/b/mattm/archive/2008/12/30/api-sample-ado-net-source.aspx
There is a similar page showing what to do for OleDB connections too.
Creating the source tables is easy. The available ODBC Metadata collections accessible should be retrieved with GetSchema("MetaDataCollections"). This will return a list of the available schema collections available for that particular ODBC driver.
Next, you'll want to see the data types returned from GetSchema("DataTypes"), so you can correctly interpret the data types for each column retrieved from GetSchema("Columns") to make your SQL Server create table script (which I'm assuming you've done).
To at least figure out which tables have primary keys, you'll need to loop over each table returned from GetSchema("Tables") in order to work with GetSchema("Indexes"). There's a bug that requires you to query the Indexes one table at a time. It is easy to google this - create a string array to pass in as the 3rd parameter: GetSchema("Indexes", tblName, resultArray[])
What I did was got the Tables and Columns collections into object variables in my parent SSIS package. Because Timberline is so fast (not), it seemed more efficient to pull all the columns down and filter them locally...which I do to create the tables in SQL Server, if necessary.
Once that is done, use the local copy of Tables again to manipulate a SSIS package in a Script task in "design mode" (change source and destination target tables, and redo the column mappings), and execute the now-in-memory SSIS package.
For me it took awhile to figure out. Both above URLs were required. I found and copied the .Net 2.0 Dts.PipelineWrap and Dts.RuntimeWrap .dlls to Microsoft.Net\FrameworkV2.0xxxxx folder, then referenced these in each script task wanting to use them, before setting up my "using DtsPW = Microsoft.SqlServer.Dts.Pipeline.Wrapper", etc.
Of note, because Timberline is 32-bit ODBC, I think it's necessary to build the SSIS package to use "X86", and target the script tasks to use .Net 2.0 framework.
I used the Derived Column code because I needed to copy multiple Timberline DBs into one SQL Server DB. Derived Column adds a "CompanyID" value to the output pipeline to SQL Server.
In the end, map the Destination's Virtual Input columns to its External Metadata columns, based off of the pipeline the Destination is attached to:
foreach (DtsPW.IDTSVirtualInputColumn100 vColumn in destVirtInput.VirtualInputColumnCollection)
{
var vCol = destInst.SetUsageType(destInput.ID, destVirtInput, vColumn.LineageID, DtsPW.DTSUsageType.UT_READWRITE);
destInst.MapInputColumn(destInput.ID, vCol.ID, destInput.ExternalMetadataColumnCollection[vColumn.Name].ID);
}
Anyways, that code will make more sense in the context of the bifuture.blogspot.com page.
The EzApi library could help with this too, but the AdoNet connection source for it is coded as a virtual class, so you'd need to implement specific classes to use. My C# kungfu is not strong enough for that in the time I have...
Also, CozyRoc sells a toolset with custom SSIS controls (data flow Source and Destination controls...) that looks like it does this on the fly input-to-output column mapping as well.
My package seems to work good enough now... Oh, and one more, I did not have luck trying to use DSN-less ODBC connections to Timberline, just: Dsn=dsnname;Uid=user;Pwd=pwd;
SSIS packages running in 64-bit land cannot see 32-bit DSNs on 64-bit OS, it seems...at least, it didn't work for me (win7-64, 32-bit Text ODBC DSN).
I have a directory of databases (*.mdb)s which are linked to several other *.mdb in other locations.
We've split the original database from one file into two partitions. The databases in the directory point to the original database file (and a few other databases as well). Now I need to re-link the tables from each database in the directory to the correct partition of the original (now split) database.
I've been going through manually and re-linking the tables in each database's Linked Table Manager, but this is grossly inefficient, and if I could just query the Linked Table Manager somehow, I could easily find out if I have changed the correct number of tables.
Is there any way to query the Linked Table manager? Through VB or even using the system tables and SQL using table name and file location?
Note I am opening the files in MS Access 2003, but MS Access 2003 is opening them and reporting Access 2000 format.
Per Remou's suggestion here is some code I wrote to relink the tables:
Sub RelinkLinks()
Dim db As Database
Dim tdf As TableDef
Dim OldLoc As String
OldLoc = ";DATABASE=\\lois\_DB\DB\BE.mdb"
Dim partition1Loc As String
Dim partition2Loc As String
partition1Loc = ";DATABASE=\\lois\_DB\DB\Partition 1\BE.mdb"
partition2Loc = ";DATABASE=\\lois\_DB\DB\Partition 2\BE.mdb"
Set db = CurrentDb
For Each tdf In db.TableDefs
' Only cycle through the locations
' that are equal to the old one...
If tdf.Connect = OldLoc Then
' Debug.Print tdf.Name & ":" & tdf.Connect
Dim whichLoc As String
If tdf.Name = "T_PAR_2_TBL_1" Or tdf.Name = "T_PAR_2_TBL_2" Then
' Only set the tables to partition 2 that were listed as in Partition 2 by the database splitter
Debug.Print "Setting linked table " & tdf.Name & " FROM " & tdf.Connect & " TO " & partition2Loc
whichLoc = partition2Loc
Else
' If the name was not listed as in partition 2, set it to partition 1
Debug.Print "Setting linked table " & tdf.Name & " FROM " & tdf.Connect & " TO " & partition1Loc
whichLoc = partition1Loc
End If
'We will uncomment this when we take the safety off...
'tdf.Connect = whichLoc
'tdf.RefreshLink
End If
Next tdf
End Sub
You can change and update links by referring to the TableDefs through VBA.
Set db = CurrentDB
For Each tdf In db.Tabledefs
If tdf.Connect <> Myconnect Then ''the connect property is a string
tdf.Connect = MyConnect
tdf.RefreshLink
End If
You can also link all tables in an external database with VBA by using CreateTableDef. I generally find it useful to keep a table of tables I want to link and use that.
One of my users wants to get data into Excel from SQL 2008 query/stored proc.
I never actually did it before.
I tried a sample using ADO and got data but user reasonably asked - where are the column names?
How do I connect a spreadsheet to an SQL resultset and get it with column names?
Apparently the field names are in the recordset object already.. just needed to pull them out.
i = 1
For Each objField In rs.Fields
Sheet1.Cells(1, i) = objField.Name
i = i + 1
Next objField
I don't know which version of Excel you are using but in Excel 2007 you can just connect to the SQL DB by going to Data -> From Other Sources -> From SQL Server. After you select your server and database, your connection will be created. Then you can edit it (Data -> Connections -> Properties) where in the Definition tab you change the Command type to SQL and enter your query in the Command text box. You can also create a view on the server and just point to that from Excel.
This should do it unless I misunderstood your question.