I require to create through ADO.NET a temp table, perform a BulkCopy and then a Merge on the server between the temp and the actual table.
Problem is creating the temp table dynamic using pure ADO.NET.
The schema should be the same as the existing table, but this table is created using an ORM (NHibernate or Entity Framework, we're not sure yet). This also means that the schema can change in the future.
Is there any way to create a table in the database using plain ADO.NET objects? Such as a DataTable containing the schema of the original table?
Any information pointing me in the right direction is appreciated.
You can create a temporary table using select into #somename.
connection_ = New SqlClient.SqlConnection(connection_string_)
connection_.Open()
If connection_.State = ConnectionState.Open Then
command_.Connection = connection_
command_.CommandType = CommandType.Text
command_.CommandText = "select * into #some_table from some_table where some_id = 0"
command_.ExecuteNonQuery()
Dim line_index_ As Integer = 0
Dim data_table_ As DataTable = New DataTable()
Using parser_ As FileIO.TextFieldParser = New FileIO.TextFieldParser(path_)
parser_.SetDelimiters(delimiter_)
parser_.HasFieldsEnclosedInQuotes = False
While Not parser_.EndOfData
If line_index_ = 0 Then
Dim headers_ As String() = parser_.ReadFields()
For Each header_ In headers_
data_table_.Columns.Add(header_)
Next
Else
Dim row_ As DataRow = data_table_.NewRow()
row_.ItemArray = parser_.ReadFields()
data_table_.Rows.Add(row_)
End If
line_index_ += 1
End While
End Using
Using bulkCopy_ As SqlBulkCopy = New SqlBulkCopy(connection_)
bulkCopy_.DestinationTableName = "#some_table"
bulkCopy_.WriteToServer(data_table_)
End Using
' proof
command_.CommandText = "select * from #some_table"
Dim reader_ As SqlDataReader = Nothing
reader_ = command_.ExecuteReader
line_index_ = 0
While reader_.Read
line_index_ += 0
End While
End If
I managed to create a temp table based on an existing schema.
Blogged the solution on my site.
Related
I'm using a set of user properties on DataBricks Delta Tables for metadata management. The problem is when I need to change one of those properties I'm getting the 'FAILED Error: The specified properties do not match the existing properties at /mnt/silver/...' error message.
Databricks documentation only states that an Exception will be raised and I didn't find any argument to force it to accept the new values.
Is it possible to just update table Properties?
Any Suggestions?
Sample Code:
query = f'''
CREATE TABLE if not exists {tableMetadataDBName}.{tableMetadataTableName}
(
... my columns ...
-- COMMON COLUMNS
,Hash string
,sourceFilename STRING
,HashWithFileDate string
,Surrogate_Key STRING
,SessionId STRING
,SessionRunDate TIMESTAMP
,Year INT GENERATED ALWAYS AS ( YEAR(fileDate))
,Month INT GENERATED ALWAYS AS ( MONTH(fileDate))
,fileDate DATE
)
USING DELTA
COMMENT '{tableDocumentationURL}'
LOCATION "{savePath}/parquet"
OPTIONS( "compression"="snappy")
PARTITIONED BY (Year, Month, fileDate )
TBLPROPERTIES ("DataStage"="{txtDataStage.upper()}"
,"Environment"="{txtEnvironment}"
,"source"="{tableMetadataSource}"
,"CreationDate" = "{tableMetadataCreationDate}"
,"CreatedBy" = "{tableMetadataCreatedBy}"
,"Project" = "{tableMetadataProject}"
,"AssociatedReports" = "{tableMetadataAssociatedReports}"
,"UpstreamDependencies" = "{tableMetadataUpstreamDependencies}"
,"DownstreamDependencies" = "{tableMetadataDownstreamDependencies}"
,"Source" = "{tableMetadataSource}"
,"PopulationFrequency" = "{tableMetadataPopulationFrequency}"
,"BusinessSubject" = "{tableMetadataBusinessSubject}"
,"JiraProject" = "{tableMetadataJiraProject}"
,"DevOpsProject" = "{tableMetadataDevOpsProject}"
,"DevOpsRepository" = "{tableMetadataDevOpsRepository}"
,"URL" = "{tableMetadataURL}") '''
spark.sql(query)
Yes, it's possible to change just properties - you need to use "ALTER TABLE [table_name] SET TBLPROPERTIES ..." for that:
query = f"""ALTER TABLE {table_name} SET TBLPROPERTIES (
'CreatedBy' = '{tableMetadataCreatedBy}
,'Project' = '{tableMetadataProject}'
....
)"""
spark.sql(query)
I'm trying to populate a combobox with records from a table filtered by a parent combobox.
The table I want to filter is called muni and parent table is called prov. They are related by a common field (muni has a field called gid_prov that contains prov gid for each record)
First combobox stores the prov gid and shows the prov name.
Both tables are in a database from postgresql connected to access by an DSN file using ODBC.
I have tried many options, and none is correct.
First I tried a simple option. This code is useless now
Private Sub PROV_Change()
MUNI.RowSourceType = "Table/Query"
MUNI.RowSource = "select * from memoria.muni where gid_provs =" & PROV
MUNI.Requery
MUNI.SetFocus
MUNI.Text = ""
End Sub
PROV is the name of the parent combobox that stores the prov gid. And MUNI the combobox I want to populate.
Other option I tried was creating a query with an sql statement, this is the code right now, i tried with the OpenRecordsetOutput, but is not working. I can access the database and get the fields, but I can't populate the combobox MUNI
Private Sub PROV_Change()
Dim oDb As DAO.Database
Dim oRs As DAO.Recordset
dbconnect = "ODBC;DRIVER=PostgreSQL ANSI;UID=***;PWD=***;PORT=5432;SERVER=127...;DATABASE=memoria_historica;DSN=C:\Users\***"
Set oDb = OpenDatabase("", False, False, dbconnect)
Dim sql As String
sql = "select gid,nombre from memoria.municipios m where m.gid_provs =" & PROV & "order by m.nombre;"
Set oRs = oDb.OpenRecordset(sql, dbOpenSnapshot, dbSQLPassThrough)
Me.muni2.RowSource = ""
If oRs.RecordCount > 0 Then
With oRs
.MoveFirst
.MoveNext
Do While Not .EOF
muni2.AddItem .Fields("nombre").Value
.MoveNext
Loop
End With
End Sub
Worked fine tih the support of #Thaippo. Thanks
You can you DAO library.
You could you DAO. Database and DAO. Recordset like:
Dim oDb as DAO.Database
Dim oRs as DAO.Recordset
Your second block of codes could do the work.
Set oDb = OpenDatabase("", False, False, dbconnect)
Dim sql As String
sql = "select m.nombre from memoria.muni
m,memoria.prov p where st_within(m.the_geom,p.the_geom) and p.gid =" & PROV ";"
Set oRs = oDb.Openrecordset(sql)
If oRs.Recordcount > 1 Theb
With oRs
.Movelast
.Movefirst
Do while not.EOF
PROV.Additem . Fields("nameOfFieldsYouNeed"). Value
.Movenext
Loop
End With
End if
And it's would be a good idea to clear the combox box before adding new items in it.
I have 2 access file, their names are A and B.
I've designed a form in access A for receiving data from users, and I want to insert or save data to another table in access B.
I know how insert record to access A tables, but I need to insert to another access file in another place.
Also I'm using this code for insert record to table:
Dim SER As Recordset
Set SER = CurrentDb.OpenRecordset("Tbl_A")
With SER
.AddNew
.Fields("Roz") = Me.Combo_Roz.Value
.Fields("Tarikh") = Me.Text_Tarikh.Value
.Fields("User") = Me.Text_User.Value
.Fields("Saat_Harekat") = Me.Text_Start.Value
.Fields("Saat_Residan") = Me.Text_End.Value
.Fields("Saat_Bargasht") = Me.Text_Bargasht.Value
.Fields("Moshkel") = Me.Text_Moshkel.Value
.Fields("Tozihat") = Me.Text_Reason.Value
.Update
Use this answer if you want a permanent connection to this table.
Have a Database Object open a connection to AccessB, and create the Recordset based on Tbl_B of this Data Base:
Dim SER As Recordset
Dim Dbs as DAO.Database
'Open a recordset to AccessB, and insert the data:
Set Dbs = OpenDatabase("C:\Path\accessB.accdb")
Set SER = Dbs.OpenRecordset("Tbl_B")
With SER
.AddNew
.Fields("Roz") = Me.Combo_Roz.Value
.Fields("Tarikh") = Me.Text_Tarikh.Value
.Fields("User") = Me.Text_User.Value
.Fields("Saat_Harekat") = Me.Text_Start.Value
.Fields("Saat_Residan") = Me.Text_End.Value
.Fields("Saat_Bargasht") = Me.Text_Bargasht.Value
.Fields("Moshkel") = Me.Text_Moshkel.Value
.Fields("Tozihat") = Me.Text_Reason.Value
.Update
End With
Use this answer for a more easily disposed connection to this table.
Create a Linked table to table B in AccessB, and insert data to it as if it was a table in you databas:
Dim SER As Recordset
'Create the liked table by VBA code:
DoCmd.TransferDatabase acLink, "Microsoft Access", "C:\Path\accessB.accdb", acTable, "Tbl_B", "Tbl_B"
'Open a recordset based on linked table, and insert the data:
Set SER = CurrentDb.OpenRecordset("Tbl_B")
With SER
.AddNew
.Fields("Roz") = Me.Combo_Roz.Value
.Fields("Tarikh") = Me.Text_Tarikh.Value
.Fields("User") = Me.Text_User.Value
.Fields("Saat_Harekat") = Me.Text_Start.Value
.Fields("Saat_Residan") = Me.Text_End.Value
.Fields("Saat_Bargasht") = Me.Text_Bargasht.Value
.Fields("Moshkel") = Me.Text_Moshkel.Value
.Fields("Tozihat") = Me.Text_Reason.Value
.Update
End With
I have some legacy code that I need to update, and I need to add a feature that uses the value returned from a table valued function. The issue I am having is that the function returns a table variable. The function is something like follows:
CREATE FUNCTION [dbo].[SomeFunction](#intKey INT)
RETURNS #t TABLE (strValue VARCHAR(20))
BEGIN
(....SOME CODE.....)
INSERT INTO #t (strValue) SELECT #SomeValue
RETURN
END
The code I call the function from VB is as follows:
Dim objRS As New ADODB.Recordset
Dim objCmd As New ADODB.Command
Set objRS = New ADODB.Recordset
With objCmd
.ActiveConnection = objConnection.CurrentConnection
.CommandType = adCmdText
.CommandText = "SELECT * FROM [dbo].[SomeFunction](?)"
.CommandTimeout = 0
End With
objCmd.Parameters.Append objCmd.CreateParameter("#intKey", adInteger, adParamInput)
objCmd("#intMatterID") = someObject.Key
Set objRs = objCmd.Execute
If (objRs.RecordCount > 0) Then
someString = objRs.Fields("strValue")
End If
The problem seems to be that last block of code, and I have confirmed that the Recordset is not empty. The Fields value seems to always be a NULL.
So I assume that I am doing this incorrectly and was wondering if it was possible to retrieve the value of "strValue".
It seems the issue I was experiencing was related to how I was accessing "strValue". I implemented the following and it ended up working.
If (objRs.RecordCount > 0) Then
objRs.MoveFirst
someString = Cstr(objRs("strValue"))
End If
I'm trying to figure out a way to store metadata about a column without repeating myself.
I'm currently working on a generic dimension loading SSIS package that will handle all my dimensions. It currently does :
Create a temporary table identical to the given table name in parameters (this is a generic stored procedure that receive the table name as parameter, and then do : select top 0 * into ##[INSERT ORIGINAL TABLE NAME HERE] from [INSERT ORIGINAL TABLE NAME HERE]).
==> Here we insert custom code for this particular dimension that will first query the data from a datasource and get my delta, then transform the data and finally loads it into my temporary table.
Merge the temporary table into my original table with a T-SQL MERGE, taking care of type1 and type2 fields accordingly.
My problem right now is that I have to maintain a table with all the fields in it to store a metadata to tell my scripts if this particular field is type1 or type2... this is nonsense, I can get the same data (minus type1/type2) from sys.columns/sys.types.
I was ultimately thinking about renaming my fields to include their type in it, such as :
FirstName_T2, LastName_T2, Sex_T1 (well, I know this can be type2, let's not fall into that debate here).
What do you guyz would do with that? My solution (using a table with that metadata) is currently in place and working, but it's obvious that repeating myself from the systables to a custom table is nonsense, just for a simple type1/type2 info.
UPDATE: I also thought about creating user defined types like varchar => t1_varchar, t2_varchar, etc. This sounds like something a bit sluggy too...
Everything you need should already be in INFORMATION_SCHEMA.COLUMNS
I can't follow your thinking of not using provided tables/views...
Edit: As scarpacci mentioned, this somewhat portable if needed.
I know this is bad, but I will post an answer to my own question... Thanks to GBN for the help tho!
I am now storing "flags" in the "description" field of my columns. I, for example, can store a flag this way : "TYPE_2_DATA".
Then, I use this query to get the flag back for each and every column :
select columns.name as [column_name]
,types.name as [type_name]
,extended_properties.value as [column_flags]
from sys.columns
inner join sys.types
on columns.system_type_id = types.system_type_id
left join sys.extended_properties
on extended_properties.major_id = columns.object_id
and extended_properties.minor_id = columns.column_id
and extended_properties.name = 'MS_Description'
where object_id = ( select id from sys.sysobjects where name = 'DimDivision' )
and is_identity = 0
order by column_id
Now I can store metadata about columns without having to create a separate table. I use what's already in place and I don't repeat myself. I'm not sure this is the best possible solution yet, but it works and is far better than duplicating information.
In the future, I will be able to use this field to store more metadata, where as : "TYPE_2_DATA|ANOTHER_FLAG|ETC|OH BOY!".
UPDATE :
I now store the information in separate extended properties. You can manage extended properties using sp_addextendedproperty and sp_updateextendedproperty stored procedures. I have created a simple store procedure that help me to update those values regardless if they currently exist or not :
create procedure [dbo].[UpdateSCDType]
#tablename nvarchar(50),
#fieldname nvarchar(50),
#scdtype char(1),
#dbschema nvarchar(25) = 'dbo'
as
begin
declare #already_exists int;
if ( #scdtype = '1' or #scdtype = '2' )
begin
select #already_exists = count(1)
from sys.columns
inner join sys.extended_properties
on extended_properties.major_id = columns.object_id
and extended_properties.minor_id = columns.column_id
and extended_properties.name = 'ScdType'
where object_id = (select sysobjects.id from sys.sysobjects where sysobjects.name = #tablename)
and columns.name = #fieldname
if ( #already_exists = 0 )
begin
exec sys.sp_addextendedproperty
#name = N'Scd_Type',
#value = #scdtype,
#level0type = N'SCHEMA',
#level0name = #dbschema,
#level1type = N'TABLE',
#level1name = #tablename,
#level2type = N'COLUMN',
#level2name = #fieldname
end
else
begin
exec sys.sp_updateextendedproperty
#name = N'Scd_Type',
#value = #scdtype,
#level0type = N'SCHEMA',
#level0name = #dbschema,
#level1type = N'TABLE',
#level1name = #tablename,
#level2type = N'COLUMN',
#level2name = #fieldname
end
end
end
Thanks again