If I have to do data migration and ETL testing for data in Azure SQL DB's then can I use Azure Data Factory?
If yes the please provide some links explaining how? or some tutorials or page where I can find some details?
Thanks in advance!
Sunil
Yes, you can do ETL testing by using data factory. I copied all tables from one database to another database by using below process:
I check all the tables by using below command:
select TABLE_SCHEMA, TABLE_NAME FROM information_schema.
TABLES Where TABLE_TYPE = 'BASE TABLE' and TABLE_SCHEMA = 'dbo'
I created pipeline and performed Lookup activity to retrieve the tables of database by entering below query.
Output:
I created linked service for source and sink on success of lookup activity I implemented foreach activity by enabling sequential and added copy activity to it. I created dynamic dataset by using linked service that created for source. to retrieve all the tables from database I added the data dynamically as below:
Created schema and table parameters enter dynamic content for schema is #dataset().schema and for table is #dataset().table entered values for schema is #item().TABLE_SCHEMA for table is #item().TABLE_NAME .
create dataset by using linked service that created for sink and created parameters and entered values same as source in the sink and enabled auto create table option.
Executed the pipeline. It executed successfully.
All tables are copied to target database.
In this way you can copy all the tables from one database to another database.
Related
In Azure Data Factory, I'm doing a pretty vanilla 'Copy Data' operation. One dataset to another.
Can I view the query being used to perform the copy operation? Apparently, it's a syntax error, but I've only used drag-and-drop menus. Here's the error:
ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A
database operation failed. Please search error to get more
details.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=Incorrect
syntax near the keyword 'from'.,Source=.Net SqlClient Data
Provider,SqlErrorNumber=156,Class=15,ErrorCode=-2146232060,State=1,Errors=[{Class=15,Number=156,State=1,Message=Incorrect
syntax near the keyword 'from'.,},],'
Extra context
1.Clear Schema and import Schema again.
2. This mostly happens when there are some changes to table schema after creating pipeline and datasets. verify once.
3.The schema and datasets should be refreshed when there are some changes in the SQL table schema.
4. For table name or view name used in the query, use []. ex: [dbo].[persons]
5. In datasets select table name
6. try to publish before testing.
I followed the same scenario and reproduced the same thing. I didn’t get any error as you can see, the above error mainly happens because of schema.
Source dataset:
In source dataset, I manually added and connected to SQL database with Sample SQL table.
Sink dataset:
In sink dataset, I added another SQL database with auto create table, write behavior: Upset, key column: PersonID.
Before execution there is no table on SQL database, then after the execution was successful, I got this sample output in azure SQL database.
I use Azure SQL DB (Single DB, Basic, DTU, Provisioned).
There are two different DBs, say, DB-1 and DB-2.
For DB-1, I have Admin access.
For DB-2, I have read-only access. (No access to create new table.)
The two DBs have no links. I access them using SSMS.
The requirement:
In DB-2, there is a table [EMP] with 1000 rows.
Only 250 of them to be exported and inserted into a new table in DB-1 (with all columns).
How can I achieve in SSMS?
Thanks in advance!
There is no way to do this in only SSMS. If this is an ad-hoc project, I would query the records, copy and paste them into Excel, configure them in Excel for an insert statement, then paste them into an insert statement against DB-1.
If this is something that will need to be sustainable, I'd recommend looking into Azure Data Factory.
I have a U-SQL managed table that contains schematized structured data.
CREATE TABLE [AdlaDb].[dbo].[User]
(
UserGuid Guid,
Postcode string,
Age int?
DateOfBirth DateTime?,
)
And a Azure SQL Database table.
CREATE TABLE [SqlDb].[dbo].[User]
(
UserGuid uniqueidentifier NOT NULL,
Postcode varchar(15) NULL,
Age int NULL,
DateOfBirth Date NULL,
)
I would like to transfer data from U-SQL managed table to Azure SQLDB table without losing the data types.
I'm using azure data factory, seems like I cannot
directly query the U-SQL managed table as an input dataset for data factory
do a federated write query to Azure SQLDB
Hence I'm having an intermediate step where I copy from U-SQL managed table to Azure Blob and then move to Azure SQLDB table. Doing this, I'm losing the data type and having to have type conversion/transformations later again before inserting.
Is there any better way to transfer data from U-SQL managed table to Azure SQL Database table without losing data type? Or am I missing something?
At this point you have the following option:
Export the U-SQL table into an intermediate format (e.g., CSV) in ADLS or blob storage.
Use ADF to move the file into Azure SQL DB.
I know that the ADF team has a work item to do this for you. I will ask them to reply to this thread as well.
Directly writing into a table from a U-SQL script has a lot of challenges due to the fault-tolerant retry and scale-out processing in U-SQL. This makes atomic writing in parallel into a transacted store a bit more complex (see for example http://www.vldb.org/conf/1996/P460.PDF).
There is now another option to transfer data from USQL managed table to Azure SQL Database table.
Write out the data from USQL Managed table or from a USQL script to Azure Blob Storage as a text file (.csv, .txt etc..)
And then make use of the public preview feature in Azure SQL Database - BULK INSERT - wrap this into a stored procedure
Add an Stored procedure activity in Azure Data Factory to schedule
Note: There is one thing to be aware of when creating DATABASE SCOPED CREDENTIAL, refer this Stack Overflow question
How can one copy data from one database to another database in DB2...?
In Oracle one can do this by creating database links as below
create public database link remote using 'orcl222'
where orcl222 is the source database
INSERT INTO test_rowid (ID, NAME)
SELECT ID, NAME
FROM rsext.test_rowid#remote
Can someone please tell me how can i do this in DB2?
You can use several methods:
Export / LOAD or IMPORT
Use federation to create a wrapper and a nickname, and then just perform an insert select. (Needs special license)
Writing an external Stored Procedure (C or Java) that connects to the remote database.
I am developing a windows application and using Postgres as backend database. At some point in my application i am dynamically creating table e.g Table1, then Table2 and so on. In this way i have many dynamic table in my database. Now i provide a button "Clean Database", so i need to remove all those dynamic tables using SQL query. Should some one guide me how to write SQL Query that automatically delete all such tables?
You should just be able to say
DROP TABLE {tablename}
for each dynamically created table. Try that and see if it works.