I have a requirement to execute Azure PostgreSQL stored procedure using Azure Data Factory. Is there any way to do the same. I found that Stored Procedure activity only supports SQL procedures.
Basically I have some staging tables in PostgreSQL and I want to load data into target tables(in PostgreSQL). I have written some stored procedures to apply transformations and load data. I want to run those stored procedures in ADF. Is there any other suggested option to achieve this. TIA!
There is a way to execute the PostgreSQL function from Azure Synapse Analytics or Azure data factory. use lookup activity with query as "select function-name()"
Yes, Stored Procedure activity doesn't support Azure PostgreSQL stored procedure. It only supports Azure SQL Database, Azure Synapse Analytics and SQL Server Database. So you can try to delegate the call to PostgreSQL with Azure Function as Joel said.
Related
I have a vectorwise database that contain big size multiple table, I need to copy those table into Azure Storage or Azure SQL DB or Synapse. We don't have direct connector for vectorWise in Azure data factory.
Is there any way to connect with VectorWise in ADF by creating API or something?
You will need to setup a virtual machine on a network which can reach your database, you will need to install the self-hosted integration runtime on it, you will need to install the proper ODBC driver from your database vendor, and then create an ODBC linked service.
We successfully migrated data to an Azure Virtual Machine that contains a Postgresqldb with Azure Data Factory. We now need to run on this database some post-loading scripts, like creating views, create indexes and so on.
For a normal SQL-DB I would put the scripts into a Stored Procedure and trigger them in Azure Data Factory.
What is the best way to trigger these scripts for the PostgreSQL also from Azure Data Factory?
ADF stored proc acitivity does not support PostgresSQL , I think you should use Azure function as a workaround . You can always invoke the azure function from ADF .
I know we can migrate ms access database to ms sql and then migrate to azure sql. Is there any other option to migrate MS Access database to NoSQL database in azure (like azure tables) or any cheaper database in azure ?
Please help.
Is there any other option to migrate MS Access database to NoSQL
database in azure (like azure tables)
Surely,please get an idea of Azure Data Factory copy activity which could transfer your data from Access DB to other destinations:including azure SQL DB,azure table storage etc.
Please follow above tutorial to configure Access DB as source dataset and Table Storage as sink dataset in the copy activity, then execute it in the pipeline.
From a cost point of view,azure storage is relatively cheap for no sql storage.Please see the price details of it.
I wanted to load the data from Azure Blob storage to Azure SQL Database using Databricks notebook . Could anyone help me in doing this
I'm new to this, so I cannot comment, but why use Databricks for this? It would be much easier and cheaper to use Azure Data Factory.
https://learn.microsoft.com/en-us/azure/data-factory/tutorial-copy-data-dot-net
If you really need to use Databricks, you would need to either mount your Blob Storage account, or access it directly from your Databricks notebook or JAR, as described in the documentation (https://docs.azuredatabricks.net/spark/latest/data-sources/azure/azure-storage.html).
You can then read the files into DataFrames for whatever format they are in, and use the SQL JDBC connector to create a connection for writing the data to SQL (https://docs.azuredatabricks.net/spark/latest/data-sources/sql-databases.html).
I created a stored procedure using U-SQL in Azure data lake.I want to schedule that stored procedure in Azure Data factory.
Is it possible?
I've tried following steps:
I created a stored procedure using U-SQL in Azure Data Lake.
I've created a script which executes the same procedure.
Now, I am trying to run that U-SQL script from ADF.
Is this a right way to execute a U-SQL stored procedure?
Yes, if you want to schedule the execution of a U-SQL stored procedure/script, you should be able to run the script by using the "DataLakeAnalyticsU-SQL" activity type in ADF.
You can schedule U-SQL activities in ADF. You can find detailed documentation below:
https://learn.microsoft.com/en-us/azure/data-factory/data-factory-usql-activity