I am planning to use Azure Data factory for ETL process, I would like to know if Azure Data factory uses the metamodel that is captured in the Data Catalog. Please advice
No currently you can' t reuse Metadata stored in Azure Data Catalog in Azure Data Factory directly. You could try to reuse some of the Metadata, retrieving Data Assets via the Rest API (https://learn.microsoft.com/en-us/rest/api/datacatalog/data-catalog-data-asset), but I think that it will be faster doing the setup in Azure Data Factory. Also be aware that main focus of Data Factory is on movement and orchestration. For Big Data transformations, you will use e. g. Databricks activities, for "classic" ETL integrate SSIS.
Related
Can someone let me know if there is a Azure Data Factory connector for Crunchbase?
Crunchbase is a leader in private-company data.
Alternatively, can someone let me know if its possible to connect to Crunchbase via REST in Data Factory
I have checked list of connectors given here Supported data stores, But unfortunately it does not have connector for Crunchbase.
If you need to move data to/from a data store that is not in the service built-in connector list, here are some extensible options:
For database and data warehouse, usually you can find a corresponding ODBC driver, with which you can use generic ODBC connector.
For SaaS applications:
If it provides RESTful APIs, you can use generic REST connector.
If it has OData feed, you can use generic OData connector.
If it provides SOAP APIs, you can use generic HTTP connector.
If it has ODBC driver, you can use generic ODBC connector.
For others, check if you can load data to or expose data as any supported data stores, e.g. Azure Blob/File/FTP/SFTP/etc, then let the service pick up from there. You can invoke custom data loading mechanism via Azure Function, Custom activity, Databricks/HDInsight, Web activity, etc.
Reference - https://learn.microsoft.com/en-us/azure/data-factory/connector-overview
We are trying to consume SAP ODP OData services (See Using the OData Service for Extracting ODP Data), and due to large volumes we would like to use their Delta Token approach (i.e. CDC) so that we only need to get changes after the initial load. This seems like a common use case to me but I cannot find any Azure Data Factory (ADF) documentation that addresses this.
What ADF Connector should we use?
ADF OData Connector does not appear to allow HTTP Request Headers to be sent, so we can't pass Prefer: odata.track-changes.
is there a way to connect to Azure Data factory to Salesforce Commerce Cloud ?
In Data Factory the I only see connectors to Salesforce Service & Marketing Cloud
if it's possible I'll appreciate it if someone could show me an example
Thank you !
Actually, from the Azure Data Factory connector overview, we can know that Salesforce Commerce Cloud is not supported.
The only way is that you must achieve that in code level. Then call the Function, Python or Notebook active to run it in Data Factory.
There isn't an exist code example we can provide for you. You need design it by yourself.
we may infer from the overview of the Azure Data Factory connector (https://docs.microsoft.com/en-us/azure/data-factory/connector-overview) that Salesforce Commerce Cloud is not supported. You can only accomplish that at the code level. In order to run the Function, Python, or Notebook in Data Factory, use the appropriate function. We are unable to give you an existing code example. You must create it on your own.
I would like to use Azure to retrieve JSON data from a REST api then store that data into a table. Data retrieval would occur daily and a parameter would be passed to the api to restrict the results to the prior day's data.
Which Azure component/mechanism should I use for calling the api?
The data would be the foundation for a data warehouse. Should I use Azure SQL table or Azure table?
I have recently begun exploring Azure and am not sure how to do this.
I look forward to feedback.
Thank you.
Take a look at Azure Functions. You can create an Azure Function that is periodically invoked, it has input bindings for different sources (or you can add some C# code to read from URL) and then place results into Azure Database.
Here is example of Azure Function that sends JSON to stored procedure:
https://www.codeproject.com/Articles/1169531/Sending-events-from-Azure-Event-Hub-to-Azure-SQL-D
I have a data transformation task at hand and am currently in need of implementing an SSIS class package using AWS Data Pipeline. Is it possible to do custom code using its SDK to retrieve data from third party SOAP based web services?
I obviously need to pull data from third party SOAP Service and then do a lot of data massaging of my own before I can dump that data on an Amazon S3 storage.
Any help in this direction is welcome.