Can someone let me know if there is a Azure Data Factory connector for Crunchbase?
Crunchbase is a leader in private-company data.
Alternatively, can someone let me know if its possible to connect to Crunchbase via REST in Data Factory
I have checked list of connectors given here Supported data stores, But unfortunately it does not have connector for Crunchbase.
If you need to move data to/from a data store that is not in the service built-in connector list, here are some extensible options:
For database and data warehouse, usually you can find a corresponding ODBC driver, with which you can use generic ODBC connector.
For SaaS applications:
If it provides RESTful APIs, you can use generic REST connector.
If it has OData feed, you can use generic OData connector.
If it provides SOAP APIs, you can use generic HTTP connector.
If it has ODBC driver, you can use generic ODBC connector.
For others, check if you can load data to or expose data as any supported data stores, e.g. Azure Blob/File/FTP/SFTP/etc, then let the service pick up from there. You can invoke custom data loading mechanism via Azure Function, Custom activity, Databricks/HDInsight, Web activity, etc.
Reference - https://learn.microsoft.com/en-us/azure/data-factory/connector-overview
Related
We are trying to consume SAP ODP OData services (See Using the OData Service for Extracting ODP Data), and due to large volumes we would like to use their Delta Token approach (i.e. CDC) so that we only need to get changes after the initial load. This seems like a common use case to me but I cannot find any Azure Data Factory (ADF) documentation that addresses this.
What ADF Connector should we use?
ADF OData Connector does not appear to allow HTTP Request Headers to be sent, so we can't pass Prefer: odata.track-changes.
I am planning to use Azure Data factory for ETL process, I would like to know if Azure Data factory uses the metamodel that is captured in the Data Catalog. Please advice
No currently you can' t reuse Metadata stored in Azure Data Catalog in Azure Data Factory directly. You could try to reuse some of the Metadata, retrieving Data Assets via the Rest API (https://learn.microsoft.com/en-us/rest/api/datacatalog/data-catalog-data-asset), but I think that it will be faster doing the setup in Azure Data Factory. Also be aware that main focus of Data Factory is on movement and orchestration. For Big Data transformations, you will use e. g. Databricks activities, for "classic" ETL integrate SSIS.
I have some tables from Postgres database to be integrated into Salesforce as external objects. I went through some video tutorials and documentations where I was recommended to use Salesforce Connect which supports providers with "OData" protocol support. Is it possible to integrate Postgres tables into Salesforce as external objects without Salesforce Connect?
Thanks.
Be careful with the phrase "external objects". To me, the use of those particular words implies the specific implementation of external data access/federation delivered with Salesforce Connect. I don't believe that there is any alternative if your goal is to create "real" external objects (named "objectname__x") within Salesforce.
There are, though, Salesforce integration solutions from the likes of Progress, Jitterbit, Mulesoft, and Informatica and others that can be used to access PostgreSQL, with varying degrees of coding being required. You won't get "external objects", but you will be able to access data residing off-cloud in a PostgreSQL database from your Salesforce system.
Hope this helps.
Currently the way to integrate data from external storages (Postgres in your case) without Salesforce Connect is implement your custom logic for synchronization using REST or SOAP API, Apex classes and triggers, Salesforce Workflows and Flows. Also you will need to implement appropriate interfaces on side of your data storage. Complexity of all these steps depends on complexity of your existing data model and infrastructure around it.
I would like to use a REST API as a data source for tableau. Is this possible in any way? The API service returns standard JSON, and I would think that would be an ideal scenario for tableau. Rather than connecting to some backend database behind the API.
Thanks
The web data connector does this and is in beta right now as noted by inox. See: https://www.tableau.com/web-data-connector
"The Web Data Connector will allow you to create a connection to almost any data that is accessible over HTTP. This can include internal web services, JSON data, REST APIs, and many other sources. It will not be supported in the initial launch of Tableau 9.0. Beta participants will be able to use a dedicated product key to continue building and testing connectors until the Web Data Connector officially launches."
Also, you can use the Extract API to create Tableau Extracts from an API: https://www.tableau.com/learn/tutorials/on-demand/extract-api-introduction
I have a data transformation task at hand and am currently in need of implementing an SSIS class package using AWS Data Pipeline. Is it possible to do custom code using its SDK to retrieve data from third party SOAP based web services?
I obviously need to pull data from third party SOAP Service and then do a lot of data massaging of my own before I can dump that data on an Amazon S3 storage.
Any help in this direction is welcome.