How to configure elasticsearch to bigquery pipeline using data fusion please share steps or Articles
Related
I know Data Export Service has a SQL storage target where as Export to Data Lake is Gen2 but seeing that Dataverse (aka Common Data Service) is structured data, I can't see why you'd use Export to Data Lake option in Powerapps, as Gen2 is for un-structured and semi-structured data!
Am I missing something here? Could they both be used e.g. Gen2 to store images data?
Data Export service is v1 used to replicate the Dynamics CRM online data to Azure SQL or Azure IaaS SQL server in near real time.
Export to Datalake is similar to v2, for the same replication purpose with new trick :) snapshot is advantage here.
There is a v3 coming, almost similar to v2 but additionally with Azure synapse linkage.
These are happening very fast and not sure how community is going to adapt.
We have an app hosted on Azure using mongoDB (running on a VM) and Azure SQL dbs. The idea is to build a basic data analysis pipeline to "join" the data between both these DBs and visually display the same using power BI.
For instance we have a "user" table in SQL with a unique "id" and a "data" table in mongo that has a reference of "id" + other tables in SQL that have reference of 'id'. So we wish to analyse the contents of data based on user and possibly join that further with other tables as needed.
Is azure data lake + power BI enough to implement this case? Or we need azure data analytics or azure synapse for this?
Azure Data Lake (ADL) and Power BI on its own is not going to be able to build a pipeline, ADL it is just a storage area and Power BI is a very much a lightweight ETL tool limited by features and capacity.
It would be highly recommended that you have some better compute power behind it using, as you mentioned Azure Synapse. That will be able to have a defined pipeline to orchestrate data movement into the data lake, then do the processing to transform the data.
Power BI on it own will not be able to do this, as you will still be limited by the Dataflow and Dataset size of 1GB if running Pro. Azure Synapse does contain Azure Data Factory Pipelines, Apache Spark and Azure SQL Data Warehouse so you can choose between Spark and SQL for your data transformational steps as both will connect to the Data Lake.
Note: Azure Data Lake Analytics (ADLA) (and USQL) is not a major focus for MS, and never widely used. Azure Databricks and Azure Synapse with Spark has replaced ADLA in all of the modern data pipeline and architectures examples for MS.
I want to read data from MariaDB and SAP-HANA and load in BigQuery using Data Fusion. Is it possible to read using jdbc driver?
Currently you can use the Database plugin with MariaDB and SAP HANA drivers to load data into BigQuery using Cloud Data Fusion. Here is some documentation on how to use JDBC drivers with Cloud Data Fusion - https://cloud.google.com/data-fusion/docs/how-to/using-jdbc-drivers.
By EOY, Cloud Data Fusion will also have specific plugins for MariaDB and SAP HANA which will still need the drivers, but which will significantly improve the configuration process.
Is it possible to import a recipe exported from dataprep into a pipeline in Data Fusion?
No unfortunately they are currently not compatible and you cannot export a recipe from Cloud Dataprep (Wrangler) and import to Cloud Data Fusion (Wrangler). Two distinct engines/services, Cloud Dataprep based on Trifacta and Cloud Data Fusion from OSS CDAP
I want to migrate AWS PostgreSQL to google cloud SQL. I can perform such by some basic strategy such as extract the AWS data, Create Database in GCP and Restore the extracted data in GCP. But I was wondering is there any more sophisticated way to so such as using terraform or similar.
Yes. See https://cloud.google.com/solutions/migrating-postgresql-to-gcp/
For migrating MySQL there are more options available, however at the time of the writing, these only apply to MySQL:
https://cloud.google.com/sql/docs/mysql/migrate-data
https://cloud.google.com/sql/docs/mysql/replication/replication-from-external