In my storage account , there are two types of data .
jsons
Delta Table (databricks)
which storing the different folder .
In Purview , I register the resource for the storage account and scan these two folders .
however after the scan , I can only see the folder name , but not the schema of the json , or the delta table .
[![screen shot of the folder][1]][1]
thanks and please help
[1]: https://i.stack.imgur.com/rWaiS.png
Related
I have json files sitting under differnt subfolders.
folder structures are like this below
/UserData/data/json/A/2021/01/28/03/
A_2021_01_28_03_file1.json
A_2021_01_28_03_file2.json
A_2021_01_28_03_file3.json
/UserData/data/json/A/2021/01/28/02/
A_2021_01_28_02_file1.json
A_2021_01_28_02_file2.json
/UserData/data/json/B/2021/03/27/02/
A_2021_03_27_02_file1.json
A_2021_03_27_02_file2.json
/UserData/data/json/C/2021/04/21/01/
A_2021_04_21_01_file1.json
A_2021_04_21_01_file2.json
I want to merge all the files available under A folder/B folder/C folder
and ingest them as A table, B table and C table in Azure Data Explorer.
the schema is
name string
timestamp date
value string
I don't see merge feature in copy activity, how could I achieve this?
appreciate your help
You need 3 copy activities.
In each copy activity, in source -> select "Wildcard file path"
and choose * to select all files.(see attached picture).
it will copy all the files under the specific folder
please read more: https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-storage?tabs=data-factory
I have a question, hopefully someone in the forum could give some help here. I am able to pull data from Soap API call to SQL Server table (xml data type field actually) via Copy Data Activity. The pipeline that runs this process is metadata driven, so how could I write other parameters in the same SQL Server table for the same run? I am using a Copy Data Activity to load XML data to SQL Server table but in Mapping tab I am not able to select other parameters in order to point them to others SQL table columns.
In addition, I am using a ForEach Activity in order the Copy Data Activity iterates for several values of one column on SQL Server table.
I will appreciate any advice on this.
Thanks
David
Thank you for your interest, I will try to be more explicit with this image: Hopefully this clarify a little bit. Given the current escenario, how could I pass StoreId and CustomerNumber parameters to the table Stage.XmlDataTable?
Taking in to account in the mapping step I am just able to map XML data from the current API call and then write it into Stage.XmlDataTable - XmlData column.
Thanks in advance David
You can add your parameters using Additional Columns in the Copy data activity Source.
When you import schema in mapping you can see the additional columns added in source.
Refer to this MS document for more details on adding additional columns during the copy.
I'm trying to upload a dataset to be viewed in Superset.
I created a postgres databased and able to connect via the URI: postgresql://user:password#localhost:port
I created a table called NYC Taxi with a table called nyctaxi.
However, when I tried to add the table to superset, I get the following error:
Table [nyctaxi] could not be found, please double check your database connection, schema, and table name, error: nyctaxi
if you have the data in CSV you can load it using the UPLOAD CSV from the source menu.
I have an small demo here
I have a CSV file.
I need to read each row and access the column values and for each row I need to call a foreach activity.
With what activity can I achieve this?
Assuming that the CSV file is in a cloud storage , you can use the lookup activity . Please beware that lookup activity has a limitation of 5000 at this time . Once you have done that you can use a FE loop and iterate through it .
Hope this helps
I have over 100 JSON files which is nested and I am trying to Load the JSON files via Data FactoryV2 into SQL Data Warehouse. I have created the Data FactoryV2 and everything seems fine the connection below seems fine and the Data Preview seems fine also.
When I run the Data Factory I get this error:
I am not sure what the issue is. I have tried to re-create the Data Factory several times.
The error message is clear enough when it says "All columns of the table must be specified...". This means that the table in the data warehouse has more columns than what you are seeing in the preview of the json file. You will need to create a table in the data warehouse with the same columns that are shown in the preview of the json files.
If you need to insert them in a table with more fields, create a "staging" table with the same columns as the json file, and then call a stored procedure to insert the content of this staging table in the corresponding table.
Hope this helped!