I'm trying to copy files from Az Storage blob to ADX using Data factory, but I can't find a solution to do this using json datasets (not binary), so I can map schema. I would love to have a template to do this.
I have tried to follow the resolution mentioned here (Get metadata from Blob storage with "folder like structure" using Azure Data Factory pipeline), but I'm lacking some more guidance (this is my first project using ADF)
enter image description here
enter image description here
enter image description here
Now I am getting another error as shown below
I'm actually also looking for a COMPLETE guide setting this up.
Here is my overall use-case target https://learn.microsoft.com/en-us/azure/sentinel/store-logs-in-azure-data-explorer?tabs=azure-storage-azure-data-factory - but the documentation is missing the detailed steps - in step 6 Create a data pipeline with a copy activity, based on when the blob properties were last modified. This step requires an extra understanding of Azure Data Factory. For more information, see Copy activity in Azure Data Factory and Azure Synapse Analytics.
enter image description here
enter image description here
It seems as I was going in a wrong direction.
I have actually found a simple solution for setting this up. I have tried the copy data tool, and it seems to be doing what I want. So case closed :-)
From the error message you have shared, it seems like your dynamic expression for passing the childItems of your get metadata activity to ForEach activity items is causing this problem.
You might be using Items = #activity('GetMetadataActivity').output instead of #activity('GetMetadataActivity').output.childItems.
Please use Items = #activity('GetMetadataActivity').output.childItems in your ForEach activity which should help resolve your error.
Here is a video demonstration by a community volunteer where this error has been explained in detail: ADF Error Function Length expects its parameter to be array or string
Related
I am doing a SQL Azure Database as a datasource in a mapping dataflow. The parameter KeyVaultConnectionString should be dynamic. Se picture 1.
My problem is that I am unable to fiugure out how to provide the value from mapping dataflow. I have tried different things like #pipeline.parameters.ConnString but thats just crap.
Thanks.
***** UPDATE (28-09-2022) *****
I forgot to explain that the flow is a Flowlet. The picture below shows the Flowlet that uses a SQL Server as a Source. I choose Inline dataset but I am not able to pass variable from the "outer" mapping data flow to the datasource.
For inline dataset as source in dataflows, the parameter option is not showing as a separate section. I tried to use the following linked service where I am using the linked service parameter to give my blob storage name.
Now when I use this in the dataflow source as inline dataset and try to test connection, it fails.
However, complete configuring the dataflow with required transformations and use dataflow activity in azure data factory pipeline. Here it shows the parameter to which the necessary value should be passed to linked service.
When you are in design/debug mode from the data flow designer, parameters will show in the right-panel when you click "Debug Settings" on the top of your designer window.
I need to extract a table with test steps that correspond to each test case, from AzureDevops to PowerBI.
I was able to retrieve a list of tables that I can extract with odata, but none of them contains test steps. I’m attaching the metadata request and an extract of its results. extract
I’ve read that another possibility would be you to use an api query, but I’m not sure which one.
Does anyone know a possible solution?
Thank you.
According to the note in this documentation,
You can’t add fields with a data type of Plain Text (long text) or HTML (rich-text). These fields aren’t available from Analytics for the purposes of reporting.
And the type of the Steps field is Text (multiple lines), so they cannot be extract with odata.
You can try to use this REST API to get the detail of the testcase which will contain the Steps detail. Please refer the screenshot:
If I manually upload txt or csv files in azure blob storage, when using the get metadata activity, I always get "contentMD5": null, while the other fields of the output are always populated. I also tried to copy it from on prem to blob storage using azcopy,but I have same issue. I am using ADF v2.
screen shot here null ContentMD5
Any idea why would this happen?
Thanks
Actually, that's not the Data Factory error.
Please check the file property in your Blob Storage. I also have one file without content-MD5:
That why the result of get metadata contentMD5 will be null.
How to solve this problem?
I just download the test.csv to my computer, delete it(Also delete blob snapshots) in the container. Then re-upload it, we can see the CONTENT-MD5 now:
Run the get metadata activity, check the out put:
Hope this helps.
I am working on a pipeline where our data sources are csv files stored in Azure data lake. I was able to process all the files using get meta data and for each activity. Now I need to find the number of files available in the Azure data lake? How can we achieve that. I couldn't find any itemcount argument in the Get Meta Data activity. I have noticed that the input of For each activity contains an itemscount value. Is there anyway to access this?
Regards,
Sandeep
Since the output of a child_items Get Metadata activity is a list of objects, why not just get the length of this list?
#{length(activity('Get Metadata1').output.childItems)}
I'm attempting to pull data from the Square Connect v1 API using ADF. I'm utilizing a Copy Activity with a REST source. I am successfully pulling back data, however, the results are unexpected.
The endpoint is /v1/{location_id}/payments. I have three parameters, shown below.
I can successfully pull this data via Postman.
The results are stored in a Blob and are as if I did not specify any parameters whatsoever.
Only when I hardcode the parameters into the relative path
do I get correct results.
I feel I must be missing a setting somewhere, but which one?
You can try setting the values you want into a setVariable activity, and then have your copyActivity reference those variables. This will tell you whether it is an issue with the dynamic content or not. I have run into some unexpected behavior myself. The benefit of the intermediate setVariable activity is twofold. Firstly it coerces the datatype, secondly, it lets you see what the value is.
My apologies for not using comments. I do not yet have enough points to comment.