Related
I have a requirement as ADLS container contains a folder with multiple files
Assume
Product.csv
Market.csv
3.Sales.csv
I need to consider file names Product, Market, Sales and form as Product_Market_Sales.csv in a destination path.
I tried multiple ways to achieve this.
Can anyone help me.
Thanks
Shanu
If all of your files are having same schema, then the below approach will work for you.
These are my variables in the pipeline.
First use Get Meta data activity to get the ChildItems list from the folder.
It will give the array like below in an alphabetical order of file names.
Then, Give this array to a ForEach activity. Inside ForEach, split each file name with '.' or '.c' and store the first element of split in an append variable activity to res_filename variable using below dynamic content.
#split(item().name,'.')[0]
Now, join this array with '_' and store it in a string variable.
#concat(join(variables('names'),'_'),'.csv')
After this, use copy activity and give wild card path *.csv in the source. Use dataset parameters for the sink dataset and give the above variable as file name and give the Copy behavior as Merge files.
This is my Pipeline JSON
{
"name": "filenames",
"properties": {
"activities": [
{
"name": "Get Metadata1",
"type": "GetMetadata",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"dataset": {
"referenceName": "Sourcefiles",
"type": "DatasetReference"
},
"fieldList": [
"childItems"
],
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
}
},
{
"name": "ForEach1",
"type": "ForEach",
"dependsOn": [
{
"activity": "Get Metadata1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#activity('Get Metadata1').output.childItems",
"type": "Expression"
},
"isSequential": true,
"activities": [
{
"name": "Append variable1",
"type": "AppendVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "names",
"value": {
"value": "#split(item().name,'.')[0]",
"type": "Expression"
}
}
}
]
}
},
{
"name": "Set variable1",
"type": "SetVariable",
"dependsOn": [
{
"activity": "ForEach1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "res_filename",
"value": {
"value": "#concat(join(variables('names'),'_'),'.csv')",
"type": "Expression"
}
}
},
{
"name": "Copy data1",
"type": "Copy",
"dependsOn": [
{
"activity": "Set variable1",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "DelimitedTextSource",
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"wildcardFileName": "*.csv",
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
},
"sink": {
"type": "DelimitedTextSink",
"storeSettings": {
"type": "AzureBlobFSWriteSettings",
"copyBehavior": "MergeFiles"
},
"formatSettings": {
"type": "DelimitedTextWriteSettings",
"quoteAllText": true,
"fileExtension": ".txt"
}
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": true,
"treatBooleanAsNumber": false
}
}
},
"inputs": [
{
"referenceName": "Sourcefiles",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "target",
"type": "DatasetReference",
"parameters": {
"targetfilename": {
"value": "#variables('res_filename')",
"type": "Expression"
}
}
}
]
}
],
"variables": {
"names": {
"type": "Array"
},
"res_filename": {
"type": "String"
}
},
"annotations": []
}
}
Result file:
If you want to merge the above with different source schemas, then as per my knowledge its better do that by code as suggested in comments with databricks or Azure functions.
I am getting query results that determine if user story hasn't been changed (changedate) in the last one day.
I'm following this article to build the logic app as the intention is similar
For some reason, despite the query returning a valid response (at least 1 user story result), the foreach expression is throwing this error:
ExpressionEvaluationFailed. The execution of template action 'For_each' failed: the result of the evaluation of 'foreach' expression '#body('Parse_JSON')?['body']?['value']' is of type 'Null'. The result must be a valid array.
How is it NULL when clearly there is a user story returned?
Get query results:
OUTPUTS:
[
{
"System.Id": 12345,
"System.WorkItemType": "User Story",
"System.State": "New",
"System.Title": "Experiment"
}
]
Parse JSON:
Inputs:
Content:
{
"value": [
{
"System.Id": 12345,
"System.WorkItemType": "User Story",
"System.State": "New",
"System.Title": "Experiment"
}
],
"#odata.nextLink": null
}
Schema
{
"type": "object",
"properties": {
"body": {
"type": "object",
"properties": {
"value": {
"type": "array",
"items": {
"type": "object",
"properties": {
"System.AssignedTo": {
"type": "string"
},
"System.Id": {
"type": "integer"
},
"System.State": {
"type": "string"
},
"System.Tags": {
"type": "string"
},
"System.Title": {
"type": "string"
},
"System.WorkItemType": {
"type": "string"
}
},
"required": [
"System.Id",
"System.WorkItemType",
"System.State",
"System.AssignedTo",
"System.Title"
]
}
},
"#odata.nextLink": {}
}
},
"headers": {
"type": "object",
"properties": {
"Cache-Control": {
"type": "string"
},
"Content-Length": {
"type": "string"
},
"Content-Type": {
"type": "string"
},
"Date": {
"type": "string"
},
"Expires": {
"type": "string"
},
"Pragma": {
"type": "string"
},
"Set-Cookie": {
"type": "string"
},
"Strict-Transport-Security": {
"type": "string"
},
"Timing-Allow-Origin": {
"type": "string"
},
"Transfer-Encoding": {
"type": "string"
},
"Vary": {
"type": "string"
},
"X-Content-Type-Options": {
"type": "string"
},
"X-Frame-Options": {
"type": "string"
},
"x-ms-apihub-cached-response": {
"type": "string"
},
"x-ms-apihub-obo": {
"type": "string"
},
"x-ms-request-id": {
"type": "string"
}
}
},
"statusCode": {
"type": "integer"
}
}
}
Outputs:
{
"value": [
{
"System.Id": 12345,
"System.WorkItemType": "User Story",
"System.State": "New",
"System.Title": "Experiment"
}
],
"#odata.nextLink": null
}
Using the Value from the Get Query Results directly works.
I have a pipeline that will iterate over files and copy them to a storage location.
The baseURL and relativeURL are stored in a json file.
I can read in this file and it is valid.
I have parameterized the linked service baseURL and this works when testing from the linked service, and from the dataset.
When I try to debug the pipeline however, I get an error:
"code":"BadRequest"
"message":null
"target":"pipeline//runid/310b8ac1-2ce6-4c7c-a1ad-433ee9019e9b"
"details":null
"error":null
It appears that from the activity in the pipeline, a null value is being passed instead of the baseURL.
I have iterated over the values from my config file and it is being read and the values are correct. It really seems like the pipeline is not passing in the correct value for baseURL.
Do I have to modify the json code behind the pipeline to get this to work?
If it helps, the json for the linked service, data set and pipeline are below:
--Linked Service:
{
"name": "ls_http_opendata_ecdc_europe_eu",
"properties": {
"parameters": {
"baseURL": {
"type": "string"
}
},
"annotations": [],
"type": "HttpServer",
"typeProperties": {
"url": "#linkedService().baseURL",
"enableServerCertificateValidation": true,
"authenticationType": "Anonymous"
}
}
}
--dataset
{
"name": "ds_ecdc_raw_csv_http",
"properties": {
"linkedServiceName": {
"referenceName": "ls_http_opendata_ecdc_europe_eu",
"type": "LinkedServiceReference"
},
"parameters": {
"relativeURL": {
"type": "string"
},
"baseURL": {
"type": "string"
}
},
"annotations": [],
"type": "DelimitedText",
"typeProperties": {
"location": {
"type": "HttpServerLocation",
"relativeUrl": {
"value": "#dataset().relativeURL",
"type": "Expression"
}
},
"columnDelimiter": ",",
"escapeChar": "\\",
"firstRowAsHeader": true,
"quoteChar": "\""
},
"schema": []
}
}
--pipeline
{
"name": "pl_ingest_ecdc_data",
"properties": {
"activities": [
{
"name": "lookup ecdc filelist",
"type": "Lookup",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "JsonSource",
"storeSettings": {
"type": "AzureBlobStorageReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "JsonReadSettings"
}
},
"dataset": {
"referenceName": "ds_ecdc_file_list",
"type": "DatasetReference"
},
"firstRowOnly": false
}
},
{
"name": "execute copy for every record",
"type": "ForEach",
"dependsOn": [
{
"activity": "lookup ecdc filelist",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#activity('lookup ecdc filelist').output.value",
"type": "Expression"
},
"activities": [
{
"name": "Copy data1",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "DelimitedTextSource",
"storeSettings": {
"type": "HttpReadSettings",
"requestMethod": "GET"
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
},
"sink": {
"type": "DelimitedTextSink",
"storeSettings": {
"type": "AzureBlobFSWriteSettings"
},
"formatSettings": {
"type": "DelimitedTextWriteSettings",
"quoteAllText": true,
"fileExtension": ".txt"
}
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": true,
"treatBooleanAsNumber": false
}
}
},
"inputs": [
{
"referenceName": "DelimitedText1",
"type": "DatasetReference",
"parameters": {
"sourceBaseURL": {
"value": "#item().sourceBaseURL",
"type": "Expression"
},
"sourceRelativeURL": {
"value": "#item().sourceRelativeURL",
"type": "Expression"
}
}
}
],
"outputs": [
{
"referenceName": "ds_ecdc_raw_csv_dl",
"type": "DatasetReference",
"parameters": {
"fileName": {
"value": "#item().sinkFileName",
"type": "Expression"
}
}
}
]
}
]
}
}
],
"concurrency": 1,
"annotations": []
}
}
I reproduced your error.
{"code":"BadRequest","message":null,"target":"pipeline//runid/abd35329-3625-490b-85cf-f6d0de3dac86","details":null,"error":null}
It is because you didn't pass your baseURL to link service in Source Dataset. Please do this:
And Dataset JSON code should be like this:
{
"name": "ds_ecdc_raw_csv_http",
"properties": {
"linkedServiceName": {
"referenceName": "ls_http_opendata_ecdc_europe_eu",
"type": "LinkedServiceReference",
"parameters": {
"baseURL": {
"value": "#dataset().baseURL",
"type": "Expression"
}
}
},
"parameters": {
"relativeURL": {
"type": "string"
},
"baseURL": {
"type": "string"
}
},
"annotations": [],
"type": "DelimitedText",
"typeProperties": {
"location": {
"type": "HttpServerLocation",
"relativeUrl": {
"value": "#dataset().relativeURL",
"type": "Expression"
}
},
"columnDelimiter": ",",
"escapeChar": "\\",
"firstRowAsHeader": true,
"quoteChar": "\""
},
"schema": []
}
}
I'm creating an openapi.json (version 3.0.3) schema and I'm modelling a post request. The body can look like this:
{
type: "A",
aParam: "string",
sharedParam1: "string",
sharedParam2: "integer",
sharedParam3: "string"
}
where type is one of A or B. If the type is A, the parameter aParam is required if the type is B aParam must be left out. Basically, the other way the request can look is:
{
type: "B",
sharedParam1: "string",
sharedParam2: "integer",
sharedParam3: "string"
}
How can I model this?
Here is what I tried:
{
"requestBody": {
"content": {
"application/json": {
"schema": {
"oneOf": [
{
"type": "object",
"properties": {
"type": {
"type": "string",
"enum": ["A"]
},
"aParam": {
"type": "string"
},
"sharedParam1": {
"type": "string"
},
"sharedParam2": {
"type": "string"
},
"sharedParam3": {
"type": "string"
}
}
},
{
"type": "object",
"properties": {
"type": {
"type": "string",
"enum": ["B"]
},
"sharedParam1": {
"type": "string"
},
"sharedParam2": {
"type": "string"
},
"sharedParam3": {
"type": "string"
}
}
}
]
}
}
}
}
}
Basically, I "overloaded" the request body by using oneOf but that has a lot of duplication.
You may extract the shared properties to a base schema. It won't make the definition much less verbose but at least will remove duplicated properties definitions making them more maintainable:
"components": {
"schemas": {
"baseRequestBody": {
"type": "object",
"required": [
"type",
"sharedParam1",
"sharedParam2",
"sharedParam3"
],
"properties": {
"type": {
"type": "string",
"enum": [
"A",
"B"
]
},
"sharedParam1": {
"type": "string"
},
"sharedParam2": {
"type": "integer"
},
"sharedParam3": {
"type": "string"
}
}
},
"requestBodyA": {
"allOf": [
{
"$ref": "#/components/schemas/baseRequestBody"
},
{
"type": "object",
"required": [
"aParam"
],
"properties": {
"aParam": {
"type": "string"
}
}
}
]
},
"requestBodyB": {
"allOf": [
{
"$ref": "#/components/schemas/baseRequestBody"
}
]
}
}
}
Additionally, you might want to use Discriminator which can be used by some tools like code generators:
"requestBody": {
"content": {
"application/json": {
"schema": {
"oneOf": [
{
"$ref": "#/components/schemas/requestBodyA"
},
{
"$ref": "#/components/schemas/requestBodyB"
}
],
"discriminator": {
"propertyName": "type",
"mapping": {
"A": "#/components/schemas/requestBodyA",
"B": "#/components/schemas/requestBodyB"
}
}
}
}
}
}
How to attach latest blob content in sendgrid service using logic app?
You can create a Blob trigger LA with SendGrid Action as shown below:
{
"$connections": {
"value": {
"azureblob": {
"connectionId": "/subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroup>/providers/Microsoft.Web/connections/azureblob",
"connectionName": "azureblob",
"id": "/subscriptions/<SubscriptionId>/providers/Microsoft.Web/locations/southeastasia/managedApis/azureblob"
},
"sendgrid": {
"connectionId": "/subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroup>/providers/Microsoft.Web/connections/sendgrid",
"connectionName": "sendgrid",
"id": "/subscriptions/<SubscriptionId>/providers/Microsoft.Web/locations/southeastasia/managedApis/sendgrid"
}
}
},
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Get_blob_content_using_path": {
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['azureblob']['connectionId']"
}
},
"method": "get",
"path": "/datasets/default/GetFileContentByPath",
"queries": {
"inferContentType": true,
"path": "#triggerBody()?['Path']",
"queryParametersSingleEncoded": true
}
},
"runAfter": {},
"type": "ApiConnection"
},
"Send_email_(V2)": {
"inputs": {
"body": {
"file": "#{base64(body('Get_blob_content_using_path'))}",
"filename": "#triggerBody()?['Name']",
"from": "<FromEmail>",
"ishtml": true,
"subject": "test attach met",
"text": "PFA",
"to": "<ToEmail>"
},
"host": {
"connection": {
"name": "#parameters('$connections')['sendgrid']['connectionId']"
}
},
"method": "post",
"path": "/mail/send"
},
"runAfter": {
"Get_blob_content_using_path": [
"Succeeded"
]
},
"type": "ApiConnection"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {
"$connections": {
"defaultValue": {},
"type": "Object"
}
},
"triggers": {
"When_a_blob_is_added_or_modified_(properties_only)": {
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['azureblob']['connectionId']"
}
},
"method": "get",
"path": "/datasets/default/triggers/batch/onupdatedfile",
"queries": {
"folderId": "JTJmbGFjb250YWluZXI=",
"maxFileCount": 10
}
},
"metadata": {
"JTJmbGFjb250YWluZXI=": "/lacontainer"
},
"recurrence": {
"frequency": "Minute",
"interval": 3
},
"splitOn": "#triggerBody()",
"type": "ApiConnection"
}
}
}
}