I have the following REST configuration in Azure Data Factory
As you can I'm getting the error:
'item' is not a recognized function
The full configuration is
convert?q=USD_#{item().Currency}&compact=ultra&apiKey=xxxxxxxxxxxxxxxxxxx
Do I need to configure #item in Parameters?
The guide suggests I need to following these steps
Based on your code in the dynamic context, you are using this REST resource inside a ForEach as above it has item() function. You can get item().<"Value"> in a ForEach using a lookup.
item() is a ForEach function and can be used inside a ForEach which is used inside a ADF pipeline. You are using the ForEach function inside Dataset which is not known for the dataset. That's why it is giving a warning. When you use that dataset only for that pipeline it will give you the result without any error. But for any other pipeline It will give you the warning as error.
To use a pipeline function in the Dataset, the best practice is to create a Dataset parameter and give the value for this in the pipeline like below.
Create a Dataset Parameter with string type and a Default value:
Give this parameter in the Dataset dynamic context:
Now you can give pipeline function values for this Parameter inside ForEach or inside Pipeline:
Here I have used Copy activity for sample and given the value as per my URL. You can give your Relative URL with item() function in dynamic context.
Based on the item().Currency values it will give the REST page URL in each iteration.
Related
I'm trying to parametrize a pipeline in Azure Data Factory in order to enable a certain functionality to mulptiple environments. The idea is that the current environment is always available through a global parameter. I'd like to use this parameter to look up an array of environments to process data to. Example:
targetEnvs = [{ "dev": ["dev"], "test": ["dev", "test"], "acc": [], "prod": ["acc", "prod"] }]
Then one should be able to select the targetEnv array with something like targetEnvs[environment] or targetEnvs.environment. Subsequently a ForEach is used to execute some logic on these target environments.
I tried setting this up with targetEnvs as a pipeline parameter (with default value mapping each env directly to targetEnv, as follows: {"dev": ["dev"], "test": ["test"]}) Then I have a Set variable step to take value from the targetEnvs parameter, as follows:.
I'm now looking for a way to use the current environment (stored in a global parameter) instead of hardcoding "dev" in the Set Variable expression, but I'm not sure how to do this.
.
Using this expression won't even start the pipeline.
.
Question: how do I select this attribute of the object? Any other suggestions on how to do tackle this problem are welcome as well!
(Python analogy would be to have a dictionary target_envs and taking a value from it by using the key "current_env": target_envs[current_env].)
When I tried to access the object same as you, the same error occurred. I have taken the parameter targetEnv (given array) and global parameter environment with value as dev.
You can use the following dynamic content to access the key value.
#pipeline().parameters.targetEnv[0][pipeline().globalParameters.environment]
My requirement is to extract a string from filenames using a ADF variable, I need to extract the string until the final underscore '_' and the number of underscores vary in every filename as seen in the below example.
abc_xyz_20221221.txt --> abc_xyz
abc_xyz_a1_20221221.txt --> abc_xyz_a1
abc_c_ab_a1_20221221.txt --> abc_c_ab_a1
abc_c_ab_a1_a11_20221221.txt --> abc_c_ab_a1_a11
I tried to get it done using indexof() to get the position of the final underscore but it does not accept negative values, so I got the below logic which works in KQL (Azure Data Explorer) but fails in ADF because there is no CountOf() in this tool. Is there any equivalent function in ADF or can you please suggest me how to achieve the same in ADF?
substring("abc_xyz_20221221.txt", 0,
indexof("abc_xyz_20221221.txt", "_", 0,
strlen("abc_xyz_20221221.txt"),
countof("abc_xyz_20221221.txt", '_')))
You can try like this also using split and join inside ForEach activity.
Array for ForEach activity:
["abc_xyz_20221221.txt","abc_xyz_a1_20221221.txt","abc_c_ab_a1_20221221.txt","abc_c_ab_a1_a11_20221221.txt"]
Append variable inside ForEach:
#join(take(split(item(), '_'),add(length(split(item(), '_')),-1)),'_')
Result in an array variable:
As mentioned by #Joel Cochran, use the below expression in the append variable inside ForEach with lastIndexOf().
#substring(item(),0,lastindexof(item(),'_'))
This is a just a simpler form of what #Rakesh called out above . The only difference being , his implementation is iterating . In my case the file name is stored in a variable named foo
#substring(variables('foo'),0,lastindexof(variables('foo'),'_'))
output
I have an array HeaderList with a list of names. I have a look up activity to look at a CSV file header. Then, I have a IF activity to compare the first element. the expression in If activity is like this:
#equals(activity('Lookup2').output.firstRow.Prop_0,variables('HeaderList')[0])
That does not work. If I change it to this:
#equals(activity('Lookup2').output.firstRow.Prop_0,'XYZ'), then it works. How do I reference an array element in expression?
Thanks
#equals(activity('Lookup2').output.firstRow.Prop_0,variables('HeaderList')[0])
What does it mean?
I have got the same error in the if condition activity. But when the pipeline is debugged, it did not throw any error. I have repro'd the same in my ADF environment. Below are the steps.
Lookup activity is taken, and it refers to a csv file.
An array variable 'HeaderList' is taken and values for the variable is set using set variables activity.
Then If Condition activity is taken and below expression is given as a dynamic content.
#equals(activity('Lookup1').output.firstRow.prop_0,variables('HeaderList')[0])
The same error is produced.
Error: Cannot fit unknown into function parameter any.
When pipeline is debugged, it did not throw any error. It is successful.
I have a pipeline with a copy activity from storage.
I'm using the concat method to combine number of parameters to create the folder path in the Storage.
I have a wildcardFolderPath field which gets its data from the parameters file.
Part of the data is string and the other is a pipeline parameter
"wildcardFolderPath": {
"value": "[concat(parameters('folderPath'), '/', parameters('folderTime')]",
"type": "Expression"
}
When the pipeline runs, the string param folderPath is retrieved as is but the value of folderTime is not evaluated and this is what I see.
formatDateTime(pipeline().parameters.currentScheduleDateTime) instead of the datetime string.
I also tried using:
#concat(parameters('folderPath'), '/', parameters('folderTime')
and
#{concat(parameters('folderPath'), '/', parameters('folderTime')}
but I get: The workflow parameter 'folderPath' is not found.
Anyone encountered such an issue?
Create a parameter at pipeline level and pass in the expression builder with the following syntax.
#pipeline().parameters.parametername
Example:
You can add the parameter inside Add dynamic content if its not created before and select the parameters created to build an expression.
#concat(pipeline().parameters.Folderpath, '/', pipeline().parameters.Filedate)
Code:
I have a concat expression defined in the Function Name setting of an Azure Function in my pipeline, where it concatenates the API Query with the current filename that I want to run on this function. When I debug the pipeline, it fails without giving me any feedback. It just says "AzureFunction failed:"
If I manually insert the string, it works fine.
the concat expression is:
#concat('HttpTrigger?filename=', variables('filename'))
I'm new to Azure, any way I can debug this?
try this way:
#concat(variables('FirstName') ,variables('LastName'))
You could use Set Variable Activity with your Azure Function Activity.
In the Variable Activity, set the value of the variable.
Then refer the variable in the Azure Function Activity:
#concat('HttpTriggerJS1?name=',variables('name'))