I would like to add an additional column during a copy activity. I cannot use the Getmetadata activity because it is through a hhtp linked service.
However, I am using a parameter called filename in order to specify the file. Would it be possible to input the above parameter in the additional column.
enter image description here
You cannot view the Dataset parameter in the pipeline directly.
Alternately you can use a pipeline parameter to provide input to Dataset parameter and use the same pipeline parameter in the Additional Column value.
Related
Is there a way to query a DB table as a one time activity, so that the values can be used to drive a repeating pipeline activity.
Let's say I have a set of values that varies based on the environment(DEV/TEST/PROD). Instead of passing the values corresponding to the environment as parameters, can I configure these values in a DB table and read them the first time the Data Factory runs, so that a repeating Orchestrator task that runs every five minutes can fetch value obtained from the table?
You can use a Lookup activity for your case.
Specify your query in the Lookup activity to get the row you want to query for your environment value. You may also wants to check the "First row only" option for your case.
To access the value returned from DB, you can get the value from the output of the Lookup. It would be in the "firstRow" object of the output.
For the conditional/switch handling for your use case, put in #activity('Lookup config table').output.firstRow.VALUE for your expression in the Switch's dynamic content.
you can use lookup activity from azure data factory to query values from db and then use parameters to store them to use in next activities, please check this
I have a row and in that we have a column called data. The below json is present in the data column. So I want to retrieve single object from json like name present inside the survey node.
{"build":{"version":"8.3.4.23"},"survey":{"database":"abc","id":952,"name":"ravi","server":"ABCD0045","group":"JKLMN"}}
I will provide survey and name as the input parameters and my output has to be ravi. is there any syntax to get this in azure data factory dataflow activity.
You can use Parse transformation in Data Flow to parse your data column.
Screenshot:
Then you can use this expression json.survey.name to get the value ravi.
I am trying to do schema compliance of an input file in ADF. I have tried the below.
Get Metadata Activity
The schema validation that is available in source activity
But the above seems to only check if a particular field is present or not in the specified position. Also Azure by default takes the datatype of all these fields as string since the input is flat file.
I want to check the position and datatype as well. for eg:-
empid,name,salary
1,abc,10
2,def,20
3,ghi,50
xyz,jkl,10
The row with empid as xyz needs to be rejected as it is not of number data type. Any help is appreciated.
You can use data flow and create a filter to achieve this.
Below is my test:
1.create a source
2.create a filter and use this expression:regexMatch(empid,'(\\d+)')
3.Output:
Hope this can help you.
I am coping data from a rest api to an azure SQL database. The copy is working find but there is a column which isn't being return within the api.
What I want to do is to add this column to the source. I've got a variable called symbol which I want to use as the source column. However, this isn't working:
Mapping
Any ideas?
This functionality is available using the "Additional Columns" feature of the Copy Activity.
If you navigate to the "Source" area, the bottom of the page will show you an area where you can add Additional Columns. Clicking the "New" button will let you enter a name and a value (which can be dynamic), which will be added to the output.
Source(s):
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview#add-additional-columns-during-copy
Per my knowledge, the copy activity may can't meet your requirements.Please see the error conditions in the link:
Source data store query result does not have a column name that is
specified in the input dataset "structure" section.
Sink data store (if with pre-defined schema) does not have a column
name that is specified in the output dataset "structure" section.
Either fewer columns or more columns in the "structure" of sink
dataset than specified in the mapping.
Duplicate mapping.
I think Mapping Data Flow is your choice.You could add a derived column before the sink dataset and create a parameter named Symbol.
Then set the derived column as the value of Symbol.
You can use the Copy Activity with a stored proc sink to do that. See my answer here for more info.
I have to call a webservice with multiple times with a diferent parameter value.
therefore i created a foreach task containing a copy activity.
Now i want to save the output of this call along with the parameter value.
Cann i somehow add a new additional field in the copy activity mapping to my parameter value? (#item().value)
what is your copy source type? You might use a query to include the item().value
You could also reference this post.
Add file name as column in data factory pipeline destination