I am trying to schedule the single dataflow will aggregate source to destination environment but our requirement is single data flow pipeline if we pass the parameter while trigger schedule. which means if I schedule A, i can pass the source raw folder details and other necessary information. similarly i can pass another schedule trigger B, i can pass the different parameter value. is it possible but i can see the failure. please advise.
Trigger (A) source ---> Dataflow ----> stage folder ---> copy activity(dynamic) ---> sql(on premise)
Trigger (B) source ---> Dataflow ----> stage folder ---> copy activity(dynamic) ---> sql(on premise)
When you debug for say stage folder 1 as parameter , are you setting the mapping ? Since its SQL as sink , I am assuming that you are . Now in the debug mode pass parameter for stage folder 2 details , unless schema is same in the both the case , it will fail with the same error which you mentioned .
Related
Error when connecting to Azure dataflow
I have given in Dataflow expression builder in sync
value of parameterdomain is coming from source column name
File Name Option: output to single file
File name : concat($parameterdomain,'.csv')
During Debug i get below error
Failure type User configuration issue
Details
Job failed due to reason: at Sink 'sinksource'(Line 17/Col 12): Column operands are not allowed in literal expressions
Source Pipeline
Make sure you have checked the expression checkbox in the parameters tab of dataflow activity.
We have an ADF pipeline with Copy activity to transfer data from Azure Table Storage to a JSON file in an Azure Blob Storage container. When the data transfer is in progress, other pipelines that use this dataset as a source fail with the following error "Job failed due to reason: Path does not resolve to any file(s)".
The dataset has a property that indicates the container directory. This property is populated by the trigger time of the pipeline copying the data, so it writes to a different directory in each run. The other failing pipelines use a directory corresponding to an earlier run of the pipeline copying the data and I have confirmed that the path does exist.
Anyone knows why this is happening and how to solve it?
Probably your expression in directory and file textbox inside the dataset is not correct.
Check this link : Azure data flow not showing / in path to data source
In TALEND, I want to search a file on a specificities directory using wildcard search.
eg. /directory/Abc_2021*
And after this I need to put "if" triggers according to the presence of file. If the file is available then one subjob will get executed else another subjob will get executed.
How to do this ?
Use tFileList, you specify the folder and the wildcard. Connect to a tJava with a OnSubjobOk connector. Connect the tJava to a tRunJob and specify the job to run if a file is present. Use a RunIf connector with the condition
((Integer)globalMap.get("tFileList_1_NB_FILE")) > 0
Add another tRunJob, specify the job to run if no file found, connect to the tJava with a RunIf connector with the condition
((Integer)globalMap.get("tFileList_1_NB_FILE")) == 0
I am using ADF to copy the files from a file server to Azure Blob storage. The files in the directory have the same structure without headers and I need to merge them to a single file in the Blob storage.
I created a ADF pipeline which uses get metadata to fetch the childItems and For each activity to loop through the files one by one
Inside the For each activity there is a copy data activity where I use the file name from the get metadata activity
In the sink setting, I use Merge files as the copy behaviour
When I execute the pipeline, the copy activity gets executed for 3 times and the file in the blob storage gets overwritten with the last file. How do I merge all the 3 files ?
I know we can use a wildcard pattern to select files. Suppose I have 3 files to begin with, when I run the get metadata activity and by the time the control comes to copy job activity and by this time if there is an addition of 4th file in the folder then with the wildcard pattern I will process all the 4 files and get metadata activity gives me the file names of the 3 files which I will use for archiving which is not right
Any help is appreciated
You don't need a for each for this. Just one copy activity that Marges all three files.
The trick would be to identify the source files using file path wildcards. if the requirement is to merge all file from source dataset, then merge behaviour in copy activity should be sufficient.
in Talend(data integration) i am trying to copy local directory to remote directory but when i am running the job only i can copy files but not folders from directory.please help me with this job.
In my talend job i am using local connection and remote connection components->
tfilelist->tfileproperties(to store path and name in one table)->tmssqlinput(extracting path from last table)->iteration-> tssh(if directory s not available then create)->finally sending it to tftpput to connect and copy to remote directory.
when i am storing in one table using tfileproperties in that for files it will generate some size but when folder s coming the size will be zero,using this condition m creating the directory using tssh component but unable to create folders,please help me.
Do you get an error message?
I believe the output of the TMSSqlInput should be a row based, rather than iteration. That might be the source of the problem.
tMSqlInput docs
tMSSqlInput executes a DB query with a strictly defined order which
must correspond to the schema definition. Then it passes on the field
list to the next component via a Main row link.