In TALEND, I want to search a file on a specificities directory using wildcard search.
eg. /directory/Abc_2021*
And after this I need to put "if" triggers according to the presence of file. If the file is available then one subjob will get executed else another subjob will get executed.
How to do this ?
Use tFileList, you specify the folder and the wildcard. Connect to a tJava with a OnSubjobOk connector. Connect the tJava to a tRunJob and specify the job to run if a file is present. Use a RunIf connector with the condition
((Integer)globalMap.get("tFileList_1_NB_FILE")) > 0
Add another tRunJob, specify the job to run if no file found, connect to the tJava with a RunIf connector with the condition
((Integer)globalMap.get("tFileList_1_NB_FILE")) == 0
Related
I am trying to schedule the single dataflow will aggregate source to destination environment but our requirement is single data flow pipeline if we pass the parameter while trigger schedule. which means if I schedule A, i can pass the source raw folder details and other necessary information. similarly i can pass another schedule trigger B, i can pass the different parameter value. is it possible but i can see the failure. please advise.
Trigger (A) source ---> Dataflow ----> stage folder ---> copy activity(dynamic) ---> sql(on premise)
Trigger (B) source ---> Dataflow ----> stage folder ---> copy activity(dynamic) ---> sql(on premise)
When you debug for say stage folder 1 as parameter , are you setting the mapping ? Since its SQL as sink , I am assuming that you are . Now in the debug mode pass parameter for stage folder 2 details , unless schema is same in the both the case , it will fail with the same error which you mentioned .
I am writing restful API with Yii, but I am getting an SQL error in create function. My purpose is to add new data to the news table, but it asks me for the author_id. How can I do it without crushing the default create method?
Solution 1. Run this below query on mysql/phpmyadmin and restart server
SET GLOBAL sql_mode = 'NO_ENGINE_SUBSTITUTION';
Solution 2.
Open the my.ini or my.cnf file for editing (the file you have depends on whether you are running Windows or Linux).
Find the following line:
sql_mode = "STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
Replace it with the line below:
If the line is not found, insert the line under the [mysqld] section (if there is no [mysqld] section, create it).
sql_mode= ""
Restart the MySQL service for the change to take effect.
If restarting is not a feasible option at the moment, you may log into the database server and execute the below command for the changes to take effect immediately. However, the change will be discarded the next time the MySQL service restarts unless the above process is performed.
set global sql_mode='';
in Talend(data integration) i am trying to copy local directory to remote directory but when i am running the job only i can copy files but not folders from directory.please help me with this job.
In my talend job i am using local connection and remote connection components->
tfilelist->tfileproperties(to store path and name in one table)->tmssqlinput(extracting path from last table)->iteration-> tssh(if directory s not available then create)->finally sending it to tftpput to connect and copy to remote directory.
when i am storing in one table using tfileproperties in that for files it will generate some size but when folder s coming the size will be zero,using this condition m creating the directory using tssh component but unable to create folders,please help me.
Do you get an error message?
I believe the output of the TMSSqlInput should be a row based, rather than iteration. That might be the source of the problem.
tMSqlInput docs
tMSSqlInput executes a DB query with a strictly defined order which
must correspond to the schema definition. Then it passes on the field
list to the next component via a Main row link.
I create a step of database script which access to SQL Database Service in Workload Scheduler Service. When I run the process, the step encountered the error below.
error message
AWKDBE018E Cannot access required JDBC Driver folder
message information
http://www-01.ibm.com/support/knowledgecenter/SSGSPN_9.2.0/com.ibm.tivoli.itws.doc_9.2/common/src_ms/awsmsawkdbe.htm?lang=en
AWKDBE018E Cannot access required JDBC Driver folder
Explanation
The job was not able to access a JDBC Driver folder, you might not
have enough permissions.
System action
The operation is not performed.
Operator response
Verify that you have enough permissions.
This message seems to ask me to grant the proper authority to the job user. But there is no property to specify the job user of Workload Automation Agent. I use a Workload Automation Agent provisioned by Bluemix automatically.
Could you teach me which parameters are needed ?
Database script step information
JDBC driver class path info
I checked the path by the following "ls -lR" command step's log.
it seems to have a problem with the agent, I tried to replicate the same job type but it is not working with the same error message (even using different solutions for jdbc driver path).
If you are using the Workload Automation Agent that is created for you then you could open a support ticket to have the Workload team look at that agent.
Edit after having support from service team:
in the jar classpath field for a predefined workload scheduler process you have to put only the path to the directory containing jar files, without putting the jar file name to use.
So, according to current Workload Scheduler documentation, you have to use the following value:
/home/wauser/utils
By this way the database script works fine.
(screenshot added)
It looks like it is having issues referencing the location to the JDBC class path for DB2. Can you please double check the location for the class path for the DB2 driver?
Even though old, I wanted to make some quick checks.
This is tested on a 9.5 FP1 dynamic agent, part of the container delivery. The path values are the standard values for the container.
Try 1 - full path - SUCCESS
<jsdldatabase:driverPath>/opt/wa/TWS/jdbcdrivers/db2/</jsdldatabase:driverPath>
= Status Message: Success
= Exit Status : 0
Try 2 - relative path - FAIL
<jsdldatabase:driverPath>./jdbcdrivers/db2/</jsdldatabase:driverPath>
Job status : FAIL
===============================================================
AWKDBE018E Cannot access required JDBC Driver folder
===============================================================
Try3 - variable in path - FAIL
<jsdldatabase:driverPath>${UNISONHOME}/jdbcdrivers/db2/</jsdldatabase:driverPath>
===============================================================
AWKDBE018E Cannot access required JDBC Driver folder
===============================================================
Try4 - variable in path - FAIL
<jsdldatabase:driverPath>$UNISONHOME/jdbcdrivers/db2/</jsdldatabase:driverPath>
===============================================================
AWKDBE018E Cannot access required JDBC Driver folder
===============================================================
So put short you need an absolute path into that parameter.
BUT, you can set the path in a config file global to the agent
Try5 - variable in agent config -
Inside IWSDATA Home : wadata/JavaExt/cfg/DatabaseJobExecutor.properties, write the following line
jdbcDriversPath=/opt/wa/TWS/jdbcdrivers
then remove the xml element about driver from the job, so no line
<jsdldatabase:driverPath>/opt/wa/TWS/jdbcdrivers/db2/</jsdldatabase:driverPath>
===============================================================
= Exit Status : 0
Note that in this case the jdbcdrivers/db2 is not needed. It will search for subdirectories.
When using the COPY command to load Amazon Redshift with a manifest, suppose one of the files contains an error.
Is there a way to just log the error for that file, but continue loading the other files?
The manifest file indicates whether a file is mandatory and whether an error should be generated if a file is not found. (Using a Manifest to Specify Data Files)
The COPY command will retry if it cannot read a file. (Errors When Reading Multiple Files)
The COPY command can specify a MAXERRORS parameter that permits a certain number of errors before the COPY command fails. (MAXERROR)
When loading data from files, Amazon Redshift will report any errors in the STL_LOAD_ERRORS table. (STL_LOAD_ERRORS)
As said above, the maxerror property should satisfy the above requirement.
In addition, copy-noload property checks the validity of the data without loading. Running with NOLOAD parameter is much faster as it only parses the file