I'm using Talend Open Studio for ESB ver.6.3.1 and created the jobs which is used to pull the data from ALM to MongoDB.In that i used tRESTClient with query parameter(date) for pulling data from ALM & tMongoDBOutput for inserting ALM data. After that i've build the job & imported into eclipse as a java project. I tried to run the program with option of 'Run as Java Application'.It is working fine.
The above job i gave query parameter value directly like 'tRESTClient --> Basic settings --> Query parameters --> name = "query" & value = "{last-modified[>=(2017-04-19 13:02:15)]}" ', so this job will pull the records based on the query parameters value.
Now i generated the eclipse talend job as a runnable jar file & trying to pass query parameter value from CMD as a parameter value.
How to pass query parameters value as parameter from CMD?
In windows command promote you can write the following command form.
for example if you have a jar file named test.jar and you have two parameters name & query, then you can pass the command as
javaw -jar test.jar -n 'name' -q 'your query'
Then you can handle these two parameters i.e name and query in your main method.
Related
I am using the below flow.
ODATA -> Blob storage (JSON)
JSON -> Snowflake table
Copy Data -> Copy Data - Lookup
Both copy data is working fine.
In the lookup (query), i have given. (Need to add 1 value in table, its a variant column)
Update T1 set source_json = object_insert(source_json,device_type,web_browser,TRUE);)
When i use the above query in snowflake database it works fine, the table has 25K rows.
When run from pipeline, it gives the below error.
Multiple SQL statements in a single API call are not supported; use one API call per statement instead.
Any suggestions please.
Some of the workarounds are provided below.
Execute multiple SQL files using SnowSql (command line utility) as described below:
snowsql -c cc -f file1.sql -f file2.sql -f file3.sql
Once we have downloaded and installed the snowsql tool, we can wrap up all our SQL queries in a single .sql file and call that file using bash.
For example, suppose that we have written all the queries which we would like to run around in a file named abc.sql stored in /tmp.
We can then run the following command:
snowsql -a enter_accountname -u enter_your_username -f /tmp/abc.sql
For reference:
Workaround for multiple sql statement in a single api call are not supported
Multiple single api call are not supported use one api call per statement instead
Thanks for the reply. Requirement got changed.
Our flow
Lookup1 -> Copy data -> Copy data > Lookup2
We passed the values from the lookup1 and ran the stored procedure.
I have a 1st Jenkins job which generates some report in the form of HTM (rows and columns). I have to fetch required values from the report and pass as input parameter for 2nd jenkin job. How can this be done?
Jenkins jobs have their "workspace" configured as a default inside /workspace/ on the server/slave where your are executing the job.
What you can do is generate the output and store it in "workspace" of 1st job and access this path in 2nd job.
Workspace is a default location where output of the job is placed. However, this is totally configurable.
If you only want some values from report and not entire report, get the values and store it in some property file and configure "build downstream job" to load parameters from the mentioned property file.
My use case is the following, I am building an Oozie Pipeline and i need to pass it an argument.
Indeed my spark job must receive a string date as an argument and it would be great to pass the argument to the Oozie Workflow in order to use it in the Spark Submit. Anyone got any idea ? I didn't find the answer on Google
Thanks
Create workflow.xml that references some variable inputDate
Create
file job.properties that defines default value for inputDate
Run
your job using CLI, overriding default value when is needed:
oozie job -run -config job.properties -DinputDate=2017-08-19
I would like to capture the output of some variables to be used elsewhere in the job using Jenkins Powershell plugin.
Is this possible?
My goal is to build the latest tag somehow and the powershell script was meant to achieve that, outputing to a text file would not help and environment variables can't be used because the process is seemingly forked unfortunately
Besides EnvInject the another common approach for sharing data between build steps is to store results in files located at job workspace.
The idea is to skip using environment variables altogether and just write/read files.
It seems that the only solution is to combine with EnvInject plugin. You can create a text file with key value pairs from powershell then export them into the build using the EnvInject plugin.
You should make the workspace persistant for this job , then you can save the data you need to file. Other jobs can then access this persistant workspace or use it as their own as long as they are on the same node.
Another option would be to use jenkins built in artifact retention, at the end of the jobs configure page there will be an option to retain files specified by a match (e.g *.xml or last_build_number). These are then given a specific address that can be used by other jobs regardless of which node they are on , the address can be on the master or the node IIRC.
For the simple case of wanting to read a single object from Powershell you can convert it to a JSON string in Powershell and then convert it back in Groovy. Here's an example:
def pathsJSON = powershell(returnStdout: true, script: "ConvertTo-Json ((Get-ChildItem -Path *.txt) | select -Property Name)");
def paths = [];
if(pathsJSON != '') {
paths = readJSON text: pathsJSON
}
I am using oozie to run my map-reduce job. I want to create the output file according to the date. But it takes date as a string and ends up printing instead of taking date as the value :
/user/skataria/geooutput/$(date +"%m%d%Y%H%M%S")
Here is the oozie properties file:
nameNode=hdfs://localhost:8020
jobTracker=localhost:8021
date=(date +"%m%d%Y%H%M%S")
oozie.wf.application.path=${nameNode}/services/advert/sid
inputDir=${nameNode}/user/${user.name}/geoinput/testgeo
outputDir=${nameNode}/user/${user.name}/geooutput/${date}
Somehow i cant have oozie as tag because my reputation is under 1500
It looks like you're trying to use a linux shell command (date +"%m%d%Y%H%M%S") in a java properties file - this isn't going to resolve.
One work around, assuming this is part of a manually submitted Workflow job (as opposed to a Coordinator job) is to provide the date property from the command line using the -D key=value option, and linux shell back quotes to resolve the output of a command inline
oozie job -run -config job.properties -D date=`date +"%m%d%Y%H%M%S"`
You'll need to make sure your version of Oozie support the -D key=value option
Yes I agree the shell option works. But that does not solve my usecase. I want to run my map-reduce job daily and schedule this thru Hue. The output directory needs to be parameterized as an job property to Oozie.
By the way I find that Oozie has Expression language Functions,
Unfortunately the function timestamp() returns the UTC current date and time in W3C format down to the second (YYYY-MM-DDThh:mm:ss.sZ). i.e.: 1997-07-16T19:20:30.45Z and completely unusable for creating a sub-directory name in HDFS
So for now,
I have a workaround. I am using the Workflow EL Function wf:id()
In workflow.xml
<property>
<name>mapreduce.output.fileoutputformat.outputdir</name>
<value>/user/sasubramanian/impressions/output/outpdir/${yyyy_mm_dd}/${wf:id()}</value>
</property>
This creates a output directory with subdirectory as,
/user/foouser/subdir1/output/outpdir/0000006-130321100837625-oozie-oozi-W
NOTE: You must specify this in the workflow.xml. This will not work if you specified it in the job.properties