I want to add param date in my controlm job using descriptor.
How should i do please??
Below my attempt
##TASKTYPE:Command
##CMDLINE: sh /applis/file.ksh -d ”%%ODATE”
##PARAM:%%PARM1=ODATE
Related
I don't have much experience with PowerShell and this simple issue has been driving me up the wall. I'm hoping someone can point me in the right direction.
I have a CSV-file with IP-range values
I wish to pass these IP values as a parameter to a Bicep template
The parameter is of type array, see code snippets below
CSV-file:
IP,Comment
10.0.0.1, Comment blabla
10.0.0.52, Comment more blabla
I wish to pass the IP-values into a Azure Bicep template with the following parameter:
param ipArray array
The cli command is as follows:
az deployment group validate -g test-rg -f .\main.bicep -p ipArray=$ipRange
I am unable to populate $ipRange properly. I have tested the following and know it works:
az deployment group validate -g test-rg -f .\main.bicep -p ipArray="['10.0.0.1','10.0.0.52']"
So I need to figure out how to build my Powershell variable according to above syntax
$ipRange = ((Get-Content .\ip_list.csv) | ConvertFrom-Csv).IP
Failed to parse string as JSON:
10.0.0.1 10.0.0.52
Error detail: Extra data: line 1 column 6 (char 5)
Any nudge in the right direction will be greatly appreciated
Thanks!
This code will convert the ip range as you asked for:
$ipRange = ((Get-Content C:\Temp\ip.csv) | ConvertFrom-Csv).IP | ConvertTo-Json
$ipRange = $ipRange.ToString() -replace '"',"'"
$ipRange
one final thing, in your param, it is mentioned as vlkIpArray and in deployment it is mentioned as ipArray. is this a typo error?
param vlkIpArray array and
az deployment group validate -g test-rg -f .\main.bicep -p ipArray="['10.0.0.1','10.0.0.52']"
I have a requirement wherein I am checking if source file which is in CSV format has only header rows i.e. 1 row only then I need to fail the informatica workflow. Informatica is installed in Windows server so only Command task is supported not Unix or bash.
I am using below code to count lines in source file using Command Task in workflow.
for /f "usebackq" %%b in (type $$outputfile ^| find "" /v /c)do (
echo line count is %%b> $$count_file.txt
)
)
Here $$outfile and $$count_file paths and filenames are picked from param files.
There is an ABORT() function that you can use in expression transformation.
Create a dummy column and put a sorter, aggregator right after the source qualifier. in the aggregator, get a count of all data and then join it back to the main flow. After joiner, put an expression transformation with below condition-
IIF( cnt_all > 1, NULL, ABORT( 'Only header exists in the input file! Session will be aborted.'))
Whole mapping should look like this --
SQ -- EXP(add dummy_col) -->SRT on dummy_col -->AGG on dummy_col, calculate Count(*)->|
|--------------------------------------------> JNR on dummy_col -->EXP (abort if count <=1) --> existing mapping logic...
EDIT :
From command task, you can call pmcmd abortworkflow when your condition satisfies. Normal syntax is below -
pmcmd abortworkflow -service service -user username -password password -f folder workflow
I need assistance with understanding why my hivevar is not being set in my query?
This is my beeline statement in a shell script:
Start_Date="20180423"
End_Date="20180424"
beeline -u 'jdbc:hive2://#####/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=######;' -f ${my_queries}/Count_Query --showHeader=false --outputformat=csv2 --silent=false --hivevar start_date=$Start_Date --hivevar end_date=$End_Date 1>${my_data}/Data_File 2>${my_log}/Log_File
The Query
use sample_db;
select count(*) from sample_table where data_dt>=${start_date} and data_dt<${end_date};
When I look at the data file, which provides a dump of the query, the variables are not properly set to the values.
0: jdbc:hive2://####> use sample_db;
0: jdbc:hive2://####> select count(*) from sample_table where data_dt>=${start_date} and data_dt<${end_date};
The issue is following part
**--hivevar start_date=$Start_Date --hivevar end_date=$End_Date**
Remove ** and you are good to go.
Shell Script.
Start_Date="20180423"
End_Date="20180424"
beeline_cmd="beeline -u 'jdbc:hive2://#####/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=######;' --showHeader=false --outputformat=csv2 --silent=false"
${beeline_cmd} -f ${my_queries}/Count_Query --hivevar start_date=${Start_Date} --hivevar end_date=${End_Date} 1>${my_data}/Data_File 2>${my_log}/Log_File
Hive Query
I'm using Buildbot V.0.9.0rc3
My Buildbot triggers when I send a change via command line or if I receieve an http Post request to the correct address.
Currently I'm sending changes to Buildbot in two different ways:
$ buildbot sendchange -m localhost:9999 -a example-user:pass -W me -C default
or
curl -X POST -d author=aalvz -d comments=mycomment -d project=my_project -d category=default -d repository=some http://192.168.33.20:8020/change_hook/base
My schedulers are defined like this:
c['schedulers'].append(schedulers.SingleBranchScheduler(
name="waiter",
builderNames=["runtests"],
change_filter=util.ChangeFilter(category='default')))
c['www'] = dict(port=8020,
plugins=dict(waterfall_view={}, console_view={}),
change_hook_dialects={
'base': True,
'somehook': {'option1':True,
'option2':False}})
And my Step in factory cloning a repo looks like this:
factory.addStep(steps.Git(repourl='git#github.com:AAlvz/my_repo.git', mode='full', workdir='newFolder', branch='my_branch', submodules=True, clobberOnFailure=True))
I would like to receive a POST with some data and use that data to trigger different commands. Something like: (using $ to make the variables noticeable)
factory.addStep(steps.Git(repourl=$myjson.name, mode='full', workdir=$myjson.path, branch=$myjson.branch, submodules=True, clobberOnFailure=True))
That way I could send a JSON like:
{myjson: {name: github/myrepo.git, path: /tmp/my/path, branch: my_branch}}
and be able to clone the repository provided by the JSON.
Thanks in advance! I hope the question is clear enough. I can provide with logs or any needed configuration. Thanks!
This is solved Using Buildbot Properties.
You cand send them via command line (with PBChangeSource) using the flag
buildbot sendchange ... --properties=my_property:myvalue
The flag can be used multiple timpes if multiple properties are needed.
We have a data set of 15k classified tweets with which we need to perform sentiment analysis. I would like to test against a test set of 5k classified tweets. Due to Weka needing the same attributes within the header of the test set as exist in the header of training set, I will have to use batch filtering if I want to be able to run my classifier against this 5k test set.
However, there are several filters that I need to run my training set through, so I figured the running a multifilter against the training set would be a good idea. The multifilter works fine when not running the batch argument, but when I try to batch filter I get an error from the CLI as it tried to execute the first filter within the multi-filter:
CLI multiFilter command w/batch argument:
java weka.filters.MultiFilter -F "weka.filters.supervised.instance.Resample -B 1.0 -S 1 -Z 15.0 -no-replacement" \
-F "weka.filters.unsupervised.attribute.StringToWordVector -R first-last -W 100000 -prune-rate -1.0 -N 0 -S -stemmer weka.core.stemmers.NullStemmer -M 2 -tokenizer weka.core.tokenizers.AlphabeticTokenizer" \
-F "weka.filters.unsupervised.attribute.Reorder -R 2-last,first"\
-F "weka.filters.supervised.attribute.AttributeSelection -E \"weka.attributeSelection.InfoGainAttributeEval \" -S \"weka.attributeSelection.Ranker -T 0.0 -N -1\"" \
-F weka.filters.AllFilter \
-b -i input\Train.arff -o output\Train_b_out.arff -r input\Test.arff -s output\Test_b_out.arff
Here is the resultant error from the CLI:
weka.core.UnassignedClassException: weka.filters.supervised.instance.Resample: Class attribute not set!
at weka.core.Capabilities.test(Capabilities.java:1091)
at weka.core.Capabilities.test(Capabilities.java:1023)
at weka.core.Capabilities.testWithFail(Capabilities.java:1302)
at weka.filters.Filter.testInputFormat(Filter.java:434)
at weka.filters.Filter.setInputFormat(Filter.java:452)
at weka.filters.SimpleFilter.setInputFormat(SimpleFilter.java:195)
at weka.filters.Filter.batchFilterFile(Filter.java:1243)
at weka.filters.Filter.runFilter(Filter.java:1319)
at weka.filters.MultiFilter.main(MultiFilter.java:425)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at weka.gui.SimpleCLIPanel$ClassRunner.run(SimpleCLIPanel.java:265)
And here are the headers with a portion of data for both the training and test input arffs:
Training:
#RELATION classifiedTweets
#ATTRIBUTE ##sentence## string
#ATTRIBUTE ##class## {1,-1,0}
#DATA
"Conditioning be very important for curly dry hair",0
"Combine with Sunday paper coupon and",0
"Price may vary by store",0
"Oil be not really moisturizers",-1
Testing:
#RELATION classifiedTweets
#ATTRIBUTE ##sentence## string
#ATTRIBUTE ##class## {1,-1,0}
#DATA
"5",0
"I give the curl a good form and discipline",1
"I have be cowashing every day",0
"LOL",0
"TITLETITLE Walgreens Weekly and Midweek Deal",0
"And then they walk away",0
Am I doing something wrong here? I know that supervised resampling requires the class attribute to be on the bottom of the attribute list within the header, and it is... within both the test and training input files.
EDIT:
Further testing reveals that this error does not occur with relationship to the batch filtering, it occurs whenever I run the supervised resample filter from the CLI... The data that I use works on every other filter I've tried within the CLI, so I don't understand why this filter is any different... resampling the data in the GUI works fine as well...
Update:
This also happens with the SMOTE filter instead of the resample filter
Could not get the batch filter to work with any resampling filter. However, our workaround was to simply resample (and then randomize) the training data as step 1. From this reduced set we ran batch filters for everything else we wanted on the test set. This seemed to work fine.
You could have used the multifilter along with the ClassAssigner method to make it work:
java -classpath $jcp weka.filters.MultiFilter
-F "weka.filters.unsupervised.attribute.ClassAssigner -C last"
-F "weka.filters.supervised.instance.Resample -B 1.0 -S 1 -Z 66.0"