Using JobExecutionDecider with createOptionFlowUsingSteps - spring-batch

I'm using createOptionFlowUsingSteps in my JobConfiguration and I've got 2 flows defined.
The first flow should always execute
If the first flow produces an empty output file, the job should end
If the file has contents, the second flow should begin
PROBLEM: the job is ending if the file is empty but if the file has contents, the job repeats the first flow and never performs the second flow.
Config
As you can see, I have startSteps and endSteps. Thank you for the help!
List<Step> startSteps = Arrays.asList( //
shp294SetupStep, // 0
deleteFileStep, // 1
callM204Step// 2
);
List<Step> endSteps = Arrays.asList( //
putFtpStep, // 0
cdtx500SetupStep, // 1
getFtpStep, // 2
callCobolStep, // 3
callSystemwareStep, //4
cdtx510SetupStep, // 5
getFtpStep, // 6
callCobolStep, // 7
putFtpStep// 8
);
List<Integer> emptyFileRequiredSteps = Arrays.asList(0);
List<Integer> requiredSteps = Arrays.asList(1, 5);
Flow startStepFlow = shpcdwrdStepHelper.createOptionalFlowUsingSteps(startSteps, emptyFileRequiredSteps);
Flow endStepFlow = shpcdwrdStepHelper.createOptionalFlowUsingSteps(endSteps, requiredSteps);
return jobFactory.get(JOB_NAME) //
.incrementer(new RunIdIncrementer()) //
.listener(new JobResultListener()) //
.start(startStepFlow) //
.next(optionalFlowDecider()).on(OptionalFlowDecider.ON_EMPTY_FILE).end() //
.from(optionalFlowDecider()).on(FlowExecutionStatus.COMPLETED.getName()).to(endStepFlow)
.build() //
.build(); //

In your flow definition, you should use the flow method to start your flow and define all outcomes from each transition, something like:
return jobFactory.get(JOB_NAME) //
.incrementer(new RunIdIncrementer()) //
.listener(new JobResultListener()) //
.flow(startStepFlow) //
.on(ExitStatus.COMPLETED.getExitCode()).to(optionalFlowDecider())
.from(optionalFlowDecider()).on(OptionalFlowDecider.ON_EMPTY_FILE).end() //
.from(optionalFlowDecider()).on(FlowExecutionStatus.COMPLETED.getName()).to(endStepFlow)
.build() //
.build(); //

Related

Extend HBase Put to avoid original Row Check in add method

HBase Need to export data from one cluster and import it to another with slight modification in row key
As I have referred in above post, need to export the HBase data of table from one cluster and import it into the another cluster by changing row key based on our match pattern
In the "org.apache.hadoop.hbase.mapreduce.Import" we have option to change the ColumnFamily using the args "HBASE_IMPORTER_RENAME_CFS"
I have slightly modified the Import code to support row key change.My code is available in Pastebin
https://pastebin.com/ticgeBb0
Changed the row key using the below code.
private static Cell convertRowKv(Cell kv, Map<byte[], byte[]> rowkeyReplaceMap) {
if (rowkeyReplaceMap != null) {
byte[] oldrowkeyName = CellUtil.cloneRow(kv);
String oldrowkey = Bytes.toString(oldrowkeyName);
Set<byte[]> keys = rowkeyReplaceMap.keySet();
for (byte[] key : keys) {
if (oldrowkey.contains(Bytes.toString(key))) {
byte[] newrowkeyName = rowkeyReplaceMap.get(key);
ByteBuffer buffer = ByteBuffer.wrap(oldrowkeyName);
buffer.get(key);
ByteBuffer newbuffer = buffer.slice();
ByteBuffer bb = ByteBuffer.allocate(newrowkeyName.length + newbuffer.capacity());
byte[] newrowkey = bb.array();
kv = new KeyValue(newrowkey, // row buffer
0, // row offset
newrowkey.length, // row length
kv.getFamilyArray(), // CF buffer
kv.getFamilyOffset(), // CF offset
kv.getFamilyLength(), // CF length
kv.getQualifierArray(), // qualifier buffer
kv.getQualifierOffset(), // qualifier offset
kv.getQualifierLength(), // qualifier length
kv.getTimestamp(), // timestamp
KeyValue.Type.codeToType(kv.getTypeByte()), // KV
// Type
kv.getValueArray(), // value buffer
kv.getValueOffset(), // value offset
kv.getValueLength()); // value length
}
}
}
return kv;
}
Executed the Import
hbase org.apache.hadoop.hbase.mapreduce.ImportWithRowKeyChange -DHBASE_IMPORTER_RENAME_ROW=123:123456 import file:///home/nshsh/export/
The row key has been successfully changed. But while put the Cell in the HBase table, using
"org.apache.hadoop.hbase.client.Put.add(Cell)" we have check as
"the row of the kv is the same as the put as we are changing row key"
Here it fails.
Then I have commented the check in Put class and updated the hbase-client.jar. Also I have tried to write HBasePut which extends Put
public class HBasePut extends Put {
public HBasePut(byte[] row) {
super(row);
// TODO Auto-generated constructor stub
}
public Put add(Cell kv) throws IOException{
byte [] family = CellUtil.cloneFamily(kv);
System.err.print(Bytes.toString(family));
List<Cell> list = getCellList(family);
//Checking that the row of the kv is the same as the put
/*int res = Bytes.compareTo(this.row, 0, row.length,
kv.getRowArray(), kv.getRowOffset(), kv.getRowLength());
if (res != 0) {
throw new WrongRowIOException("The row in " + kv.toString() +
" doesn't match the original one " + Bytes.toStringBinary(this.row));
}*/
list.add(kv);
familyMap.put(family, list);
return this;
}
}
In the Mapreduce, the task always fails with the below exception
2020-07-24 13:37:15,105 WARN [htable-pool1-t1] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period"
2020-07-24 13:37:15,122 INFO [LocalJobRunner Map Task Executor #0] client.AsyncProcess: , tableName=import
2020-07-24 13:37:15,178 INFO [htable-pool1-t1] client.AsyncProcess: #2, table=import, attempt=18/35 failed=7ops, last exception: org.apache.hadoop.hbase.client.WrongRowIOException: org.apache.hadoop.hbase.client.WrongRowIOException: The row in \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00/vfrt:con/1589541180643/Put/vlen=225448/seqid=0 doesn't match the original one 123_abcf
at org.apache.hadoop.hbase.client.Put.add(Put.java:330)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toPut(ProtobufUtil.java:574)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:744)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2196)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
I don't know where the old Put Class has been referred in the task.
Can someone please help to fix this.

How to implement tLoop in Talend?

I'm new to Talend and need an example job to implement tLoop. I want to run a job 10 times if it fails. I've looked at the documents, but I can't seem to figure this out.
This answer has 2 sections
Creating a loop with tJava
Retying a failed connection to a data source 5 times (with adding tJavaFlex)
___________________________________
SECTION 1 : Creating a loop with tJava
-----------------------------------------------------------
I just write a tJava component and then iterate to false.
Like this
Step 1: create a context variable
Step 2: write some java code in tJava (tJava1)
// setting loop flag
context.continueLooping = true;
//log.info("Starting job...");
then connect On Component Ok
Step 3: Create the tLoop
in the loop condition put your context context.continueLooping
which should be true by the first iteration.
then iterate
to the next tJava (tJava2)
if ( ((Integer)globalMap.get("tLoop_1_CURRENT_ITERATION")) == 1)
{
// code
}
else if(((Integer)globalMap.get("tLoop_1_CURRENT_ITERATION")) == 2)
{
// code
}
else if (((Integer)globalMap.get("tLoop_1_CURRENT_ITERATION")) == 3)
{
// code
}
else if (((Integer)globalMap.get("tLoop_1_CURRENT_ITERATION")) == 4)
{
// code
}
else if (((Integer)globalMap.get("tLoop_1_CURRENT_ITERATION")) == 5)
{
// code
context.continueLooping = false;
// log.info("DONE");
}
else
{
context.continueLooping = false;
// log.error("out of bounds...");
}
this tJava runs different code for each iteration till it reaches 5
I use this area to count stuff and load value to other contexts and more.
Then it runs the nest part n times till the context value is set to false.
___________________________
SECTION 2 : TO Retry Failed Connections
___________________________
if you need to retry a DB connection.
add a tJavaFlex between tLoop1 and tJava2 like so
and add the following code in the 3 sections
Start:
// start part of your Java code
try{
Main:
// here is the main part of the component,
// a piece of code executed in the row
// loop
if ( ((Integer)globalMap.get("tLoop_1_CURRENT_ITERATION")) > 1)
{
Thread.sleep(10000);
}
End:
// end of the component, outside/closing the loop
}catch (Exception e) {
if ( ((Integer)globalMap.get("tLoop_1_CURRENT_ITERATION")) > 5)
{
context.continueLooping = false;
}
else
{
System.out.println("Connection failed. Retrying...next");
}
}
and add On Component Ok tJava with the code to stop looping on the success (tJava3)
context.continueLooping = false;

How to use simulation time to triger action in modules?

I want to create simulation with server and 2 nodes. Nodes have defined vector that contain times for turn on/off.(example timersOnOff = 5,10,13,25 … nod will turn on in 5th second of beginning simulation, and then will be shutdown in 10th seconds etc). How to trigger action at certain time to send msg to node to "turn on" or "turn off".?
Let's assume that these times are written in timersOnOff declared as:
std::vector<simtime_t> timersOnOff;
In initialize() add the following code:
for (int i = 0; i < timersOnOff.size(); i = i + 2) {
simtime_t timeOn = timersOnOff[i];
simtime_t timeOff = timersOnOff[i+1];
cMessage * msgOn = new cMessage("nodeOn"); // (1)
cMessage * msgOff = new cMessage("nodeOff"); // (2)
scheduleAt (timeOn, msgOn);
scheduleAt (timeOff, msgOff);
}
The above code schedules all ON and OFF events.
Then, in handleMessage() add:
if (msg->isSelfMessage()) {
if (msg->isName("nodeOn")) { // the same name as in (1)
delete msg;
// turning on the node
} else if (msg->isName("nodeOff")) { // the same name as in (2)
delete msg;
// turning off the node
}
} else {
// ...
}

DataProvider in TestNG to pass data from excel

I started learning Selenium2 (WebDriver) with Eclipse and TestNG. I have a question about DataProvider. I have a login page having user, password and login button for example. I have written a test in TestNG. I have used pageobject for UI objects (have separate class) and actual test in another class.
Here glogin is a class and login is the function where finding elements and sending keys is done and this is called in another class gtest(which is main test) which has TestNG annotations.
I access that class in main script which will take values.
#test(Dataprovide = "test")
public void glogin(String user, String pass)
{
glogin log1 = new login;
log1.login(user,pass);
}
I have the following excel sheet
user pass
John Smith
Carson Black
Carla ck23
test test4
When I use dataprovider and get data from the excel sheet as array and use it in Test then the following error is displayed:
org.testng.TestNGException:
The data provider is trying to pass 4 parameters but the method plus.gmail#glogin takes 2
at org.testng.internal.Invoker.injectParameters(Invoker.java:1337)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1225)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:128)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
at org.testng.TestRunner.privateRun(TestRunner.java:767)
at org.testng.TestRunner.run(TestRunner.java:617)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
at org.testng.SuiteRunner.run(SuiteRunner.java:240)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1203)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1128)
at org.testng.TestNG.run(TestNG.java:1036)
at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)
Any help is really appreciated.
Here is code for method annotated with Dataprovider and
#DataProvider(name="test")
public Object[][] createdata1()throws Exception
{
Object[][] retobj = getexcel();
return retobj;
}
private String[][] getexcel() throws Exception
{
// TODO Auto-generated method stub
String[][] tabarray = null;
try {
Workbook wb1 = Workbook.getWorkbook(new
File("F:/testdata.xls"));
Sheet sheet = wb1.getSheet("userlogin");
Cell tablestart = sheet.findCell("login");
int startrow = tablestart.getRow();
int startcol = tablestart.getColumn();
Cell tableend = sheet.findCell("login",startcol+1,startrow+1,
100, 64000, false);
int endrow = tableend.getRow();
int endcol = tableend.getColumn();
System.out.println("startRow="+startrow+", endRow="+endrow+",
" + "startCol="+startcol+", endCol="+endcol);
tabarray = new String[endrow - startrow + 1][endcol -
startcol + 1];
int ci = 0;
for(int i = startrow +1 ;i<endrow;i++,ci++)
{
int cj = 0;
for(int j = startcol + 1;j<endcol;j++,cj++)
{
tabarray[ci][cj] = sheet.getCell(j,
i).getContents();
System.out.println(tabarray[ci][cj]);
}
}
} catch (BiffException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return tabarray;
}
test(Dataprovider = "test")
public void glogins(String user, String pass)
{
glogin log1 = new glogin(driver);
log1.login(user,pass);
}
When I executed the test, I received data from excel as
john
smith
carson
Black
carla
ck23
test
test4
as output
Isn't the error message self-explanatory?
The data provider is trying to pass 4 parameters but the method plus.gmail#glogin takes 2
Use a debugger and figure out why your data provider is returning 4 parameters instead of just 2.
Use
tabarray = new String[endrow - 1][endcol - 1]; instead of //tabarray = new String[endrow - startrow + 1][endcol - startcol + 1];
Because you intended to return only 2*2 array
Replace the line
tabarray = new String[endrow - startrow + 1][endcol - startcol + 1];
with
tabarray = new String[endrow - startrow - 1][endcol - startcol - 1];
I agree with the response posted by Kalyan; the reason being the first and last column of the data provider excel sheet is also counted in as arguments / parameters by the function; therefore please use tabArray=new String[endRow-startRow-1][endCol-startCol-1]; Hope this helps and happy testing...

Drools: Having trouble with drools event processing

I am quite new to drools.
I am working on an application where my drools engine will get a series of event every second. I need to see if all the events in last 10 seconds has attribute value below 10, if the condition is true, I have to do some processing. Here is the example code which I tried, Please help me understand and resolve the issue.
My Rule file.....
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
declare Employee
#role (event)
#expires(10s)
end
// Using timer to ensure rule processing starts only after 10 secs,
//else processing starts as soon as first event comes in
rule "Test Timer"
no-loop true
10timer(int: 5s)
when
$E : Employee()
$total : Number( doubleValue < 1 )
from accumulate( Employee( Age > 10 ), count() )
then
System.out.println( $E.getName() + " is crossing the threshold of 20");
retract($E);
end
And Main class
// import classes removed from here...
public class MainClass {
/**
* #param args
*/
public static void main(String[] args){
//Create KnowledgeBase...
KnowledgeBase knowledgeBase = createKnowledgeBase();
//Create a stateful session
StatefulKnowledgeSession session = knowledgeBase.newStatefulKnowledgeSession();
// KnowledgeRuntimeLogger logger = KnowledgeRuntimeLoggerFactory.newConsoleLogger(session);
try {
// Using random generator to simulate the data.
int randomInt=0;
Random randomGenerator = new Random();
DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
Date date = null;
while (true) {
Thread.sleep(1000);
date = new Date();
randomInt = randomGenerator.nextInt(12);
//Create Facts and insert them
Employee emp = new Employee();
emp.setName("Anurag" + randomInt);
emp.setAge(randomInt);
//LOAD THE FACT AND FIREEEEEEEEEEEEEEEEEEE............
System.out.println(dateFormat.format(date)+ " => Random no " + randomInt);
session.insert(emp);
session.fireAllRules();
}
} catch (Exception e) {
e.printStackTrace();
}finally {
session.dispose();
}
}
/**
* Create new knowledge base
*/
private static KnowledgeBase createKnowledgeBase() {
KnowledgeBuilder builder = KnowledgeBuilderFactory.newKnowledgeBuilder();
//Add drl file into builder
File drl = new File("D:\\eclipse\\worspace\\Research\\misc\\testforall.drl");
builder.add(ResourceFactory.newFileResource(drl), ResourceType.DRL);
if (builder.hasErrors()) {
System.out.println(builder.getErrors().toString());
//throw new RuntimeException(builder.getErrors().toString());
}
KnowledgeBase knowledgeBase = KnowledgeBaseFactory.newKnowledgeBase();
//Add to Knowledge Base packages from the builder which are actually the rules from the drl file.
knowledgeBase.addKnowledgePackages(builder.getKnowledgePackages());
KnowledgeBaseConfiguration config = KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
config.setOption( EventProcessingOption.STREAM );
return knowledgeBase;
}
}
public class Employee {
private String Name;
private int Age;
// getter - setters
}
did you check the Drools Fusion documentation?
First of all, Employee doesn't sounds as a good idea for an Event. Events are meaningful changes of states of something related with your domain. So, an event could be the time of arrival of an Employee to the office, or the time of departure, but the Employee itself is a domain entity (or a fact for the rule engine) more than an event.
If you are interested in using Drools fusion temporal operators I recommend you to read about sliding windows (temporal ones) which will allow you to see what happen in the last ten seconds all the time. You don't need to use timers for that.
Cheers
You forgot telling what happened when you ran it, if you did.
If your entities set is not very large, I think this problem can be solved very easily with the base Drools distribution.
I have a similar app to yours and works for me:
rule "Clear only auxiliar fact"
salience 1
when
af: AuxFact()
then
DroolsRepository.retractFact(af);
end
rule "Clear auxiliar fact and an old meaningful fact"
salience 1000
when
af: AuxFact()
mf: MeaningfulFact()
then
if(DroolsRepository.getCurrentTimeMillis() - tmf.getCreationDate().getTime() > 5000){
DroolsRepository.retractFact(af);
DroolsRepository.retractFact(mf);
// YOUR MEANINGFUL CODE
}
else{
DroolsRepository.retractFact(af);}
end
query "getAllFacts"
$result: Fact()
end
and
// Boot rules re-executing thread.
new Thread(new Runnable(){
public void run(){
do{
kSession.insert(new AuxFact());
try{
Thread.sleep(99);}
catch(InterruptedException e){
e.printStackTrace();}}
while(true);}
}).start();
A similar approach could be simpler and effective.