While using Thread sleep on ParallelFlux its not waiting for sleep thread to get completed and executing onComplete() function - webclient

While using Parallel on Flux i am stopping thread for some time using thread sleep, but the problem is that flux not waiting till thread sleep time and executed on onComplete on subscribe.
List str = new ArrayList<>();
str.add("spring");
str.add("webflux");
str.add("example");
AtomicInteger num = new AtomicInteger();
ParallelFlux<Object> names = Flux.fromIterable(str)
.log()
.parallel(2)
.runOn(Schedulers.boundedElastic())
.map( s-> {
if(s.equalsIgnoreCase("webflux")) {
try {
System.out.println("waiting...");
Thread.sleep(1000);
System.out.println("done...");
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
return s+" "+num.incrementAndGet();
});
names.subscribe(s -> {
System.out.println("value "+s+" thread : "+Thread.currentThread().getName());
});
Output:
19:35:24.870 [main] INFO reactor.Flux.Iterable.1 - | onSubscribe([Synchronous Fuseable] FluxIterable.IterableSubscription)
19:35:24.896 [main] INFO reactor.Flux.Iterable.1 - | request(256)
19:35:24.897 [main] INFO reactor.Flux.Iterable.1 - | onNext(spring)
19:35:24.898 [main] INFO reactor.Flux.Iterable.1 - | onNext(webflux)
19:35:24.898 [main] INFO reactor.Flux.Iterable.1 - | onNext(example)
waiting...
value spring 1 thread : boundedElastic-1
value example 2 thread : boundedElastic-1
19:35:24.899 [main] INFO reactor.Flux.Iterable.1 - | onComplete()

Related

When TaskExecutor concurrencyLimit is less than the number of flow steps, job will be blocked

I'm using Spring Batch 4.1.2.RELEASE. I have a problem about Parallel Steps
(using Split Flow). When SplitFlow's concurrencyLimit (or ThreadPoolTaskExecutor.corePoolSize) is less than the number of A split flow's steps. the job never stops, and no Exception thrown.
I know the solution is to increase the concurrencyLimit or decrease the number of steps in each flow. But I want to make sure whether there is a problem with job's TaskExecutor and task's TaskExecutor or my code is wrong.
Without consideration of SplitFlow, I found that if if the number of Jobs (as simple as possible) submitted to jobLauncher is more than its TaskExecutor.corePoolSize(assume 1), the job will be executed one by one. This is the expected result.
#Bean
public TaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor executor = new SimpleAsyncTaskExecutor("tsk-Exec-");
executor.setConcurrencyLimit(2);
return executor;
}
#SpringBootApplication
#EnableBatchProcessing
#EnableWebMvc
public class BatchJobApplication {
public static void main(String[] args) {
SpringApplication.run(BatchJobApplication.class, args);
}
}
The code below create a single job, contains a split flow with 4 tasklet step.
#Autowired
private TaskExecutor taskExecutor;
public JobExecution experiment(Integer flowId) {
String dateFormat = LocalDate.now(ZoneId.of("+8")).format(DateTimeFormatter.BASIC_ISO_DATE);
JobBuilder job1 = this.jobBuilderFactory.get("Job_" + flowId + "_" + dateFormat);
List<TaskletStep> taskletSteps = Lists.newArrayList();
for (int i = 0; i < 4; i++) {
taskletSteps.add(this.stepBuilderFactory.get("step:" + i).tasklet(
(contribution, chunkContext) -> {
Thread.sleep(3000);
return RepeatStatus.FINISHED;
}).build());
}
JobExecution run = null;
FlowBuilder.SplitBuilder<SimpleFlow> splitFlow = new FlowBuilder<SimpleFlow>("splitFlow").split(taskExecutor);
FlowBuilder<SimpleFlow> lastFlowNode = null;
for (TaskletStep taskletStep : taskletSteps) {
SimpleFlow singleNode = new FlowBuilder<SimpleFlow>("async-fw-" + taskletStep.getName()).start(taskletStep).build();
lastFlowNode = splitFlow.add(singleNode);
}
Job build = job1.start(lastFlowNode.end()).build().build();
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder();
jobParametersBuilder.addDate("parameterGenerated", new Date());
try {
run = jobLauncher.run(build, jobParametersBuilder.toJobParameters());
} catch (JobExecutionAlreadyRunningException e) {
e.printStackTrace();
} catch (JobRestartException e) {
e.printStackTrace();
} catch (JobInstanceAlreadyCompleteException e) {
e.printStackTrace();
} catch (JobParametersInvalidException e) {
e.printStackTrace();
}
return run;
}
Now It's blocked.
2019-07-29 18:08:10.321 INFO 24416 --- [ job-Exec-1] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=Job_2124_20190729]] launched with the following parameters: [{parameterGenerated=1564394890193}]
2019-07-29 18:08:13.392 DEBUG 24416 --- [ job-Exec-1] cTaskExecutor$ConcurrencyThrottleAdapter : Entering throttle at concurrency count 0
2019-07-29 18:08:13.393 DEBUG 24416 --- [ job-Exec-1] cTaskExecutor$ConcurrencyThrottleAdapter : Entering throttle at concurrency count 1
2019-07-29 18:08:13.393 DEBUG 24416 --- [ tsk-Exec-2] cTaskExecutor$ConcurrencyThrottleAdapter : Concurrency count 2 has reached limit 2 - blocking
2019-07-29 18:08:13.425 INFO 24416 --- [ tsk-Exec-1] o.s.batch.core.job.SimpleStepHandler : Executing step: [step:3]
2019-07-29 18:08:16.466 DEBUG 24416 --- [ tsk-Exec-1] cTaskExecutor$ConcurrencyThrottleAdapter : Returning from throttle at concurrency count 1
2019-07-29 18:08:16.466 DEBUG 24416 --- [ tsk-Exec-2] cTaskExecutor$ConcurrencyThrottleAdapter : Entering throttle at concurrency count 1
2019-07-29 18:08:16.466 DEBUG 24416 --- [ tsk-Exec-2] cTaskExecutor$ConcurrencyThrottleAdapter : Concurrency count 2 has reached limit 2 - blocking
2019-07-29 18:08:16.484 INFO 24416 --- [ tsk-Exec-3] o.s.batch.core.job.SimpleStepHandler : Executing step: [step:2]
2019-07-29 18:08:19.505 DEBUG 24416 --- [ tsk-Exec-3] cTaskExecutor$ConcurrencyThrottleAdapter : Returning from throttle at concurrency count 1
2019-07-29 18:08:19.505 DEBUG 24416 --- [ tsk-Exec-2] cTaskExecutor$ConcurrencyThrottleAdapter : Entering throttle at concurrency count 1
2019-07-29 18:08:19.506 DEBUG 24416 --- [ tsk-Exec-4] cTaskExecutor$ConcurrencyThrottleAdapter : Concurrency count 2 has reached limit 2 - blocking
I think I have found the answer yesterday evening.
When The TaskExecutor in SplitFlow with a few concurrencyLimit or ThreadPoolTaskExecutor.corePoolSize. According to the code, It's very likely to happen that All the Thread is blocked by future.get(), but no Thread available has chance to run taskletStep.
//SplitState.java:114
results.add(task.get());
In addition, the Threads created by TaskExecutor in JobLauncher doesn't have to wait results of future. So TaskExcutor always have enough free Thread to accept jobs, no one need to wait any condition.

app is alive but stops consuming messages after a while

spring-kafka consumer stops consuming messages after a while. The stoppage happens every time, but never at the same duration. When app is no longer consuming, in the end of the log always I see the statement that consumer sent LEAVE_GROUP signal. If I am not seeing any errors or exceptions, why is the consumer leaving the group?
org.springframework.boot:spring-boot-starter-parent:2.0.4.RELEASE
spring-kafka:2.1.8.RELEASE
org.apache.kafka:kafka-clients:1.0.2
I've set logging as
logging.level.org.apache.kafka=DEBUG
logging.level.org.springframework.kafka=INFO
other settings
spring.kafka.listener.concurrency=5
spring.kafka.listener.type=single
spring.kafka.listener.ack-mode=record
spring.kafka.listener.poll-timeout=10000
spring.kafka.consumer.heartbeat-interval=5000
spring.kafka.consumer.max-poll-records=50
spring.kafka.consumer.fetch-max-wait=10000
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.properties.security.protocol=SSL
spring.kafka.consumer.retry.maxAttempts=3
spring.kafka.consumer.retry.backoffperiod.millisecs=2000
ContainerFactory setup
#Bean
public ConcurrentKafkaListenerContainerFactory<String, String> recordsKafkaListenerContainerFactory(RetryTemplate retryTemplate) {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory);
factory.setConcurrency(listenerCount);
factory.getContainerProperties().setAckMode(AbstractMessageListenerContainer.AckMode.RECORD);
factory.getContainerProperties().setPollTimeout(pollTimeoutMillis);
factory.getContainerProperties().setErrorHandler(new SeekToCurrentErrorHandler());
factory.getContainerProperties().setAckOnError(false);
factory.setRetryTemplate(retryTemplate);
factory.setStatefulRetry(true);
factory.getContainerProperties().setIdleEventInterval(60000L);
return factory;
}
Listener configuration
#Component
public class RecordsEventListener implements ConsumerSeekAware {
private static final org.slf4j.Logger LOG = org.slf4j.LoggerFactory.getLogger(RecordsEventListener.class);
#Value("${mode.replay:false}")
public void setModeReplay(boolean enabled) {
this.isReplay = enabled;
}
#KafkaListener(topics = "${event.topic}", containerFactory = "RecordsKafkaListenerContainerFactory")
public void handleEvent(#Payload String payload) throws RecordsEventListenerException {
try {
//business logic
} catch (Exception e) {
LOG.error("Process error for event: {}",payload,e);
if(isRetryableException(e)) {
LOG.warn("Retryable exception detected. Going to retry.");
throw new RecordsEventListenerException(e);
}else{
LOG.warn("Dropping event because non retryable exception");
}
}
}
private Boolean isRetryableException(Exception e) {
return binaryExceptionClassifier.classify(e);
}
#Override
public void registerSeekCallback(ConsumerSeekCallback callback) {
//do nothing
}
#Override
public void onPartitionsAssigned(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) {
//do this only once per start of app
if (isReplay && !partitonSeekToBeginningDone) {
assignments.forEach((t, p) -> callback.seekToBeginning(t.topic(), t.partition()));
partitonSeekToBeginningDone = true;
}
}
#Override
public void onIdleContainer(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) {
//do nothing
LOG.info("Container is IDLE; no messages to pull.");
assignments.forEach((t,p)->LOG.info("Topic:{}, Partition:{}, Offset:{}",t.topic(),t.partition(),p));
}
boolean isPartitionSeekToBeginningDone() {
return partitonSeekToBeginningDone;
}
void setPartitonSeekToBeginningDone(boolean partitonSeekToBeginningDone) {
this.partitonSeekToBeginningDone = partitonSeekToBeginningDone;
}
}
When app is no longer consuming, in the end of the log always I see the statement that consumer sent LEAVE_GROUP signal.
2019-05-02 18:31:05.770 DEBUG 9548 --- [kafka-coordinator-heartbeat-thread | app] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=app] Sending Heartbeat request to coordinator x.x.x.com:9093 (id: 2147482638 rack: null)
2019-05-02 18:31:05.770 DEBUG 9548 --- [kafka-coordinator-heartbeat-thread | app] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=app] Using older server API v0 to send HEARTBEAT {group_id=app,generation_id=6,member_id=consumer-1-98d28e69-b0b9-4c2b-82cd-731e53b74b87} with correlation id 5347 to node 2147482638
2019-05-02 18:31:05.872 DEBUG 9548 --- [kafka-coordinator-heartbeat-thread | app] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=app] Received successful Heartbeat response
2019-05-02 18:31:10.856 DEBUG 9548 --- [kafka-coordinator-heartbeat-thread | app] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=app] Sending Heartbeat request to coordinator x.x.x.com:9093 (id: 2147482638 rack: null)
2019-05-02 18:31:10.857 DEBUG 9548 --- [kafka-coordinator-heartbeat-thread | app] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=app] Using older server API v0 to send HEARTBEAT {group_id=app,generation_id=6,member_id=consumer-1-98d28e69-b0b9-4c2b-82cd-731e53b74b87} with correlation id 5348 to node 2147482638
2019-05-02 18:31:10.958 DEBUG 9548 --- [kafka-coordinator-heartbeat-thread | app] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=app] Received successful Heartbeat response
2019-05-02 18:31:11.767 DEBUG 9548 --- [kafka-coordinator-heartbeat-thread | app] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=app] Sending LeaveGroup request to coordinator x.x.x.com:9093 (id: 2147482638 rack: null)
2019-05-02 18:31:11.767 DEBUG 9548 --- [kafka-coordinator-heartbeat-thread | app] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=app] Using older server API v0 to send LEAVE_GROUP {group_id=app,member_id=consumer-1-98d28e69-b0b9-4c2b-82cd-731e53b74b87} with correlation id 5349 to node 2147482638
2019-05-02 18:31:11.768 DEBUG 9548 --- [kafka-coordinator-heartbeat-thread | app] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-1, groupId=app] Disabling heartbeat thread
Full log
Thanks to all who replied. Turns out, it was indeed the broker dropping the consumer on session timeout. The broker a very old version (0.10.0.1) did not accommodate the newer features as outlined in KIP-62 that spring-kafka version we used could make use of.
Since we could not dictate the upgrade to the broker or changes to session timeout, we simply modified our processing logic so as to finish the task under the session timeout.

Standalone Spark Task Hangs When Inserting Into DB

I have a standalone spark 1.4.1 job that runs on a Red Hat box I submit via spark-submit that sometimes hangs during insertion of data from an RDD. I have auto-commit on the connection turned off and commit the transactions in batches of insertions. What the logs show me before it hangs:
16/03/25 14:00:05 INFO Executor: Finished task 3.0 in stage 138.0 (TID 915). 1847 bytes result sent to driver
16/03/25 14:00:05 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(StatusUpdate(915,FINISHED,java.nio.HeapByteBuffer[pos=0 lim=1847 cap=1
16/03/25 14:00:05 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(StatusUpdate(915,FINISHED,java.nio.HeapByteBuffer[pos=0 lim=1847 cap=1847
16/03/25 14:00:05 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_138, runningTasks: 1
16/03/25 14:00:05 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.118 ms) AkkaMessage(StatusUpdate(915,FINISHED,java.nio.HeapByteBuffer[pos=621 li
16/03/25 14:00:05 INFO TaskSetManager: Finished task 3.0 in stage 138.0 (TID 915) in 7407 ms on localhost (23/24)
16/03/25 14:00:05 TRACE DAGScheduler: Checking for newly runnable parent stages
16/03/25 14:00:05 TRACE DAGScheduler: running: Set(ResultStage 138)
16/03/25 14:00:05 TRACE DAGScheduler: waiting: Set()
16/03/25 14:00:05 TRACE DAGScheduler: failed: Set()
16/03/25 14:00:10 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7ed52306,BlockManagerId(driver, local
16/03/25 14:00:10 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7ed52306,BlockManagerId(driver, localhos
16/03/25 14:00:10 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.099 ms) AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7ed52306,BlockManagerId(dri
And then it just repeats the last 3 lines with this intermittently:
16/03/25 14:01:04 TRACE HeartbeatReceiver: Checking for hosts with no recent heartbeats in HeartbeatReceiver.
The kicker is that I can't take a look at the web UI due to some firewall issues on these machines. What I noticed is that this issue was more prevalent when I was inserting with batches of 1000 than with 100. This is the scala code that looks to be the culprit.
//records should have up to INSERT_BATCH_SIZE entries
private def insertStuff(records: Seq[(String, (String, Stuff1, Stuff2, Stuff3))]) {
if (!records.isEmpty) {
//get statement used for insertion (instantiated in an array of statements)
val stmt = stuffInsertArray(//stuff)
logger.info("Starting insertions on stuff" + table + " for " + time + " with " + records.length + " records")
try {
records.foreach(record => {
//get vals from record
...
//perform sanity checks
if (//validate stuff)
{
//log stuff because it didn't validate
}
else
{
stmt.setInt(1, //stuff)
stmt.setLong(2, //stuff)
...
stmt.addBatch()
}
})
//check if connection is still valid
if (!connInsert.isValid(VALIDATE_CONNECTION_TIMEOUT))
{
logger.error("Insertion connection is not valid while inserting stuff.")
throw new RuntimeException(s"Insertion connection not valid while inserting stuff.")
}
logger.debug("Stuff insertion executing batch...")
stmt.executeBatch()
logger.debug("Stuff insertion execution complete. Committing...")
//commit insert batch. Either INSERT_BATCH_SIZE insertions planned or the last batch to be done
insertCommit() //this does the commit and resets some counters
logger.debug("stuff insertion commit complete.")
} catch {
case e: Exception => throw new RuntimeException(s"insertStuff exception ${e.getMessage}")
}
}
}
And here's how it gets called:
//stuffData is an RDD
stuffData.foreachPartition(recordIt => {
//new instance of the object of whose member function we're currently in
val obj = new Obj(clusterInfo)
recordIt.grouped(INSERT_BATCH_SIZE).foreach(records => obj.insertStuff(records))
})
All the extra logging and connection checking I put in just to isolate the issue but since I write for every batch of insertions, the logs get convoluted. If I serialize the insertions, the issue still persists. Any idea why the last task (out of 24) doesn't finish? Thanks.

Async Spring Batch job fails to process file

I'm trying to process a file, and upload it into a database, using spring batch, right after uploading it. However the job completes right after it's started, and I'm not too sure of the exact reason. I think it's not doing what it should, in the tasklet.execute. Below is the DEBUG output:
22:25:09.823 [http-nio-127.0.0.1-8080-exec-2] DEBUG o.s.b.c.c.a.SimpleBatchConfiguration$ReferenceTargetSource - Initializing lazy target object
22:25:09.912 [SimpleAsyncTaskExecutor-1] INFO o.s.b.c.l.support.SimpleJobLauncher - Job: [FlowJob: [name=moneyTransactionImport]] launched with the following parameters: [{targetFile=C:\Users\test\AppData\Local\Temp\tomcat.1435325122308787143.8080\uploads\test.csv}]
22:25:09.912 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.job.AbstractJob - Job execution starting: JobExecution: id=95, version=0, startTime=null, endTime=null, lastUpdated=Tue Sep 16 22:25:09 BST 2014, status=STARTING, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=52, version=0, Job=[transactionImport]], jobParameters=[{targetFile=C:\Users\test\AppData\Local\Temp\tomcat.1435325122308787143.8080\uploads\test.csv}]
22:25:09.971 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.job.flow.support.SimpleFlow - Resuming state=transactionImport.step with status=UNKNOWN
22:25:09.972 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.job.flow.support.SimpleFlow - Handling state=transactionImport.step
22:25:10.018 [SimpleAsyncTaskExecutor-1] INFO o.s.batch.core.job.SimpleStepHandler - Executing step: [step]
22:25:10.019 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.step.AbstractStep - Executing: id=93
22:25:10.072 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.scope.StepScope - Creating object in scope=step, name=scopedTarget.reader
22:25:10.117 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.scope.StepScope - Registered destruction callback in scope=step, name=scopedTarget.reader
22:25:10.136 [SimpleAsyncTaskExecutor-1] WARN o.s.b.item.file.FlatFileItemReader - Input resource does not exist class path resource [C:/Users/test/AppData/Local/Temp/tomcat.1435325122308787143.8080/uploads/test.csv]
22:25:10.180 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Starting repeat context.
22:25:10.181 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Repeat operation about to start at count=1
22:25:10.181 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.s.c.StepContextRepeatCallback - Preparing chunk execution for StepContext: org.springframework.batch.core.scope.context.StepContext#5d85b879
22:25:10.181 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.s.c.StepContextRepeatCallback - Chunk execution starting: queue size=0
22:25:12.333 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Starting repeat context.
22:25:12.333 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Repeat operation about to start at count=1
22:25:12.334 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Repeat is complete according to policy and result value.
22:25:12.334 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.s.item.ChunkOrientedTasklet - Inputs not busy, ended: true
22:25:12.334 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.core.step.tasklet.TaskletStep - Applying contribution: [StepContribution: read=0, written=0, filtered=0, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
22:25:12.337 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.core.step.tasklet.TaskletStep - Saving step execution before commit: StepExecution: id=93, version=1, name=step, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
22:25:12.358 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.repeat.support.RepeatTemplate - Repeat is complete according to policy and result value.
22:25:12.358 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.step.AbstractStep - Step execution success: id=93
22:25:12.419 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.step.AbstractStep - Step execution complete: StepExecution: id=93, version=3, name=step, status=COMPLETED, exitStatus=COMPLETED, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0
22:25:12.442 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.job.flow.support.SimpleFlow - Completed state=transactionImport.step with status=COMPLETED
22:25:12.443 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.job.flow.support.SimpleFlow - Handling state=transactionImport.COMPLETED
22:25:12.443 [SimpleAsyncTaskExecutor-1] DEBUG o.s.b.c.job.flow.support.SimpleFlow - Completed state=transactionImport.COMPLETED with status=COMPLETED
22:25:12.445 [SimpleAsyncTaskExecutor-1] DEBUG o.s.batch.core.job.AbstractJob - Job execution complete: JobExecution: id=95, version=1, startTime=Tue Sep 16 22:25:09 BST 2014, endTime=null, lastUpdated=Tue Sep 16 22:25:09 BST 2014, status=COMPLETED, exitStatus=exitCode=COMPLETED;exitDescription=, job=[JobInstance: id=52, version=0, Job=[transactionImport]], jobParameters=[{targetFile=C:\Users\test\AppData\Local\Temp\tomcat.1435325122308787143.8080\uploads\test.csv}]
22:25:12.466 [SimpleAsyncTaskExecutor-1] INFO o.s.b.c.l.support.SimpleJobLauncher - Job: [FlowJob: [name=transactionImport]] completed with the following parameters: [{targetFile=C:\Users\test\AppData\Local\Temp\tomcat.1435325122308787143.8080\uploads\test.csv}] and the following status: [COMPLETED]
My config is as follows:
#Configuration
#EnableBatchProcessing
public class BatchConfiguration {
#Inject
private TransactionRepository transactionRepository;
#Inject
private JobRepository jobRepository;
#Bean
#StepScope
public FlatFileItemReader<MoneyTransaction> reader(#Value("#{jobParameters[targetFile]}") String file) {
FlatFileItemReader<MoneyTransaction> reader = new FlatFileItemReader<>();
reader.setResource(new ClassPathResource(file));
reader.setLineMapper(new DefaultLineMapper<MoneyTransaction>() {
{
setLineTokenizer(new DelimitedLineTokenizer() {
{
setNames(new String[]{"Number", "Date", "Account", "Payee", "Cleared", "Amount", "Category", "Subcategory", "Memo"});
}
}
);
setFieldSetMapper(new BeanWrapperFieldSetMapper<MoneyTransaction>() {
{
setTargetType(MoneyTransaction.class);
}
});
}
}
);
reader.setStrict(false);
reader.setLinesToSkip(1);
return reader;
}
#Bean
public ItemProcessor<MoneyTransaction, Transaction> processor() {
return new TransactionProcessor();
}
#Bean
public RepositoryItemWriter writer() {
RepositoryItemWriter writer = new RepositoryItemWriter();
writer.setRepository(transactionRepository);
writer.setMethodName("save");
return writer;
}
#Bean
public Step step(StepBuilderFactory stepBuilderFactory, ItemReader<MoneyTransaction> reader,
ItemWriter<Transaction> writer, ItemProcessor<MoneyTransaction, Transaction> processor) {
return stepBuilderFactory.get("step")
.<MoneyTransaction, Transaction>chunk(100)
.reader(reader)
.processor(processor)
.writer(writer)
.build();
}
#Bean
public SimpleAsyncTaskExecutor taskExecutor() {
SimpleAsyncTaskExecutor executor = new SimpleAsyncTaskExecutor();
executor.setConcurrencyLimit(1);
return executor;
}
#Bean
public SimpleJobLauncher jobLauncher() {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(jobRepository);
jobLauncher.setTaskExecutor(taskExecutor());
return jobLauncher;
}
}
And I save the file, and start processing in the following way:
public JobExecution processFile(String name, MultipartFile file) {
if (!file.isEmpty()) {
try {
byte[] bytes = file.getBytes();
String rootPath = System.getProperty("catalina.home");
File uploadDirectory = new File(rootPath.concat(File.separator).concat("uploads"));
if (!uploadDirectory.exists()) {
uploadDirectory.mkdirs();
}
File uploadFile = new File(uploadDirectory.getAbsolutePath() + File.separator + file.getOriginalFilename());
BufferedOutputStream stream =
new BufferedOutputStream(new FileOutputStream(uploadFile));
stream.write(bytes);
stream.close();
return startImportJob(uploadFile, "transactionImport");
} catch (Exception e) {
logger.error(String.format("Error processing file '%s'.", name), e);
throw new MoneyException(e);
}
} else {
throw new MoneyException("There was no file to process.");
}
}
/**
* #param file
*/
private JobExecution startImportJob(File file, String jobName) {
logger.debug(String.format("Starting job to import file '%s'.", file));
try {
Job job = jobs.get(jobName).incrementer(new MoneyRunIdIncrementer()).flow(step).end().build();
return jobLauncher.run(job, new JobParametersBuilder().addString("targetFile", file.getAbsolutePath()).toJobParameters());
} catch (JobExecutionAlreadyRunningException e) {
logger.error(String.format("Job for processing file '%s' is already running.", file), e);
throw new MoneyException(e);
} catch (JobParametersInvalidException e) {
logger.error(String.format("Invalid parameters for processing of file '%s'.", file), e);
throw new MoneyException(e);
} catch (JobRestartException e) {
logger.error(String.format("Error restarting job, for processing file '%s'.", file), e);
throw new MoneyException(e);
} catch (JobInstanceAlreadyCompleteException e) {
logger.error(String.format("Job to process file '%s' has already completed.", file), e);
throw new MoneyException(e);
}
}
I'm kind of stumped at the minute, and any help would be greatly received.
Thanks.
Found the issue. The problem was with the type of resource ClassPathResource(file), combined with the fact that I was setting the strict property to false.
reader.setResource(new ClassPathResource(file));
I should have used
reader.setResource(new FileSystemResource(file));
Which makes complete sense, as I wasn't uploading the file as a class path resource.

JMeter - XMPP Authentication

I am building a test plan to test XMPP with JMeter. But I always meet an error when I send a authentication string to server even the authentication string is correct. Does anybody have the same problem or know how to fix this issue? Thanks.
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: Running the test!
2014/07/04 10:23:22 INFO - jmeter.gui.util.JMeterMenuBar: setRunning(true,*local*)
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: Starting ThreadGroup: 1 : Thread Group
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: Starting 1 threads for group Thread Group.
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: Thread will continue on error
2014/07/04 10:23:22 INFO - jmeter.threads.ThreadGroup: Starting thread group number 1 threads 1 ramp-up 1 perThread 1000.0 delayedStart=false
2014/07/04 10:23:22 INFO - jmeter.threads.ThreadGroup: Started thread group number 1
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: All thread groups have been started
2014/07/04 10:23:22 INFO - jmeter.threads.JMeterThread: Thread started: Thread Group 1-1
2014/07/04 10:23:22 ERROR - ru.yandex.jmeter.XMPPClientImpl: Error reading data java.lang.RuntimeException: Retries more than 1000, aborting read
at ru.yandex.jmeter.XMPPClientImpl.read(XMPPClientImpl.java:116)
at org.apache.jmeter.protocol.tcp.sampler.TCPSampler.sample(TCPSampler.java:414)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:428)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
at java.lang.Thread.run(Unknown Source)
2014/07/04 10:23:22 ERROR - jmeter.protocol.tcp.sampler.TCPSampler: java.lang.RuntimeException: Error reading data
at ru.yandex.jmeter.XMPPClientImpl.read(XMPPClientImpl.java:152)
at org.apache.jmeter.protocol.tcp.sampler.TCPSampler.sample(TCPSampler.java:414)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:428)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: Retries more than 1000, aborting read
at ru.yandex.jmeter.XMPPClientImpl.read(XMPPClientImpl.java:116)
... 4 more
2014/07/04 10:23:22 INFO - jmeter.threads.JMeterThread: Thread finished: Thread Group 1-1
2014/07/04 10:23:22 INFO - jmeter.engine.StandardJMeterEngine: Notifying test listeners of end of test
2014/07/04 10:23:22 INFO - jmeter.gui.util.JMeterMenuBar: setRunning(false,*local*)
I had tried using XMPPClientImpl plugin but always got the same error ("Retries more than 1000, aborting read"), so I decided to leave it and write my own code.
I use a BeanShell Sampler in which I run the following code (using smack library) to connect to XMPP server.
String CLASS_PATH = "C:/JMeter/apache-jmeter-2.13/lib/ext/smack/";
addClassPath(CLASS_PATH + "smack-android-extensions-4.1.3.jar");
addClassPath(CLASS_PATH + "smack-tcp-4.1.3.jar");
addClassPath(CLASS_PATH + "smack-android-4.1.3.jar");
// explicitly import every class you need
import org.jivesoftware.smack.XMPPConnection;
import org.jivesoftware.smack.ConnectionConfiguration;
import org.jivesoftware.smack.ConnectionListener;
import org.jivesoftware.smack.tcp.XMPPTCPConnection;
import org.jivesoftware.smack.tcp.XMPPTCPConnectionConfiguration;
import org.jivesoftware.smack.SmackException;
import org.jivesoftware.smack.XMPPException;
import org.jivesoftware.smack.XMPPException.XMPPErrorException;
String jabberId = "...";
String jabberPass = "...";
String SERVER_ADDRESS = "...";
int PORT = 5222; // or any other port
XMPPTCPConnection getConnection() {
XMPPTCPConnectionConfiguration config =
XMPPTCPConnectionConfiguration.builder()
.setUsernameAndPassword(jabberId, jabberPass)
.setHost(SERVER_ADDRESS)
.setServiceName(SERVER_ADDRESS)
.setPort(DEFAULT_PORT)
// .setSecurityMode(ConnectionConfiguration.SecurityMode.disabled)
.setSendPresence(true)
// .setDebuggerEnabled(YouMe.DEBUG)
.build();
XMPPTCPConnection con = new XMPPTCPConnection(config);
int REPLY_TIMEOUT = 50000; // 50 seconds, but can be shorter
con.setPacketReplyTimeout(REPLY_TIMEOUT);
return con;
}
Don't forget to add the smack path (e.g. C:\JMeter\apache-jmeter-2.13\lib\ext\smack) to the library field (under "Add directory or jar classpath") in the Test Plan node of your test plan.
To connect -
con = getConnection();
con.connect();
To login -
con.login(jabberId, jabberPass);
You can also add connection listener -
ConnectionListener listener = new ConnectionListener() {
public void connected(XMPPConnection xmppConnection) {
// run main code incl. the login code
runMain();
}
public void authenticated(XMPPConnection xmppConnection, boolean resumed) {
}
public void connectionClosed() {
}
public void connectionClosedOnError(Exception e) {
}
public void reconnectingIn(int i) {
}
public void reconnectionSuccessful() {
}
public void reconnectionFailed(Exception e) {
}
};
con.addConnectionListener(listener);
// connect
con.connect();
runMain() {
con.login(jabberId, jabberPass);
// ...
}