project reactor always runs in the main thread when the logs get prints - reactive-programming

I am new to project reactor, while I am studying the project reactor I observed one thing, always project rector logs get print as the main tread. if reactor runs without caring the thread, how that happens? and how I get verified reactive codes run on different different threads?

as default Reactor runs base on the thread that you subscribed. if you subscribe from your own thread, the reactor runs base on that thread. which means that not every time the main thread.
you can run this test code to verify that. here is the flux created on the main thread. and that flux subscribed from another thread that was newly created. after running the application, see the logs. the logs will prove it. the logs have been run on the new thread that we newly created.
public void testTheThread() throws InterruptedException {
//the flux created on the main thread.
Flux<String> stringFlux = Flux
.fromArray(new String[]{"a", "b"})
.map(String::toUpperCase)
.log();
//the subscriber runs on another new thread. [my-new-thread]
Thread newThread = new Thread(() -> {
stringFlux.subscribe(s -> {
System.out.println("String is = " + s);
});
});
newThread.setName("my-new-thread");
newThread.start();
//sleep the main thread until get the data from my-new-thread
//otherwise the log will not be printed.
Thread.sleep(1000);
}
the logs will be like this.
17:04:30.867 [my-new-thread] INFO reactor.Flux.MapFuseable.1 - | onSubscribe([Fuseable] FluxMapFuseable.MapFuseableSubscriber)
17:04:30.874 [my-new-thread] INFO reactor.Flux.MapFuseable.1 - | request(unbounded)
17:04:30.874 [my-new-thread] INFO reactor.Flux.MapFuseable.1 - | onNext(A)
s = A
17:04:30.880 [my-new-thread] INFO reactor.Flux.MapFuseable.1 - | onNext(B)
s = B
17:04:30.881 [my-new-thread] INFO reactor.Flux.MapFuseable.1 - | onComplete()
Process finished with exit code 0

Related

How to add custom shutdown hook to Kafka Connect?

Is there a way to add custom shutdown hook to kafka connect which I can put in the classpath using the plugins.path property?
Use Case:
The Kafka Connect cluster tries to connect to Kafka Cluster.
If it fails it logs and shutsdown immediately
The error logs does not reach to the remote log server like Splunk
I need to delay the shutdown so that the log collector agent can push the logs to the target log server.
I am using Confluent Platform v 6.1
The best way to accomplish what you are looking for is write the log file to a location that outlives the container. This could be a persistent volume like #OneCricketeer mentioned previously, or a network based logging service.
If for some reason you can't do that, you can create a JVM shutdown hook using a Java Agent. This hook could delay the JVM's shutdown long enough (risky) or it could force a flush of the logging library, or any other cleanup. Since the agent is configured as a JVM command line argument, it should work for your kafka-connect workers. You just need to modify the command line that runs the workers.
There are good instructions for creating a Java Agent here and an example for setting up a shutdown hook here: Java shutdown hook
Here is a super simple example class that has both the applications main() method and the Agent's premain() in the same class:
public class App
{
public static void main( String[] args ) throws InterruptedException {
System.out.println( System.currentTimeMillis() + ": Main Started!" );
Thread.sleep(1000);
System.out.println( System.currentTimeMillis() + ": Main Ended!" );
}
public static void premain(String agentArgs, Instrumentation inst) {
System.out.println(System.currentTimeMillis() + ": Agent Started");
Runtime.getRuntime()
.addShutdownHook(
new Thread() {
#Override
public void run() {
System.out.println(System.currentTimeMillis() + ": Shutdown Hook is running!");
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// do nothing
}
System.out.println(System.currentTimeMillis() + ": Shutdown hook is completed");
}
});
}
}
Note that in your case, you only need the premain method, as the main method is implemented by the connect worker.
Running the above class with the following command line:
java -javaagent:<MY_AGENT_FILE>.jar -classpath <MY_APP_FILE>.jar
org.example.App
generates the following output:
1630187996652: Agent Started
1630187996672: Main Started!
1630187997672: Main Ended!
1630187997673: Shutdown Hook is running!
1630188002675: Shutdown hook is completed
So you have your delay.

twisted reactor not running properly inside celery

System/Dependencies details:
CPU --> 4
requirements --> celery==4.3.0, twisted==19.7.0 , python3.7
Below is the celery setup I have
from threading import Thread
from celery import Celery
from twisted.internet import threads, reactor, defer
from twisted.web.error import Error
from celery import signals
app = Celery('tasks', broker='pyamqp://guest#localhost//')
#signals.worker_process_init.connect
def configure_infrastructure(**kwargs):
Thread(target=reactor.run, name="reactor.run", args=(False,)).start()
print('started new thread')
#signals.worker_process_shutdown.connect()
def shutdown_reactor(**kwargs):
"""
This is invoked when the individual workers shut down. It just stops the twisted reactor
#param kwargs:
#return:
"""
reactor.callFromThread(reactor.stop)
print('REACTOR SHUTDOWN')
def getPage(inp):
print(inp)
return inp
def inThread():
print('inside inthread method')
try:
result = threads.blockingCallFromThread(
reactor, getPage, "http://twistedmatrix.com/")
except Exception as exc:
print(exc)
else:
print(result)
#app.task
def add(x, y):
print('inside add method')
inThread()
return x + y
Running celery worker like below:
celery -A run worker --loglevel=info
Logs when celery start:
(2_env) ubuntu#gpy:~/app/env/src$ celery -A run worker --loglevel=info
[tasks]
. run.add
[2020-04-09 07:25:29,357: WARNING/Worker-1] started new thread
[2020-04-09 07:25:29,362: WARNING/Worker-4] started new thread
[2020-04-09 07:25:29,362: WARNING/Worker-3] started new thread
[2020-04-09 07:25:29,364: WARNING/Worker-2] started new thread
[2020-04-09 07:25:29,367: INFO/MainProcess] Connected to amqp://guest:**#127.0.0.1:5672//
calling method like below:
>>> run.add.delay(1,2)
<AsyncResult: d41680fd-7cc1-4e75-81be-6496bad0cc16>
>>>
sometimes I can see it is working fine.
[2020-04-09 07:27:17,998: INFO/MainProcess] Received task: run.add[00934769-48c4-48b8-852c-8b746bdd5e03]
[2020-04-09 07:27:17,999: WARNING/Worker-4] inside add method
[2020-04-09 07:27:17,999: WARNING/Worker-4] inside inthread method
[2020-04-09 07:27:18,000: WARNING/Worker-4] http://twistedmatrix.com/
[2020-04-09 07:27:18,000: WARNING/Worker-4] http://twistedmatrix.com/
[2020-04-09 07:27:18,000: INFO/MainProcess] Task run.add[00934769-48c4-48b8-852c-8b746bdd5e03] succeeded in 0.00144551398989s: 3
Sometimes I can see it's not able to call getPage method and got hung like below logs
[2020-04-09 07:27:22,318: INFO/MainProcess] Received task: run.add[d41680fd-7cc1-4e75-81be-6496bad0cc16]
[2020-04-09 07:27:22,319: WARNING/Worker-2] inside add method
[2020-04-09 07:27:22,319: WARNING/Worker-2] inside inthread method
is there any issue in using reactor.run inside Thread?
UPDATE
I put print into *twisted.internet.threads.blockingCallFromThread* .
def blockingCallFromThread(reactor, f, *a, **kw):
queue = Queue.Queue()
def _callFromThread():
print('inside _callFromThread')
result = defer.maybeDeferred(f, *a, **kw)
result.addBoth(queue.put)
print('before calling _callFromThread')
reactor.callFromThread(_callFromThread)
print('after calling _callFromThread')
result = queue.get()
if isinstance(result, failure.Failure):
result.raiseException()
return result
I can see that the celery worker got hung only when _callFromThread method is not get called in reactor.callFromThread(_callFromThread) but when I manually stop the worker with CTRL + c then I can it get called.
Everytime I stop worker where the job was hung, it starts processing job.
Update:27 April 2020
It got solved if I use crochet to run the twisted reactor. I update the below function.
#signals.worker_process_init.connect
def configure_infrastructure(**kwargs):
from crochet import setup
setup()
print('started new thread')
With some care, which you seem to have taken, you can run the Twisted reactor in one thread. However, you will not be able to run it in more than one thread which I suppose is what is happening when you use it with Celery. It has both instance and global state which will get stomped on if it is run in more than one thread.
Instead, try using crochet to coordinate calls onto the reactor running in a single non-main thread from as many other threads as you like.

Why does subscribe work and block doesn't in Spring reactive Mongo?

I created a project fresh out of the Spring Initializr by choosing Kotlin, Gradle, M7 and Web-reactive.
I made a small project:
data class Person (val id: String)
#Component class PersonHandler(val template: ReactiveMongoTemplate)
{
init
{
println("Initializing")
val jim: Mono<Person> = template.save(Person("Jim"))
val john: Mono<Person> = template.save(Person("John"))
val jack: Mono<Person> = template.save(Person("Jack"))
launch(jim)
launch(john)
launch(jack)
println("Finished Initializing")
}
fun launch(mono: Mono<Person>)
{
mono.subscribe({println(it.id)}, {println("Error")}) // This works
// mono.block() This just hangs
}
}
I try to save three persons into the database. The save method returns just a Mono which needs to be executed. If I try to execute it by simply subscribing, everything works nice:
Initializing
Finished Initializing
2017-12-21 13:14:39.513 INFO 17278 --- [ Thread-13] org.mongodb.driver.connection : Opened connection [connectionId{localValue:3, serverValue:158}] to localhost:27017
2017-12-21 13:14:39.515 INFO 17278 --- [ Thread-12] org.mongodb.driver.connection : Opened connection [connectionId{localValue:4, serverValue:159}] to localhost:27017
2017-12-21 13:14:39.520 INFO 17278 --- [ Thread-14] org.mongodb.driver.connection : Opened connection [connectionId{localValue:5, serverValue:160}] to localhost:27017
Jim
Jack
John
However, when I use block instead of subscribe the application hangs:
Initializing
2017-12-21 13:16:47.200 INFO 17463 --- [ Thread-14] org.mongodb.driver.connection : Opened connection [connectionId{localValue:3, serverValue:163}] to localhost:27017
If I query the database manually, I see that Jim has been saved, but not Jack and John.
Is this a bug, or am I doing something wrong? I would like to have the guarantee that the users are in the database before the code goes any further so I would really like to use block.
I am not sure if it is relevant, but I get a compiler warning
Accessing nonfinal property template in constructor
There is a minimal working example. It contains two branches. One is a workaround for the issue.
https://github.com/martin-drozdik/spring-mongo-bug-example
I think this might be a Spring Framework bug / usability issue.
First, let me underline the difference between subscribe and block:
the subscribe method kicks off the work and returns immediately. So you get no guarantee that the operation is done when other parts of your application run.
block is a blocking operation: it triggers the operation and waits for its completion.
For initialisation work, composing operations and calling block once is probably the best choice:
val jim: Mono<Person> = template.save(Person("Jim"))
val john: Mono<Person> = template.save(Person("John"))
val jack: Mono<Person> = template.save(Person("Jack"))
jim.then(john).then(jack).block();
As you've stated, using block hangs the application. I suspect this might be an Spring context initialisation issue - If I remember correctly, this process might assume a single thread in some parts and using a reactive pipeline there schedules work on many threads.
Could you create a minimal sample application (using just Java/Spring Boot/Spring Data Reactive Mongo) and report that on https://jira.spring.io?
I had a similar situation where by calling "reactiveMongoTemplate.save(model).block()" the application was hanging.
The issue was caused by the #PostConstruct in one of my classes designed to create my system users after the application initialization. I think somehow It was invoked before full Spring context initialization.
#Configuration
public class InitialDataPostLoader {
private Logger logger = LogManager.getLogger(this.getClass());
#PostConstruct
public void init() {
logger.info(String.format(MSG_SERVICE_JOB, "System Metadata initialization"));
createDefaultUsers();
}
By replacing #PostConstruct with ContextRefreshEvent listener the issues got resolved.
#Configuration
public class InitialDataPostLoader implements
ApplicationListener<ContextRefreshedEvent> {
private Logger logger = LogManager.getLogger(this.getClass());
#Override
public void onApplicationEvent(ContextRefreshedEvent arg0) {
logger.info(String.format(MSG_SERVICE_JOB, "System Metadata initialization"));
createDefaultUsers();
}

How to stop and resume a spring batch job

Goal : I am using spring batch for data processing and I want to have an option to stop/resume (where it left off).
Issue: I am able to send a stop signal to a running job and it gets stopped successfully. But when I try to send start signal to same job its creating a new instance of the job and starts as a fresh job.
My question is how can we achieve a resume functionality for a stopped job in spring batch.
You just have to run it with the same parameters. Just make sure you haven't marked the job as non-restarted and that you're not using RunIdIncrementer or similar to automatically generate unique job parameters.
See for instance, this example. After the first run, we have:
INFO: Job: [SimpleJob: [name=myJob]] completed with the following parameters: [{}] and the following status: [STOPPED]
Status is: STOPPED, job execution id 0
#1 step1 COMPLETED
#2 step2 STOPPED
And after the second:
INFO: Job: [SimpleJob: [name=myJob]] completed with the following parameters: [{}] and the following status: [COMPLETED]
Status is: COMPLETED, job execution id 1
#3 step2 COMPLETED
#4 step3 COMPLETED
Note that stopped steps will be re-executed. If you're using chunk-oriented steps, make sure that at least the ItemReader implements ItemStream (and does it with the correct semantics).
Steps marked with allowRestartWithComplete will always be re-run.

find if a job is running in Quartz1.6

I would like to clarify details of the scheduler.getCurrentlyExecutingJobs() method in Quartz1.6. I have a job that should have only one instance running at any given moment. It can be triggered to "run Now" from a UI but if a job instance already running for this job - nothing should happen.
This is how I check whether there is a job running that interests me:
List<JobExecutionContext> currentJobs = scheduler.getCurrentlyExecutingJobs();
for (JobExecutionContext jobCtx: currentJobs){
jobName = jobCtx.getJobDetail().getName();
groupName = jobCtx.getJobDetail().getGroup();
if (jobName.equalsIgnoreCase("job_I_am_looking_for_name") &&
groupName.equalsIgnoreCase("job_group_I_am_looking_for_name")) {
//found it!
logger.warn("the job is already running - do nothing");
}
}
then, to test this, I have a unit test that tries to schedule two instances of this job one after the other. I was expecting to see the warning when trying to schedule the second job, however, instead, I'm getting this exception:
org.quartz.ObjectAlreadyExistsException: Unable to store Job with name:
'job_I_am_looking_for_name' and group: 'job_group_I_am_looking_for_name',
because one already exists with this identification.
When I run this unit test in a debug mode, with the break on this line:
List currentJobs = scheduler.getCurrentlyExecutingJobs();
I see the the list is empty - so the scheduler does not see this job as running , but it still fails to schedule it again - which tells me the job was indeed running at the time...
Am I missing some finer points with this scheduler method?
Thanks!
Marina
For the benefit of others, I'm posting an answer to the issue I was having - I got help from the Terracotta Forum's Zemian Deng: posting on Terracotta's forum
Here is the re-cap:
The actual checking of the running jobs was working fine - it was just timing in the Unit tests, of course. I've added some sleeping in the job, and tweaked unit tests to schedule the second job while the first one is still running - and verified that I could indeed find the first job still running.
The exception I was getting was because I was trying to schedule a new job with the same name, rather than try to trigger the already stored in the scheduler job. The following code worked exactly as I needed:
List<JobExecutionContext> currentJobs = scheduler.getCurrentlyExecutingJobs();
for (JobExecutionContext jobCtx: currentJobs){
jobName = jobCtx.getJobDetail().getName();
groupName = jobCtx.getJobDetail().getGroup();
if (jobName.equalsIgnoreCase("job_I_am_looking_for_name") && groupName.equalsIgnoreCase("job_group_I_am_looking_for_name")) {
//found it!
logger.warn("the job is already running - do nothing");
return;
}
}
// check if this job is already stored in the scheduler
JobDetail emailJob;
emailJob = scheduler.getJobDetail("job_I_am_looking_for_name", "job_group_I_am_looking_for_name");
if (emailJob == null){
// this job is not in the scheduler yet
// create JobDetail object for my job
emailJob = jobFactory.getObject();
emailJob.setName("job_I_am_looking_for_name");
emailJob.setGroup("job_group_I_am_looking_for_name");
scheduler.addJob(emailJob, true);
}
// this job is in the scheduler and it is not running right now - run it now
scheduler.triggerJob("job_I_am_looking_for_name", "job_group_I_am_looking_for_name");
Thanks!
Marina