In code:
val executor = new ForkJoinPool()
executor.execute(new Runnable{
def run = println("This task is run asynchronously")
})
Thread.sleep(10000)
This code prints: This task is run asynchronously
But if I remove Thread.sleep(10000), program doesn't print.
I then learnt that its so because sleep prevents daemon threads in ForkJoinPool from being terminated before they call run method on Runnable object.
So, few questions:
Does it mean threads started by ForkJoinPool are all daemon threads?Any why is it so?
How does sleep help here?
Answers:
Yes, because you are using the default thread factory and that is how it is configured. You can provide a custom thread factory to it if you wish, and you may configure the threads to be non-daemon.
Sleep helps because it prevents your program from exiting for long enough for the thread pool threads to find your task and execute it.
Related
I'm using celery 4.4.7
Some of my tasks are using too much memory and are getting killed with SIGTERM 9. I would like to retry them later since I'm running with concurrency on the machine and they might run OK again.
However, as far as I understand you can't catch WorkerLostError exception thrown within a task i.e. this won't won't work as I expect:
from billiard.exceptions import WorkerLostError
#celery_app.task(acks_late=True, max_retries=2, autoretry_for=(WorkerLostError,))
def some_task():
#task code
I also don't won't to use task_reject_on_worker_lost as it makes the tasks requeued and max_retries is not applied.
What would be the best approach to handle my use case?
Thanks in advance for your time :)
Gal
System/Dependencies details:
CPU --> 4
requirements --> celery==4.3.0, twisted==19.7.0 , python3.7
Below is the celery setup I have
from threading import Thread
from celery import Celery
from twisted.internet import threads, reactor, defer
from twisted.web.error import Error
from celery import signals
app = Celery('tasks', broker='pyamqp://guest#localhost//')
#signals.worker_process_init.connect
def configure_infrastructure(**kwargs):
Thread(target=reactor.run, name="reactor.run", args=(False,)).start()
print('started new thread')
#signals.worker_process_shutdown.connect()
def shutdown_reactor(**kwargs):
"""
This is invoked when the individual workers shut down. It just stops the twisted reactor
#param kwargs:
#return:
"""
reactor.callFromThread(reactor.stop)
print('REACTOR SHUTDOWN')
def getPage(inp):
print(inp)
return inp
def inThread():
print('inside inthread method')
try:
result = threads.blockingCallFromThread(
reactor, getPage, "http://twistedmatrix.com/")
except Exception as exc:
print(exc)
else:
print(result)
#app.task
def add(x, y):
print('inside add method')
inThread()
return x + y
Running celery worker like below:
celery -A run worker --loglevel=info
Logs when celery start:
(2_env) ubuntu#gpy:~/app/env/src$ celery -A run worker --loglevel=info
[tasks]
. run.add
[2020-04-09 07:25:29,357: WARNING/Worker-1] started new thread
[2020-04-09 07:25:29,362: WARNING/Worker-4] started new thread
[2020-04-09 07:25:29,362: WARNING/Worker-3] started new thread
[2020-04-09 07:25:29,364: WARNING/Worker-2] started new thread
[2020-04-09 07:25:29,367: INFO/MainProcess] Connected to amqp://guest:**#127.0.0.1:5672//
calling method like below:
>>> run.add.delay(1,2)
<AsyncResult: d41680fd-7cc1-4e75-81be-6496bad0cc16>
>>>
sometimes I can see it is working fine.
[2020-04-09 07:27:17,998: INFO/MainProcess] Received task: run.add[00934769-48c4-48b8-852c-8b746bdd5e03]
[2020-04-09 07:27:17,999: WARNING/Worker-4] inside add method
[2020-04-09 07:27:17,999: WARNING/Worker-4] inside inthread method
[2020-04-09 07:27:18,000: WARNING/Worker-4] http://twistedmatrix.com/
[2020-04-09 07:27:18,000: WARNING/Worker-4] http://twistedmatrix.com/
[2020-04-09 07:27:18,000: INFO/MainProcess] Task run.add[00934769-48c4-48b8-852c-8b746bdd5e03] succeeded in 0.00144551398989s: 3
Sometimes I can see it's not able to call getPage method and got hung like below logs
[2020-04-09 07:27:22,318: INFO/MainProcess] Received task: run.add[d41680fd-7cc1-4e75-81be-6496bad0cc16]
[2020-04-09 07:27:22,319: WARNING/Worker-2] inside add method
[2020-04-09 07:27:22,319: WARNING/Worker-2] inside inthread method
is there any issue in using reactor.run inside Thread?
UPDATE
I put print into *twisted.internet.threads.blockingCallFromThread* .
def blockingCallFromThread(reactor, f, *a, **kw):
queue = Queue.Queue()
def _callFromThread():
print('inside _callFromThread')
result = defer.maybeDeferred(f, *a, **kw)
result.addBoth(queue.put)
print('before calling _callFromThread')
reactor.callFromThread(_callFromThread)
print('after calling _callFromThread')
result = queue.get()
if isinstance(result, failure.Failure):
result.raiseException()
return result
I can see that the celery worker got hung only when _callFromThread method is not get called in reactor.callFromThread(_callFromThread) but when I manually stop the worker with CTRL + c then I can it get called.
Everytime I stop worker where the job was hung, it starts processing job.
Update:27 April 2020
It got solved if I use crochet to run the twisted reactor. I update the below function.
#signals.worker_process_init.connect
def configure_infrastructure(**kwargs):
from crochet import setup
setup()
print('started new thread')
With some care, which you seem to have taken, you can run the Twisted reactor in one thread. However, you will not be able to run it in more than one thread which I suppose is what is happening when you use it with Celery. It has both instance and global state which will get stomped on if it is run in more than one thread.
Instead, try using crochet to coordinate calls onto the reactor running in a single non-main thread from as many other threads as you like.
I'm trying to get a child actor in my Scala 2.11 app (that uses Akka) to send a PoisonPill to the top-level actor (Master) and shut down the entire JVM process.
I uploaded this repo so you can reproduce what I'm seeing. Just clone it and run ./gradlew run.
But essentially we have the Driver:
object Driver extends App {
println("Starting upp the app...")
lazy val cortex = ActorSystem("cortex")
cortex.registerOnTermination {
System.exit(0)
}
val master = cortex.actorOf(Props[Master], name = "Master")
println("About to fire a StartUp message at Master...")
master ! StartUp
println("Fired! Actor system is spinning up...")
}
and then in the Child:
class Child extends Actor {
override def receive = {
case MakeItHappen =>
println("Child will make it happen!")
context.parent ! PoisonPill
}
}
When I run ./gradlew run I see the following output:
./gradlew run
Starting a Gradle Daemon (subsequent builds will be faster)
:compileJava NO-SOURCE
:compileScala UP-TO-DATE
:processResources NO-SOURCE
:classes UP-TO-DATE
:run
Starting upp the app...
About to fire a StartUp message at Master...
Fired! Actor system is spinning up...
Stage Director has received a command to start up the actor system!
Child will make it happen!
<==========---> 80% EXECUTING
> :run
However the JVM process never shuts down. I would have expected the PoisonPill to kill the Master (as well as Child), exit/shutdown the actor system, and then the System.exit(0) command (which I registered upon actor system termination) to kick in and exit the JVM process.
Can anyone tell what's going on here?
You're stopping the Master, but you're not shutting down the ActorSystem. Even though Master is a top-level actor (i.e., an actor created by system.actorOf), stopping it doesn't terminate the ActorSystem (note that Master itself has a parent, so it's not the "highest" actor in the actor hierarchy).
The normal way to terminate a ActorSystem is to call terminate. If you want the entire system to shut down if the Master stops, then you could override the Master's postStop hook and call context.system.terminate() there. Calling system.terminate() in an actor's lifecycle hook is risky, however. For example, if an exception is thrown in the actor, and the actor's supervision strategy stops it, you probably don't intend to shut down the entire system at that point. Such a design is not very resilient.
Check out this pattern for shutting down an actor system "at the right time."
I have a TYPO3 extension providing a scheduler task (extension scheduler 6.2.0).
In this task I have following strage problem:
private $svm;
...
$this->svm = new \SVM();
When this line is executed during task execution (started from cron job) program hangs up and does nothing. No exception, no error. It is just waiting for something.
If this line is executed within extension - an object is created.
If I start scheduler task manually in TYPO3 back-end - an object is also created.
It looks like during CLI execution class SVM is unknown. But then there should be an error...
If the class SVMis unknown, an error would occur and nothing would wait. I propose that you debug that and check, e.g. with class_exists($this->svm) if this class exists.
Without knowing more it is hard to help more.
I have some Java code that I am unit testing using JUnit, running inside Eclipse (Neon). It so happens that some code under test has a bug in it, causing it to enter an infinite loop. The JUnit run then, of course, does not finish. How do I kill the test run?
The button for stopping the test run ("Stop JUnit Test Run") does not work well: the GUI seems to think that it has stopped the test run, but a look at the CPU activity (using top, for example), shows that a Java thread is still running. I can kill the thread myself by sending it a kill signal. But that seems a kludge and is inconvenient. Is there a better way, available within Eclipse itself?
Kill it from the console view, using the red button. This stops the process.
Stopping it from the junit view only asks it to stop.
You can handle such things with JUnit by specifying a time-out within
the #Test annotation. For example:
// Simple test-case which will always fail with time-out
#Test(timeout = 1000 * 60) throws Exception // 60 seconds
public void testSomething() {
for (int i = 0; i < 100; i++) { // 100 seconds
Thread.sleep(1000);
}
}
If your test method doesn't finish on time,
then JUnit will interrupt it and report a test failure.
For the example above it will be:
java.lang.Exception: test timed out after 60000 milliseconds
at java.lang.Thread.sleep(Native Method)
at my.package.Test1.testSomething(Test1.java:12)
First, try Matthew Farwell's answer. If that doesn't work then you have to go to your processes running (ctrl+shift+esc on windows, then processes tab) and select the java.exe task, then end process. In the event you ever need to kill eclipse, it's javaw.exe