I have a function, which indirectly calls a celery task using apply_async() internally.
def abc(...):
send_notification.apply_async(...)
where send_notification is a celery task method:
#app.task(bind=True)
def send_notification(self, ...):
...
Now, I'm writing a test case for testing abc(), where I'd want to convert invocation of send_notification.apply_async() to send_notification.s(...).apply().
Is this something thats possible using mock/monkeypatch/patch?
Related
I have a scala program that runs for a while and then terminates. I'd like to provide a library to this program that, behind the scenes, schedules an asynchronous task to run every N seconds. I'd also like the program to terminate when the main entrypoint's work is finished without needing to explicitly tell the background work to shut down (since it's inside a library).
As best I can tell the idiomatic way to do polling or scheduled work in Scala is with Akka's ActorSystem.scheduler.schedule, but using an ActorSystem makes the program hang after main waiting for the actors. I then tried and failed to add another actor that joins on the main thread, seemingly because "Anything that blocks a thread is not advised within Akka"
I could introduce a custom dispatcher; I could kludge something together with a polling isAlive check, or adding a similar check inside each worker; or I could give up on Akka and just use raw Threads.
This seems like a not-too-unusual thing to want to do, so I'd like to use idiomatic Scala if there's a clear best way.
I don't think there is an idiomatic Scala way.
The JVM program terminates when all non-daemon thread are finished. So you can schedule your task to run on a daemon thread.
So just use Java functionality:
import java.util.concurrent._
object Main {
def main(args: Array[String]): Unit = {
// Make a ThreadFactory that creates daemon threads.
val threadFactory = new ThreadFactory() {
def newThread(r: Runnable) = {
val t = Executors.defaultThreadFactory().newThread(r)
t.setDaemon(true)
t
}
}
// Create a scheduled pool using this thread factory
val pool = Executors.newSingleThreadScheduledExecutor(threadFactory)
// Schedule some function to run every second after an initial delay of 0 seconds
// This assumes Scala 2.12. In 2.11 you'd have to create a `new Runnable` manually
// Note that scheduling will stop, if there is an exception thrown from the function
pool.scheduleAtFixedRate(() => println("run"), 0, 1, TimeUnit.SECONDS)
Thread.sleep(5000)
}
}
You can also use guava to create a daemon thread factory with new ThreadFactoryBuilder().setDaemon(true).build().
If you use Akka scheduler you will be relying on highly tuned and optimized implementation that is well tested. Bringing up an actor system is a bit heavy weight though, I agree. Additionally you have to bring in a dependency on akka. If you are ok with that you can explicitly call system.shutdown from main when you are done, or wrap it in a function that will do it for you.
Alternatively, you could try something along these lines:
import scala.concurrent._
import ExecutionContext.Implicits.global
object Main extends App {
def repeatEvery[T](timeoutMillis: Int)(f: => T): Future[T] = {
val p = Promise[T]()
val never = p.future
f
def timeout = Future {
Thread.sleep(timeoutMillis)
throw new TimeoutException
}
val failure = Future.firstCompletedOf(List(never, timeout))
failure.recoverWith { case _ => repeatEvery(timeoutMillis)(f) }
}
repeatEvery(1000) {
println("scheduled job called")
}
println("main started doing its work")
Thread.sleep(10000)
println("main finished")
}
Prints:
scheduled job called
main started doing its work
scheduled job called
scheduled job called
scheduled job called
scheduled job called
scheduled job called
scheduled job called
scheduled job called
scheduled job called
scheduled job called
main finished
I don't like that it uses Thread.sleep, but that is done to avoid using any other 3rd party schedulers and Scala Future does not provide timeout options. So you'll be wasting one thread on that scheduling task, but that's what Akka scheduler seems to do anyway. The difference is that perhaps you want a single scheduler for the whole JVM not to waste too many threads. The code I provided albeit simpler will waste a thread per job.
I am minimally using pytest as a generic test runner for large automated integration tests against various API products at work, and I've been trying to find an equally generic example of a teardown function that runs on completion of any test, regardless of success or failure.
My typical use pattern is super linear and usually goes something like this:
def test_1():
<logic>
assert something
def test_2():
<logic>
assert something
def test_3():
<logic>
assert something
Occasionally, when it makes sense to do so, at the top of my script I toss in a setup fixture with an autouse argument set to "True" that runs on the launch of every script:
#pytest.fixture(scope="session", autouse=True)
def setup_something():
testhelper = TestHelper
testhelper.create_something(host="somehost", channel="somechannel")
def test_1():
<logic>
assert something
def test_2():
<logic>
assert something
def test_3():
<logic>
assert something
Up until recently, disposable docker environments have allowed me to get away with skipping the entire teardown process, but I'm in a bit of pinch where one of those is not available right now. Ideally, without diverting from the same linear pattern I've already been using, how would I implement another pytest fixture that does something like:
#pytest.fixture
def teardown():
testhelper = TestHelper
testhelper.delete_something(thing=something)
when the run is completed?
Every fixture may have a tear down part:
#pytest.fixture
def something(request):
# setup code
def finalize():
# teardown code
request.addfinalizer(finalize)
return fixture_result
Or as I usually use it:
#pytest.fixture
def something():
# setup code
yield fixture_result
# teardown code
Note that in pytest pre-3.0, the decorator required for the latter idiom was #pytest.yield_fixture. Since 3.0, however, one can just use the regular #pytest.fixture decorator, and #pytest.yield_fixture is deprecated.
See more here
you can use these functions in your conftest.py
def pytest_runtest_setup(item):
pass
def pytest_runtest_teardown(item):
pass
see here for docs
I am building some tests for python3 code using py.test. The code accesses a Postgresql Database using aiopg (Asyncio based interface to postgres).
My main expectations:
Every test case should have access to a new asyncio event loop.
A test that runs too long will stop with a timeout exception.
Every test case should have access to a database connection.
I don't want to repeat myself when writing the test cases.
Using py.test fixtures I can get pretty close to what I want, but I still have to repeat myself a bit in every asynchronous test case.
This is how my code looks like:
#pytest.fixture(scope='function')
def tloop(request):
# This fixture is responsible for getting a new event loop
# for every test, and close it when the test ends.
...
def run_timeout(cor,loop,timeout=ASYNC_TEST_TIMEOUT):
"""
Run a given coroutine with timeout.
"""
task_with_timeout = asyncio.wait_for(cor,timeout)
try:
loop.run_until_complete(task_with_timeout)
except futures.TimeoutError:
# Timeout:
raise ExceptAsyncTestTimeout()
#pytest.fixture(scope='module')
def clean_test_db(request):
# Empty the test database.
...
#pytest.fixture(scope='function')
def udb(request,clean_test_db,tloop):
# Obtain a connection to the database using aiopg
# (That's why we need tloop here).
...
# An example for a test:
def test_insert_user(tloop,udb):
#asyncio.coroutine
def insert_user():
# Do user insertion here ...
yield from udb.insert_new_user(...
...
run_timeout(insert_user(),tloop)
I can live with the solution that I have so far, but it can get cumbersome to define an inner coroutine and add the run_timeout line for every asynchronous test that I write.
I want my tests to look somewhat like this:
#some_magic_decorator
def test_insert_user(udb):
# Do user insertion here ...
yield from udb.insert_new_user(...
...
I attempted to create such a decorator in some elegant way, but failed. More generally, if my test looks like:
#some_magic_decorator
def my_test(arg1,arg2,...,arg_n):
...
Then the produced function (After the decorator is applied) should be:
def my_test_wrapper(tloop,arg1,arg2,...,arg_n):
run_timeout(my_test(),tloop)
Note that some of my tests use other fixtures (besides udb for example), and those fixtures must show up as arguments to the produced function, or else py.test will not invoke them.
I tried using both wrapt and decorator python modules to create such a magic decorator, however it seems like both of those modules help me create a function with a signature identical to my_test, which is not a good solution in this case.
This can probably solved using eval or a similar hack, but I was wondering if there is something elegant that I'm missing here.
I’m currently trying to solve a similar problem. Here’s what I’ve come up with so far. It seems to work but needs some clean-up:
# tests/test_foo.py
import asyncio
#asyncio.coroutine
def test_coro(loop):
yield from asyncio.sleep(0.1)
assert 0
# tests/conftest.py
import asyncio
#pytest.yield_fixture
def loop():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
yield loop
loop.close()
def pytest_pycollect_makeitem(collector, name, obj):
"""Collect asyncio coroutines as normal functions, not as generators."""
if asyncio.iscoroutinefunction(obj):
return list(collector._genfunctions(name, obj))
def pytest_pyfunc_call(pyfuncitem):
"""If ``pyfuncitem.obj`` is an asyncio coroutinefunction, execute it via
the event loop instead of calling it directly."""
testfunction = pyfuncitem.obj
if not asyncio.iscoroutinefunction(testfunction):
return
# Copied from _pytest/python.py:pytest_pyfunc_call()
funcargs = pyfuncitem.funcargs
testargs = {}
for arg in pyfuncitem._fixtureinfo.argnames:
testargs[arg] = funcargs[arg]
coro = testfunction(**testargs) # Will no execute the test yet!
# Run the coro in the event loop
loop = testargs.get('loop', asyncio.get_event_loop())
loop.run_until_complete(coro)
return True # TODO: What to return here?
So I basically let pytest collect asyncio coroutines like normal functions. I also intercept text exectuion for functions. If the to-be-tested function is a coroutine, I execute it in the event loop. It works with or without a fixture creating a new event loop instance per test.
Edit: According to Ronny Pfannschmidt, something like this will be added to pytest after the 2.7 release. :-)
Every test case should have access to a new asyncio event loop.
The test suite of asyncio uses unittest.TestCase. It uses setUp() method to create a new event loop. addCleanup(loop.close) is close automatically the event loop, even on error.
Sorry, I don't know how to write this with py.test if you don't want to use TestCase. But if I remember correctly, py.test supports unittest.TestCase.
A test that runs too long will stop with a timeout exception.
You can use loop.call_later() with a function which raises a BaseException as a watch dog.
I'm using RxJava in my Scala project and I need to execute my Observable in a separate thread. I know in order to achieve this I need to call observeOn method on it and pass an instance of rx.lang.scala.Scheduler as an argument.
But how can I create that instance? I did not find any apparent ways of instantiating of rx.lang.scala.Scheduler trait. For example, I have this code:
Observable.from(List(1,2,3)).observeOn(scheduler)
Can someone provide an example of working scheduler variable that will do the trick?
A trait is not instantiable.
You need to use one of the subclasses of the trait listed under "Known Subclasses" in the API documentation.
All schedulers are in the package
import rx.lang.scala.schedulers._
For blocking IO operations, use IO scheduler
Observable.from(List(1,2,3)).observeOn(IOScheduler())
For computational work, use computation scheduler
Observable.from(List(1,2,3)).observeOn(ComputationScheduler())
To execute on the current thread
Observable.from(List(1,2,3)).observeOn(ImmediateScheduler())
To execute on a new thread
Observable.from(List(1,2,3)).observeOn(NewThreadScheduler())
To queues work on the current thread to be executed after the current one
Observable.from(List(1,2,3)).observeOn(TrampolineScheduler())
If you want to use your own custom thread pool
val threadPoolExecutor = Executors.newFixedThreadPool(2)
val executionContext = ExecutionContext.fromExecutor(threadPoolExecutor)
val customScheduler = ExecutionContextScheduler(executionContext)
Observable.from(List(1,2,3)).observeOn(customScheduler)
Lets say a single task is enough for a machine to stay very busy for a few minutes.
I want to get the result of the task, then depending on the result, have the worker perform the same task again.
The question I cannot find an answer to is this: Can I keep data in memory on the worker machine in order to use it on the next task?
Yes you can. The documentation (http://docs.celeryproject.org/en/latest/userguide/tasks.html#instantiation) is a bit vague and I'm not sure if this is the best way, but you can do something like this:
This is what you'd like to do, but doesn't work:
# This doesn't work
a = 0
#celery.task
def mytask(x):
a += x
return a
This is how to make it work:
from celery import Task, registry
#celery.task
class MyTask(Task):
def __init__(self):
self.a = 0
def run(self, x):
self.a += x
return self.a
mytask = registry.tasks[MyTask.name]
According to the docs:
A task is not instantiated for every request, but is registered in the task registry as a global instance. This means that the __init__ constructor will only be called once per process, and that the task class is semantically closer to an Actor. ... If you have a task, ... And you route every request to the same process, then it will keep state between requests."
I've had success with this but I don't know if there are better methods.