I am fairly new to decorators but am experiencing unexpected behavior revolving around autoreload in an interactive workflow with decorated functions. Its best explained by example (note these are all cells in a jupyter notebook):
The decorator:
%%file testdec.py
def decorated(func):
print("decorating")
def wrapped(*args, **kwargs):
return func(*args, **kwargs)
return wrapped
Where the decorator is used:
%%file testmod.py
from testdec import decorated
#decorated
def thisfunc():
print("old output")
def _thisfunc1():
print("old output 1")
thisfunc1 = decorated(_thisfunc1)
I would use the following to call the decorated functions:
from testmod import *
thisfunc()
thisfunc1()
outputs:
decorating
decorating
old output
old output 1
Now updating testmod.py with:
%%file testmod.py
from testdec import decorated
#decorated
def thisfunc():
print("new output")
def _thisfunc1():
print("new output 1")
thisfunc1 = decorated(_thisfunc1)
and calling the functions again:
thisfunc()
thisfunc1()
gives the following, note the old output from the first method:
decorating
decorating
old output
new output 1
However, explicitly reimporting from this module:
from testmod import *
thisfunc()
thisfunc1()
results in:
new output
new output 1
Ideally the #decorated function (e.g. with the # and not the second method) would autoreload transparently as the second method does. Is there something I can do to achieve this? What am I missing for decorated functions. For now we're manually disabling decorators when editing interactively in order to have the benefits of autoreload.
Thanks.
Related
I want to separate DB configuration before test and test itself.
Let's say I have fixture_1 (user object) with method get_friends.
get_friends making calls to DB, well I should mock some calls beforehead.
I did the decorator below for this purpose, but I can't pass required fixture parameters (*a, **kw) to this function.
Pytest just not triggering on it, how I can do it?
The second decoration is just to avoid typing and wrapping decorator for every test_db_func.
P.S. please tell me if it's a bad idea.
I found this one question How do I make pytest fixtures work with decorated functions? but this will force me to wrap every test_db_func.
def decorator(test_db_func):
#functools_wraps(test_db_func)
def wrapper(unit_func, *a, **kw):
#functools_wraps(unit_func)
def unit_func_params(*unit_func_args, **unit_func_kwargs):
# Prepare DB for test
test_db_func(*a, **kw) # Should pass fixtures to this func
unit_func(*unit_func_args, **unit_func_kwargs)
test_db_func(*a, **kw) # Assertions after yield
return unit_func_params
return wrapper
#test_db_func
def test_x(fixture_1, fixture_2):
assert fixture_1.get_friends(fixture_2)
#decorator
def test_db_func(db_fixture):
db_fixture.return_value = 'foo'
yield
db_fixture.assert_called_once ...
In my test suite, I have certain data-generation fixtures which are used with many parameterized tests. Some of these tests would want these fixtures to run only once per session, while others need them to run every function. For example, I may have a fixture similar to:
#pytest.fixture
def get_random_person():
return random.choice(list_of_people)
and 2 parameterized tests, one which wants to use the same person for each test condition and one which wants a new person each time. Is there any way for this fixture to have scope="session" for one test and scope="function" for another?
James' answer is okay, but it doesn't help if you yield from your fixture code. This is a better way to do it:
# Built In
from contextlib import contextmanager
# 3rd Party
import pytest
#pytest.fixture(session='session')
def fixture_session_fruit():
"""Showing how fixtures can still be passed to the different scopes.
If it is `session` scoped then it can be used by all the different scopes;
otherwise, it must be the same scope or higher than the one it is used on.
If this was `module` scoped then this fixture could NOT be used on `fixture_session_scope`.
"""
return "apple"
#contextmanager
def _context_for_fixture(val_to_yield_after_setup):
# Rather long and complicated fixture implementation here
print('SETUP: Running before the test')
yield val_to_yield_after_setup # Let the test code run
print('TEARDOWN: Running after the test')
#pytest.fixture(session='function')
def fixture_function_scope(fixture_session_fruit):
with _context_for_fixture(fixture_session_fruit) as result:
yield result
#pytest.fixture(scope='class')
def fixture_class_scope(fixture_session_fruit):
with _context_for_fixture(fixture_session_fruit) as result:
yield result
#pytest.fixture(scope='module')
def fixture_module_scope(fixture_session_fruit):
with _context_for_fixture(fixture_session_fruit) as result:
yield result
#pytest.fixture(scope='session')
def fixture_session_scope(fixture_session_fruit):
with _context_for_fixture(fixture_session_fruit) as result:
# NOTE if the `_context_for_fixture` just did `yield` without any value,
# there should still be a `yield` here to keep the fixture
# inside the context till it is done. Just remove the ` result` part.
yield result
This way you can still handle contextual fixtures.
Github issue for reference: https://github.com/pytest-dev/pytest/issues/3425
One way to do this to separate out the implementation and then have 2 differently-scoped fixtures return it. So something like:
def _random_person():
return random.choice(list_of_people)
#pytest.fixture(scope='function')
def get_random_person_function_scope():
return _random_person()
#pytest.fixture(scope='session')
def get_random_person_session_scope():
return _random_person()
I've been doing this:
def _some_fixture(a_dependency_fixture):
def __some_fixture(x):
return x
yield __some_fixture
some_temp_fixture = pytest.fixture(_some_fixture, scope="function")
some_module_fixture = pytest.fixture(_some_fixture, scope="module")
some_session_fixture = pytest.fixture(_some_fixture, scope="session")
Less verbose than using a context manager.
Actually there is a workaround for this using the request object.
You could do something like:
#pytest.fixture(scope='class')
def get_random_person(request):
request.scope = getattr(request.cls, 'scope', request.scope)
return random.choice(list_of_people)
Then back at the test class:
#pytest.mark.usefixtures('get_random_person')
class TestSomething:
scope = 'function'
def a_random_test():
def another_test():
However, this only works properly for choosing between 'function' and 'class' scope and particularly if the fixture starts as class-scoped (and then changes to 'function' or is left as is).
If I try the other way around (from 'function' to 'class') funny stuff happen and I still can't figure out why.
I have a use case where I may use fixture multiple times inside a test in a "context manager" way. See example code below:
in conftest.py
class SomeYield(object):
def __enter__(self):
log.info("SomeYield.__enter__")
def __exit__(self, exc_type, exc_val, exc_tb):
log.info("SomeYield.__exit__")
def generate_name():
name = "{current_time}-{uuid}".format(
current_time=datetime.now().strftime("%Y-%m-%d-%H-%M-%S"),
uuid=str(uuid.uuid4())[:4]
)
return name
#pytest.yield_fixture
def some_yield():
name = generate_name()
log.info("Start: {}".format(name))
yield SomeYield()
log.info("End: {}".format(name))
in test_some_yield.py
def test_some_yield(some_yield):
with some_yield:
pass
with some_yield:
pass
Console output:
INFO:conftest:Start: 2017-12-06-01-50-32-5213
INFO:conftest:SomeYield.__enter__
INFO:conftest:SomeYield.__exit__
INFO:conftest:SomeYield.__enter__
INFO:conftest:SomeYield.__exit__
INFO:conftest:End: 2017-12-06-01-50-32-5213
Questions:
If I have some setup code in SomeYield.enter and cleanup code in
SomeYield.exit, is this the right way to do it using fixture for
multiple calls in my test?
Why didn't I see three occurrences of
enter and exit? Is this expected?
New to Scala and looking for pointers to an idiomatic solution, if there is one.
I'd like to have arbitrary user-supplied Scala functions (which are allowed to reference functions/classes I have defined in my code) applied to some data.
For example: I have foo(s: String): String and bar(s: String): String functions defined in my myprog.scala. The user runs my program like this:
$ scala myprog data.txt --func='(s: Str) => foo(bar(s)).reverse'
This would run line by line through the data file and emit the result of applying the user-specified function to that line.
For extra points, can I ensure that there are no side-effects in the user-defined function? If not, can I restrict the function to use only a restricted subset of functions (which I can assure to be safe)?
#kenjiyoshida has a nice gist that shows how to eval Scala code. Note that when using Eval from that gist, not specifying a return value will result in a runtime failure when Scala defaults to inferring Nothing.
scala> Eval("println(\"Hello\")")
Hello
java.lang.ClassCastException: scala.runtime.BoxedUnit cannot be cast to scala.runtime.Nothing$
... 42 elided
vs
scala> Eval[Unit]("println(\"Hello\")")
Hello
It nicely handles whatever's in scope as well.
object Thing {
val thing: Int = 5
}
object Eval {
def apply[A](string: String): A = {
val toolbox = currentMirror.mkToolBox()
val tree = toolbox.parse(string)
toolbox.eval(tree).asInstanceOf[A]
}
def fromFile[A](file: File): A =
apply(scala.io.Source.fromFile(file).mkString(""))
def fromFileName[A](file: String): A =
fromFile(new File(file))
}
object Thing2 {
val thing2 = Eval[Int]("Thing.thing") // 5
}
Twitter's util package used to have util-eval, but that seems to have been deprecated now (and also triggers a compiler bug when compiled).
As for the second part of your question, the answer seems to be no. Even if you disable default Predef and imports yourself, a user can always get to those functions with the fully qualified package name. You could perhaps use Scala's scala.tools.reflect.ToolBox to first parse your string and then compare against a whitelist, before passing to eval, but at that point things could get pretty hairy since you'll be manually writing code to sanitize the Scala AST (or at the very least reject dangerous input). It definitely doesn't seem to be an "idiomatic solution."
This should be possible by using the standard Java JSR 223 Scripting Engine
see https://issues.scala-lang.org/browse/SI-874
(also mentions using scala.tools.nsc.Interpreter but not sure this is still available)
import javax.script.*;
ScriptEngine e = new ScriptEngineManager().getEngineByName("scala");
e.getContext().setAttribute("label", new Integer(4), ScriptContext.ENGINE_SCOPE);
try {
engine.eval("println(2+label)");
} catch (ScriptException ex) {
ex.printStackTrace();
}
I have a group of objects which I am creating a class for that I want to store each object as its own text file. I would really like to store it as a Python class definition which subclasses the main class I am creating. So, I did some poking around and found a Python Code Generator on effbot.org. I did some experimenting with it and here's what I came up with:
#
# a Python code generator backend
#
# fredrik lundh, march 1998
#
# fredrik#pythonware.com
# http://www.pythonware.com
#
# Code taken from http://effbot.org/zone/python-code-generator.htm
import sys, string
class CodeGeneratorBackend:
def begin(self, tab="\t"):
self.code = []
self.tab = tab
self.level = 0
def end(self):
return string.join(self.code, "")
def write(self, string):
self.code.append(self.tab * self.level + string)
def indent(self):
self.level = self.level + 1
def dedent(self):
if self.level == 0:
raise SyntaxError, "internal error in code generator"
self.level = self.level - 1
class Point():
"""Defines a Point. Has x and y."""
def __init__(self, x, y):
self.x = x
self.y = y
def dump_self(self, filename):
self.c = CodeGeneratorBackend()
self.c.begin(tab=" ")
self.c.write("class {0}{1}Point()\n".format(self.x,self.y))
self.c.indent()
self.c.write('"""Defines a Point. Has x and y"""\n')
self.c.write('def __init__(self, x={0}, y={1}):\n'.format(self.x, self.y))
self.c.indent()
self.c.write('self.x = {0}\n'.format(self.x))
self.c.write('self.y = {0}\n'.format(self.y))
self.c.dedent()
self.c.dedent()
f = open(filename,'w')
f.write(self.c.end())
f.close()
if __name__ == "__main__":
p = Point(3,4)
p.dump_self('demo.py')
That feels really ugly, is there a cleaner/better/more pythonic way to do this? Please note, this is not the class I actually intend to do this with, this is a small class I can easily mock up in not too many lines. Also, the subclasses don't need to have the generating function in them, if I need that again, I can just call the code generator from the superclass.
We use Jinja2 to fill in a template. It's much simpler.
The template looks a lot like Python code with a few {{something}} replacements in it.
This is pretty much the best way to generate Python source code. However, you can also generate Python executable code at runtime using the ast library. You can build code using the abstract syntax tree, then pass it to compile() to compile it into executable code. Then you can use eval() to run the code.
I'm not sure whether there is a convenient way to save the compiled code for use later though (ie. in a .pyc file).
Just read your comment to wintermute - ie:
What I have is a bunch of planets that
I want to store each as their own text
files. I'm not particularly attached
to storing them as python source code,
but I am attached to making them
human-readable.
If that's the case, then it seems like you shouldn't need subclasses but should be able to use the same class and distinguish the planets via data alone. And in that case, why not just write the data to files and, when you need the planet objects in your program, read in the data to initialize the objects?
If you needed to do stuff like overriding methods, I could see writing out code - but shouldn't you just be able to have the same methods for all planets, just using different variables?
The advantage of just writing out the data (it can include label type info for readability that you'd skip when you read it in) is that non-Python programmers won't get distracted when reading them, you could use the same files with some other language if necessary, etc.
I'm not sure whether this is especially Pythonic, but you could use operator overloading:
class CodeGenerator:
def __init__(self, indentation='\t'):
self.indentation = indentation
self.level = 0
self.code = ''
def indent(self):
self.level += 1
def dedent(self):
if self.level > 0:
self.level -= 1
def __add__(self, value):
temp = CodeGenerator(indentation=self.indentation)
temp.level = self.level
temp.code = str(self) + ''.join([self.indentation for i in range(0, self.level)]) + str(value)
return temp
def __str__(self):
return str(self.code)
a = CodeGenerator()
a += 'for a in range(1, 3):\n'
a.indent()
a += 'for b in range(4, 6):\n'
a.indent()
a += 'print(a * b)\n'
a.dedent()
a += '# pointless comment\n'
print(a)
This is, of course, far more expensive to implement than your example, and I would be wary of too much meta-programming, but it was a fun exercise. You can extend or use this as you see fit; how about:
adding a write method and redirecting stdout to an object of this to print straight to a script file
inheriting from it to customise output
adding attribute getters and setters
Would be great to hear about whatever you go with :)
From what I understand you are trying to do, I would consider using reflection to dynamically examine a class at runtime and generate output based on that. There is a good tutorial on reflection (A.K.A. introspection) at http://diveintopython3.ep.io/.
You can use the dir() function to get a list of names of the attributes of a given object. The doc string of an object is accessible via the __doc__ attribute. That is, if you want to look at the doc string of a function or class you can do the following:
>>> def foo():
... """A doc string comment."""
... pass
...
>>> print foo.__doc__
A doc string comment.