I'm using pyro5, and I want to make a remote object function as a worker class, where a client can ask it to perform various tasks by sending it functions to execute.
For this to work, I need to serialize functions, but the serializers available through Pyro5 does not support serializing functions.
Instead, I intend to use Dill, and somehow send the already serialized function through Pyro5.
However, when trying to serialize a function, I receive errors which say that I can't serialize X, where X is the enclosing class where the function lives. How can I prevent this from happening?
Actually, I was not able to reproduce this in a small script with a class and a function. I am running the dill serializer in a Qt app, not sure if that's the problem?
Alternatively, does anyone have a better idea on how to approach this?
I'm the dill author. There are several ways to approach this that may work for you. It's hard to tell without more information on your class in question, so I'll have to make assumptions.
If assume you are working with a class that you are importing from an installed package, then you can do something like this:
>>> # Mock a unserializable builtin class "Foo"
>>> import ctypes
>>> PyObjectType = ctypes.py_object(lambda :None)
>>> class Foo(PyObjectType.__class__):
... def __init__(self, x):
... self.x = x
... def bar(self, y):
... return y + self.x
...
>>> f = Foo(1)
>>> f.bar(2)
3
>>> # serializing the method fails
>>> # dill.dumps(f.bar) #FAILS
>>> # so we create a wrapping function
>>> bar = lambda x:f.bar(x)
>>>
>>> import dill
>>> dill.dumps(bar)
b'\x80\x03cdill._dill\n_create_function\nq\x00(cdill._dill\n_create_code\nq\x01(K\x01K\x00K\x01K\x03KCC\nt\x00\xa0\x01|\x00\xa1\x01S\x00q\x02N\x85q\x03X\x01\x00\x00\x00fq\x04X\x03\x00\x00\x00barq\x05\x86q\x06X\x01\x00\x00\x00xq\x07\x85q\x08X\x07\x00\x00\x00<stdin>q\tX\x08\x00\x00\x00<lambda>q\nK\x01C\x00q\x0b))tq\x0cRq\rc__builtin__\n__main__\nh\nNNtq\x0eRq\x0f}q\x10}q\x11X\x0f\x00\x00\x00__annotations__q\x12}q\x13s\x86q\x14b.'
This works, as long as the class is available for import wherever you are going to use dill.loads to restore the function.
Or if you need to ship the class as well, you could also create a wrapping class (i.e. a derived class) that includes a __reduce__ method which informs dill how to serialize the class. https://docs.python.org/3/library/pickle.html#object.__reduce__
There are other approaches as well, but it would help to have more information on what you are dealing with.
Related
What is the benefit of importing from __future__ import annotations? When I understand it right I should stop unnecessary typing import in runtime.
In my example HelloWorld is only needed for typing. But with this code the output always is:
Should this happen?
x = World!
When I remove from hello import HelloWorld the typing help in PyCharm does not longer work (I can understand this, because it does not understand where HelloWorld is from).
from __future__ import annotations
from hello import HelloWorld
if __name__ == '__main__':
def hello(x: str, hello: HelloWorld = None):
if hello is not None:
print('hello.data_01 =', hello.data_01)
print('x =', x)
hello('World!')
hello.py
from dataclasses import dataclass
#dataclass
class HelloWorld:
data_01: str
data_02: str
print("Should this happen?")
So my question is if I still need to do from hello import HelloWorld what benefits do I get from from __future__ import annotations?
The from __future__ import annotations import has one core advantage: it makes using forward references cleaner.
For example, consider this (currently broken) program.
# Error! MyClass has not been defined yet in the global scope
def foo(x: MyClass) -> None:
pass
class MyClass:
# Error! MyClass is not defined yet, in the class scope
def return_copy(self) -> MyClass:
pass
This program will actually crash when you try running it at runtime: you've tried using 'MyClass' before it's actually ever defined. In order to fix this before, you had to either use the type-comment syntax or wrap each 'MyClass' in a string to create a forward reference:
def foo(x: "MyClass") -> None:
pass
class MyClass:
def return_copy(self) -> "MyClass":
pass
Although this works, it feels very janky. Types should be types: we shouldn't need to have to manually convert certain types into strings just to make types play nicely with the Python runtime.
We can fix this by including the from __future__ import annotations import: that line automatically makes all types a string at runtime. This lets us write code that looks like the first example without it crashing: since each type hint is actually a string at runtime, we're no longer referencing something that doesn't exist yet.
And typecheckers like mypy or Pycharm won't care: to them, the type looks the same no matter how Python itself chooses to represent it.
One thing to note is that this import does not, by itself, let us avoid importing things. It simply makes it cleaner when doing so.
For example, consider the following:
from expensive_to_import_module import MyType
def blah(x: MyType) -> None: ...
If expensive_to_import_module does a lot of startup logic, that might mean it takes a non-negligible amount of time to import MyType. This won't really make a difference once the program is actually running, but it does make the time-to-start slower. This can feel particularly bad if you're trying to write short-lived command-line style programs: the act of adding type hints can sometimes make your program feel more sluggish when starting up.
We could fix this by making MyType a string while hiding the import behind an if TYPE_CHECKING guard, like so:
from typing import TYPE_CHECKING
# TYPE_CHECKING is False at runtime, but treated as True by type checkers.
# So the Python interpreters won't do the expensive import, but the type checker
# will still understand where MyType came from!
if TYPE_CHECKING:
from expensive_to_import_module import MyType
# This is a string reference, so we don't attempt to actually use MyType
# at runtime.
def blah(x: "MyType") -> None: ...
This works, but again looks clunky. Why should we need to add quotes around the last type? The annotations future import makes the syntax for doing this a little smoother:
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from expensive_to_import_module import MyType
# Hooray, no more quotes!
def blah(x: MyType) -> None: ...
Depending on your point-of-view, this may not seem like a huge win. But it does help make using type hints much more ergonomic, makes them feel more "integrated" into Python, and (once these become enabled by default) removes a common stumbling block newcomers to PEP 484 tend to trip over.
The following example fails because the definition for Stuff can't be found:
package com.example
import javax.script.ScriptEngineManager
object Driver5 extends App {
case class Stuff(s: String, d: Double)
val e = new ScriptEngineManager().getEngineByName("scala")
println(e.eval("""import Driver5.Stuff; Stuff("Hello", 3.14)"""))
}
I'm unable to find any import statement that allows me to use my own classes inside of the eval statement. Am I doing something wrong? How does one import classes to be used during eval?
EDIT: Clarified example code to elicit more direct answers.
The Scripting engine does not know the context. It surely can't access all the local variables and imports in the script, since they are not available in the classfiles. (Well, variable names may be optionally available as a debug information, but it is virtually impossible to use them for this purpose.)
I am not sure if there is a special API for that. Imports are different across various languages, so bringing an API that should fit them all can be difficult.
You should be able to add the imports to the eval-ed String. I am not sure if there is a better way to do this.
I'm using a utility library for dimensional analysis that i'd like to extend with my own units, and I'd like to be able to write
import my.util.units._
in files in my project. My thought was to define
package my.util
object units {
import squants._
[... other definitions ...]
}
and I expected import my.util.units._ to have the same effect as import squants._, plus the other definitions. But it seems importing units._ doesn't end up adding squants._ to the scope.
Is there a way to do this in scala?
We've dealt with this a little bit at work, and we've tried to resolve this a few ways. Here's an example of how we import rabbitmq types throughout scala-amqp:
package com.bostontechnologies
package object amqp {
type RabbitShutdownListener = com.rabbitmq.client.ShutdownListener
type RabbitConnection = com.rabbitmq.client.Connection
type RabbitChannel = com.rabbitmq.client.Channel
type RabbitConsumer = com.rabbitmq.client.Consumer
type RabbitAddress = com.rabbitmq.client.Address
...
}
So now when we import com.bostontechnologies.amqp._ we get access to the rabbitmq types that we've defined. I know it requires quite a bit of duplication, however we've found it to be somewhat useful, especially since it gives us granularity over type names.
Also, you don't need to use a package object, we mainly use it for convenience of automatically importing our types around a package. You could just use a normal object as well.
Imports are not transitive in Java or Scala. Probably the closest you are going to get to what you seek is to create an object (perhaps a package object) with a type definition for each type of interest.
Hello Pythoneers: the following code is only a mock up of what I'm trying to do, but it should illustrate my question.
I would like to know if this is dirty trick I picked up from Java programming, or a valid and Pythonic way of doing things: basically I'm creating a load of instances, but I need to track 'static' data of all the instances as they are created.
class Myclass:
counter=0
last_value=None
def __init__(self,name):
self.name=name
Myclass.counter+=1
Myclass.last_value=name
And some output of using this simple class , showing that everything is working as I expected:
>>> x=Myclass("hello")
>>> print x.name
hello
>>> print Myclass.last_value
hello
>>> y=Myclass("goodbye")
>>> print y.name
goodbye
>>> print x.name
hello
>>> print Myclass.last_value
goodbye
So is this a generally acceptable way of doing this kind of thing, or an anti-pattern ?
[For instance, I'm not too happy that I can apparently set the counter from both within the class(good) and outside of it(bad); also not keen on having to use full namespace 'Myclass' from within the class code itself - just looks bulky; and lastly I'm initially setting values to 'None' - probably I'm aping static-typed languages by doing this?]
I'm using Python 2.6.2 and the program is single-threaded.
Class variables are perfectly Pythonic in my opinion.
Just watch out for one thing. An instance variable can hide a class variable:
x.counter = 5 # creates an instance variable in the object x.
print x.counter # instance variable, prints 5
print y.counter # class variable, prints 2
print myclass.counter # class variable, prints 2
Do. Not. Have. Stateful. Class. Variables.
It's a nightmare to debug, since the class object now has special features.
Stateful classes conflate two (2) unrelated responsibilities: state of object creation and the created objects. Do not conflate responsibilities because it "seems" like they belong together. In this example, the counting of created objects is the responsibility of a Factory. The objects which are created have completely unrelated responsibilities (which can't easily be deduced from the question).
Also, please use Upper Case Class Names.
class MyClass( object ):
def __init__(self, name):
self.name=name
def myClassFactory( iterable ):
for i, name in enumerate( iterable ):
yield MyClass( name )
The sequence counter is now part of the factory, where the state and counts should be maintained. In a separate factory.
[For folks playing Code Golf, this is shorter. But that's not the point. The point is that the class is no longer stateful.]
It's not clear from question how Myclass instances get created. Lacking any clue, there isn't much more than can be said about how to use the factory. An iterable is the usual culprit. Perhaps something that iterates through a list or a file or some other iterable data structure.
Also -- for folks just of the boat from Java -- the factory object is just a function. Nothing more is needed.
Since the example on the question is perfectly unclear, it's hard to know why (1) two unique objects are created with (2) a counter. The two unique objects are already two unique objects and a counter isn't needed.
For example, the static variables in the Myclass are never referenced anywhere. That makes it very, very hard to understand the example.
x, y = myClassFactory( [ "hello", "goodbye" ] )
If the count or last value where actually used for something, then a perhaps meaningful example could be created.
You can solve this problem by splitting the code into two separate classes.
The first class will be for the object you are trying to create:
class MyClass(object):
def __init__(self, name):
self.Name = name
And the second class will create the objects and keep track of them:
class MyClassFactory(object):
Counter = 0
LastValue = None
#classmethod
def Build(cls, name):
inst = MyClass(name)
cls.Counter += 1
cls.LastValue = inst.Name
return inst
This way, you can create new instances of the class as needed, but the information about the created classes will still be correct.
>>> x = MyClassFactory.Build("Hello")
>>> MyClassFactory.Counter
1
>>> MyClassFactory.LastValue
'Hello'
>>> y = MyClassFactory.Build("Goodbye")
>>> MyClassFactory.Counter
2
>>> MyClassFactory.LastValue
'Goodbye'
>>> x.Name
'Hello'
>>> y.Name
'Goodbye'
Finally, this approach avoids the problem of instance variables hiding class variables, because MyClass instances have no knowledge of the factory that created them.
>>> x.Counter
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'MyClass' object has no attribute 'Counter'
You don't have to use a class variable here; this is a perfectly valid case for using globals:
_counter = 0
_last_value = None
class Myclass(obj):
def __init__(self, name):
self.name = name
global _counter, _last_value
_counter += 1
_last_value = name
I have a feeling some people will knee-jerk against globals out of habit, so a quick review may be in order of what's wrong--and not wrong--with globals.
Globals traditionally are variables which are visible and changeable, unscoped, from anywhere in the program. This is a problem with globals in languages like C. It's completely irrelevant to Python; these "globals" are scoped to the module. The class name "Myclass" is equally global; both names are scoped identically, in the module they're contained in. Most variables--in Python equally to C++--are logically part of instances of objects or locally scoped, but this is cleared shared state across all users of the class.
I don't have any strong inclination against using class variables for this (and using a factory is completely unnecessary), but globals are how I'd generally do it.
Is this pythonic? Well, it's definitely more pythonic than having global variables for a counter and the value of the most recent instance.
It's said in Python that there's only one right way to do anything. I can't think of a better way to implement this, so keep going. Despite the fact that many will criticize you for "non-pythonic" solutions to problems (like the needless object-orientation that Java coders like or the "do-it-yourself" attitude that many from C and C++ bring), in most cases your Java habits will not send you to Python hell.
And beyond that, who cares if it's "pythonic"? It works, and it's not a performance issue, is it?
I have a script where I do some magic stuff to dynamically load a module, and instantiate the first class found in the module. But I can't use types.ClassType anymore in Python 3. What is the correct way to do this now?
I figured it out. It seems that classes are of type "type". Here is an example of how to distinguish between classes and other objects at runtime.
>>> class C: pass
...
>>> type(C)
<class 'type'>
>>> isinstance(C, type)
True
>>> isinstance('string', type)
False
It was used for classic classes. In Python 3 they're gone.
I suppose you could use something like:
issubclass(ClassName, object)