I have an in-house C++ library that I've successfully exposed to Python using Boost.Python. It accepts a user-created Python object and then uses some methods within that object to perform certain tasks, and it works quite well for the most part.
The Python use of the library looks like:
class Foo(object):
def __init__(self, args):
"""create an instance of this class with instance-specific attributes"""
def Bar1(self, a, b, c):
"""do something with the given integers a, b and c"""
return a + (b*c)
def Bar2(self, a, b, c):
"""do something else with the given integers a, b and c"""
print (a*b) + c
import mylib
cheese = mylib.Wine()
qux = Foo()
cheese.setup(qux)
cheese.do_something(1)
cheese.do_something(2)
The "Wine" object in C++ looks like:
#include <boost/python.h>
#include <Python.h>
class Wine {
public:
Wine() {};
~Wine() {};
void setup(boost::python::object baz) {
baz_ = baz;
};
static void do_something(boost::python::object pyreq) {
int request = boost::python::extract<int>(pyreq);
int a = 1;
int b = 2;
int c = 3;
if (request == 1) {
int d = boost::python::extract<int>(baz_.attr("Bar1")(a, b, c));
};
else if (request == 2) {
baz_.attr("Bar2")(a, b, c);
};
};
private:
static boost::python::object baz_;
};
BOOST_PYTHON_MODULE(mylib)
{
using namespace boost::python;
class_<Wine>("Wine")
.def("do_something", &Wine::do_something)
.staticmethod("do_something")
.def("setup", &Wine::setup)
;
};
The problem is that, after successfully executing all of the tasks, the program terminates with a SegFault. This isn't really a huge deal because the code that I need to execute still executes, and the tasks that I need to perform are all performed. The SegFault only occurs on the destruction of this C++ "Wine" object. Still, it's an inelegant outcome and I'd like to fix the problem.
What I could gather from an online search implied that this is a problem with improper declaration of ownership to Python. The end result is that the C++ destructor gets called twice, and the second call causes a SegFault.
Unfortunately I haven't been able to remedy the problem so far. Available documentation only covers the basics and I haven't been able to replicate some success others have had using boost smart pointers and some fancy declaration/destruction tricks in C++ with it. Any guidance would be much appreciated.
The problem is that the static Wine::baz_ object is being destroyed during static/global destruction. This is after the Python runtime has been finalized, but since boost::python::object uses the Python C-API, its destruction requires a valid Python runtime (though possible not if the object refers to None.) By arranging for baz_ to be destroyed before Python finalization, you should be able to avoid the segfault. The cleanest approach overall might be to make baz_ a non-static member variable.
Related
I have a class in c++ that I'm wrapping into python with pybind11. That class has a std::function, and I'd like to control how the arguments to that function are dealt with (like return value policies). I just can't find the syntax or examples to do this...
class N {
public:
using CallbackType = std::function<void(const OtherClass*)>;
N(CallbackType callback): callback(callback) { }
CallbackType callback;
void doit() {
OtherClass * o = new OtherClass();
callback(o);
}
}
wrapped with
py::class_<OtherClass>(...standard stuff...);
py::class_<N>(m, "N")
.def(py::init<N::CallbackType>(),
py::arg("callback"));
I all works: I can do this in python:
def callback(o):
dosomethingwith(o)
k = N(callback)
, but I'd like to be able to control what happens when callback(o); is called - whether python then will take ownership of the wrapped o variable or not, basically.
I put a printout in the destructor of OtherClass, and as far as I can tell, it never gets called.
OK, I think I figured it out:
Instead of std::function, use a pybind11::function:
using CallbackType = pybind11::function
and then
void doit(const OtherClass &input) {
if (<I want to copy it>) {
callback(pybind11::cast(input, pybind11::return_value_policy::copy));
} else {
callback(pybind11::cast(input, pybind11::return_value_policy::reference));
}
}
I see nothing in pybind11/functional that allows you to change the ownership of the parameters at the point of call, as the struct func_wrapper used is function local, so can not be specialized. You could provide another wrapper yourself, but in the code you can't know whether the callback is a Python function or a bound C++ function (well, technically you can if that bound C++ function is bound by pybind11, but you can't know in general). If the function is C++, then changing Python ownership in the wrapper would be the wrong thing to do, as the temporary proxy may destroy the object even as its payload is stored by the C++ callback.
Do you have control over the implementation of class N? The reason is that by using std::shared_ptr all your ownership problems will automagically evaporate, regardless of whether the callback function is C++ or Python and whether it stores the argument or not. Would work like so, expanding on your example above:
#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
namespace py = pybind11;
class OtherClass {};
class N {
public:
using CallbackType = std::function<void(const std::shared_ptr<OtherClass>&)>;
N(CallbackType callback): callback(callback) { }
CallbackType callback;
void doit() {
auto o = std::make_shared<OtherClass>();
callback(o);
}
};
PYBIND11_MODULE(example, m) {
py::class_<OtherClass, std::shared_ptr<OtherClass>>(m, "OtherClass");
py::class_<N>(m, "N")
.def(py::init<N::CallbackType>(), py::arg("callback"))
.def("doit", &N::doit);
}
The newtypes tutorial shows you how to inherit from a base python class. Can you inherit from your own python class? Something like this?
PyObject *mod = PyImport_AddModule("foomod");
PyObject *o = PyObject_GetAttrString(mod, "BaseClass");
PyTypeObject *t = o->ob_type;
FooType.tp_base = t;
if (PyType_Ready(&FooType ) < 0) return NULL;
though you need to define your struct with the base class as the first member per the documentation so it sounds like this is not possible? ie how would I setup the Foo struct?
typedef struct {
PyListObject list;
int state;
} SubListObject;
What I'm really trying to do is subclass _UnixSelectorEventLoop and it seems like my only solution is to define a python class that derives from my C class and from _UnixSelectorEventLoop with my C class listed first so that it can override methods in the other base class.
I think you're basically right on your assessment:
it seems like my only solution is to define a python class that derives from my C class and from _UnixSelectorEventLoop with my C class listed first so that it can override methods in the other base class.
You can't define a class that inherits from a Python class because it'd need to start with a C struct of basically arbitrary size.
There's a couple of other options that you might like to consider:
You could create a class the manual way by calling PyType_Type. See this useful answer on a question about multiple inheritance which is another sort of inheritance that the C API struggles with. This probably limits you too much, since you can't have C attributes, but you can have C functions.
You could do "inheritance by composition" - i.e. have you _UnixSelectorEventLoop as part of the object, then forward __getattr__ and __setattr__ to it in the event of unknown attributes. It's probably easier to see what I mean with Python code (which is simply but tediously transformed into C API code)
class YourClass:
def __init__(self,...):
self.state = 0
self._usel = _UnixSelectorEventLoop()
def __getattr__(self, name):
return getattr(self._usel, 'name')
def __setattr__(self, name, value):
if name in self.__dict__:
object.__setattr__(self, name, value)
else:
setattr(self._usel, name, value)
# maybe __hasattr__ and __delattr__ too?
I'm hoping to avoid having to write this C API code myself, but the slots are tp_getattro and tp_setattro. Note that __getattr__ will need to be more comprehensive in the C version, since it acts closer to the __getattribute__ in Python. The flipside is that isinstance and issubclass will fail, which may or may not be an issue for you.
I used to do C++ development several years ago and back then I found it difficult to combine template programming with OOP. Currently I program in Swift and I tried doing some of the things I struggled with then.
This Swift code will illustrate the problem:
// protocol is like Java interface or C++ pure virtual base class
protocol Log {
// want to able to add elements from a collection of Ints, but
// it should be any sort of collection that
// can be treated as a sequence
func add<T: SequenceType where T.Generator.Element == Int>(values: T)
}
class DiscreteLog: Log {
var vals: [Int] = []
func add<T: SequenceType where T.Generator.Element == Int>(values: T) {
for v in values {
vals.append(v)
}
}
}
class ContinousLog: Log {
var vals: [Double] = []
func add<T: SequenceType where T.Generator.Element == Int>(values: T) {
for v in values {
vals.append(Double(v))
}
}
}
// I don't have to know whether the log is Continuous or Discrete
// I can still add elements to it
var log: Log = ContinousLog()
log.add([1, 2, 3])
// and elements can come from any kind of sequence, it does not need
// to be an array
log.add(["four": 4, "five: 5].values)
So the problem is that if the C++ code defined as as:
virtual void add(vector<Int> elements>)
Then sure I could have multiple subclasses implement this method, but I could never provide anything but vectors as arguments.
I could try changing it to something more generic using iterator:
virtual void add(vector<Int>::iterator elements>)
But I am still limited to using vector iterators. So I guess I would have to write something like:
template<typename Iterator>
virtual void add(Iterator elements>)
But that will give compile errors as template based arguments are not allowed for virtual methods.
Anyway I wondered if this sort of thing is possible in modern C++.
C++ templates and C#/Swift/Java generics are different things.
They are both "pattern code" in a sense (they are patterns that generate code), but C#/Swift/Java generics use type erasure and "forget" almost everything about the types they work with, while C++ templates are elephants. And elephants never forget.
It turns out that can make an elephant forget, but you have to tell it to. The technique of "forgetting" about details of a type is known as "type erasure" or "run time concepts".
So you want to type erase down to the concept of "a sequence of integers". You want to take any type, so long as it is a sequence of integers, and be able to iterate over it. Seems fair.
boost has such type erasures. But who wants to always rely on boost?
First, type erase an input iterator:
template<class T>
struct input_iterator:
std::iterator<
std::input_iterator_tag, // category
T, // value
std::ptrdiff_t, // distance
T*, // pointer
T // reference
>
{
struct erase {
virtual void advance() = 0;
virtual erase* clone() const = 0;
virtual T get() const = 0;
virtual bool equal(erase const& o) = 0;
virtual ~erase() {}
};
std::unique_ptr<erase> pimpl;
input_iterator(input_iterator&&)=default;
input_iterator& operator=(input_iterator&&)=default;
input_iterator()=default;
input_iterator& operator++() {
pimpl->advance();
return *this;
}
input_iterator operator++(int) {
auto copy = *this;
++*this;
return copy;
}
input_iterator(input_iterator const& o):
pimpl(o.pimpl?o.pimpl->clone():nullptr)
{}
input_iterator& operator=(input_iterator const&o) {
if (!o.pimpl) {
if (pimpl) pimpl->reset();
return *this;
}
pimpl = std::unique_ptr<erase>(o.pimpl->clone());
return *this;
}
T operator*() const {
return pimpl->get();
}
friend bool operator==( input_iterator const& lhs, input_iterator const& rhs ) {
return lhs.pimpl->equal(*rhs.pimpl);
}
friend bool operator!=( input_iterator const& lhs, input_iterator const& rhs ) {
return !(lhs==rhs);
}
template<class It>
struct impl:erase{
It it;
impl(impl const&)=default;
impl(It in):it(std::move(in)){}
virtual void advance() override { ++it; }
virtual erase* clone() const override { return new impl(*this); }
virtual T get() const override { return *it; }
virtual bool equal(erase const& o) override {
return static_cast<impl const&>(o).it == it;
}
};
template<
class It,
class=std::enable_if<
std::is_convertible<
typename std::iterator_traits<It>::reference,
T
>{}
>
>
input_iterator(It it):pimpl( new impl<It>{it} ) {}
}; // input_iterator
Next, have a range template. This is a container that stores non-type erased iterators, and exposes enough to iterate over those iterators.
template<class It>
struct range {
It b; It e;
It begin() const { return b; }
It end() const { return e; }
range() = default;
range(It start, It finish):b(std::move(start)),e(std::move(finish)) {};
range(range&&)=default;
range(range const&)=default;
range& operator=(range&&)=default;
range& operator=(range const&)=default;
template<class R,
class R_It=std::decay_t<decltype(std::begin(std::declval<R>()))>,
class=std::enable_if< std::is_convertible<R_It, It>{} >
>
range( R&& r ):
range(std::begin(r), std::end(r))
{} // TODO: enable ADL begin lookup
};
The above type is really basic: C++1z has better ones, as does boost, as do I have in my own code base. But it is enough to handle for(:) loops, and implicit conversion from containers with compatible iterators.
Finally our sequence type:
template<class T>
using sequence_of = range<input_iterator<T>>;
Wait, that's it? Nice, those types compose well!
And barring errors, we are done.
Your code now would take a sequence_of<int>, and they could pass a std::vector<int> or std::list<int> or whatever.
The input_iterator type-erasure type-erases any iterator down to getting a T via *, ==, copy, and ++ advance, which is enough for a for(:) loop.
The range<input_iterator<int>> will accept any iterable range (including containers) whose iterators can be converted to an input_iterator<int>.
The downside? We just introduced a bunch of overhead. Each method goes through virtual dispatch, from ++ to * to ==.
This is (roughly) what generics do -- they type-erase down to the requirements you give it in the generic clause. This means they are working with abstract objects, not concrete objects, so they unavoidably suffer performance penalties of this indirection.
C++ templates can be used to generate type erasure, and there are even tools (boost has some) to make it easier. What I did above is a half-assed manual one. Similar techniques are used in std::function<R(Args...)>, which type-erases down to (conceptually) {copy, call with (Args...) returning R, destroy} (plus some incidentals).
live example.
(The code above freely uses C++14.)
So the C++ equivalent Log is:
struct Log {
virtual void add(sequence_of<int>) = 0;
virtual ~Log() {}
};
Now, the type erasure code above is a bit ugly. To be fair, I just implemented a language feature in C++ without direct language support for it.
I've seen some proposals to make type erasure easier in C++. I do not know the status of those proposals.
If you want to do your own, here is an "easy" way to do type erasure in 3 steps:
First, determine what operations you want to erase. Write the equivalent of input_iterator<T> -- give it a bunch of methods and operators that do what you want. Be sparse. Call this the "external type". Ideally nothing in this type is virtual, and it should be a Regular or Semi-regular type (ie, it should behave value-like, or move-only-value-like). Don't implement anything but the interface yet.
Second, write an inner class erase. It provides a pure-virtual interface to a set of functions that could provide what you need in your external type.
Store a unique_ptr<erase> pimpl; within the external type. Forward the methods you expose in the external type to the pimpl;.
Third, write an inner template<class X> class impl<X>:erase. It stores a variable X x;, and it implements everything in erase by interacting with X. It should be constructable from an X (with optional perfect forwarding).
You then create a perfect forwarding constructor for the external type that creates its pimpl via a new impl<X>(whatever). Ideally it should check that its argument is a valid one via SFINAE techniques, but that is just a qualify of implementation issue.
Now the external type "erases" the type of any object it is constructed from "down to" the operations you exposed.
Now, for your actual problem, I'd write array_view or steal std::experimental::array_view, and restrict my input to be any kind of contiguous buffer of data of that type. This is more performant, and accepting any sequence is over engineering unless you really need it.
I've just started making libraries in Arduino. I've made a library named inSerialCmd. I want to call a function named delegate() that is defined in the main program file, stackedcontrol.ino, after the inSerialCmd library is included.
When I try to compile, one error is thrown:
...\Arduino\libraries\inSerialCmd\inSerialCmd.cpp: In member function
'void inSerialCmd::serialListen()':
...\Arduino\libraries\inSerialCmd\inSerialCmd.cpp:32: error:
'delegate' has not been declared
After doing a bit of searching, it seemed that adding the scope resolution operator might do the trick. So I added the "::" before delegate(), now "::delegate()", but the same error is thrown.
Now I'm stumped.
You cannot and should not directly call a function in a program from a library. Keep in mind a key aspect that makes a library into a library:
A library does not depend on the specific application. A library can be fully compiled and packaged into the .a file without the existence of a program.
So there is a one way dependency, a program depends on a library. This at first glance may seem to prevent you from achieving what you want. You can achieve the functionality you are asking about through what is sometimes referred to as a callback. The main program would provide to the library at runtime a pointer to the function to execute.
// in program somwehere
int myDelegate(int a, int b);
// you set this to the library
setDelegate( myDelegate );
You see this in the arduino if you look at how interrupt handlers are installed. This same concept exists in many environments - event listeners, action adapters - all with the same goal of allowing a program to define the specific action that a library cannot know.
The library would store and call the function via the function pointer. Here is a rough sketch of what this looks like:
// in the main program
int someAction(int t1, int t2) {
return 1;
}
/* in library
this is the delegate function pointer
a function that takes two int's and returns an int */
int (*fpAction)(int, int) = 0;
/* in library
this is how an application registers its action */
void setDelegate( int (*fp)(int,int) ) {
fpAction = fp;
}
/* in libary
this is how the library can safely execute the action */
int doAction(int t1, int t2) {
int r;
if( 0 != fpAction ) {
r = (*fpAction)(t1,t2);
}
else {
// some error or default action here
r = 0;
}
return r;
}
/* in program
The main program installs its delegate, likely in setup() */
void setup () {
...
setDelegate(someAction);
...
(note: this is related to Usage preference between a struct and a class in D language but for a more specific use case)
When writing a D interface to, say, C++ code, SWIG and others do something like this:
class A{
private _A*ptr;//defined as extern(C) elsewhere
this(){ptr=_A_new();}//ditto
this(string s){ptr=_A_new(s);} //ditto
~this(){_A_delete(ptr);} //ditto
void fun(){_A_fun(ptr);}
}
Let's assume no inheritance is needed.
My question is: wouldn't it be preferable to use a struct instead of a class for this?
The pros being:
1) efficiency (stack allocation)
2) ease-of-use (no need to write new everywhere, eg: auto a=A(B(1),C(2)) vs auto a=new A(new B(1),new C(2)) )?
The cons being:
require additional field is_own to handle aliasing via postblit.
What would be the best way to do so?
Is there anything else to worry about?
Here's an attempt:
struct A{
private _A*ptr;
bool is_own;//required for postblit
static A opCall(){//cannot write this() for struct
A a;
a.ptr=_A_new();
a.is_own=true;
return a;
}
this(string s){ptr=_A_new(s); is_own=true;}
~this(){if(is_own) _A_delete(ptr);}
void fun(){_A_fun(ptr);}
this(this){//postblit;
//shallow copy: I don't want to call the C++ copy constructor (expensive or unknown semantics)
is_own=false; //to avoid _A_delete(ptr)
}
}
Note the postblit is necessary for cases when calling functions such as:
myfun(A a){}
I suggest that you read this page. The only functions on C++ classes that you can call in D are virtual functions. That means that
D cannot call C++ special member functions, and vice versa. These include constructors, destructors, conversion operators, operator overloading, and allocators.
And when you declare a C++ class in D, you use an extern(C++) interface. So, your class/struct would look like this
extern(C++) interface A
{
void fun();
}
However, you'd need another extern(C++) function to allocate any objects of type A, since it's C++ code that has to do that as the D code doesn't have access to any of the constructors. You'd also need a way to pass it back to C++ code to be deleted when you're done with it.
Now, if you want to wrap that interface in a type which is going to call the extern(C++) function to construct it and the extern(C++) function to delete it (so that you don't have to worry about doing that manually), then whether you use a class or struct depends entirely on what you're trying to do with it.
A class would be a reference type, which mirrors what the C++ class actually is. So, passing it around would work without you having to do anything special. But if you wanted a guarantee that the wrapped C++ object was freed, you'd have to do so manually, because there's no guarantee that the D class' finalizer would ever be run (and presumably, that's where you'd put the code for calling the C++ function to delete the C++ object). You'd have to either use clear (which will actually be renamed to destroy in the next release of the compiler - dmd 2.060) to destroy the D object (i.e. call its finalizer and handle the destruction of any of its member variables which are value types), or you'd have to call a function on the D object which called the C++ function to delete the C++ object. e.g.
extern(C++) interface A
{
void fun();
}
extern(C++) A createA();
extern(C++) void deleteA(A a);
class Wrapper
{
public:
this()
{
_a = createA();
}
~this()
{
deleteA(_a);
}
auto opDispatch(string name, Args...)(Args args)
{
return mixin("_a." ~ name ~ "(args)");
}
private:
A _a;
}
void main()
{
auto wrapped = new Wrapper();
//do stuff...
//current
clear(wrapped);
//starting with dmd 2.060
//destroy(wrapped);
}
But that does have the downside that if you don't call clear/destroy, and the garbage collector never collects your wrapper object, deleteA will never be called on the C++ object. That may or may not matter. It depends on whether the C++ object really needs its destructor to be called before the program terminates or whether it can just let its memory return to the OS (without its destructor being called) when the program terminates if the GC never needs to collect the wrapper object.
If you want deterministic destruction, then you need a struct. That means that you'll need to worry about making the struct into a reference type. Otherwise, if it gets copied, when one of them is destroyed, the C++ object will be deleted, and the other struct will point to garbage (which it will then try and delete when it gets destroyed). To solve that, you could use std.typecons.RefCounted. Then you get something like
extern(C++) interface A
{
void fun();
}
extern(C++) A createA();
extern(C++) void deleteA(A a);
struct Wrapper
{
public:
static Wrapper opCall()
{
Wrapper retval;
retval._a = createA();
return retval;
}
~this()
{
if(_a !is null)
{
deleteA(_a);
_a = null;
}
}
auto opDispatch(string name, Args...)(Args args)
{
return mixin("_a." ~ name ~ "(args)");
}
private:
A _a;
}
void main()
{
auto wrapped = RefCounted!Wrapper();
//do stuff...
}
You could also define the wrapper so that it has the ref-counting logic in it and avoid RefCounted, but that would definitely be more complicated.
Regardless, I would definitely advise against your suggestion of using a bool to mark whether the wrapper owns the C++ object or not, because if the original wrapper object gets destroyed before all of the copies do, then your copies will point to garbage.
Another option if you did want the C++ object's copy constructor to be used (and therefore treat the C++ object as a value type) would be to add an extern(C++) function which took the C++ object and returned a copy of it and then use it in a postblit.
extern(C++) A copyA(A a);
this(this)
{
if(_a !is null)
_a = copyA(a);
}
Hopefully that makes things clear enough.