How to convert a python 2 C extension module to python 3 - python-c-api

Question

I am trying to compile an extension module where I define a custom type. However, I got hold of some python 2 code for this purpose, but I cannot for the life of me convert it into a python 3 module.
So far, I have tweaked the original code I found in the book 'Python in a nutshell' in order to attempt to compile it for python 3, by merging some bits from https://docs.python.org/3/extending/newtypes_tutorial.html,but I am still getting nowhere. I know the 3rd edition of this book has a python 3 version of the code I am trying to compile but I don't have access to that edition of the book.
So, here is the module source code test.c:
#include "Python.h"
#include "structmember.h"
/* per-instance data structure */
typedef struct {
PyObject_HEAD
int first, second;
} intpair;
static int
intpair_init(PyObject *self, PyObject *args, PyObject *kwds)
{
static char* nams[] = {"first","second",NULL};
int first, second;
if(!PyArg_ParseTupleAndKeywords(args, kwds, "ii", nams, &first, &second))
return -1;
((intpair*)self)->first = first;
((intpair*)self)->second = second;
return 0;
}
static void
intpair_dealloc(PyObject *self)
{
self->ob_type->tp_free(self);
}
static PyObject* nothing_method(PyObject* self, PyObject* args)
{
printf("This does literally nothing!\n");
Py_RETURN_NONE;
}
static PyMemberDef intpair_members[] = {
{"first", T_INT, offsetof(intpair, first), 0, "first item" },
{"second", T_INT, offsetof(intpair, second), 0, "second item" },
{NULL}
};
static PyTypeObject t_intpair = {
PyObject_HEAD_INIT(0) /* tp_head */
0, /* tp_internal */
"test.intpair", /* tp_name */
sizeof(intpair), /* tp_basicsize */
0, /* tp_itemsize */
intpair_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_compare */
0, /* tp_as_number */
0, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
PyObject_GenericGetAttr, /* tp_getattro */
PyObject_GenericSetAttr, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT,
"two ints (first,second)",
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
0, /* tp_methods */
intpair_members, /* tp_members */
0, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
intpair_init, /* tp_init */
PyType_GenericAlloc, /* tp_alloc */
PyType_GenericNew, /* tp_new */
PyObject_Del, /* tp_free */
};
static PyMemberDef Noddy_members[] = {
{"PI", T_INT, 3, 0, "noddy number"},
{NULL, NULL, 0, NULL}
};
static PyMethodDef CosMethods[] =
{
{"does_nothing", nothing_method, METH_VARARGS, "This really does nothing"},
{NULL, NULL, 0, NULL}
};
static PyModuleDef testmodule = {
PyModuleDef_HEAD_INIT,
.m_name = "test",
.m_doc = "Example module that creates an extension type.",
.m_size = -1,
CosMethods,
};
PyMODINIT_FUNC
PyInit_test(void)
{
PyObject *m;
if (PyType_Ready(&t_intpair) < 0)
return NULL;
m = PyModule_Create(&testmodule);
if (m == NULL)
return NULL;
Py_INCREF(&t_intpair);
PyModule_AddObject(m, "Custom", (PyObject *) &t_intpair);
return m;
}
And the file setup.py:
from distutils.core import setup, Extension
setup(name="test", version="1.0",
ext_modules=[
Extension("test", ["test.c"]),
])
I can successfully compile this module using python3 setup.py build_ext --inplace, however, when I import this module in python 3 I get the following error:
Python 3.7.2 (default, Jan 10 2019, 23:51:51)
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
SystemError: Type does not define the tp_name field.
I do not understand this error message, since there is a tp_name field "test.intpair" defined in PyTypeObject. What am I missing here?
Also, what is necessary to change in this code to make it compile for python 3?
Thanks!

Answer 1

I'm pretty sure the flaw is the line 0, /* tp_internal */ in:
PyObject_HEAD_INIT(0) /* tp_head */
0, /* tp_internal */
"test.intpair", /* tp_name */
If you search for this line the only references to it on the internet are this question and the book you're using - it's completely non-standard and I have no idea where it's come from. I half suspect that the Python 2 code uses PyVarObject and then uses tp_internal to fill in the size rather than PyVarObject_HEAD_INIT, but I can't see the relevant page of the book...
The current documentation recommends using C99 initialization like
static PyTypeObject t_intpair = {
PyObject_HEAD_INIT(0)
.tp_name = "test.intpair",
// etc
};
so that getting the order of the attributes right no longer matters (anything you don't specify is 0 initialized).
Building your code provides some pretty clear warnings about mismatched types:
warning: initialization of ‘long int’ from ‘char *’ makes integer from pointer without a cast [-Wint-conversion]
"test_mod.intpair", /* tp_name */`
You should pay attention to these

Answer 2

Not a direct answer to your question, so posting this is probably against site rules. I hit this page when looking for advice about how to do a similar thing, converting a python2 - C extension module to python3 - C.
This is generally not a trivial endeavor, as the interface changed significantly. I think you will find the resources at this link helpful:
https://docs.python.org/3/howto/cporting.html