I created a small C wrapper function which requires two bytes variables:
static PyObject * wrapperfunction(PyObject * self, PyObject * args) {
const unsigned char * data;
Py_ssize_t datalen;
const unsigned char * otherdata;
Py_ssize_t otherlen;
if (!PyArg_ParseTuple(args, "y#y#", &data, &datalen, &otherdata, &otherlen))
return NULL;
some_function(data, datalen, otherdata, otherlen);
}
But I noticed that on Linux 64bit the function would fail in certain cases (I could not really narrow them down to a special case) and segfault inside some_function as data was a not readable address.
usually this address would be 0x7fff00000001
I would not see why this was happening but changed the code to use Py_buffer instead - which works perfectly:
static PyObject * wrapperfunction(PyObject * self, PyObject * args) {
Py_buffer data;
Py_buffer otherdata;
if (!PyArg_ParseTuple(args, "y*y*", &data, &otherdata))
return NULL;
some_function((unsigned char *)data.buf, data.len, (unsigned char *)otherdata, otherdata.len);
}
As far as I can tell, the python documentation only says that y* is the preferred method, but not that y# would only work once.
Is there any reason why the method using y# fails?
I'm using Python 3.5.3 on debian stretch amd64.
On a Windows machine (python 3.6.4 / x64), the same code never produced a segfault.
Related
I'm converting a pure-Python module to a C-extension to familiarize myself with the C API.
The Python implementation is as follows:
_CRC_TABLE_ = [0] * 256
def initialize_crc_table():
if _CRC_TABLE_[1] != 0: # Safeguard against re-initialization
return
# snip
def calculate_crc(data: bytes, initial: int = 0) -> int:
if _CRC_TABLE_[1] == 0: # In case user forgets to initialize first
initialize_crc_table()
# snip
# additional non-CRC methods trimmed
My C-extension thus far works:
#include <Python.h>
static Py_ssize_t CRC_TABLE_LEN = 256;
PyObject *_CRC_TABLE_;
static PyObject *method_initialize_crc_table(PyObject *self, PyObject *args) {
// snip
}
static PyMethodDef module_methods[] = {
{"initialize_crc_table", method_initialize_crc_table, METH_VARARGS, NULL},
{NULL, NULL, 0, NULL}
};
void _allocate_table_() {
_CRC_TABLE = PyList_New(CRC_TABLE_LEN);
PyObject *zero = Py_BuildValue("i", 0);
for (int i = 0; i < CRC_TABLE_LEN; i++) {
PyList_SetItem(_CRC_TABLE_, i, zero);
}
}
#if PY_MAJOR_VERSION >= 3
static struct PyModuleDef module_utilities = {
PyModuleDef_HEAD_INIT,
"utilities",
NULL,
-1,
module_methods,
};
PyMODINIT_FUNC PyInit_utilities() {
PyObject *module = PyModule_Create(&module_utilities);
_allocate_table_();
PyModule_AddObject(module, "_CRC_TABLE", _CRC_TABLE_);
return module;
}
#else
PyMODINIT_FUNC initutilities() {
PyObject *module = Py_InitModule3("utilities", module_methods, NULL);
_allocate_table_();
PyModule_AddObject(module, "_CRC_TABLE", _CRC_TABLE_);
}
I am able to access utilities._CRC_TABLE_ from the C-extension in the interpreter and values match the Python-equivalent when invoking utilities.intialize_crc_table.
Now I'm trying to call initialize_crc_table at the start of calculate_crc, performing the same check as used in the Python implementation. I'm returning None for now:
static PyObject *method_calculate_crc(PyObject *self, PyObject *args) {
if (!(uint)PyLong_AsUnsignedLong(PyList_GetItem(_CRC_TABLE_, (Py_ssize_t) 1))) {
PyObject *call_initialize_crc_table = PyObject_GetAttrString(self, "initialize_crc_table");
PyObject_CallObject(call_initialize_crc_table, NULL);
Py_DECREF(call_initialize_crc_table);
}
Py_RETURN_NONE;
}
I've added this to module_methods[] and it compiles without warnings or errors. When I run this method within the interpreter, I get a segfault. I assume it's because self isn't the module as an object.
I can do this as an alternative, which appears to work without issue:
static PyObject *method_calculate_crc(PyObject *self, PyObject *args) {
if (!(uint)PyLong_AsUnsignedLong(PyList_GetItem(_CRC_TABLE_, (Py_ssize_t) 1))) {
method_initialize_crc_table(self, NULL);
}
Py_RETURN_NONE;
}
However, I am not certain if I should be passing self, NULL, or something else to the method.
What is the proper way of invoking method_initialize_crc_table from method_calculate_crc?
There was a "gotcha" here that I must clarify on. While the code was intended for Python 3, development was initially done in Python 2 as the development files were not yet available on the machine I was using. This shed some light on some differences in how each version handles things. David's comments helped lead to this clarification.
If a method is defined as METH_VARARGS but is defined for a module (versus a class), Python 2 does not pass anything for the PyObject *self parameter. This is noted in the documentation but is easy to overlook if you're not careful. Python 3, however, does pass a pointer to the module. As DavidW recommended, I implemented a global variable to hold a reference to the module. Assuming his claims of Python handling the de-referencing at exit are correct, we can safely use this for accessing module global attributes.
With our issue of PyObject *self solved, we no longer get a segfault. We can then address the question of which approach is (seemingly more) correct for calling a method within the local scope of the module. Do we do this:
if (/* conditional */)
PyObject_CallMethod(module, "initialize_crc_table", NULL);
Or this:
if (/* conditional */)
method_initialize_crc_table(self, args, kwargs);
Benchmarks seem to provide an answer here. Using Python's built-in timeit module, we can see a very clear difference in terms of performance. Note that so far in our implementation, .calculate_crc accesses ._CRC_TABLE_ and checks if it's initialized, but no processing occurs. Performance compared to Python 2 and 3 were identical and thus ignored.
The command is as follows:
python3 -m timeit "import utilities; utilities.calculate_crc(0)"
PyObject_CallMethod: 874 nsec per loop
method_initialize_crc_table: 44.3 usec per loop
Using the PyObject_ function is reported as 50x faster, quite a significant difference. Benchmarks alone do not facilitate what is "more correct" but with no clear guidance it may be a sufficient justification for our use. Therefore, I will be using PyObject_ calls for this project.
I build a code in Xcode console with C++ project works perfectly before:
#include "SerialPort.hpp"
#include "TypeAbbreviations.hpp"
#include <iostream>
int main(int argc, const char * argv[]) {
//* Open port, and connect to a device
const char devicePathStr[] = "/dev/tty.usbserial-A104RXG4";
const int baudRate = 9600;
int sfd = openAndConfigureSerialPort(devicePathStr, baudRate);
if (sfd < 0) {
if (sfd == -1) {
printf("Unable to connect to serial port.\n");
}
else { //sfd == -2
printf("Error setting serial port attributes.\n");
}
return 0;
}
// * Read using readSerialData(char* bytes, size_t length)
// * Write using writeSerialData(const char* bytes, size_t length)
// * Remember to flush potentially buffered data when necessary
// * Close serial port when done
const char dataToWrite[]="abcd";
char databuffer[1024];
while(1){
readSerialData(databuffer, 4);
sleep(2);
writeSerialData(databuffer, 4);
sleep(2);
}
printf("end.\n");
return 0;
}
After this build, I tried to migrate it to my Xcode cocoa application with C++ wrappers below.
I am pretty sure my Wrapper works fine with test C++ code. That means, I can call C++ function from my ViewController.swift.
But there's one strange thing happened. I am not able to open connection with the following code:
sfd = open(portPath, (O_RDWR | O_NOCTTY | O_NDELAY));
if (sfd == -1) {
printf("Unable to open serial port: %s at baud rate: %d\n", portPath, baudRate);
printf("%s", std::strerror(errno));
return sfd;
}
There error message returns :
Unable to open serial port: /dev/tty.usbserial-A104RXG4 at baud rate: 9600
Operation not permitted
I've tried to change app sandbox configuration, set up my system preference to grant access to my app, also I disabled my rootless. (csrutil disable with command + R)
But the problem still persists:
&
I want to ask that:
1. Why my code on Xcode C++ project works fine but fail on swift's cocoa app on Xcode?
2. How to solve the "Operation not permitted" Issue.
My Xcode version is 11.3.1 and Mac OS is 10.14.6 Mojave.
I figure it out myself.
It's APP sandbox is bothering me.
All you need to do is turn off sandbox
Turn off it by click X on the mouse point.
If you want to add it back, just click +Capability and put it back on.
https://i.stack.imgur.com/ZOc18.jpg
reference : https://forums.developer.apple.com/thread/94177#285075
I had to deal with a really old codebase in my company which had C++ apis exposed via perl.
In on of the code reviews, I suggested it was necessary to garbage collect memory which was being allocated in c++.
Here is the skeleton of the code:
char* convert_to_utf8(char *src, int length) {
.
.
.
length = get_utf8_length(src);
char *dest = new char[length];
.
.
// No delete
return dest;
}
Perl xs definition:
PROTOTYPE: ENABLE
char * _xs_convert_to_utf8(src, length)
char *src
int length
CODE:
RETVAL = convert_to_utf8(src, length)
OUTPUT:
RETVAL
so, I had a comment that the memory created in the c++ function will not garbage collected by Perl. And 2 java developers think it will crash since perl will garbage collect the memory allocated by c++. I suggested the following code.
CLEANUP:
delete[] RETVAL
Am I wrong here?
I also ran this code and showed them the increasing memory utilization, with and without the CLEANUP section. But, they are asking for exact documentation which proves it and I couldn't find it.
Perl Client:
use ExtUtils::testlib;
use test;
for (my $i=0; $i<100000000;$i++) {
my $a = test::hello();
}
C++ code:
#define PERL_NO_GET_CONTEXT
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"
#include <stdio.h>
char* create_mem() {
char *foo = (char*)malloc(sizeof(char)*150);
return foo;
}
XS code:
MODULE = test PACKAGE = test
char * hello()
CODE:
RETVAL = create_mem();
OUTPUT:
RETVAL
CLEANUP:
free(RETVAL);
I'm afraid that the people who wrote (and write) the Perl XS documentation probably consider it too obvious that Perl cannot magically detect memory allocation made in other languages (like C++) to document that explicitly. There's a bit in the perlguts documentation page that says that all memory to be used via the Perl XS API must use Perl's macros to do so that may help you argue.
When you write XS code, you're writing C (or sometimes C++) code. You still need to write proper C/C++, which includes deallocating allocated memory when appropriate.
The glue function you desire XS to create is the following:
void hello() {
dSP; // Declare and init SP, the stack pointer used by mXPUSHs.
char* mem = create_mem();
mXPUSHs(newSVpv(mem, 0)); // Create a scalar, mortalize it, and push it on the stack.
free(mem); // Free memory allocated by create_mem().
XSRETURN(1);
}
newSVpv makes a copy of mem rather than taking possession of it, so the above clearly shows that free(mem) is needed to deallocate mem.
In XS, you could write that as
void hello()
CODE:
{ // A block is needed since we're declaring vars.
char* mem = create_mem();
mXPUSHs(newSVpv(mem, 0));
free(mem);
XSRETURN(1);
}
Or you could take advantage of XS features such as RETVAL and CLEANUP.
SV* hello()
char* mem; // We can get rid of the block by declaring vars here.
CODE:
mem = create_mem();
RETVAL = newSVpv(mem, 0); // Values returned by SV* subs are automatically mortalized.
OUTPUT:
RETVAL
CLEANUP: // Happens after RETVAL has been converted
free(mem); // and the converted value has been pushed onto the stack.
Or you could also take advantage of the typemap, which defines how to convert the returned value into a scalar.
char* hello()
CODE:
RETVAL = create_mem();
OUTPUT:
RETVAL
CLEANUP:
free(RETVAL);
All three of these are perfectly acceptable.
A note on mortals.
Mortalizing is a delayed reference count decrement. If you were to decrement the SV created by hello before hello returns, it would get deallocated before hello returns. By mortalizing it instead, it won't be deallocated until the caller has a chance to inspect it or take possession of it (by increasing its reference count).
I'm attempting to port my zeronconf-enabled C/C++ app to Linux, however I'm getting D-BUS related segfaults. I'm not sure if this is a bug in Avahi, my misuse of Avahi, or a bug in my code.
I am using a ZeroconfResolver object that encapsulates an AvahiClient,
AvahiSimplePoll, and AvahiServiceResolver. The ZeroconfResolver has a
Resolve function that first instantiates the AvahiSimplePoll, then
AvahiClient, and finally the AvahiServiceResolver. At each
instantiation I am checking for errors before continuing to the next.
After the AvahiServiceResolver has been successfully created it calls
avahi_simple_poll_loop with the AvahiSimplePoll.
This whole process works great when done synchronously but fails with
segfaults when multiple ZeroconfResolvers are being used at the same
time asynchronously (i.e I have multiple threads creating their own
ZeroconfResolver objects). A trivial adaptation of the object that
reproduces the segfaults can be seen in the code below (may not produce a
segfault right away, but in my use case it happens frequently).
I understand that "out of the box" Avahi is not thread safe, but
according to my interpretation of [1] it is safe to have multiple
AvahiClient/AvahiPoll objects in the same process as long as they are
not 'accessed' from more than one thread. Each ZeroconfResolver has
its own set of Avahi objects that do not interact with each other
across thread boundaries.
The segfaults occur in seemingly random functions within the Avahi
library. In general they happen within the avahi_client_new or
avahi_service_resolver_new functions referencing dbus. Does the Avahi wiki
mean to imply that the 'creation' of AvahiClient/AvahiPoll objects is
also not thread safe?
[1] http://avahi.org/wiki/RunningAvahiClientAsThread
#include <dispatch/dispatch.h>
#include <cstdio>
#include <sys/types.h>
#include <netinet/in.h>
#include <avahi-client/lookup.h>
#include <avahi-client/client.h>
#include <avahi-client/publish.h>
#include <avahi-common/alternative.h>
#include <avahi-common/simple-watch.h>
#include <avahi-common/malloc.h>
#include <avahi-common/error.h>
#include <avahi-common/timeval.h>
void resolve_reply(
AvahiServiceResolver *r,
AVAHI_GCC_UNUSED AvahiIfIndex interface,
AVAHI_GCC_UNUSED AvahiProtocol protocol,
AvahiResolverEvent event,
const char *name,
const char *type,
const char *domain,
const char *host_name,
const AvahiAddress *address,
uint16_t port,
AvahiStringList *txt,
AvahiLookupResultFlags flags,
void * context) {
assert(r);
if (event == AVAHI_RESOLVER_FOUND)
printf("resolve_reply(%s, %s, %s, %s)[FOUND]\n", name, type, domain, host_name);
avahi_service_resolver_free(r);
avahi_simple_poll_quit((AvahiSimplePoll*)context);
}
int main() {
// Run until segfault
while (true) {
// Adding block to conccurent GCD queue (managed thread pool)
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), [=]{
char name[] = "SomeHTTPServerToResolve";
char domain[] = "local.";
char type[] = "_http._tcp.";
AvahiSimplePoll * simple_poll = NULL;
if ((simple_poll = avahi_simple_poll_new())) {
int error;
AvahiClient * client = NULL;
if ((client = avahi_client_new(avahi_simple_poll_get(simple_poll), AVAHI_CLIENT_NO_FAIL, NULL, NULL, &error))) {
AvahiServiceResolver * resolver = NULL;
if ((resolver = avahi_service_resolver_new(client, AVAHI_IF_UNSPEC, AVAHI_PROTO_UNSPEC, name, type, domain, AVAHI_PROTO_UNSPEC, AVAHI_LOOKUP_NO_ADDRESS, (AvahiServiceResolverCallback)resolve_reply, simple_poll))) {
avahi_simple_poll_loop(simple_poll);
printf("Exit Loop(%p)\n", simple_poll);
} else {
printf("Resolve(%s, %s, %s)[%s]\n", name, type, domain, avahi_strerror(avahi_client_errno(client)));
}
avahi_client_free(client);
} else {
printf("avahi_client_new()[%s]\n", avahi_strerror(error));
}
avahi_simple_poll_free(simple_poll);
} else {
printf("avahi_simple_poll_new()[Failed]\n");
}
});
}
// Never reached
return 0;
}
One solution that seems to work fine is to add your own synchronization (a common mutex) around avahi_client_new, avahi_service_resolver_new and the corresponding free operations. It seems avahi does not claim those operation to be internally synchronized.
What is claimed is that independent objects do not interfere.
I had success with this approach, using a helper class with a static mutex. To be specific, a static member function (or free function) like this:
std::mutex& avahi_mutex(){
static std::mutex mtx;
return mtx;
}
and a lock around any section of code (as small as possible) doing free or new:
{
std::unique_lock<std::mutex> alock(avahi_mutex());
simple_poll = avahi_simple_poll_new()
}
I have just updated to Lion and now my app is crashing which was working fine in older version. It crash on memset function with no logs.
unsigned char *theValue;
add(theValue, someotherValues);
I have passed theValue reference to function
add(unsigned char *inValue, some other perameter) {
memset(inValue,0,sizeOf(inValue)); // **here it is crashing**
}
Is there really no code between the declaration of theValue and the call to add()? If so, then that's your problem. You are passing a random value as the first parameter to memset().
For this code to make sense, you have to allocate a block of memory for theValue and pass its size to add(), like so:
unsigned char *theValue = new unsigned char[BUFSIZE]; // Or malloc
add(theValue, BUFSIZE, ...);
void add(unsigned char *inValue, size_t bufsize, ...) {
memset(inValue, 0, bufsize);
...
}
Do you allocate memory for inValue?
1)
add(unsigned char *inValue, some other perameter) {
inValue = (unsigned char*)malloc(sizeof(inValue));
memset(inValue,0,sizeOf(inValue)); // **here it is crashing**
}
2)
theValue = (unsigned char*)malloc(sizeof(inValue));
add(theValue, ...)
unsigned char *theValue;
This points to a random bit of memory (or 0). Until you call malloc you don't own what it's pointing at so you can't really memset it.