I'm writing some common DML code that contains a fairly complex method, something like:
saved uint32 checksum_ini;
method calculate_checksum(bytes_t data) -> (uint32 sum) {
uint32 result = checksum_ini;
for (int i = 0; i < data.size; ++i) {
result = f(result, data.data[i]);
}
return result;
}
My device calls the function indirectly by reading and writing some registers, which makes it cumbersome to unit test all corner cases of the checksum algorithm.
How can I efficiently write a unit test for my checksum implementation?
One approach is to create a dedicated test module, say test-checksum, containing a test device, say test_checksum_dev, that imports only your common code, and exposes the calculate_checksum method to Python, where it is easy to write tests. This is done in two steps: First, expose the method to C:
dml 1.4;
device test_checksum_dev;
import "checksum-common.dml";
// Make DML method calculate_checksum available as extern C symbol "calculate_checksum"
// The signature will be:
// uint64 calculate_checksum(conf_object_t *obj, bytes_t data)
export calculate_checksum as "calculate_checksum";
The second step is to expose it to Python. Create checksum.h:
#ifndef CHECKSUM_H
#define CHECKSUM_H
#include <simics/base/types.h>
#include <simics/pywrap.h>
extern uint32 calculate_checksum(conf_object_t *obj, bytes_t data);
#endif /* CHECKSUM_H */
(if you also add header %{ #include "checksum.h" %} to the DML file, you will get a hard check that signatures stay consistent).
Now add the header file to IFACE_FILES in your module makefile to create a Python wrapping:
SRC_FILES = test-checksum.dml
IFACE_FILES = checksum.h
include $(MODULE_MAKEFILE)
You can now call the DML method directly from your test:
SIM_load_module('test-checksum')
from simmod.test_checksum.checksum import calculate_checksum
obj = SIM_create_object('test_checksum_dev', 'dev', checksum_ini=0xdeadbeef)
assert calculate_checksum(obj, b'hello world') == (0xda39ba47).to_bytes(4, 'little')
Related
I'm converting a pure-Python module to a C-extension to familiarize myself with the C API.
The Python implementation is as follows:
_CRC_TABLE_ = [0] * 256
def initialize_crc_table():
if _CRC_TABLE_[1] != 0: # Safeguard against re-initialization
return
# snip
def calculate_crc(data: bytes, initial: int = 0) -> int:
if _CRC_TABLE_[1] == 0: # In case user forgets to initialize first
initialize_crc_table()
# snip
# additional non-CRC methods trimmed
My C-extension thus far works:
#include <Python.h>
static Py_ssize_t CRC_TABLE_LEN = 256;
PyObject *_CRC_TABLE_;
static PyObject *method_initialize_crc_table(PyObject *self, PyObject *args) {
// snip
}
static PyMethodDef module_methods[] = {
{"initialize_crc_table", method_initialize_crc_table, METH_VARARGS, NULL},
{NULL, NULL, 0, NULL}
};
void _allocate_table_() {
_CRC_TABLE = PyList_New(CRC_TABLE_LEN);
PyObject *zero = Py_BuildValue("i", 0);
for (int i = 0; i < CRC_TABLE_LEN; i++) {
PyList_SetItem(_CRC_TABLE_, i, zero);
}
}
#if PY_MAJOR_VERSION >= 3
static struct PyModuleDef module_utilities = {
PyModuleDef_HEAD_INIT,
"utilities",
NULL,
-1,
module_methods,
};
PyMODINIT_FUNC PyInit_utilities() {
PyObject *module = PyModule_Create(&module_utilities);
_allocate_table_();
PyModule_AddObject(module, "_CRC_TABLE", _CRC_TABLE_);
return module;
}
#else
PyMODINIT_FUNC initutilities() {
PyObject *module = Py_InitModule3("utilities", module_methods, NULL);
_allocate_table_();
PyModule_AddObject(module, "_CRC_TABLE", _CRC_TABLE_);
}
I am able to access utilities._CRC_TABLE_ from the C-extension in the interpreter and values match the Python-equivalent when invoking utilities.intialize_crc_table.
Now I'm trying to call initialize_crc_table at the start of calculate_crc, performing the same check as used in the Python implementation. I'm returning None for now:
static PyObject *method_calculate_crc(PyObject *self, PyObject *args) {
if (!(uint)PyLong_AsUnsignedLong(PyList_GetItem(_CRC_TABLE_, (Py_ssize_t) 1))) {
PyObject *call_initialize_crc_table = PyObject_GetAttrString(self, "initialize_crc_table");
PyObject_CallObject(call_initialize_crc_table, NULL);
Py_DECREF(call_initialize_crc_table);
}
Py_RETURN_NONE;
}
I've added this to module_methods[] and it compiles without warnings or errors. When I run this method within the interpreter, I get a segfault. I assume it's because self isn't the module as an object.
I can do this as an alternative, which appears to work without issue:
static PyObject *method_calculate_crc(PyObject *self, PyObject *args) {
if (!(uint)PyLong_AsUnsignedLong(PyList_GetItem(_CRC_TABLE_, (Py_ssize_t) 1))) {
method_initialize_crc_table(self, NULL);
}
Py_RETURN_NONE;
}
However, I am not certain if I should be passing self, NULL, or something else to the method.
What is the proper way of invoking method_initialize_crc_table from method_calculate_crc?
There was a "gotcha" here that I must clarify on. While the code was intended for Python 3, development was initially done in Python 2 as the development files were not yet available on the machine I was using. This shed some light on some differences in how each version handles things. David's comments helped lead to this clarification.
If a method is defined as METH_VARARGS but is defined for a module (versus a class), Python 2 does not pass anything for the PyObject *self parameter. This is noted in the documentation but is easy to overlook if you're not careful. Python 3, however, does pass a pointer to the module. As DavidW recommended, I implemented a global variable to hold a reference to the module. Assuming his claims of Python handling the de-referencing at exit are correct, we can safely use this for accessing module global attributes.
With our issue of PyObject *self solved, we no longer get a segfault. We can then address the question of which approach is (seemingly more) correct for calling a method within the local scope of the module. Do we do this:
if (/* conditional */)
PyObject_CallMethod(module, "initialize_crc_table", NULL);
Or this:
if (/* conditional */)
method_initialize_crc_table(self, args, kwargs);
Benchmarks seem to provide an answer here. Using Python's built-in timeit module, we can see a very clear difference in terms of performance. Note that so far in our implementation, .calculate_crc accesses ._CRC_TABLE_ and checks if it's initialized, but no processing occurs. Performance compared to Python 2 and 3 were identical and thus ignored.
The command is as follows:
python3 -m timeit "import utilities; utilities.calculate_crc(0)"
PyObject_CallMethod: 874 nsec per loop
method_initialize_crc_table: 44.3 usec per loop
Using the PyObject_ function is reported as 50x faster, quite a significant difference. Benchmarks alone do not facilitate what is "more correct" but with no clear guidance it may be a sufficient justification for our use. Therefore, I will be using PyObject_ calls for this project.
In the fftw3 documentation the standard example is:
#include <fftw3.h>
...
{
fftw_complex *in, *out;
fftw_plan p;
...
in = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * N);
out = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * N);
p = fftw_plan_dft_1d(N, in, out, FFTW_FORWARD, FFTW_ESTIMATE);
...
fftw_execute(p); /* repeat as needed */
...
fftw_destroy_plan(p);
fftw_free(in); fftw_free(out);
}
The following syntax also works:
#include <fftw3.h>
{
fftw_complex *in, *out;
fftw_plan p;
...
in = new fftw_complex[N];
out = new fftw_complex[N];
p = fftw_plan_dft_1d(N, in, out, FFTW_FORWARD, FFTW_ESTIMATE);
...
fftw_execute(p); /* repeat as needed */
...
fftw_destroy_plan(p);
delete [] in;
delete [] out;
}
I would like to use smart pointers instead, perhaps something like:
#include <fftw3.h>
...
{
fftw_plan p;
...
auto *in = std::make_shared<fftw_complex[N]>();
auto *out = std::make_shared<fftw_complex[N]>();
p = fftw_plan_dft_1d(N, in, out, FFTW_FORWARD, FFTW_ESTIMATE);
...
fftw_execute(p); /* repeat as needed */
...
}
But I can't seem to get the syntax worked out. Has anyone used smart pointers along with the FFTW3 library?
I would advise caution in using shared pointers in this context. Yes, you can give a raw pointer to the FFTW planning function (details below). However, this will not increment the reference counter to the shared pointer. This is problematic because the FFTW plan will know about the memory, but the shared pointer thinks it nobody needs the memory such that the deleter may be called. You'll get a segfault.
Shared pointers have no implicit conversion to the raw pointer, you'd want to use the '.get()' method. I.e.
p = fftw_plan_dft_1d(N, in.get(), out.get(), FFTW_FORWARD, FFTW_ESTIMATE);
But don't do this
When trying to get transaction log information on a DB2 LUW database through an external UDF calling db2ReadLog a SQL0487N (attempted to execute on SQL statement) is returned. Using the same shared library as a stored procedure works without error.
The reason it needs to be a UDF is to be able to eventually create a table valued UDF which will return actual entries from the transaction log.
Here is the C source for the UDF to determine the initial LRI:
#include <sqlca.h>
#include <db2ApiDf.h>
#include <sqludf.h>
#include <string.h>
#include <stdio.h>
#include <inttypes.h>
#ifdef __plusplus
extern "C"
#endif
SQL_API_RC SQL_API_FN get_initial_lri(SQLUDF_CHAR *lri, SQLUDF_NULLIND
*lri_null_ind, SQLUDF_TRAIL_ARGS)
{
struct sqlca sqlca;
db2ReadLogStruct read_log_params;
db2ReadLogInfoStruct info;
SQL_API_RC rc;
char state_msg[1024], error_msg[1024];
memset(&sqlca, 0, sizeof sqlca);
memset(&read_log_params, 0, sizeof read_log_params);
memset(&info, 0, sizeof info);
read_log_params.iCallerAction = DB2READLOG_QUERY;
read_log_params.iFilterOption = DB2READLOG_FILTER_OFF;
read_log_params.poReadLogInfo = &info;
rc = db2ReadLog(db2Version1058, &read_log_params, &sqlca);
if (rc < 0) {
memcpy(SQLUDF_STATE, "38TA0", SQLUDF_SQLSTATE_LEN);
strncpy(SQLUDF_MSGTX, "Could not query log for initial LRI", SQLUDF_MSGTEXT_LEN);
goto error;
} else if (sqlca.sqlcode < 0) {
strncpy(SQLUDF_STATE, sqlca.sqlstate, SQLUDF_SQLSTATE_LEN);
SQLUDF_MSGTX[0] = '\0';
rc = sqlaintp(error_msg, sizeof error_msg, 80, &sqlca);
if (rc > 0) {
strncpy(SQLUDF_MSGTX, error_msg, SQLUDF_MSGTEXT_LEN);
}
strncat(SQLUDF_MSGTX, "|", SQLUDF_MSGTEXT_LEN);
rc = sqlogstt(state_msg, sizeof state_msg, 80, sqlca.sqlstate);
if (rc > 0) {
strncat(SQLUDF_MSGTX, state_msg, SQLUDF_MSGTEXT_LEN);
}
goto error;
} else {
snprintf(lri, 101, "%" PRIx64 ":%" PRIx64 ":%" PRIx64, info.nextStartLRI.lriType, info.nextStartLRI.part1, info.nextStartLRI.part2);
}
return 0;
error:
return SQLZ_DISCONNECT_PROC;
}
The following SQL is used for registering the C function as an external UDF:
create or replace function get_initial_lri()
returns varchar(100)
language c
external name 'get_initial_lri_0!get_initial_lri'
parameter style sql
fenced not threadsafe
reads sql data
no external action
no scratchpad
no final call
disallow parallel
no dbinfo
And for registering the same code as a stored procedure the SQL is:
create or replace procedure get_initial_lri_sp(out lri varchar(100))
language c
dynamic result sets 0
reads sql data
not deterministic
external name 'get_initial_lri_0!get_initial_lri'
fenced not threadsafe
no external action
program type sub
no dbinfo
parameter style sql
The C source code was compiled with:
gcc -o get_initial_lri_0 get_initial_lri.c -L ~/sqllib/lib64 -ldb2 -shared -fpic -D_REENTRANT -I ~/sqllib/include
The version of DB2 is v10.5.0.8.
Is it at all possible to call the db2ReadLog API from a user defined function?
The documentation states that "connection level APIs cannot be called from within external functions or external methods". And the db2ReadLog API requires a connection. The documentation link is here.
There is another API called db2ReadLogNoconn, maybe you should try that to compare its behaviour, although it may be subject to a different restriction. It is documented here.
Otherwise, a non-external UDF can call a stored procedure (subject to some restrictions) so you may be able to wrap the stored procedure, maybe you can investigate.
I had to deal with a really old codebase in my company which had C++ apis exposed via perl.
In on of the code reviews, I suggested it was necessary to garbage collect memory which was being allocated in c++.
Here is the skeleton of the code:
char* convert_to_utf8(char *src, int length) {
.
.
.
length = get_utf8_length(src);
char *dest = new char[length];
.
.
// No delete
return dest;
}
Perl xs definition:
PROTOTYPE: ENABLE
char * _xs_convert_to_utf8(src, length)
char *src
int length
CODE:
RETVAL = convert_to_utf8(src, length)
OUTPUT:
RETVAL
so, I had a comment that the memory created in the c++ function will not garbage collected by Perl. And 2 java developers think it will crash since perl will garbage collect the memory allocated by c++. I suggested the following code.
CLEANUP:
delete[] RETVAL
Am I wrong here?
I also ran this code and showed them the increasing memory utilization, with and without the CLEANUP section. But, they are asking for exact documentation which proves it and I couldn't find it.
Perl Client:
use ExtUtils::testlib;
use test;
for (my $i=0; $i<100000000;$i++) {
my $a = test::hello();
}
C++ code:
#define PERL_NO_GET_CONTEXT
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"
#include <stdio.h>
char* create_mem() {
char *foo = (char*)malloc(sizeof(char)*150);
return foo;
}
XS code:
MODULE = test PACKAGE = test
char * hello()
CODE:
RETVAL = create_mem();
OUTPUT:
RETVAL
CLEANUP:
free(RETVAL);
I'm afraid that the people who wrote (and write) the Perl XS documentation probably consider it too obvious that Perl cannot magically detect memory allocation made in other languages (like C++) to document that explicitly. There's a bit in the perlguts documentation page that says that all memory to be used via the Perl XS API must use Perl's macros to do so that may help you argue.
When you write XS code, you're writing C (or sometimes C++) code. You still need to write proper C/C++, which includes deallocating allocated memory when appropriate.
The glue function you desire XS to create is the following:
void hello() {
dSP; // Declare and init SP, the stack pointer used by mXPUSHs.
char* mem = create_mem();
mXPUSHs(newSVpv(mem, 0)); // Create a scalar, mortalize it, and push it on the stack.
free(mem); // Free memory allocated by create_mem().
XSRETURN(1);
}
newSVpv makes a copy of mem rather than taking possession of it, so the above clearly shows that free(mem) is needed to deallocate mem.
In XS, you could write that as
void hello()
CODE:
{ // A block is needed since we're declaring vars.
char* mem = create_mem();
mXPUSHs(newSVpv(mem, 0));
free(mem);
XSRETURN(1);
}
Or you could take advantage of XS features such as RETVAL and CLEANUP.
SV* hello()
char* mem; // We can get rid of the block by declaring vars here.
CODE:
mem = create_mem();
RETVAL = newSVpv(mem, 0); // Values returned by SV* subs are automatically mortalized.
OUTPUT:
RETVAL
CLEANUP: // Happens after RETVAL has been converted
free(mem); // and the converted value has been pushed onto the stack.
Or you could also take advantage of the typemap, which defines how to convert the returned value into a scalar.
char* hello()
CODE:
RETVAL = create_mem();
OUTPUT:
RETVAL
CLEANUP:
free(RETVAL);
All three of these are perfectly acceptable.
A note on mortals.
Mortalizing is a delayed reference count decrement. If you were to decrement the SV created by hello before hello returns, it would get deallocated before hello returns. By mortalizing it instead, it won't be deallocated until the caller has a chance to inspect it or take possession of it (by increasing its reference count).
I'm attempting to port my zeronconf-enabled C/C++ app to Linux, however I'm getting D-BUS related segfaults. I'm not sure if this is a bug in Avahi, my misuse of Avahi, or a bug in my code.
I am using a ZeroconfResolver object that encapsulates an AvahiClient,
AvahiSimplePoll, and AvahiServiceResolver. The ZeroconfResolver has a
Resolve function that first instantiates the AvahiSimplePoll, then
AvahiClient, and finally the AvahiServiceResolver. At each
instantiation I am checking for errors before continuing to the next.
After the AvahiServiceResolver has been successfully created it calls
avahi_simple_poll_loop with the AvahiSimplePoll.
This whole process works great when done synchronously but fails with
segfaults when multiple ZeroconfResolvers are being used at the same
time asynchronously (i.e I have multiple threads creating their own
ZeroconfResolver objects). A trivial adaptation of the object that
reproduces the segfaults can be seen in the code below (may not produce a
segfault right away, but in my use case it happens frequently).
I understand that "out of the box" Avahi is not thread safe, but
according to my interpretation of [1] it is safe to have multiple
AvahiClient/AvahiPoll objects in the same process as long as they are
not 'accessed' from more than one thread. Each ZeroconfResolver has
its own set of Avahi objects that do not interact with each other
across thread boundaries.
The segfaults occur in seemingly random functions within the Avahi
library. In general they happen within the avahi_client_new or
avahi_service_resolver_new functions referencing dbus. Does the Avahi wiki
mean to imply that the 'creation' of AvahiClient/AvahiPoll objects is
also not thread safe?
[1] http://avahi.org/wiki/RunningAvahiClientAsThread
#include <dispatch/dispatch.h>
#include <cstdio>
#include <sys/types.h>
#include <netinet/in.h>
#include <avahi-client/lookup.h>
#include <avahi-client/client.h>
#include <avahi-client/publish.h>
#include <avahi-common/alternative.h>
#include <avahi-common/simple-watch.h>
#include <avahi-common/malloc.h>
#include <avahi-common/error.h>
#include <avahi-common/timeval.h>
void resolve_reply(
AvahiServiceResolver *r,
AVAHI_GCC_UNUSED AvahiIfIndex interface,
AVAHI_GCC_UNUSED AvahiProtocol protocol,
AvahiResolverEvent event,
const char *name,
const char *type,
const char *domain,
const char *host_name,
const AvahiAddress *address,
uint16_t port,
AvahiStringList *txt,
AvahiLookupResultFlags flags,
void * context) {
assert(r);
if (event == AVAHI_RESOLVER_FOUND)
printf("resolve_reply(%s, %s, %s, %s)[FOUND]\n", name, type, domain, host_name);
avahi_service_resolver_free(r);
avahi_simple_poll_quit((AvahiSimplePoll*)context);
}
int main() {
// Run until segfault
while (true) {
// Adding block to conccurent GCD queue (managed thread pool)
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), [=]{
char name[] = "SomeHTTPServerToResolve";
char domain[] = "local.";
char type[] = "_http._tcp.";
AvahiSimplePoll * simple_poll = NULL;
if ((simple_poll = avahi_simple_poll_new())) {
int error;
AvahiClient * client = NULL;
if ((client = avahi_client_new(avahi_simple_poll_get(simple_poll), AVAHI_CLIENT_NO_FAIL, NULL, NULL, &error))) {
AvahiServiceResolver * resolver = NULL;
if ((resolver = avahi_service_resolver_new(client, AVAHI_IF_UNSPEC, AVAHI_PROTO_UNSPEC, name, type, domain, AVAHI_PROTO_UNSPEC, AVAHI_LOOKUP_NO_ADDRESS, (AvahiServiceResolverCallback)resolve_reply, simple_poll))) {
avahi_simple_poll_loop(simple_poll);
printf("Exit Loop(%p)\n", simple_poll);
} else {
printf("Resolve(%s, %s, %s)[%s]\n", name, type, domain, avahi_strerror(avahi_client_errno(client)));
}
avahi_client_free(client);
} else {
printf("avahi_client_new()[%s]\n", avahi_strerror(error));
}
avahi_simple_poll_free(simple_poll);
} else {
printf("avahi_simple_poll_new()[Failed]\n");
}
});
}
// Never reached
return 0;
}
One solution that seems to work fine is to add your own synchronization (a common mutex) around avahi_client_new, avahi_service_resolver_new and the corresponding free operations. It seems avahi does not claim those operation to be internally synchronized.
What is claimed is that independent objects do not interfere.
I had success with this approach, using a helper class with a static mutex. To be specific, a static member function (or free function) like this:
std::mutex& avahi_mutex(){
static std::mutex mtx;
return mtx;
}
and a lock around any section of code (as small as possible) doing free or new:
{
std::unique_lock<std::mutex> alock(avahi_mutex());
simple_poll = avahi_simple_poll_new()
}