Calling db2ReadLog from external UDF - db2

When trying to get transaction log information on a DB2 LUW database through an external UDF calling db2ReadLog a SQL0487N (attempted to execute on SQL statement) is returned. Using the same shared library as a stored procedure works without error.
The reason it needs to be a UDF is to be able to eventually create a table valued UDF which will return actual entries from the transaction log.
Here is the C source for the UDF to determine the initial LRI:
#include <sqlca.h>
#include <db2ApiDf.h>
#include <sqludf.h>
#include <string.h>
#include <stdio.h>
#include <inttypes.h>
#ifdef __plusplus
extern "C"
#endif
SQL_API_RC SQL_API_FN get_initial_lri(SQLUDF_CHAR *lri, SQLUDF_NULLIND
*lri_null_ind, SQLUDF_TRAIL_ARGS)
{
struct sqlca sqlca;
db2ReadLogStruct read_log_params;
db2ReadLogInfoStruct info;
SQL_API_RC rc;
char state_msg[1024], error_msg[1024];
memset(&sqlca, 0, sizeof sqlca);
memset(&read_log_params, 0, sizeof read_log_params);
memset(&info, 0, sizeof info);
read_log_params.iCallerAction = DB2READLOG_QUERY;
read_log_params.iFilterOption = DB2READLOG_FILTER_OFF;
read_log_params.poReadLogInfo = &info;
rc = db2ReadLog(db2Version1058, &read_log_params, &sqlca);
if (rc < 0) {
memcpy(SQLUDF_STATE, "38TA0", SQLUDF_SQLSTATE_LEN);
strncpy(SQLUDF_MSGTX, "Could not query log for initial LRI", SQLUDF_MSGTEXT_LEN);
goto error;
} else if (sqlca.sqlcode < 0) {
strncpy(SQLUDF_STATE, sqlca.sqlstate, SQLUDF_SQLSTATE_LEN);
SQLUDF_MSGTX[0] = '\0';
rc = sqlaintp(error_msg, sizeof error_msg, 80, &sqlca);
if (rc > 0) {
strncpy(SQLUDF_MSGTX, error_msg, SQLUDF_MSGTEXT_LEN);
}
strncat(SQLUDF_MSGTX, "|", SQLUDF_MSGTEXT_LEN);
rc = sqlogstt(state_msg, sizeof state_msg, 80, sqlca.sqlstate);
if (rc > 0) {
strncat(SQLUDF_MSGTX, state_msg, SQLUDF_MSGTEXT_LEN);
}
goto error;
} else {
snprintf(lri, 101, "%" PRIx64 ":%" PRIx64 ":%" PRIx64, info.nextStartLRI.lriType, info.nextStartLRI.part1, info.nextStartLRI.part2);
}
return 0;
error:
return SQLZ_DISCONNECT_PROC;
}
The following SQL is used for registering the C function as an external UDF:
create or replace function get_initial_lri()
returns varchar(100)
language c
external name 'get_initial_lri_0!get_initial_lri'
parameter style sql
fenced not threadsafe
reads sql data
no external action
no scratchpad
no final call
disallow parallel
no dbinfo
And for registering the same code as a stored procedure the SQL is:
create or replace procedure get_initial_lri_sp(out lri varchar(100))
language c
dynamic result sets 0
reads sql data
not deterministic
external name 'get_initial_lri_0!get_initial_lri'
fenced not threadsafe
no external action
program type sub
no dbinfo
parameter style sql
The C source code was compiled with:
gcc -o get_initial_lri_0 get_initial_lri.c -L ~/sqllib/lib64 -ldb2 -shared -fpic -D_REENTRANT -I ~/sqllib/include
The version of DB2 is v10.5.0.8.
Is it at all possible to call the db2ReadLog API from a user defined function?

The documentation states that "connection level APIs cannot be called from within external functions or external methods". And the db2ReadLog API requires a connection. The documentation link is here.
There is another API called db2ReadLogNoconn, maybe you should try that to compare its behaviour, although it may be subject to a different restriction. It is documented here.
Otherwise, a non-external UDF can call a stored procedure (subject to some restrictions) so you may be able to wrap the stored procedure, maybe you can investigate.

Related

How can I unit test a specific DML method?

I'm writing some common DML code that contains a fairly complex method, something like:
saved uint32 checksum_ini;
method calculate_checksum(bytes_t data) -> (uint32 sum) {
uint32 result = checksum_ini;
for (int i = 0; i < data.size; ++i) {
result = f(result, data.data[i]);
}
return result;
}
My device calls the function indirectly by reading and writing some registers, which makes it cumbersome to unit test all corner cases of the checksum algorithm.
How can I efficiently write a unit test for my checksum implementation?
One approach is to create a dedicated test module, say test-checksum, containing a test device, say test_checksum_dev, that imports only your common code, and exposes the calculate_checksum method to Python, where it is easy to write tests. This is done in two steps: First, expose the method to C:
dml 1.4;
device test_checksum_dev;
import "checksum-common.dml";
// Make DML method calculate_checksum available as extern C symbol "calculate_checksum"
// The signature will be:
// uint64 calculate_checksum(conf_object_t *obj, bytes_t data)
export calculate_checksum as "calculate_checksum";
The second step is to expose it to Python. Create checksum.h:
#ifndef CHECKSUM_H
#define CHECKSUM_H
#include <simics/base/types.h>
#include <simics/pywrap.h>
extern uint32 calculate_checksum(conf_object_t *obj, bytes_t data);
#endif /* CHECKSUM_H */
(if you also add header %{ #include "checksum.h" %} to the DML file, you will get a hard check that signatures stay consistent).
Now add the header file to IFACE_FILES in your module makefile to create a Python wrapping:
SRC_FILES = test-checksum.dml
IFACE_FILES = checksum.h
include $(MODULE_MAKEFILE)
You can now call the DML method directly from your test:
SIM_load_module('test-checksum')
from simmod.test_checksum.checksum import calculate_checksum
obj = SIM_create_object('test_checksum_dev', 'dev', checksum_ini=0xdeadbeef)
assert calculate_checksum(obj, b'hello world') == (0xda39ba47).to_bytes(4, 'little')

Can the equivalent of Tcl "chan push" be implemented in C code?

I have an imbedded Tcl interpreter and want to redirect its stderr and stdout to a console widget in the application.
Using a chan push command for stderr seems to work (not much testing yet), like explained here:
TCL: Redirect output of proc to a file
I could have a file with the required tcl namespace definition, etc, and do a Tcl_Eval to source that script after creating an interp with Tcl_CreateInterp.
Can I do the same thing using Tcl C library calls instead of running the Tcl commands via a Tcl_Eval?
To implement a channel transformation in C, you first have to define a Tcl_ChannelType structure. Such a structure specifies a name for the transformation and pointers to functions for the different operations that may be done on a channel. Next, you implement the functions that perform those operations. The most important ones are inputProc and outputProc. You also have to implement a watchProc. The pointers for other operations can be set to NULL, if you don't need them.
For your example it may look something like:
static const Tcl_ChannelType colorChannelType = {
"color",
TCL_CHANNEL_VERSION_5,
NULL,
ColorTransformInput,
ColorTransformOutput,
NULL, /* seekProc */
NULL, /* setOptionProc */
NULL, /* getOptionProc */
ColorTransformWatch,
NULL, /* getHandleProc */
NULL, /* close2Proc */
NULL, /* blockModeProc */
NULL, /* flushProc */
NULL, /* handlerProc */
NULL, /* wideSeekProc */
NULL,
NULL
};
Then, when you want to push the transformation onto a channel:
chan = Tcl_StackChannel(interp, &colorChannelType, clientData,
Tcl_GetChannelMode(channel), channel);
For a complete example from the Tcl sources, see tclZlib.c
Not really an answer to my question, but maybe it will help someone to see what works by using a Tcl_Eval to show the tcl code that does the redirection.
proc redir_stdout {whichChan args} {
switch -- [lindex $args 0] {
initialize {
return {initialize write finalize}
}
write {
::HT_puts $whichChan [lindex $args 2]
}
finalize {
}
}
}
chan push stderr [list redir_stdout 1]
chan push stdout [list redir_stdout 2]
Both the chan push commands use the same proc, but pass an different identifier (1 or 2) to indicate whether stdout or stderr was the originator of the output.
HT_puts is an extension provided by the C code:
Tcl_CreateObjCommand(interp,"HT_puts",putsCmd,(ClientData) NULL,NULL);
int TclInterp::putsCmd(ClientData ,Tcl_Interp *,int objcnt,Tcl_Obj * CONST *objv)
{
if (objcnt != 3)
return TCL_ERROR;
int length;
int whichChan;
Tcl_GetIntFromObj(interp,objv[1],&whichChan);
//qDebug() << "Channel is $whichChan";
QString out =Tcl_GetStringFromObj(objv[2],&length);
QColor textColor;
if (whichChan==1)
textColor = QColor(Qt::red);
else
textColor = QColor(Qt::white);
console->putData(out.toUtf8(),textColor);
//qDebug() << out;
return TCL_OK;
}
Text forwarded from stderr gets colored red and text from stdout gets colored white.
And, as I mentioned above, each subsequent command that gets executed via Tcl_Eval needs to have the Tcl_Eval return value processed something like this:
if (rtn != TCL_OK)
{
QString output = Tcl_GetVar(interp, "errorInfo", TCL_GLOBAL_ONLY);
console->putData(output.toUtf8(),QColor(Qt::red));
//qDebug("Failed Tcl_Eval: %d \n%s\n", rtn,
}
To get what's normally printed to stderr by tclsh on a TCL_ERROR into the console (instead of the app's stderr).
I was planning to do the equivalent in C to eliminate the need to run Tcl code in the interpreter for the redirect. But, really there's no need for that.
The Tcl_Eval that does the redirection is done right after doing the Tcl_CreateInterp. Any subsequent Tcl_Evals using that interp will have stdout and stderr redirected to my application's console.
Besides, I'm having trouble understanding how to use Tcl_StackChannel and can't find an example I can follow.
Honestly, can't say that I completely understand the Tcl implementation. I made some assumptions on what gets passed to the proc used in the "chan push" command based on the referenced thread.
It looks like the proc is called with the list specified in the chan push command AND an args list. The first element of the args list is a name like "write" or "initialize". The third element looks like the string to be printed.
Still trying to find a definition of what's passed without having to dig into something like namespace ensemble.
So, it's likely that this Tcl code isn't the best implementation but it's working so far (with limited testing).

pcap_getnonblock() returns -3

I am quite new to using pcap lib, so please bear with me.
I am trying to use pcap_getnonblock function, the documentation says the following:
pcap_getnonblock() returns the current 'non-blocking' state of
the capture descriptor; it always returns 0 on 'savefiles' . If
there is an error, PCAP_ERROR is returned and errbuf is filled in
with an appropriate error message.
errbuf is assumed to be able to hold at least PCAP_ERRBUF_SIZE
chars.
I got -3 returned and the errbuf is an empty string, I couldn't understand the meaning of such result.
I believe this caused a socket error: 10065.
This problem happened only once and I could not reproduce it, but still it would be great to find its causing to prevent it in future executions.
Thanks in advance.
pcap_getnonblock() can return -3 - that's PCAP_ERROR_NOT_ACTIVATED. Unfortunately, that's not documented; I'll fix that.
Here's a minimal reproducible example that demonstrates this:
#include <pcap/pcap.h>
#include <stdio.h>
int
main(int argc, char **argv)
{
pcap_t *pcap;
char errbuf[PCAP_ERRBUF_SIZE];
if (argc != 2) {
fprintf(stderr, "Usage: this_program <interface_name>\n");
return 1;
}
pcap = pcap_create(argv[1], errbuf);
if (pcap == NULL) {
fprintf(stderr, "this_program: pcap_create(%s) failed: %s\n",
argv[1], errbuf);
return 2;
}
printf("pcap_getnonblock() returns %d on non-activated pcap_t\n",
pcap_getnonblock(pcap, errbuf));
return 0;
}
(yes, that's minimal, as 1) names of interfaces are OS-dependent, so it has to be a command-line argument and 2) if you don't run the program correctly, it should let you know what's happening, so you know what you have to do in order to reproduce the problem).
Perhaps pcap_getnonblock() and pcap_setnonblock() should be changed so that you can set non-blocking mode before activating the pcap_t, so that, when activated, it will be in non-blocking mode. It doesn't work that way currently, however.
I.e., you're allocating a pcap_t with pcap_create(), but you're not activating it with pcap_activate(). You need to do both in order to have a pcap_t on which you can capture.

PostgreSQL C function: libpq doesn't link?

I want to write a C function for PostgreSQL. For this function I will need to query some data using libpq, so I started by writing a dummy function to test this part:
#define _XOPEN_SOURCE
#include <libpq-fe.h>
#include <postgres.h>
#include <fmgr.h>
#include <funcapi.h>
#include <executor/executor.h>
#include "test.h"
PG_FUNCTION_INFO_V1(getAnnotation);
Datum getAnnotation(PG_FUNCTION_ARGS) {
// Connection to the database
PGconn *conn = PQsetdbLogin("localhost",
"5432",
"",
"",
"postgres",
"postgres",
"password");
// Databases names
PGresult *res = PQexec (conn, "SELECT user FROM activity LIMIT 1;");
VarChar* i = PQgetvalue(res, 0, 0);
PG_RETURN_VARCHAR_P(i);
}
It is just supposed to return the first column of the first line of one of my tables. Pretty simple right ? Well it doesn't work.
When I try to use it within psql, it says :
ERROR: could not load library "/usr/local/lib/postgresql/test.so": Error relocating /usr/local/lib/postgresql/test.so: PQexec: symbol not found
The weird part is that both PQsetdbLogin and PQexec are in the libpq-fe.h file, but only the second one causes an error. If I comment the PQexec line, then PQsetdbLogin raises an error as well.
Here is the Makefile I use to build the code:
PG_CPPFLAGS = -I$(libpq_srcdir)
LDFLAGS_INTERNAL = -L$(libdir)
SHLIB_LINK_INTERNAL = $(libpq)
SHLIB_PREREQS = submake-libpq
EXTENSION = test
DATA = test--0.1.sql
MODULES = test
# REGRESS = ... # Script for tests
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)
As you can see, I linked libpq therem so everything should work... But it doesn't and I don't know why.
I am using PostgreSQL 9.6 in a docker container.
If you want to link with external libraries, you need SHLIB_LINK, but that only works if you use MODULE_big instead of MODULES.
A working Makefile would be
PG_CPPFLAGS = -I$(libpq_srcdir)
SHLIB_LINK = $(libpq)
EXTENSION = test
DATA = test--0.1.sql
MODULE_big = test
OBJS = test.o
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)
But there are other problems with your function:
VarChar is a varlena, but PQgetvalue returns a char *.
You'd have to convert the value to a varlena with code similar to this:
VarChar *result;
char *c = PQgetvalue(res, 0, 0);
result = (VarChar *) palloc(strlen(c) + VARHDRSZ);
strncpy(VARDATA(result), c, strlen(c));
SET_VARSIZE(result, strlen(c) + VARHDRSZ);
It is usually the wrong idea to write client code in a server function.
If all you need is to run a query inside the current session, use the Server Programming Interface.
I guess your function is only sample code, but you need to close the connection before the backend terminates, else you have a connection leak.

multi-threaded avahi resolving causes segfault

I'm attempting to port my zeronconf-enabled C/C++ app to Linux, however I'm getting D-BUS related segfaults. I'm not sure if this is a bug in Avahi, my misuse of Avahi, or a bug in my code.
I am using a ZeroconfResolver object that encapsulates an AvahiClient,
AvahiSimplePoll, and AvahiServiceResolver. The ZeroconfResolver has a
Resolve function that first instantiates the AvahiSimplePoll, then
AvahiClient, and finally the AvahiServiceResolver. At each
instantiation I am checking for errors before continuing to the next.
After the AvahiServiceResolver has been successfully created it calls
avahi_simple_poll_loop with the AvahiSimplePoll.
This whole process works great when done synchronously but fails with
segfaults when multiple ZeroconfResolvers are being used at the same
time asynchronously (i.e I have multiple threads creating their own
ZeroconfResolver objects). A trivial adaptation of the object that
reproduces the segfaults can be seen in the code below (may not produce a
segfault right away, but in my use case it happens frequently).
I understand that "out of the box" Avahi is not thread safe, but
according to my interpretation of [1] it is safe to have multiple
AvahiClient/AvahiPoll objects in the same process as long as they are
not 'accessed' from more than one thread. Each ZeroconfResolver has
its own set of Avahi objects that do not interact with each other
across thread boundaries.
The segfaults occur in seemingly random functions within the Avahi
library. In general they happen within the avahi_client_new or
avahi_service_resolver_new functions referencing dbus. Does the Avahi wiki
mean to imply that the 'creation' of AvahiClient/AvahiPoll objects is
also not thread safe?
[1] http://avahi.org/wiki/RunningAvahiClientAsThread
#include <dispatch/dispatch.h>
#include <cstdio>
#include <sys/types.h>
#include <netinet/in.h>
#include <avahi-client/lookup.h>
#include <avahi-client/client.h>
#include <avahi-client/publish.h>
#include <avahi-common/alternative.h>
#include <avahi-common/simple-watch.h>
#include <avahi-common/malloc.h>
#include <avahi-common/error.h>
#include <avahi-common/timeval.h>
void resolve_reply(
AvahiServiceResolver *r,
AVAHI_GCC_UNUSED AvahiIfIndex interface,
AVAHI_GCC_UNUSED AvahiProtocol protocol,
AvahiResolverEvent event,
const char *name,
const char *type,
const char *domain,
const char *host_name,
const AvahiAddress *address,
uint16_t port,
AvahiStringList *txt,
AvahiLookupResultFlags flags,
void * context) {
assert(r);
if (event == AVAHI_RESOLVER_FOUND)
printf("resolve_reply(%s, %s, %s, %s)[FOUND]\n", name, type, domain, host_name);
avahi_service_resolver_free(r);
avahi_simple_poll_quit((AvahiSimplePoll*)context);
}
int main() {
// Run until segfault
while (true) {
// Adding block to conccurent GCD queue (managed thread pool)
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), [=]{
char name[] = "SomeHTTPServerToResolve";
char domain[] = "local.";
char type[] = "_http._tcp.";
AvahiSimplePoll * simple_poll = NULL;
if ((simple_poll = avahi_simple_poll_new())) {
int error;
AvahiClient * client = NULL;
if ((client = avahi_client_new(avahi_simple_poll_get(simple_poll), AVAHI_CLIENT_NO_FAIL, NULL, NULL, &error))) {
AvahiServiceResolver * resolver = NULL;
if ((resolver = avahi_service_resolver_new(client, AVAHI_IF_UNSPEC, AVAHI_PROTO_UNSPEC, name, type, domain, AVAHI_PROTO_UNSPEC, AVAHI_LOOKUP_NO_ADDRESS, (AvahiServiceResolverCallback)resolve_reply, simple_poll))) {
avahi_simple_poll_loop(simple_poll);
printf("Exit Loop(%p)\n", simple_poll);
} else {
printf("Resolve(%s, %s, %s)[%s]\n", name, type, domain, avahi_strerror(avahi_client_errno(client)));
}
avahi_client_free(client);
} else {
printf("avahi_client_new()[%s]\n", avahi_strerror(error));
}
avahi_simple_poll_free(simple_poll);
} else {
printf("avahi_simple_poll_new()[Failed]\n");
}
});
}
// Never reached
return 0;
}
One solution that seems to work fine is to add your own synchronization (a common mutex) around avahi_client_new, avahi_service_resolver_new and the corresponding free operations. It seems avahi does not claim those operation to be internally synchronized.
What is claimed is that independent objects do not interfere.
I had success with this approach, using a helper class with a static mutex. To be specific, a static member function (or free function) like this:
std::mutex& avahi_mutex(){
static std::mutex mtx;
return mtx;
}
and a lock around any section of code (as small as possible) doing free or new:
{
std::unique_lock<std::mutex> alock(avahi_mutex());
simple_poll = avahi_simple_poll_new()
}