Increasing max size of Mongo BSON Objects results in Boost library compile error - mongodb

I need to increase the max size of BSON objects in Mongo. Since the BSON object size is not directly configurable, I've attempted to do this by changing the constant in the source and recompiling.
The problem is that any attempt to change const int BSONObjMaxUserSize = 16 * 1024 * 1024; to anything other than 16 in this file causes a crash when the boost library goes to compile.
Leaving BSONObjMaxUserSize as 16 compiles perfectly, but changing it to anything else fails. I've tried numerous options (8,32, even 7 for good measure.)
I'm experiencing this in version 2.6.6 stable.
In file included from src/third_party/boost/boost/type_traits/type_with_alignment.hpp:19:0,
from src/third_party/boost/boost/smart_ptr/make_shared.hpp:17,
from src/third_party/boost/boost/make_shared.hpp:15,
from src/third_party/boost/boost/smart_ptr.hpp:28,
from src/mongo/pch.h:50,
from src/mongo/db/dur_commitjob.cpp:31:
src/third_party/boost/boost/static_assert.hpp:124:85: error: invalid application of 'sizeof' to incomplete type 'boost::STATIC_ASSERTION_FAILURE<false>'
sizeof(::boost::STATIC_ASSERTION_FAILURE< BOOST_STATIC_ASSERT_BOOL_CAST( B ) >)>\
^
src/mongo/db/dur_commitjob.cpp:147:9: note: in expansion of macro 'BOOST_STATIC_ASSERT'
BOOST_STATIC_ASSERT( sizeof(void*)==4 || UncommittedBytesLimit > BSONObjMaxInternalSize * 6 );
^
src/third_party/boost/boost/static_assert.hpp:124:86: error: template argument 1 is invalid
sizeof(::boost::STATIC_ASSERTION_FAILURE< BOOST_STATIC_ASSERT_BOOL_CAST( B ) >)>\
^
src/mongo/db/dur_commitjob.cpp:147:9: note: in expansion of macro 'BOOST_STATIC_ASSERT'
BOOST_STATIC_ASSERT( sizeof(void*)==4 || UncommittedBytesLimit > BSONObjMaxInternalSize * 6 );
^
src/mongo/db/dur_commitjob.cpp:147:102: error: invalid type in declaration before ';' token
BOOST_STATIC_ASSERT( sizeof(void*)==4 || UncommittedBytesLimit > BSONObjMaxInternalSize * 6 );
^
In file included from src/third_party/boost/boost/filesystem/v3/path_traits.hpp:23:0,
from src/third_party/boost/boost/filesystem/v3/path.hpp:25,
from src/third_party/boost/boost/filesystem/path.hpp:31,
from src/mongo/util/paths.h:21,
from src/mongo/db/storage/durable_mapped_file.h:35,
from src/mongo/db/dur.h:34,
from src/mongo/db/dur_commitjob.h:34,
from src/mongo/db/dur_commitjob.cpp:33:
src/third_party/boost/boost/system/error_code.hpp:214:36: error: 'boost::system::posix_category' defined but not used [-Werror=unused-variable]
static const error_category & posix_category = generic_category();
^
src/third_party/boost/boost/system/error_code.hpp:215:36: error: 'boost::system::errno_ecat' defined but not used [-Werror=unused-variable]
static const error_category & errno_ecat = generic_category();
^
src/third_party/boost/boost/system/error_code.hpp:216:36: error: 'boost::system::native_ecat' defined but not used [-Werror=unused-variable]
static const error_category & native_ecat = system_category();
^
cc1plus: all warnings being treated as errors
scons: *** [build/linux2/normal/mongo/db/dur_commitjob.o] Error 1
scons: building terminated because of errors.

It's not been a problem here:
cloned 0ae940b456ceafc3a0d47 from github
changed builder.h:
diff --git a/src/mongo/bson/util/builder.h b/src/mongo/bson/util/builder.h
index ae31074..8e34aee 100644
--- a/src/mongo/bson/util/builder.h
+++ b/src/mongo/bson/util/builder.h
## -63,7 +63,7 ## namespace mongo {
update $push (append) operation
various db.eval() type operations
*/
- const int BSONObjMaxUserSize = 16 * 1024 * 1024;
+ const int BSONObjMaxUserSize = 24 * 1024 * 1024;
/*
Sometimes we need objects slightly larger - an object in the replication local.oplog
Compiles cleanly with scons -j 8. Here's my system information:
OS version
/tmp/mongo$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty
kernel version
/tmp/mongo$ uname -a
Linux desktop 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
gcc version
/tmp/mongo$ gcc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.8.2-19ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1)
boost version
/tmp/mongo$ dpkg --status libboost-all-dev
Package: libboost-all-dev
Status: install ok installed
Priority: optional
Section: libdevel
Installed-Size: 26
Maintainer: Ubuntu Developers <ubuntu-devel-discuss#lists.ubuntu.com>
Architecture: amd64
Source: boost-defaults
Version: 1.54.0.1ubuntu1

Related

DBX showing incorrect values in shared library built with -g

-bash-3.2$ uname -a
SunOS b2s-sol10spr-1 5.10 Generic_147147-26 sun4v sparc sun4v
-bash-3.2$ dbx -V
Sun DBX Debugger 8.0 SunOS_sparc 2014/10/21
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 8.0' in your .dbxrc
(dbx)
I'm getting a weird issue where the debugger is just showing completely wrong values for function parameters & local/member variables.
Here's a sample backtrace
=>[1] SQLGetDiagRecW(HandleType = -28417, Handle = 0x480000000101dfc1, RecNumber = -12288, Sqlstate = (nil), NativeError = 0xffffffff7fffdf, MessageText = 0x9100000001002bbd, BufferLength = -9583, TextLength = 0x380000000101da91), line 3154 in "CInterface.cpp"
[2] extract_diag_error_w(htype = 3, handle = 0x3, connection = 0x1014d32c0, head = 0x101e5a5e0, return_code = -1, save_to_diag = 0), line 4447 in "__info.c"
[3] function_return_ex(level = 3, handle = 0x101e5a1a0, ret_code = -1, save_to_diag = 0), line 4863 in "__info.c"
[4] SQLExecDirectW(statement_handle = 0x101e5a1a0, statement_text = 0x101dacde0, text_length = -3), line 442 in "SQLExecDirectW.c"
<truncated>
Frame 1 is in the problematic shared library, while the other frames are in the unixODBC libodbc.so
If we look at the code for extract_diag_error_w (you can get the code from http://www.unixodbc.org/unixODBC-2.3.1.tar.gz, the following snippet is from DriverManager/__info.c )
ret = SQLGETDIAGRECW( connection,
head -> handle_type,
handle,
rec_number,
sqlstate,
&native,
msg1,
sizeof( msg1 ),
&len );
Note that we have, elsewhere
#define SQLGETDIAGRECW(con,typ,han,rn,st,nat,msg,bl,tlp)\
(con->functions[77].funcW)\
(typ,han,rn,st,nat,msg,bl,tlp)
in dbx, if I go to frame 2 and print head -> handle_type, I get 3, but if I go to frame 1 and print HandleType, I get -28417.
I'd expect dbx, if it can't find the value (due to optimization or w/e, even though this was a debug build), to tell me so (<value optimized out> or somesuch). Also, every value I look at seems to be wrong (I've confirmed that the debugger is lying to me via control flow, there's code like if (x < 0) throw ...; where dbx claims it's some negative number, but the exception doesn't get thrown.
The problematic shared libary was built on another machine (while unixODBC was built locally), so maybe that's involved? It was also built w/ gcc, where I used the default compiler when building unixODBC (though I think it's also w/ gcc, since they both have a dependency on libgcc_s.so.1)
Any suggestions? (I'd guess updating dbx might work, but it would be difficult in this environment).
edit: Here's an example w/ local variables
(dbx) next
t#1 (l#1) stopped in Simba::Support::ODBCStringConverter::ConvertWStringToSQLWCHARString at line 64 in file "ODBCStringConverter.cpp"
64 simba_int32 length = in_length;
(dbx) next
t#1 (l#1) stopped in Simba::Support::ODBCStringConverter::ConvertWStringToSQLWCHARString at line 66 in file "ODBCStringConverter.cpp"
66 if (0 == in_length)
(dbx) print length
length = 11
(dbx) print in_length
in_length = 419430399
(dbx) next
t#1 (l#1) stopped in Simba::Support::ODBCStringConverter::ConvertWStringToSQLWCHARString at line 73 in file "ODBCStringConverter.cpp"
73 else if (SIMBA_NTS == in_length)
(dbx) next
t#1 (l#1) stopped in Simba::Support::ODBCStringConverter::ConvertWStringToSQLWCHARString at line 79 in file "ODBCStringConverter.cpp"
79 simba_int32 bufferBytes = length;
(dbx) next
t#1 (l#1) stopped in Simba::Support::ODBCStringConverter::ConvertWStringToSQLWCHARString at line 80 in file "ODBCStringConverter.cpp"
80 if (!in_isLengthInBytes)
(dbx) print bufferBytes
bufferBytes = 1744830464
(dbx)
edit 2: Here's how the troublesome library was built (need to rebuild unixODBC, will do that later).
Example command for compiling one of the .cpp files (I removed a bunch of include directories)
/opt/csw/gcc4/bin/g++ -DSIZEOF_LONG_INT=8 -DSQL_WCHART_CONVERT -DHAVE_MEMMOVE -m64 -fPIC -pthread -Wall -Wno-unknown-pragmas -DSIMBA -D_REENTRANT -DCLUNIX -DNDEBUG -D_POSIX_PTHREAD_SEMANTICS -O0 -g -D_DEBUG -c Common/QSTableMetadataFile.cpp -o Common/QSTableMetadataFile_solaris10sparc_gcc4_9_debug64.cpp.o
Command used to link (removed a bunch of .o files)
/opt/csw/gcc4/bin/g++ -DSIMBA -D_REENTRANT -m64 -fPIC -pthread -Wall -Wno-unknown-pragmas -lrt -O0 -g -shared -L/bamboo/bamboo-agent-home/xml-data/build-dir/ThirdParty/icu/53.1.x/solaris10sparc/gcc4_9/release64/lib -lstdc++ -licudata_sb64 -licui18n_sb64 -licuuc_sb64 -lpthread -lm -lsocket -lnsl -Wl,-M,exports_SunOS.map -Wl,-zallextract,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libSimbaDSI.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libSimbaSupport.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libAEProcessor.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libCore.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libDSIExt.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libExecutor.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libParser.a,/bamboo/bamboo-agent-home/xml-data/build-dir/SimbaEngine/Maintenance/10.1/Product/Lib/solaris10sparc/gcc4_9/debug64/libSimbaODBC.a -Wl,-zweakextract -o ../Bin/solaris10sparc/gcc4_9/debug64/libQuickstart64.so

ExecError: error invoking 'nvcc --version': [Errno 2] No such file or directory: 'nvcc': 'nvcc'

I am trying this code on my spyder(python3).
import pycuda.autoinit
import pycuda.driver as drv
import numpy
from pycuda.compiler import SourceModule
mod = SourceModule("""
__global__ void multiply_them(float *dest, float *a, float *b)
{
const int i = threadIdx.x;
dest[i] = a[i] * b[i];
}
""")
multiply_them = mod.get_function("multiply_them")
a = numpy.random.randn(400).astype(numpy.float32)
b = numpy.random.randn(400).astype(numpy.float32)
dest = numpy.zeros_like(a)
multiply_them(
drv.Out(dest), drv.In(a), drv.In(b),
block=(400,1,1), grid=(1,1))
print (dest-a*b)
But couldn't run the code.
ExecError: error invoking 'nvcc --version': [Errno 2] No such file or
directory: 'nvcc': 'nvcc'
My nvcc -- version is
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61
which nvcc
/usr/local/cuda-8.0/bin/nvcc
which spyder
/home/anaconda3/bin/spyder
$ which conda
/home/anaconda3/bin/conda
Device query :
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce 940MX"
CUDA Driver Version / Runtime Version 9.0 / 8.0
echo $PATH
/home/anaconda3/bin:/usr/local/cuda-8.0/bin:/home/bin:/home/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin
I have tried running the code on terminal. Then its running , but from the sourceModule line itself it shows errors.
"/home/anaconda3/lib/python3.6/site-packages/pycuda-2017.1-py3.6-linux-x86_64.egg/pycuda/compiler.py", line 255, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/home/anaconda3/lib/python3.6/site-packages/pycuda-2017.1-py3.6-linux-x86_64.egg/pycuda/compiler.py", line 137, in compile_plain
stderr=stderr.decode("utf-8", "replace"))
pycuda.driver.CompileError: nvcc compilation of /tmp/tmpbdkxu33o/kernel.cu failed
[command: nvcc --cubin -arch sm_50 -I/home/anaconda3/lib/python3.6/site-packages/pycuda-2017.1-py3.6-linux-x86_64.egg/pycuda/cuda kernel.cu]
[stderr:
In file included from /usr/local/cuda-8.0/bin/..//include/cuda_runtime.h:78:0,
from <command-line>:0:
/usr/local/cuda-8.0/bin/..//include/host_config.h:119:2: error: #error -- unsupported GNU version! gcc versions later than 5 are not supported!
#error -- unsupported GNU version! gcc versions later than 5 are not supported!
^~~~~
]
Also i have tried on my Jupiter notebook and on other Cuda installed system. But couldn't identify the problem. Any help is appreciable.
Its working on python2 virtual environment through terminal.

snmpget “No such object available on this agent at this OID”

I have been working on trying to get a value with a custom MIB through SNMP following this tutorial. I'm using net-snmp 5.7.3. I have used this MIB along with the corresponding codes with this.
My setup: I have two VM's, both Ubuntu 16, one is snmp-server with IP:192.168.5.20 and the other snmp-agent with IP:192.168.5.21.
I did MIBS=ALL and export MIBS. On the agent, I have successfully compiled my MIB and got the expected output. My code files are in the path /home/snmp-agent/Desktop/proj/net-snmp-5.7.3/agent/mibgroup/. I reconfigure mthe net-snmp like this:
./configure --with-mib-modules="pingCtlTable pingResultsTable pingProbeHistoryTable"
make
make install
Everything worked fine. And when I do snmptranslate on the agent I get:
root#snmpagent:~# snmptranslate -On DISMAN-PING-MIB::pingResultsTable
.1.3.6.1.2.1.80.1.3
root#snmpagent:~# snmptranslate -On DISMAN-PING-MIB::pingCtlTable
.1.3.6.1.2.1.80.1.2
root#snmpagent:~# snmptranslate -On DISMAN-PING-MIB::pingProbeHistoryTable
.1.3.6.1.2.1.80.1.4
Output on the server, when I do this:
snmpget -v2c -c public 192.168.5.21 sysDescr.0
SNMPv2-MIB::sysDescr.0 = STRING: Linux snmp-agent 4.10.0-33-generic #37~16.04.1-Ubuntu SMP Fri Aug 11 14:07:24 UTC 2017 x86_64
But when I run snmpget, from the server, I get this error for all the three:
snmpget -v2c -c public 192.168.5.21 DISMAN-PING-MIB::pingCtlTable.0
DISMAN-PING-MIB::pingCtlTable = No Such Object available on this agent at this OID
Output when specified the OID's:
snmpget -v2c -c public 192.168.5.21 .1.3.6.1.2.1.80.1.2
DISMAN-PING-MIB::pingCtlTable = No Such Object available on this agent at this OID
NOTE: I have already seen these posts with no luck for me.
add new mib master agent
snmpget-no-such-object... and snmpget-returns-no-such-object...
UPDATE 2:
When I do snmpwalk from the server, I get:
root#snmp-server:# snmp-server:# snmpwalk -v 2c -c ncs -m DISMAN-PING-MIB 192.168.5.22 .1.3.6.1.2.1.80
DISMAN-PING-MIB::pingObjects.0 = INTEGER: 1
DISMAN-PING-MIB::pingFullCompliance.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = STRING: "/bin/echo"
DISMAN-PING-MIB::pingMinimumCompliance.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = ""
DISMAN-PING-MIB::pingCompliances.4.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = ""
DISMAN-PING-MIB::pingCompliances.5.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = INTEGER: 5
DISMAN-PING-MIB::pingCompliances.6.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = INTEGER: 1
DISMAN-PING-MIB::pingCompliances.7.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = INTEGER: 1
DISMAN-PING-MIB::pingCompliances.20.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = INTEGER: 4
DISMAN-PING-MIB::pingCompliances.21.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = INTEGER: 1
DISMAN-PING-MIB::pingIcmpEcho.1.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = ""
DISMAN-PING-MIB::pingIcmpEcho.2.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = ""
DISMAN-PING-MIB::pingIcmpEcho.3.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = INTEGER: 1
DISMAN-PING-MIB::pingIcmpEcho.4.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = INTEGER: 0
DISMAN-PING-MIB::pingMIB.4.1.2.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48.1 = ""
So accordingly when I follow the output of snmpwalk, I get:
root#snmp-server:# snmpget 192.168.5.22 DISMAN-PING-MIB::pingFullCompliance.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48
DISMAN-PING-MIB::pingFullCompliance.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 = Wrong Type (should be INTEGER): STRING: "/bin/echo"
I don't understand why I have such a long OID? What is pingFullCompliance.15.46.49.46.51.46.54.46.49.46.50.46.49.46.56.48 ? Where am I going wrong? Can anyone please point me in the right direction... Thank you.

Systemtap %M printf format only returns one character

I'm trying to print the data received on a socket - the contents of ubuf on the return of sys_recv. I cant get the %M format specifier to work properly. Can someone please explain how to use it properly. Thanks
stap -L 'kernel.function("sys_recv#net/socket.c")'
kernel.function("sys_recv#net/socket.c:1800") $fd:int $ubuf:void* $size:size_t $flags:unsigned int
using this probe:
[laris#kakitis stap]$ cat socket-recv.stp
#! /usr/bin/env stap
probe kernel.function("sys_recv#net/socket.c").return {
if (pid() == target())
printf ("%s fd %d size %d ubuf %p %10M \n ", ppfunc(),$fd,$size,$ubuf,$ubuf)
}
From my reading of the man page the format %10M should return 10 bytes from the location pointed to by $ubuf:void but I only get 1. Adjusting the parameter 10 shifts the one character output rather than showing more or less memory
[root#kakitis stap]# stap -x 16796 socket-recv.stp
sys_recv fd 13 size 64071 ubuf 0x86ceca0 70
sys_recv fd 13 size 62679 ubuf 0x86cf210 50
Changing 10 to 2 gives this
[root#kakitis stap]# stap -x 16796 socket-recv.stp
sys_recv fd 13 size 64071 ubuf 0x86ceca0 70
sys_recv fd 13 size 62679 ubuf 0x86cf210 50
System particulars are:
[laris#kakitis stap]$ stap --version
Systemtap translator/driver (version 2.1/0.154, rpm 2.1-2.fc17)
Copyright (C) 2005-2013 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBRPM LIBSQLITE3 NSS TR1_UNORDERED_MAP NLS
[laris#kakitis stap]$ uname -a
Linux kakitis 3.4.33 #1 SMP Tue Jan 7 14:15:58 EST 2014 i686 i686 i386 GNU/Linux
[laris#kakitis stap]$ cat /etc/redhat-release
Fedora release 17 (Beefy Miracle)
Don't confuse the output-width and precision parameters for printf(). The following will do what you meant:
printf ("%33.10M", $pointer)
to print 10 bytes (20 hex characters) in a 33-character-wide output field. One or both numbers can be replaced by *, so that the respective widths are passed as parameters before the $pointer. The upstream man page has been updated with an example.

"error: variable length array of non-POD element type" when compiling re::engine::RE2

I'm trying to install the re::engine::RE2 module with cpan.
When building, the compiler emits the error message:
re2_xs.cc:254:25: error: variable length array of non-POD element type
're2::StringPiece'
re2::StringPiece res[re->nparens + 1];
^
What does that mean? How can I fix that? I'm using a 2011 Macbook Air and Perl 5.12.
re::engine::RE2 was never successfully tested on darwin/Perl 5.12.
Googling for the error message suggests that this is a flaw in old clang that ships with Xcode, and not present in newer versions.
Upgrade your software, you're out of support, perhaps use Macports to get a fresh gcc.
True, re::engine::RE2 never compiled on OS X. But you can get it to compile. Locate the cpan directory. For me it is ~/.cpan/build/re-engine-RE2-0.13-BY20k3/. Then within there, you change two C++ source files.
$ diff -p re2_xs.cc.old re2_xs.cc
*** re2_xs.cc.old 2015-04-20 20:20:15.000000000 +0200
--- re2_xs.cc 2015-04-20 20:22:24.000000000 +0200
*************** RE2_exec(pTHX_ REGEXP * const rx, char *
*** 229,236 ****
RE2 * ri = (RE2*) SvANY(rx)->pprivate;
regexp * re = SvANY(rx);
- re2::StringPiece res[re->nparens + 1];
-
#ifdef RE2_DEBUG
Perl_warner(aTHX_ packWARN(WARN_MISC), "RE2: Matching '%s' (%p, %p) against '%s'", stringarg, strbeg, stringarg, RX_WRAPPED(rx));
#endif
--- 229,234 ----
*************** RE2_exec(pTHX_ REGEXP * const rx, char *
*** 241,246 ****
--- 239,246 ----
return 0;
}
+ re2::StringPiece *res = new re2::StringPiece[re->nparens + 1];
+
bool ok = ri->Match(
re2::StringPiece(strbeg, strend - strbeg),
stringarg - strbeg,
*************** RE2_exec(pTHX_ REGEXP * const rx, char *
*** 250,255 ****
--- 250,256 ----
/* Matching failed */
if (!ok) {
+ delete [] res;
return 0;
}
*************** RE2_exec(pTHX_ REGEXP * const rx, char *
*** 266,271 ****
--- 267,274 ----
}
}
+ delete [] res;
+
return 1;
}
Attempting to compile after that will get you to a tr1 issue.
/usr/bin/clang -o obj/util/arena.o -xc++ -O3 -DHAVE_PTHREAD -pthread -Wno-sign-compare -c -I. -DNDEBUG util/arena.cc
In file included from util/arena.cc:5:
./util/util.h:45:10: fatal error: 'tr1/unordered_set' file not found
I'm sure there are simpler -D fixes for the Makefile but I changed the code.
# diff -p re2/util/util.h.old re2/util/util.h
*** re2/util/util.h.old 2015-04-20 20:29:01.000000000 +0200
--- re2/util/util.h 2015-04-20 20:29:26.000000000 +0200
*************** using std::make_pair;
*** 42,49 ****
#if defined(__GNUC__) && !defined(USE_CXX0X)
! #include <tr1/unordered_set>
! using std::tr1::unordered_set;
#else
--- 42,51 ----
#if defined(__GNUC__) && !defined(USE_CXX0X)
! //#include <tr1/unordered_set>
! //using std::tr1::unordered_set;
! #include <unordered_set>
! using std::unordered_set;
#else
Now just
make && make install
and you now have re::engine::RE2 available for OS X.
The answer by Douglas Scofield is correct, but one additional change is needed. In re2_xs.cc you also must change line 244 as follows:
bool ok = ri->Match(
re2::StringPiece(strbeg, strend - strbeg),
stringarg - strbeg,
strend - strbeg,
RE2::UNANCHORED,
! res, re->nparens + 1);
Douglas Scofield's answer changes the array res to pointer *res to avoid the compiler error of variable length array. However, after this change the line
sizeof res / sizeof *res
does not work because you would be taking the size of the pointer and dividing by the size of the array. Therefore, you have to make the change above to
re->nparens + 1
to get the size of the array pointed to by *res, which is what was desired in the original code.
If you don't make the change then capture groups will not work in re2. You will get errors that your capture groups like $1, $2, etc. are uninitialized.
So great answer above, just needs one addendum.