I'm following these instructions to upgrade my Mongo config servers to a replica set. Everything seems to work initially, but then when connecting to a Mongos and running sh.status, I get the following error message and stack.
Of importance: "Detected upgrade from mirrored (SCCC) config servers to replica set (CSRS) config servers. setShardVersion was given: csReplSet/mongodb-config-1:27019 for the config server connection string, but has stored: mongodb-config-1:27019,mongodb-config-2:27019,mongodb-config-3:27019
This causes the PRIMARY and one of the SECONDARY config servers to crash. I checked that all mongos have the right connection string and that all mongod and mongos are running the same version of MongoDB (3.2.11). I also tried firing up config servers from scratch (at the expense of losing data), but to no avail.
Any ideas?
2016-12-13T08:27:47.825+0000 I SHARDING [conn849] Detected upgrade from mirrored (SCCC) config servers to replica set (CSRS) config servers. setShardVersion was given: csReplSet/mo\
ngodb-config-1:27019 for the config server connection string, but has stored: mongodb-config-1:27019,mongodb-config-2:27019,mongodb-config-3:27019
2016-12-13T08:27:47.825+0000 F - [conn849] Invalid access at address: 0x108
2016-12-13T08:27:47.831+0000 F - [conn849] Got signal: 11 (Segmentation fault).
0x132f902 0x132ea59 0x132edd8 0x7f288d434330 0x7f288d42e404 0x1140949 0xfa1480 0xbd2653 0xbd34d9 0xb2c1f0 0xce12f5 0x9be81c 0x12dc0c5 0x7f288d42c184 0x7f288d15937d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F2F902","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F2EA59"},{"b":"400000","o":"F2EDD8"},{"b":"7F288D424000","o":"10330"},{"b":"7F288D4\
24000","o":"A404","s":"pthread_mutex_lock"},{"b":"400000","o":"D40949","s":"_ZN5mongo24ForwardingCatalogManager37scheduleReplaceCatalogManagerIfNeededENS_14CatalogManager16ConfigSer\
verModeERKNS_16ConnectionStringE"},{"b":"400000","o":"BA1480"},{"b":"400000","o":"7D2653","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyB\
uilderInterfaceE"},{"b":"400000","o":"7D34D9","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"400000\
","o":"72C1F0","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"400000","o":"8E12F5","s":"_ZN5mongo16assembleResp\
onseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"400000","o":"5BE81C","s":"_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessag\
ingPortE"},{"b":"400000","o":"EDC0C5","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"7F288D424000","o":"8184"},{"b":"7F288D05F000","o":"FA37D","s":"clone"}],"proces\
sInfo":{ "mongodbVersion" : "3.2.11", "gitVersion" : "009580ad490190ba33d1c6253ebd8d91808923e4", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-53-gener\
ic", "version" : "#74~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "55994D9F4E3C47C30959A86BC9F3\
3188CB6ADF21" }, { "b" : "7FFCEDF32000", "elfType" : 3, "buildId" : "AA5EEB9DFDD3AABAC9F493EC883C381016A1C312" }, { "b" : "7F288E346000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1\
.0.0", "elfType" : 3, "buildId" : "15A6A69B862FAC0AA589921DB1B0DF8A673FEC81" }, { "b" : "7F288DF6A000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId"\
: "607C4A6F3428B6F065D55691AFDC4E343786A8A5" }, { "b" : "7F288DD62000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC681\
92AB7" }, { "b" : "7F288DB5E000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F288D858000", "path\
" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F288D642000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1\
", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F288D424000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E\
9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F288D05F000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }\
, { "b" : "7F288E5A5000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x132f902]
mongod(+0xF2EA59) [0x132ea59]
mongod(+0xF2EDD8) [0x132edd8]
libpthread.so.0(+0x10330) [0x7f288d434330]
libpthread.so.0(pthread_mutex_lock+0x4) [0x7f288d42e404]
mongod(_ZN5mongo24ForwardingCatalogManager37scheduleReplaceCatalogManagerIfNeededENS_14CatalogManager16ConfigServerModeERKNS_16ConnectionStringE+0x39) [0x1140949]
mongod(+0xBA1480) [0xfa1480]
mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x473) [0xbd2653]
mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x3F9) [0xbd34d9]
mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x1F0) [0xb2c1f0]
mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xB45) [0xce12f5]
mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xEC) [0x9be81c]
mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x325) [0x12dc0c5]
libpthread.so.0(+0x8184) [0x7f288d42c184]
libc.so.6(clone+0x6D) [0x7f288d15937d]
----- END BACKTRACE -----
2016-12-13T08:27:47.831+0000 F - [conn849] /proc/self/maps:
00400000-02008000 r-xp 00000000 08:01 2125 /usr/bin/mongod
2016-12-13T08:27:47.831+0000 F - [conn849] 02008000-020db000 rw-p 01c07000 08:01 2125 /usr/bin/mongod
2016-12-13T08:27:47.831+0000 F - [conn849] 020db000-0214b000 rw-p 00000000 00:00 0
2016-12-13T08:27:47.831+0000 F - [conn849] 02885000-03486000 rw-p 00000000 00:00 0 [heap]
2016-12-13T08:27:47.831+0000 F - [conn849] 03486000-08972000 rw-p 00000000 00:00 0 [heap]
2016-12-13T08:27:47.831+0000 F - [conn849] 7f28689ba000-7f28689bb000 ---p 00000000 00:00 0
2016-12-13T08:27:47.831+0000 F - [conn849] 7f28689bb000-7f2868abb000 rw-p 00000000 00:00 0
2016-12-13T08:27:47.831+0000 F - [conn849] 7f2868abb000-7f2868abc000 ---p 00000000 00:00 0
2016-12-13T08:27:47.831+0000 F - [conn849] 7f2868abc000-7f2868bbc000 rw-p 00000000 00:00 0
2016-12-13T08:27:47.831+0000 F - [conn849] 7f2868bbc000-7f2868bbd000 ---p 00000000 00:00 0
2016-12-13T08:27:47.831+0000 F - [conn849] 7f2868bbd000-7f2868cbd000 rw-p 00000000 00:00 0
2016-12-13T08:27:47.831+0000 F - [conn849] 7f2868cbd000-7f2868cbe000 ---p 00000000 00:00 0
.
.
.
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288c85e000-7f288c85f000 ---p 00000000 00:00 0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288c85f000-7f288d05f000 rw-p 00000000 00:00 0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d05f000-7f288d219000 r-xp 00000000 08:01 3079 /lib/x86_64-linux-gnu/libc-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d219000-7f288d419000 ---p 001ba000 08:01 3079 /lib/x86_64-linux-gnu/libc-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d419000-7f288d41d000 r--p 001ba000 08:01 3079 /lib/x86_64-linux-gnu/libc-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d41d000-7f288d41f000 rw-p 001be000 08:01 3079 /lib/x86_64-linux-gnu/libc-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d41f000-7f288d424000 rw-p 00000000 00:00 0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d424000-7f288d43d000 r-xp 00000000 08:01 3042 /lib/x86_64-linux-gnu/libpthread-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d43d000-7f288d63c000 ---p 00019000 08:01 3042 /lib/x86_64-linux-gnu/libpthread-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d63c000-7f288d63d000 r--p 00018000 08:01 3042 /lib/x86_64-linux-gnu/libpthread-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d63d000-7f288d63e000 rw-p 00019000 08:01 3042 /lib/x86_64-linux-gnu/libpthread-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d63e000-7f288d642000 rw-p 00000000 00:00 0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d642000-7f288d658000 r-xp 00000000 08:01 3111 /lib/x86_64-linux-gnu/libgcc_s.so.1
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d658000-7f288d857000 ---p 00016000 08:01 3111 /lib/x86_64-linux-gnu/libgcc_s.so.1
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d857000-7f288d858000 rw-p 00015000 08:01 3111 /lib/x86_64-linux-gnu/libgcc_s.so.1
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d858000-7f288d95d000 r-xp 00000000 08:01 3084 /lib/x86_64-linux-gnu/libm-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288d95d000-7f288db5c000 ---p 00105000 08:01 3084 /lib/x86_64-linux-gnu/libm-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288db5c000-7f288db5d000 r--p 00104000 08:01 3084 /lib/x86_64-linux-gnu/libm-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288db5d000-7f288db5e000 rw-p 00105000 08:01 3084 /lib/x86_64-linux-gnu/libm-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288db5e000-7f288db61000 r-xp 00000000 08:01 3076 /lib/x86_64-linux-gnu/libdl-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288db61000-7f288dd60000 ---p 00003000 08:01 3076 /lib/x86_64-linux-gnu/libdl-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288dd60000-7f288dd61000 r--p 00002000 08:01 3076 /lib/x86_64-linux-gnu/libdl-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288dd61000-7f288dd62000 rw-p 00003000 08:01 3076 /lib/x86_64-linux-gnu/libdl-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288dd62000-7f288dd69000 r-xp 00000000 08:01 3002 /lib/x86_64-linux-gnu/librt-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288dd69000-7f288df68000 ---p 00007000 08:01 3002 /lib/x86_64-linux-gnu/librt-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288df68000-7f288df69000 r--p 00006000 08:01 3002 /lib/x86_64-linux-gnu/librt-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288df69000-7f288df6a000 rw-p 00007000 08:01 3002 /lib/x86_64-linux-gnu/librt-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288df6a000-7f288e11d000 r-xp 00000000 08:01 3012 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e11d000-7f288e31c000 ---p 001b3000 08:01 3012 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e31c000-7f288e337000 r--p 001b2000 08:01 3012 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e337000-7f288e342000 rw-p 001cd000 08:01 3012 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e342000-7f288e346000 rw-p 00000000 00:00 0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e346000-7f288e39b000 r-xp 00000000 08:01 3059 /lib/x86_64-linux-gnu/libssl.so.1.0.0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e39b000-7f288e59b000 ---p 00055000 08:01 3059 /lib/x86_64-linux-gnu/libssl.so.1.0.0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e59b000-7f288e59e000 r--p 00055000 08:01 3059 /lib/x86_64-linux-gnu/libssl.so.1.0.0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e59e000-7f288e5a5000 rw-p 00058000 08:01 3059 /lib/x86_64-linux-gnu/libssl.so.1.0.0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e5a5000-7f288e5c8000 r-xp 00000000 08:01 3010 /lib/x86_64-linux-gnu/ld-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e62d000-7f288e62f000 rw-p 00000000 00:00 0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e62f000-7f288e7b8000 r--p 00000000 08:01 28741 /usr/lib/locale/locale-archive
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e7be000-7f288e7c5000 r--s 00000000 08:01 29034 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e7c5000-7f288e7c7000 rw-p 00000000 00:00 0
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e7c7000-7f288e7c8000 r--p 00022000 08:01 3010 /lib/x86_64-linux-gnu/ld-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e7c8000-7f288e7c9000 rw-p 00023000 08:01 3010 /lib/x86_64-linux-gnu/ld-2.19.so
2016-12-13T08:27:47.833+0000 F - [conn849] 7f288e7c9000-7f288e7ca000 rw-p 00000000 00:00 0
2016-12-13T08:27:47.833+0000 F - [conn849] 7ffcede0e000-7ffcede2f000 rw-p 00000000 00:00 0 [stack]
2016-12-13T08:27:47.833+0000 F - [conn849] 7ffcedf30000-7ffcedf32000 r--p 00000000 00:00 0 [vvar]
2016-12-13T08:27:47.833+0000 F - [conn849] 7ffcedf32000-7ffcedf34000 r-xp 00000000 00:00 0 [vdso]
2016-12-13T08:27:47.833+0000 F - [conn849] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
The problem was that there were still some mongos instances using the old (mirrored) connection string. It seems that even mongos instances that are not up keep on pinging the config servers which cause the error and the fatal crash. Do use config; db.mongos.find() to find all your mongos instances and make sure they are all using the updated connection string without exception.
Context Broker Orion crashes in my Cloud + CoreOS + Docker architecture, by using a centos6 based dockerfile definition and the latest rpm compilation provided.
Any thoughts on crashes like the one I paste here? After 20/30 ContextUpdates it stops. Actually, cygnus and proton are subscribed.
DEBUG#11:34:05 httpRequestSend.cpp[409]: Sending message 114 to HTTP server: sending message of 649 bytes to HTTP server
INFO#11:34:05 MongoCommonUpdate.cpp[960]: Database Operation Successful (update: { $set: { lastNotification: 1434022445 }, $inc: { count: 1 } }, query: { _id: ObjectId('55795ab89b16d6c37d911b90') })
DEBUG#11:34:05 MongoCommonUpdate.cpp[925]: blocked due to throttling, current time is:
DEBUG#11:34:05 MongoCommonUpdate.cpp[1941]: Docs found: 1
DEBUG#11:34:05 sem.cpp[255]: mongoUpdateContext gives the 'req' semaphore for 'ngsi10 update request'
DEBUG#11:34:05 restReply.cpp[67]: Response 95: responding with 734 bytes, Status Code 200
DEBUG#11:34:05 restReply.cpp[68]: Response payload: '{
"contextResponses" : [
{
"contextElement" : {
"type" : "Node",
"isPattern" : "false",
"id" : "XXXXXXXXXXXXXX",
"attributes" : [
{
"name" : "TimeInstant",
"type" : "",
"value" : ""
},
{
"name" : "batteryCharge",
"type" : "",
"value" : ""
},
{
"name" : "rows",
"type" : "",
"value" : ""
},
{
"name" : "columns",
"type" : "",
"value" : ""
}
]
},
"statusCode" : {
"code" : "200",
"reasonPhrase" : "OK"
}
}
]
}
'
DEBUG#11:34:05 UpdateContextResponse.cpp[74]: destroyed
DEBUG#11:34:05 RegisterContextResponse.cpp[56]: destroyed
INFO#11:34:05 rest.cpp[384]: Transaction ended
INFO#11:34:05 httpRequestSend.cpp[424]: Notification Successfully Sent to node-2:5050/notify
INFO#11:34:05 httpRequestSend.cpp[435]: Transaction ended
*** longjmp causes uninitialized stack frame ***: /usr/bin/contextBroker terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7ff44c9e4527]
/lib64/libc.so.6(+0x1024b9)[0x7ff44c9e44b9]
/lib64/libc.so.6(__longjmp_chk+0x33)[0x7ff44c9e4423]
/usr/lib64/libcurl.so.4(+0xee55)[0x7ff44df5be55]
/lib64/libc.so.6(+0x326a0)[0x7ff44c9146a0]
/lib64/libc.so.6(nanosleep+0x2d)[0x7ff44c98ea3d]
/lib64/libc.so.6(sleep+0xe0)[0x7ff44c98e8b0]
/usr/bin/contextBroker(main+0x90a)[0x49e5da]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7ff44c900d5d]
/usr/bin/contextBroker[0x483219]
======= Memory map: ========
00400000-00737000 r-xp 00000000 08:09 2107197 /usr/bin/contextBroker
00937000-00941000 rw-p 00337000 08:09 2107197 /usr/bin/contextBroker
00941000-00989000 rw-p 00000000 00:00 0
018c0000-01923000 rw-p 00000000 00:00 0 [heap]
7ff428000000-7ff428029000 rw-p 00000000 00:00 0
7ff428029000-7ff42c000000 ---p 00000000 00:00 0
7ff42f7ff000-7ff42f800000 ---p 00000000 00:00 0
7ff42f800000-7ff430000000 rw-p 00000000 00:00 0
7ff430000000-7ff430021000 rw-p 00000000 00:00 0
7ff430021000-7ff434000000 ---p 00000000 00:00 0
7ff434000000-7ff434029000 rw-p 00000000 00:00 0
7ff434029000-7ff438000000 ---p 00000000 00:00 0
7ff438000000-7ff440000000 rw-p 00000000 00:00 0
7ff440000000-7ff440021000 rw-p 00000000 00:00 0
7ff440021000-7ff444000000 ---p 00000000 00:00 0
7ff444418000-7ff444419000 ---p 00000000 00:00 0
7ff444419000-7ff444c19000 rw-p 00000000 00:00 0
7ff444c19000-7ff444c1a000 ---p 00000000 00:00 0
7ff444c1a000-7ff44541a000 rw-p 00000000 00:00 0
7ff44541a000-7ff44541b000 ---p 00000000 00:00 0
7ff44541b000-7ff445c1b000 rw-p 00000000 00:00 0 [stack:20]
7ff445c1b000-7ff445c1c000 ---p 00000000 00:00 0
7ff445c1c000-7ff44641c000 rw-p 00000000 00:00 0 [stack:19]
7ff44641c000-7ff446421000 r-xp 00000000 08:09 5529 /lib64/libnss_dns-2.12.so
7ff446421000-7ff446620000 ---p 00005000 08:09 5529 /lib64/libnss_dns-2.12.so
7ff446620000-7ff446621000 r--p 00004000 08:09 5529 /lib64/libnss_dns-2.12.so
7ff446621000-7ff446622000 rw-p 00005000 08:09 5529 /lib64/libnss_dns-2.12.so
7ff446622000-7ff44662e000 r-xp 00000000 08:09 5531 /lib64/libnss_files-2.12.so
7ff44662e000-7ff44682e000 ---p 0000c000 08:09 5531 /lib64/libnss_files-2.12.so
7ff44682e000-7ff44682f000 r--p 0000c000 08:09 5531 /lib64/libnss_files-2.12.so
7ff44682f000-7ff446830000 rw-p 0000d000 08:09 5531 /lib64/libnss_files-2.12.so
7ff446830000-7ff446832000 r-xp 00000000 08:09 5455 /lib64/libfreebl3.so
7ff446832000-7ff446a31000 ---p 00002000 08:09 5455 /lib64/libfreebl3.so
7ff446a31000-7ff446a32000 r--p 00001000 08:09 5455 /lib64/libfreebl3.so
7ff446a32000-7ff446a33000 rw-p 00002000 08:09 5455 /lib64/libfreebl3.so
7ff446a33000-7ff446a50000 r-xp 00000000 08:09 5560 /lib64/libselinux.so.1
7ff446a50000-7ff446c4f000 ---p 0001d000 08:09 5560 /lib64/libselinux.so.1
7ff446c4f000-7ff446c50000 r--p 0001c000 08:09 5560 /lib64/libselinux.so.1
7ff446c50000-7ff446c51000 rw-p 0001d000 08:09 5560 /lib64/libselinux.so.1
7ff446c51000-7ff446c52000 rw-p 00000000 00:00 0
7ff446c52000-7ff446c59000 r-xp 00000000 08:09 5445 /lib64/libcrypt-2.12.so
7ff446c59000-7ff446e59000 ---p 00007000 08:09 5445 /lib64/libcrypt-2.12.so
7ff446e59000-7ff446e5a000 r--p 00007000 08:09 5445 /lib64/libcrypt-2.12.so
7ff446e5a000-7ff446e5b000 rw-p 00008000 08:09 5445 /lib64/libcrypt-2.12.so
7ff446e5b000-7ff446e89000 rw-p 00000000 00:00 0
7ff446e89000-7ff447dce000 r-xp 00000000 08:09 2107212 /usr/lib64/libicudata.so.42.1
7ff447dce000-7ff447fcd000 ---p 00f45000 08:09 2107212 /usr/lib64/libicudata.so.42.1
7ff447fcd000-7ff447fce000 rw-p 00f44000 08:09 2107212 /usr/lib64/libicudata.so.42.1
7ff447fce000-7ff447fde000 r-xp 00000000 08:09 6734 /usr/lib64/libtasn1.so.3.1.6
7ff447fde000-7ff4481dd000 ---p 00010000 08:09 6734 /usr/lib64/libtasn1.so.3.1.6
7ff4481dd000-7ff4481de000 rw-p 0000f000 08:09 6734 /usr/lib64/libtasn1.so.3.1.6
7ff4481de000-7ff448397000 r-xp 00000000 08:09 6626 /usr/lib64/libcrypto.so.1.0.1e
7ff448397000-7ff448596000 ---p 001b9000 08:09 6626 /usr/lib64/libcrypto.so.1.0.1e
7ff448596000-7ff4485b1000 r--p 001b8000 08:09 6626 /usr/lib64/libcrypto.so.1.0.1e
7ff4485b1000-7ff4485bd000 rw-p 001d3000 08:09 6626 /usr/lib64/libcrypto.so.1.0.1e
7ff4485bd000-7ff4485c1000 rw-p 00000000 00:00 0
7ff4485c1000-7ff448623000 r-xp 00000000 08:09 6728 /usr/lib64/libssl.so.1.0.1e
7ff448623000-7ff448822000 ---p 00062000 08:09 6728 /usr/lib64/libssl.so.1.0.1e
7ff448822000-7ff448826000 r--p 00061000 08:09 6728 /usr/lib64/libssl.so.1.0.1e
7ff448826000-7ff44882d000 rw-p 00065000 08:09 6728 /usr/lib64/libssl.so.1.0.1e
7ff44882d000-7ff44882f000 r-xp 00000000 08:09 5499 /lib64/libkeyutils.so.1.3
7ff44882f000-7ff448a2e000 ---p 00002000 08:09 5499 /lib64/libkeyutils.so.1.3
7ff448a2e000-7ff448a2f000 r--p 00001000 08:09 5499 /lib64/libkeyutils.so.1.3
7ff448a2f000-7ff448a30000 rw-p 00002000 08:09 5499 /lib64/libkeyutils.so.1.3
7ff448a30000-7ff448a3a000 r-xp 00000000 08:09 5503 /lib64/libkrb5support.so.0.1
7ff448a3a000-7ff448c39000 ---p 0000a000 08:09 5503 /lib64/libkrb5support.so.0.1
7ff448c39000-7ff448c3a000 r--p 00009000 08:09 5503 /lib64/libkrb5support.so.0.1
7ff448c3a000-7ff448c3b000 rw-p 0000a000 08:09 5503 /lib64/libkrb5support.so.0.1
7ff448c3b000-7ff448c54000 r-xp 00000000 08:09 6720 /usr/lib64/libsasl2.so.2.0.23
7ff448c54000-7ff448e53000 ---p 00019000 08:09 6720 /usr/lib64/libsasl2.so.2.0.23
7ff448e53000-7ff448e54000 r--p 00018000 08:09 6720 /usr/lib64/libsasl2.so.2.0.23
7ff448e54000-7ff448e55000 rw-p 00019000 08:09 6720 /usr/lib64/libsasl2.so.2.0.23
7ff448e55000-7ff448e6b000 r-xp 00000000 08:09 5556 /lib64/libresolv-2.12.so
7ff448e6b000-7ff44906b000 ---p 00016000 08:09 5556 /lib64/libresolv-2.12.so
7ff44906b000-7ff44906c000 r--p 00016000 08:09 5556 /lib64/libresolv-2.12.so
7ff44906c000-7ff44906d000 rw-p 00017000 08:09 5556 /lib64/libresolv-2.12.so
7ff44906d000-7ff44906f000 rw-p 00000000 00:00 0
7ff44906f000-7ff44907d000 r-xp 00000000 08:09 5505 /lib64/liblber-2.4.so.2.10.2
7ff44907d000-7ff44927c000 ---p 0000e000 08:09 5505 /lib64/liblber-2.4.so.2.10.2
7ff44927c000-7ff44927d000 r--p 0000d000 08:09 5505 /lib64/liblber-2.4.so.2.10.2
7ff44927d000-7ff44927e000 rw-p 0000e000 08:09 5505 /lib64/liblber-2.4.so.2.10.2
7ff44927e000-7ff449406000 r-xp 00000000 08:09 2107220 /usr/lib64/libicui18n.so.42.1
7ff449406000-7ff449606000 ---p 00188000 08:09 2107220 /usr/lib64/libicui18n.so.42.1
7ff449606000-7ff449613000 rw-p 00188000 08:09 2107220 /usr/lib64/libicui18n.so.42.1
7ff449613000-7ff449614000 rw-p 00000000 00:00 0
7ff449614000-7ff449753000 r-xp 00000000 08:09 2107200 /usr/lib64/libicuuc.so.42.1
7ff449753000-7ff449953000 ---p 0013f000 08:09 2107200 /usr/lib64/libicuuc.so.42.1
7ff449953000-7ff449964000 rw-p 0013f000 08:09 2107200 /usr/lib64/libicuuc.so.42.1
7ff449964000-7ff449966000 rw-p 00000000 00:00 0
7ff449966000-7ff449969000 r-xp 00000000 08:09 5471 /lib64/libgpg-error.so.0.5.0
7ff449969000-7ff449b68000 ---p 00003000 08:09 5471 /lib64/libgpg-error.so.0.5.0
7ff449b68000-7ff449b69000 r--p 00002000 08:09 5471 /lib64/libgpg-error.so.0.5.0
7ff449b69000-7ff449b6a000 rw-p 00003000 08:09 5471 /lib64/libgpg-error.so.0.5.0
7ff449b6a000-7ff449bdc000 r-xp 00000000 08:09 5461 /lib64/libgcrypt.so.11.5.3
7ff449bdc000-7ff449ddb000 ---p 00072000 08:09 5461 /lib64/libgcrypt.so.11.5.3
7ff449ddb000-7ff449ddc000 r--p 00071000 08:09 5461 /lib64/libgcrypt.so.11.5.3
7ff449ddc000-7ff449ddf000 rw-p 00072000 08:09 5461 /lib64/libgcrypt.so.11.5.3
7ff449ddf000-7ff449e7b000 r-xp 00000000 08:09 2107217 /usr/lib64/libgnutls.so.26.14.12
7ff449e7b000-7ff44a07b000 ---p 0009c000 08:09 2107217 /usr/lib64/libgnutls.so.26.14.12
7ff44a07b000-7ff44a082000 rw-p 0009c000 08:09 2107217 /usr/lib64/libgnutls.so.26.14.12
7ff44a082000-7ff44a0a9000 r-xp 00000000 08:09 6727 /usr/lib64/libssh2.so.1.0.1
7ff44a0a9000-7ff44a2a8000 ---p 00027000 08:09 6727 /usr/lib64/libssh2.so.1.0.1
7ff44a2a8000-7ff44a2aa000 rw-p 00026000 08:09 6727 /usr/lib64/libssh2.so.1.0.1
7ff44a2aa000-7ff44a2ac000 r-xp 00000000 08:09 5450 /lib64/libdl-2.12.so
7ff44a2ac000-7ff44a4ac000 ---p 00002000 08:09 5450 /lib64/libdl-2.12.so
7ff44a4ac000-7ff44a4ad000 r--p 00002000 08:09 5450 /lib64/libdl-2.12.so
7ff44a4ad000-7ff44a4ae000 rw-p 00003000 08:09 5450 /lib64/libdl-2.12.so
7ff44a4ae000-7ff44a4e7000 r-xp 00000000 08:09 5526 /lib64/libnspr4.so
7ff44a4e7000-7ff44a6e7000 ---p 00039000 08:09 5526 /lib64/libnspr4.so
7ff44a6e7000-7ff44a6e8000 r--p 00039000 08:09 5526 /lib64/libnspr4.so
7ff44a6e8000-7ff44a6ea000 rw-p 0003a000 08:09 5526 /lib64/libnspr4.so
7ff44a6ea000-7ff44a6ec000 rw-p 00000000 00:00 0
7ff44a6ec000-7ff44a6f0000 r-xp 00000000 08:09 5547 /lib64/libplc4.so
7ff44a6f0000-7ff44a8ef000 ---p 00004000 08:09 5547 /lib64/libplc4.so
7ff44a8ef000-7ff44a8f0000 r--p 00003000 08:09 5547 /lib64/libplc4.so
7ff44a8f0000-7ff44a8f1000 rw-p 00004000 08:09 5547 /lib64/libplc4.so
7ff44a8f1000-7ff44a8f4000 r-xp 00000000 08:09 5548 /lib64/libplds4.so
7ff44a8f4000-7ff44aaf3000 ---p 00003000 08:09 5548 /lib64/libplds4.so
7ff44aaf3000-7ff44aaf4000 r--p 00002000 08:09 5548 /lib64/libplds4.so
7ff44aaf4000-7ff44aaf5000 rw-p 00003000 08:09 5548 /lib64/libplds4.so
7ff44aaf5000-7ff44ab1a000 r-xp 00000000 08:09 6697 /usr/lib64/libnssutil3.so
7ff44ab1a000-7ff44ad1a000 ---p 00025000 08:09 6697 /usr/lib64/libnssutil3.so
7ff44ad1a000-7ff44ad20000 r--p 00025000 08:09 6697 /usr/lib64/libnssutil3.so
7ff44ad20000-7ff44ad21000 rw-p 0002b000 08:09 6697 /usr/lib64/libnssutil3.so
7ff44ad21000-7ff44ae58000 r-xp 00000000 08:09 6691 /usr/lib64/libnss3.so
7ff44ae58000-7ff44b057000 ---p 00137000 08:09 6691 /usr/lib64/libnss3.so
7ff44b057000-7ff44b05c000 r--p 00136000 08:09 6691 /usr/lib64/libnss3.so
7ff44b05c000-7ff44b05e000 rw-p 0013b000 08:09 6691 /usr/lib64/libnss3.so
7ff44b05e000-7ff44b060000 rw-p 00000000 00:00 0
7ff44b060000-7ff44b088000 r-xp 00000000 08:09 6721 /usr/lib64/libsmime3.so
7ff44b088000-7ff44b287000 ---p 00028000 08:09 6721 /usr/lib64/libsmime3.so
7ff44b287000-7ff44b28b000 r--p 00027000 08:09 6721 /usr/lib64/libsmime3.so
7ff44b28b000-7ff44b28c000 rw-p 0002b000 08:09 6721 /usr/lib64/libsmime3.so
7ff44b28c000-7ff44b2c7000 r-xp 00000000 08:09 6730 /usr/lib64/libssl3.so
7ff44b2c7000-7ff44b4c6000 ---p 0003b000 08:09 6730 /usr/lib64/libssl3.so
7ff44b4c6000-7ff44b4c9000 r--p 0003a000 08:09 6730 /usr/lib64/libssl3.so
7ff44b4c9000-7ff44b4ca000 rw-p 0003d000 08:09 6730 /usr/lib64/libssl3.so
7ff44b4ca000-7ff44b4cb000 rw-p 00000000 00:00 0
7ff44b4cb000-7ff44b4e0000 r-xp 00000000 08:09 5576 /lib64/libz.so.1.2.3
7ff44b4e0000-7ff44b6df000 ---p 00015000 08:09 5576 /lib64/libz.so.1.2.3
7ff44b6df000-7ff44b6e0000 r--p 00014000 08:09 5576 /lib64/libz.so.1.2.3
7ff44b6e0000-7ff44b6e1000 rw-p 00015000 08:09 5576 /lib64/libz.so.1.2.3
7ff44b6e1000-7ff44b6e4000 r-xp 00000000 08:09 5444 /lib64/libcom_err.so.2.1
7ff44b6e4000-7ff44b8e3000 ---p 00003000 08:09 5444 /lib64/libcom_err.so.2.1
7ff44b8e3000-7ff44b8e4000 r--p 00002000 08:09 5444 /lib64/libcom_err.so.2.1
7ff44b8e4000-7ff44b8e5000 rw-p 00003000 08:09 5444 /lib64/libcom_err.so.2.1
7ff44b8e5000-7ff44b90e000 r-xp 00000000 08:09 5497 /lib64/libk5crypto.so.3.1
7ff44b90e000-7ff44bb0e000 ---p 00029000 08:09 5497 /lib64/libk5crypto.so.3.1
7ff44bb0e000-7ff44bb0f000 r--p 00029000 08:09 5497 /lib64/libk5crypto.so.3.1
7ff44bb0f000-7ff44bb10000 rw-p 0002a000 08:09 5497 /lib64/libk5crypto.so.3.1
7ff44bb10000-7ff44bb11000 rw-p 00000000 00:00 0
7ff44bb11000-7ff44bbec000 r-xp 00000000 08:09 5501 /lib64/libkrb5.so.3.3
7ff44bbec000-7ff44bdeb000 ---p 000db000 08:09 5501 /lib64/libkrb5.so.3.3
7ff44bdeb000-7ff44bdf5000 r--p 000da000 08:09 5501 /lib64/libkrb5.so.3.3
7ff44bdf5000-7ff44bdf7000 rw-p 000e4000 08:09 5501 /lib64/libkrb5.so.3.3
7ff44bdf7000-7ff44be38000 r-xp 00000000 08:09 5473 /lib64/libgssapi_krb5.so.2.2
7ff44be38000-7ff44c038000 ---p 00041000 08:09 5473 /lib64/libgssapi_krb5.so.2.2
7ff44c038000-7ff44c039000 r--p 00041000 08:09 5473 /lib64/libgssapi_krb5.so.2.2
7ff44c039000-7ff44c03b000 rw-p 00042000 08:09 5473 /lib64/libgssapi_krb5.so.2.2
7ff44c03b000-7ff44c088000 r-xp 00000000 08:09 5507 /lib64/libldap-2.4.so.2.10.2
7ff44c088000-7ff44c287000 ---p 0004d000 08:09 5507 /lib64/libldap-2.4.so.2.10.2
7ff44c287000-7ff44c289000 r--p 0004c000 08:09 5507 /lib64/libldap-2.4.so.2.10.2
7ff44c289000-7ff44c28b000 rw-p 0004e000 08:09 5507 /lib64/libldap-2.4.so.2.10.2
7ff44c28b000-7ff44c2bd000 r-xp 00000000 08:09 5479 /lib64/libidn.so.11.6.1
7ff44c2bd000-7ff44c4bc000 ---p 00032000 08:09 5479 /lib64/libidn.so.11.6.1
7ff44c4bc000-7ff44c4bd000 rw-p 00031000 08:09 5479 /lib64/libidn.so.11.6.1
7ff44c4bd000-7ff44c4d4000 r-xp 00000000 08:09 5552 /lib64/libpthread-2.12.so
7ff44c4d4000-7ff44c6d4000 ---p 00017000 08:09 5552 /lib64/libpthread-2.12.so
7ff44c6d4000-7ff44c6d5000 r--p 00017000 08:09 5552 /lib64/libpthread-2.12.so
7ff44c6d5000-7ff44c6d6000 rw-p 00018000 08:09 5552 /lib64/libpthread-2.12.so
7ff44c6d6000-7ff44c6da000 rw-p 00000000 00:00 0
7ff44c6da000-7ff44c6e1000 r-xp 00000000 08:09 5558 /lib64/librt-2.12.so
7ff44c6e1000-7ff44c8e0000 ---p 00007000 08:09 5558 consumer /lib64/librt-2.12.so
7ff44c8e0000-7ff44c8e1000 r--p 00006000 08:09 5558 /lib64/librt-2.12.so
7ff44c8e1000-7ff44c8e2000 rw-p 00007000 08:09 5558 /lib64/librt-2.12.so
7ff44c8e2000-7ff44ca6c000 r-xp 00000000 08:09 5437 /lib64/libc-2.12.so
7ff44ca6c000-7ff44cc6c000 ---p 0018a000 08:09 5437 /lib64/libc-2.12.so
7ff44cc6c000-7ff44cc70000 r--p 0018a000 08:09 5437 /lib64/libc-2.12.so
7ff44cc70000-7ff44cc71000 rw-p 0018e000 08:09 5437 /lib64/libc-2.12.so
7ff44cc71000-7ff44cc76000 rw-p 00000000 00:00 0
7ff44cc76000-7ff44cc8c000 r-xp 00000000 08:09 5458 /lib64/libgcc_s-4.4.7-20120601.so.1
7ff44cc8c000-7ff44ce8b000 ---p 00016000 08:09 5458 /lib64/libgcc_s-4.4.7-20120601.so.1
7ff44ce8b000-7ff44ce8c000 rw-p 00015000 08:09 5458 /lib64/libgcc_s-4.4.7-20120601.so.1
7ff44ce8c000-7ff44cf0f000 r-xp 00000000 08:09 5512 /lib64/libm-2.12.so
7ff44cf0f000-7ff44d10e000 ---p 00083000 08:09 5512 /lib64/libm-2.12.so
7ff44d10e000-7ff44d10f000 r--p 00082000 08:09 5512 /lib64/libm-2.12.so
7ff44d10f000-7ff44d110000 rw-p 00083000 08:09 5512 /lib64/libm-2.12.so
7ff44d110000-7ff44d1f8000 r-xp 00000000 08:09 6732 /usr/lib64/libstdc++.so.6.0.13
7ff44d1f8000-7ff44d3f8000 ---p 000e8000 08:09 6732 /usr/lib64/libstdc++.so.6.0.13
7ff44d3f8000-7ff44d3ff000 r--p 000e8000 08:09 6732 /usr/lib64/libstdc++.so.6.0.13
7ff44d3ff000-7ff44d401000 rw-p 000ef000 08:09 6732 /usr/lib64/libstdc++.so.6.0.13
7ff44d401000-7ff44d416000 rw-p 00000000 00:00 0
7ff44d416000-7ff44d504000 r-xp 00000000 08:09 2107203 /usr/lib64/libboost_regex-mt.so.5
7ff44d504000-7ff44d703000 ---p 000ee000 08:09 2107203 /usr/lib64/libboost_regex-mt.so.5
7ff44d703000-7ff44d70a000 rw-p 000ed000 08:09 2107203 /usr/lib64/libboost_regex-mt.so.5
7ff44d70a000-7ff44d70d000 r-xp 00000000 08:09 2107218 /usr/lib64/libboost_system-mt.so.5
7ff44d70d000-7ff44d90c000 ---p 00003000 08:09 2107218 /usr/lib64/libboost_system-mt.so.5
7ff44d90c000-7ff44d90d000 rw-p 00002000 08:09 2107218 /usr/lib64/libboost_system-mt.so.5
7ff44d90d000-7ff44d921000 r-xp 00000000 08:09 2107225 /usr/lib64/libboost_filesystem-mt.so.5
7ff44d921000-7ff44db21000 ---p 00014000 08:09 2107225 /usr/lib64/libboost_filesystem-mt.so.5
7ff44db21000-7ff44db22000 rw-p 00014000 08:09 2107225 /usr/lib64/libboost_filesystem-mt.so.5
7ff44db22000-7ff44db35000 r-xp 00000000 08:09 2107211 /usr/lib64/libboost_thread-mt.so.5
7ff44db35000-7ff44dd35000 ---p 00013000 08:09 2107211 /usr/lib64/libboost_thread-mt.so.5
7ff44dd35000-7ff44dd37000 rw-p 00013000 08:09 2107211 /usr/lib64/libboost_thread-mt.so.5
7ff44dd37000-7ff44dd4d000 r-xp 00000000 08:09 2107214 /usr/lib64/libmicrohttpd.so.10.22.0
7ff44dd4d000-7ff44df4c000 ---p 00016000 08:09 2107214 /usr/lib64/libmicrohttpd.so.10.22.0
7ff44df4c000-7ff44df4d000 rw-p 00015000 08:09 2107214 /usr/lib64/libmicrohttpd.so.10.22.0
7ff44df4d000-7ff44df9f000 r-xp 00000000 08:09 6629 /usr/lib64/libcurl.so.4.1.1
7ff44df9f000-7ff44e19e000 ---p 00052000 08:09 6629 /usr/lib64/libcurl.so.4.1.1
7ff44e19e000-7ff44e1a1000 rw-p 00051000 08:09 6629 /usr/lib64/libcurl.so.4.1.1
7ff44e1a1000-7ff44e1c1000 r-xp 00000000 08:09 5418 /lib64/ld-2.12.so
7ff44e396000-7ff44e3bb000 rw-p 00000000 00:00 0
7ff44e3bd000-7ff44e3c0000 rw-p 00000000 00:00 0
7ff44e3c0000-7ff44e3c1000 r--p 0001f000 08:09 5418 /lib64/ld-2.12.so
7ff44e3c1000-7ff44e3c2000 rw-p 00020000 08:09 5418 /lib64/ld-2.12.so
7ff44e3c2000-7ff44e3c3000 rw-p 00000000 00:00 0
7ffe2a0c0000-7ffe2a0e1000 rw-p 00000000 00:00 0 [stack]
7ffe2a1b8000-7ffe2a1ba000 r--p 00000000 00:00 0 [vvar]
7ffe2a1ba000-7ffe2a1bc000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
1 ContextElements
It seems that it does not crash without this subscriptions.
thanks!
I think that what made it crash was a ContextUpdate from CEP:Proton, or at least its relation with, which has Orion as consumer. It sent following XML on http://node-1:1026/v1/updateContext
<?xml version="1.0" encoding="UTF-8"?>
<updateContextRequest>
<contextElementList>
<contextElement>
<entityId type="Node" isPattern="false">
<id>XXXXXXXXXXXXXX</id>
</entityId>
<contextAttributeList>
<contextAttribute>
<name>Latitud</name>
<contextValue>0.0</contextValue>
</contextAttribute>
<contextAttribute>
<name>EventId</name>
<contextValue>afb3e832-6253-4816-ad31-62df35550e1b</contextValue>
</contextAttribute>
<contextAttribute>
<name>Longitud</name>
<contextValue>0.0</contextValue>
</contextAttribute>
<contextAttribute>
<name>DetectionTime</name>
<contextValue>2015-06-11T13:34:43.53000UTC</contextValue>
</contextAttribute>
<contextAttribute>
<name>Cost</name>
<contextValue>0.0</contextValue>
</contextAttribute>
<contextAttribute>
<name>Certainty</name>
<contextValue>1</contextValue>
</contextAttribute>
<contextAttribute>
<name>Name</name>
<contextValue>NodeContextUpdate</contextValue>
</contextAttribute>
<contextAttribute>
<name>presence</name>
<contextValue>0</contextValue>
</contextAttribute>
<contextAttribute>
<name>illuminance</name>
<contextValue>0.0</contextValue>
</contextAttribute>
<contextAttribute>
<name>Duration</name>
<contextValue>0</contextValue>
</contextAttribute>
<contextAttribute>
<name>batteryCharge</name>
<contextValue>67</contextValue>
</contextAttribute>
</contextAttributeList>
</contextElement>
</contextElementList>
<updateAction>UPDATE</updateAction>
</updateContextRequest>
Removing Proton subscription, Orion remains stable.
The fact is that when I manually send it it works.
The "longjmp causes uninitialized stack frame" is related with a bug recently discovered in Orion 0.22.0. We plan to solve that bug for Orion 0.23.0 but while the fix comes, the workaround is to use directly IP addresses in reference field at subscribeContext.
EDIT: the bug has been already fixed in the develop branch, so it will be available on next Orion version (i.e. 0.23.0).
Sometimes executing my script gets me a "Segmentation Fault" and sometimes it gets me the output below. Any help would be appreciated? I'm no Perl expert, so please go easy on my coding skills. You can view my code here.
*** glibc detected *** perl: free(): invalid pointer: 0x0000000002982e64 ***
======= Backtrace: =========
/lib64/libc.so.6[0x32eac714af]
/lib64/libc.so.6(cfree+0x4b)[0x32eac757ab]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_sv_clear+0x6d5)[0x3ee4895a05]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_sv_free+0x80)[0x3ee4895b50]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_av_undef+0x60)[0x3ee48895e0]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_sv_clear+0x48c)[0x3ee48957bc]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_sv_free+0x80)[0x3ee4895b50]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_leave_scope+0xc75)[0x3ee48b8ef5]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_pp_leavesub+0x11d)[0x3ee488c41d]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_runops_standard+0xe)[0x3ee488a37e]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so[0x3ee48bb02a]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_runops_standard+0xe)[0x3ee488a37e]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_call_sv+0x4d6)[0x3ee48375a6]
/home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/DBI/DBI.so(XS_DBI_dispatch+0x1f22)[0x7fb85387fc32]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_pp_entersub+0x3f6)[0x3ee4890ad6]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(Perl_runops_standard+0xe)[0x3ee488a37e]
/usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so(perl_run+0x11c)[0x3ee4837ecc]
perl(main+0x150)[0x401960]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x32eac1d9f4]
perl[0x401759]
======= Memory map: ========
00400000-00403000 r-xp 00000000 08:04 110043381 /usr/bin/perl
00602000-00604000 rw-p 00002000 08:04 110043381 /usr/bin/perl
00604000-02fc2000 rw-p 00000000 00:00 0 [heap]
30eec00000-30eed2d000 r-xp 00000000 08:04 9224244 /lib64/libcrypto.so.0.9.8e
30eed2d000-30eef2c000 ---p 0012d000 08:04 9224244 /lib64/libcrypto.so.0.9.8e
30eef2c000-30eef4d000 rw-p 0012c000 08:04 9224244 /lib64/libcrypto.so.0.9.8e
30eef4d000-30eef51000 rw-p 00000000 00:00 0
30ef000000-30ef048000 r-xp 00000000 08:04 9224343 /lib64/libssl.so.0.9.8e
30ef048000-30ef248000 ---p 00048000 08:04 9224343 /lib64/libssl.so.0.9.8e
30ef248000-30ef24e000 rw-p 00048000 08:04 9224343 /lib64/libssl.so.0.9.8e
32ea800000-32ea81c000 r-xp 00000000 08:04 9224227 /lib64/ld-2.5.so
32eaa1c000-32eaa1d000 r--p 0001c000 08:04 9224227 /lib64/ld-2.5.so
32eaa1d000-32eaa1e000 rw-p 0001d000 08:04 9224227 /lib64/ld-2.5.so
32eac00000-32ead4f000 r-xp 00000000 08:04 9224334 /lib64/libc-2.5.so
32ead4f000-32eaf4f000 ---p 0014f000 08:04 9224334 /lib64/libc-2.5.so
32eaf4f000-32eaf53000 r--p 0014f000 08:04 9224334 /lib64/libc-2.5.so
32eaf53000-32eaf54000 rw-p 00153000 08:04 9224334 /lib64/libc-2.5.so
32eaf54000-32eaf59000 rw-p 00000000 00:00 0
32eb000000-32eb082000 r-xp 00000000 08:04 9224469 /lib64/libm-2.5.so
32eb082000-32eb281000 ---p 00082000 08:04 9224469 /lib64/libm-2.5.so
32eb281000-32eb282000 r--p 00081000 08:04 9224469 /lib64/libm-2.5.so
32eb282000-32eb283000 rw-p 00082000 08:04 9224469 /lib64/libm-2.5.so
32eb400000-32eb402000 r-xp 00000000 08:04 9224462 /lib64/libdl-2.5.so
32eb402000-32eb602000 ---p 00002000 08:04 9224462 /lib64/libdl-2.5.so
32eb602000-32eb603000 r--p 00002000 08:04 9224462 /lib64/libdl-2.5.so
32eb603000-32eb604000 rw-p 00003000 08:04 9224462 /lib64/libdl-2.5.so
32eb800000-32eb816000 r-xp 00000000 08:04 9224477 /lib64/libpthread-2.5.so
32eb816000-32eba16000 ---p 00016000 08:04 9224477 /lib64/libpthread-2.5.so
32eba16000-32eba17000 r--p 00016000 08:04 9224477 /lib64/libpthread-2.5.so
32eba17000-32eba18000 rw-p 00017000 08:04 9224477 /lib64/libpthread-2.5.so
32eba18000-32eba1c000 rw-p 00000000 00:00 0
32ebc00000-32ebc14000 r-xp 00000000 08:04 9224305 /lib64/libz.so.1.2.3
32ebc14000-32ebe13000 ---p 00014000 08:04 9224305 /lib64/libz.so.1.2.3
32ebe13000-32ebe14000 rw-p 00013000 08:04 9224305 /lib64/libz.so.1.2.3
32ec000000-32ec007000 r-xp 00000000 08:04 9224484 /lib64/librt-2.5.so
32ec007000-32ec207000 ---p 00007000 08:04 9224484 /lib64/librt-2.5.so
32ec207000-32ec208000 r--p 00007000 08:04 9224484 /lib64/librt-2.5.so
32ec208000-32ec209000 rw-p 00008000 08:04 9224484 /lib64/librt-2.5.so
32ec400000-32ec415000 r-xp 00000000 08:04 9224496 /lib64/libselinux.so.1
32ec415000-32ec615000 ---p 00015000 08:04 9224496 /lib64/libselinux.so.1
32ec615000-32ec617000 rw-p 00015000 08:04 9224496 /lib64/libselinux.so.1
32ec617000-32ec618000 rw-p 00000000 00:00 0
32ec800000-32ec83b000 r-xp 00000000 08:04 9224492 /lib64/libsepol.so.1
32ec83b000-32eca3b000 ---p 0003b000 08:04 9224492 /lib64/libsepol.so.1
32eca3b000-32eca3c000 rw-p 0003b000 08:04 9224492 /lib64/libsepol.so.1
32eca3c000-32eca46000 rw-p 00000000 00:00 0
32ecc00000-32ecc0d000 r-xp 00000000 08:04 9224472 /lib64/libgcc_s-4.1.2-20080825.so.1
32ecc0d000-32ece0d000 ---p 0000d000 08:04 9224472 /lib64/libgcc_s-4.1.2-20080825.so.1
32ece0d000-32ece0e000 rw-p 0000d000 08:04 9224472 /lib64/libgcc_s-4.1.2-20080825.so.1
32ed000000-32ed015000 r-xp 00000000 08:04 9224474 /lib64/libnsl-2.5.so
32ed015000-32ed214000 ---p 00015000 08:04 9224474 /lib64/libnsl-2.5.so
32ed214000-32ed215000 r--p 00014000 08:04 9224474 /lib64/libnsl-2.5.so
32ed215000-32ed216000 rw-p 00015000 08:04 9224474 /lib64/libnsl-2.5.so
32ed216000-32ed218000 rw-p 00000000 00:00 0
32ee800000-32ee811000 r-xp 00000000 08:04 9224478 /lib64/libresolv-2.5.so
32ee811000-32eea11000 ---p 00011000 08:04 9224478 /lib64/libresolv-2.5.so
32eea11000-32eea12000 r--p 00011000 08:04 9224478 /lib64/libresolv-2.5.so
32eea12000-32eea13000 rw-p 00012000 08:04 9224478 /lib64/libresolv-2.5.so
32eea13000-32eea15000 rw-p 00000000 00:00 0
32eec00000-32eec09000 r-xp 00000000 08:04 9224480 /lib64/libcrypt-2.5.so
32eec09000-32eee08000 ---p 00009000 08:04 9224480 /lib64/libcrypt-2.5.so
32eee08000-32eee09000 r--p 00008000 08:04 9224480 /lib64/libcrypt-2.5.so
32eee09000-32eee0a000 rw-p 00009000 08:04 9224480 /lib64/libcrypt-2.5.so
32eee0a000-32eee38000 rw-p 00000000 00:00 0
32efc00000-32efc02000 r-xp 00000000 08:04 9224285 /lib64/libkeyutils-1.2.so
32efc02000-32efe01000 ---p 00002000 08:04 9224285 /lib64/libkeyutils-1.2.so
32efe01000-32efe02000 rw-p 00001000 08:04 9224285 /lib64/libkeyutils-1.2.so
32f0000000-32f0002000 r-xp 00000000 08:04 9224499 /lib64/libcom_err.so.2.1
32f0002000-32f0201000 ---p 00002000 08:04 9224499 /lib64/libcom_err.so.2.1
32f0201000-32f0202000 rw-p 00001000 08:04 9224499 /lib64/libcom_err.so.2.1
32f9400000-32f9402000 r-xp 00000000 08:04 9224482 /lib64/libutil-2.5.so
32f9402000-32f9601000 ---p 00002000 08:04 9224482 /lib64/libutil-2.5.so
32f9601000-32f9602000 r--p 00001000 08:04 9224482 /lib64/libutil-2.5.so
32f9602000-32f9603000 rw-p 00002000 08:04 9224482 /lib64/libutil-2.5.so
3ee4800000-3ee492c000 r-xp 00000000 08:04 110405822 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
3ee492c000-3ee4b2b000 ---p 0012c000 08:04 110405822 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
3ee4b2b000-3ee4b34000 rw-p 0012b000 08:04 110405822 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
3ee4b34000-3ee4b36000 rw-p 00000000 00:00 0
3fea800000-3fea891000 r-xp 00000000 08:04 157884806 /usr/lib64/libkrb5.so.3.3
3fea891000-3feaa91000 ---p 00091000 08:04 157884806 /usr/lib64/libkrb5.so.3.3
3feaa91000-3feaa95000 rw-p 00091000 08:04 157884806 /usr/lib64/libkrb5.so.3.3
3feac00000-3feac2c000 r-xp 00000000 08:04 157884807 /usr/lib64/libgssapi_krb5.so.2.2
3feac2c000-3feae2c000 ---p 0002c000 08:04 157884807 /usr/lib64/libgssapi_krb5.so.2.2
3feae2c000-3feae2e000 rw-p 0002c000 08:04 157884807 /usr/lib64/libgssapi_krb5.so.2.2
3feb000000-3feb024000 r-xp 00000000 08:04 157884805 /usr/lib64/libk5crypto.so.3.1
3feb024000-3feb223000 ---p 00024000 08:04 157884805 /usr/lib64/libk5crypto.so.3.1
3feb223000-3feb225000 rw-p 00023000 08:04 157884805 /usr/lib64/libk5crypto.so.3.1
3feb400000-3feb408000 r-xp 00000000 08:04 157884804 /usr/lib64/libkrb5support.so.0.1
3feb408000-3feb607000 ---p 00008000 08:04 157884804 /usr/lib64/libkrb5support.so.0.1
3feb607000-3feb608000 rw-p 00007000 08:04 157884804 /usr/lib64/libkrb5support.so.0.1
7fb851f89000-7fb851fd7000 r-xp 00000000 08:04 110413043 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
7fb851fd7000-7fb8521d6000 ---p 0004e000 08:04 110413043 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
7fb8521d6000-7fb8521d8000 rw-p 0004d000 08:04 110413043 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Net/SSLeay/SSLeay.so
7fb8521d8000-7fb8521da000 r-xp 00000000 08:04 110405769 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
7fb8521da000-7fb8523da000 ---p 00002000 08:04 110405769 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
7fb8523da000-7fb8523db000 rw-p 00002000 08:04 110405769 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Cwd/Cwd.so
7fb8523db000-7fb8523fc000 rw-p 00000000 00:00 0
7fb8523fc000-7fb852402000 r-xp 00000000 08:04 110412622 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
7fb852402000-7fb852601000 ---p 00006000 08:04 110412622 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
7fb852601000-7fb852602000 rw-p 00005000 08:04 110412622 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Time/HiRes/HiRes.so
7fb852602000-7fb852609000 r-xp 00000000 08:04 110405775 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
7fb852609000-7fb852809000 ---p 00007000 08:04 110405775 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
7fb852809000-7fb85280a000 rw-p 00007000 08:04 110405775 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so
7fb85280a000-7fb85280c000 r-xp 00000000 08:04 70410455 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/Clone/Clone.so
7fb85280c000-7fb852a0c000 ---p 00002000 08:04 70410455 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/Clone/Clone.so
7fb852a0c000-7fb852a0d000 rw-p 00002000 08:04 70410455 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/Clone/Clone.so
7fb852a0d000-7fb852a12000 r-xp 00000000 08:04 70410438 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/Params/Util/Util.so
7fb852a12000-7fb852c11000 ---p 00005000 08:04 70410438 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/Params/Util/Util.so
7fb852c11000-7fb852c12000 rw-p 00004000 08:04 70410438 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/Params/Util/Util.so
7fb852c12000-7fb852c30000 r-xp 00000000 08:04 110791534 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
7fb852c30000-7fb852e30000 ---p 0001e000 08:04 110791534 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
7fb852e30000-7fb852e31000 rw-p 0001e000 08:04 110791534 /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/auto/Compress/Raw/Zlib/Zlib.so
7fb852e31000-7fb852e36000 r-xp 00000000 08:04 110414506 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
7fb852e36000-7fb853035000 ---p 00005000 08:04 110414506 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
7fb853035000-7fb853036000 rw-p 00004000 08:04 110414506 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/File/Glob/Glob.so
7fb853036000-7fb85303a000 r-xp 00000000 08:04 9224292 /lib64/libnss_dns-2.5.so
7fb85303a000-7fb853239000 ---p 00004000 08:04 9224292 /lib64/libnss_dns-2.5.so
7fb853239000-7fb85323a000 r--p 00003000 08:04 9224292 /lib64/libnss_dns-2.5.so
7fb85323a000-7fb85323b000 rw-p 00004000 08:04 9224292 /lib64/libnss_dns-2.5.so
7fb85323b000-7fb853245000 r-xp 00000000 08:04 9224294 /lib64/libnss_files-2.5.so
7fb853245000-7fb853444000 ---p 0000a000 08:04 9224294 /lib64/libnss_files-2.5.so
7fb853444000-7fb853445000 r--p 00009000 08:04 9224294 /lib64/libnss_files-2.5.so
7fb853445000-7fb853446000 rw-p 0000a000 08:04 9224294 /lib64/libnss_files-2.5.so
7fb853446000-7fb85344b000 r-xp 00000000 08:04 110412590 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
7fb85344b000-7fb85364a000 ---p 00005000 08:04 110412590 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
7fb85364a000-7fb85364b000 rw-p 00004000 08:04 110412590 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Socket/Socket.so
7fb85364b000-7fb853667000 r-xp 00000000 08:04 110412428 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
7fb853667000-7fb853867000 ---p 0001c000 08:04 110412428 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
7fb853867000-7fb853868000 rw-p 0001c000 08:04 110412428 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/POSIX/POSIX.so
7fb853868000-7fb853888000 r-xp 00000000 08:04 56328843 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/DBI/DBI.so
7fb853888000-7fb853a87000 ---p 00020000 08:04 56328843 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/DBI/DBI.so
7fb853a87000-7fb853a88000 rw-p 0001f000 08:04 56328843 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/DBI/DBI.so
7fb853a88000-7fb853a94000 r-xp 00000000 08:04 56328198 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/Text/CSV_XS/CSV_XS.so
7fb853a94000-7fb853c93000 ---p 0000c000 08:04 56328198 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/Text/CSV_XS/CSV_XS.so
7fb853c93000-7fb853c94000 rw-p 0000b000 08:04 56328198 /home/yostauto/perl5/lib/perl5/x86_64-linux-thread-multi/auto/Text/CSV_XS/CSV_XS.so
7fb853c94000-7fb853c9f000 r-xp 00000000 08:04 110412967 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so
7fb853c9f000-7fb853e9e000 ---p 0000b000 08:04 110412967 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so
7fb853e9e000-7fb853e9f000 rw-p 0000a000 08:04 110412967 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/HTML/Parser/Parser.so
7fb853e9f000-7fb853ebb000 r-xp 00000000 08:04 110413015 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
7fb853ebb000-7fb8540bb000 ---p 0001c000 08:04 110413015 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
7fb8540bb000-7fb8540bc000 rw-p 0001c000 08:04 110413015 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/List/MoreUtils/MoreUtils.so
7fb8540bc000-7fb8540c8000 r-xp 00000000 08:04 110412819 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Params/Validate/Validate.so
7fb8540c8000-7fb8542c7000 ---p 0000c000 08:04 110412819 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Params/Validate/Validate.so
7fb8542c7000-7fb8542c8000 rw-p 0000b000 08:04 110412819 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/Params/Validate/Validate.so
7fb8542c8000-7fb8542cd000 r-xp 00000000 08:04 110414454 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/DateTime/DateTime.so
7fb8542cd000-7fb8544cc000 ---p 00005000 08:04 110414454 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/DateTime/DateTime.so
7fb8544cc000-7fb8544cd000 rw-p 00004000 08:04 110414454 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/DateTime/DateTime.so
7fb8544cd000-7fb8544d9000 r-xp 00000000 08:04 110412715 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
7fb8544d9000-7fb8546d8000 ---p 0000c000 08:04 110412715 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
7fb8546d8000-7fb8546d9000 rw-p 0000b000 08:04 110412715 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/List/Util/Util.so
7fb8546d9000-7fb8546e2000 r-xp 00000000 08:04 110405801 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Encode.so
7fb8546e2000-7fb8548e1000 ---p 00009000 08:04 110405801 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Encode.so
7fb8548e1000-7fb8548e2000 rw-p 00008000 08:04 110405801 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Encode/Encode.so
7fb8548e2000-7fb8548e5000 r-xp 00000000 08:04 110412420 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
7fb8548e5000-7fb854ae4000 ---p 00003000 08:04 110412420 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
7fb854ae4000-7fb854ae5000 rw-p 00002000 08:04 110412420 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/MIME/Base64/Base64.so
7fb854ae5000-7fb854ae8000 r-xp 00000000 08:04 110412398 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
7fb854ae8000-7fb854ce7000 ---p 00003000 08:04 110412398 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
7fb854ce7000-7fb854ce8000 rw-p 00002000 08:04 110412398 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so
7fb854ce8000-7fb854cec000 r-xp 00000000 08:04 110412414 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
7fb854cec000-7fb854eeb000 ---p 00004000 08:04 110412414 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
7fb854eeb000-7fb854eec000 rw-p 00003000 08:04 110412414 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/auto/IO/IO.so
7fb854eec000-7fb854f12000 rw-p 00000000 00:00 0
7fb854f28000-7fb854f2a000 rw-p 00000000 00:00 0
7fffe55a9000-7fffe55ca000 rw-p 00000000 00:00 0 [stack]
7fffe55ff000-7fffe5600000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r--p 00000000 00:00 0 [vsyscall]
/bin/sh: line 1: 15018 Aborted perl /home/user/emailservice/send.pl
Admins told us that there is some prolem in production. They noticed big jump in memory usage and in requests waiting.
I received one crash dump. I need help analysing it.
Using Debug Diagnostic Tool I found this:
Detected possible blocking or leaked critical section at 0x1e5bd320 owned by thread 141 in dllhst3g.exe__Metastorm Process Engine__PID__7444__Date__10_25_2011__Time_01_19_15PM__686__Manual Dump.dmp
Impact of this lock
11,59% of threads blocked
(Threads 97 137 142 143 144 147 207 208 211 212 213 214 215 216 217 218 219 221 222 223 224 225 226 227 228 229 230)
The following functions are trying to enter this critical section
ACECORE+20eb
The following module(s) are involved with this critical section
C:\Program Files\Common Files\Microsoft Shared\OFFICE12\ACECORE.DLL from Microsoft Corporation
From the listed threads only thread 142 is waiting for thread 141. All others are waiting for thread 142.
~141
141 Id: 1d14.1b38 Suspend: 0 Teb: 7fee4000 Unfrozen
Start: msvcrt!_endthreadex+0x2f (77bcb4bc)
Priority: 0 Priority class: 32 Affinity: ff
~141s : Edit: new stacktrace after fixing symbols
kb
2a2efdcc 7c827b89 77e6202c 00000003 2a2efe1c ntdll!KiFastSystemCallRet
2a2efdd0 77e6202c 00000003 2a2efe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
2a2efe78 7739bbd1 00000003 2a2efea0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
2a2efed4 7739ce36 00000002 2a2eff74 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141
2a2efef0 4a77cb28 00000002 2a2eff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
2a2eff84 77bcb530 33c23fe8 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
2a2effb8 77e6482f 37e3b7e8 00000000 00000000 msvcrt!_endthreadex+0xa3
2a2effec 00000000 77bcb4bc 37e3b7e8 00000000 kernel32!BaseThreadStart+0x34
~142
142 Id: 1d14.1128 Suspend: 0 Teb: 7feb9000 Unfrozen
Start: msvcrt!_endthreadex+0x2f (77bcb4bc)
Priority: 0 Priority class: 32 Affinity: ff
~142s : Edit: new stacktrace after fixing symbols
kb
31b5aa88 7c827b99 7c83d09c 00006c44 00000000 ntdll!KiFastSystemCallRet
31b5aa8c 7c83d09c 00006c44 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
31b5aac8 7c83d0e7 00006c44 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x1a3
*** ERROR: Module load completed but symbols could not be loaded for ACECORE.DLL
31b5aae8 3c9e20eb 1e5bd320 31b5ab3c 2ae13a61 ntdll!RtlEnterCriticalSection+0xa8
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 00000000 00000000 00000000 ACECORE+0x20eb
!clrstack
Shows call to System.Data.OleDb.OleDbConnection.Open()
I found it connects to xls
Provider=Microsoft.ACE.OLEDB.12.0;Data Source=c:\temp\somefile.xls;Extended Properties="Excel 8.0;HDR=Yes;"
~97
97 Id: 1d14.730 Suspend: 0 Teb: 7fee5000 Unfrozen
Start: mscorwks!CorExitProcess+0x21ef9 (79f756cf)
Priority: 0 Priority class: 32 Affinity: ff
Edit: new stacktrace after fixing symbols
~97s
kb
27e1ca10 7c827b99 77e61d1e 000018e4 00000000 ntdll!KiFastSystemCallRet
27e1ca14 77e61d1e 000018e4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
27e1ca84 77e61c8d 000018e4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
27e1ca98 7769c7ad 000018e4 ffffffff 2386d3a8 kernel32!WaitForSingleObject+0x12
27e1cab4 7778b5cb 24a1c758 2386d3a8 00000000 ole32!GetToSTA+0x7c
27e1cad4 7778c38b 27e1cb9c 27e1cc9c 22d3e674 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xcb
27e1cbb4 776c0585 22d3e674 27e1ccac 27e1cc9c ole32!CRpcChannelBuffer::SendReceive2+0xd3
27e1cc20 776c051a 22d3e674 27e1ccac 27e1cc9c ole32!CAptRpcChnl::SendReceive+0xab
27e1cc74 77ce347f 22d3e674 27e1ccac 27e1cc9c ole32!CCtxComChnl::SendReceive+0x1a9
27e1cc90 77ce352f 45e02be4 27e1ccd8 0600016e rpcrt4!NdrProxySendReceive+0x43
27e1d080 77ce35a6 776762b8 776794ba 27e1d0b8 rpcrt4!NdrClientCall2+0x206
27e1d0a0 77c65037 00000014 00000004 27e1d0d0 rpcrt4!ObjectStublessClient+0x8b
27e1d0b0 776ad951 45e02be4 00000000 27e1d89c rpcrt4!ObjectStubless+0xf
27e1d0d0 776acb4b 77794960 00000001 00000000 ole32!CProcessActivator::CCICallback+0x6d
27e1d0f0 776acafc 77794960 27e1d6f8 00000000 ole32!CProcessActivator::AttemptActivation+0x2c
27e1d12c 776ada3b 77794960 27e1d6f8 00000000 ole32!CProcessActivator::ActivateByContext+0x4f
27e1d154 776aaf9e 77794960 00000000 27e1d89c ole32!CProcessActivator::CreateInstance+0x49
27e1d194 4a777108 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d210 776aaf9e 443dd8f8 00000000 27e1d89c comsvcs!CSTAPoolActivator::CreateInstance+0x5a9
27e1d250 4a766303 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d2a0 776aaf9e 341d3168 00000000 27e1d89c comsvcs!CStdContextActivator::CreateInstance+0x221
27e1d2e0 77727f8a 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d310 776f5c55 0009a0f8 00000000 27e1d89c ole32!CSurrogateProcessActivator::CreateInstance+0xf7
27e1d344 776aaf9e 7779487c 00000000 27e1d89c ole32!CClientContextActivator::CreateInstance+0xc9
27e1d384 776ab12f 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
27e1d68c 776a67ba 361f8c54 00000000 00000015 ole32!ICoCreateInstanceEx+0x3f8
27e1d6c0 7769b9b3 361f8c54 00000000 00000000 ole32!CComActivator::DoCreateInstance+0x6a
27e1ddc8 4a7516d8 45bfeb58 361f8c54 00000000 ole32!CComActivator::StandardCreateInstance+0x7c
27e1de3c 4a751fc6 00000000 361f8c40 00000000 comsvcs!CClassFactoryWrapper::ActivateOnMachine+0xaf
27e1de74 7a078d9b 361f8c40 00000000 79edda70 comsvcs!CClassFactoryWrapper::CreateInstance+0x80
27e1e118 7a07a1e6 361f8c40 00000000 00000000 mscorwks!ComClassFactory::CreateInstanceFromClassFactory+0x102
27e1e174 7a0bf10a 1e772994 00000000 27e1e1b4 mscorwks!ComClassFactory::CreateInstance+0x91
27e1e184 7a0c870b 1e772994 1e772994 79faa672 mscorwks!AllocateComObject_ForManaged+0x1e
27e1e1b4 79e9c82b 1e772994 234b20e8 79faa737 mscorwks!AllocateObject+0x38
27e1e1c0 79faa737 05ff2627 00000001 072336fc mscorwks!MethodTable::Allocate+0x35
27e1e260 792c25c3 1e772994 27e1e278 792c257c mscorwks!CRemotingServices::AllocateUninitializedObject+0xdf
27e1e278 792c1951 0f5b1b50 27e1e2d0 03100120 mscorlib_ni+0x2025c3
27e1e290 792c143e 44194464 00000000 00000016 mscorlib_ni+0x201951
27e1e2ac 79e71e04 00000001 27e1e334 79faa569 mscorlib_ni+0x20143e
27e1e2b8 79faa569 792c1400 072336fc 00000000 mscorwks!CTPMethodTable__CallTargetHelper3+0xf
27e1e334 79faa5d8 072336fc 00000000 00000001 mscorwks!CTPMethodTable::CallTarget+0xdd
27e1e348 79faa614 790fd65c 072336fc 00000000 mscorwks!CTPMethodTable::CallTarget+0x14
27e1e368 7a04b578 1e772994 00000000 00000001 mscorwks!CRemotingServices::CreateProxyOrObject+0x38
27e1e414 1e923c99 0f5b1a58 0f5b1b1c 27e1e464 mscorwks!JIT_NewCrossContextHelper+0xa9
WARNING: Frame IP not in any known module. Following frames may be wrong.
27e1e430 776e2fae 00000000 00000064 00000001 0x1e923c99
27e1e464 79e71b4c 27e1ef70 00000064 00000001 ole32!CoGetContextToken+0x29
27e1e494 79e821b9 27e1ee70 00000008 27e1ee20 mscorwks!CallDescrWorker+0x33
27e1e514 7a0f851b 27e1ee70 00000008 27e1ee20 mscorwks!CallDescrWorkerWithHandler+0xa3
27e1e5e4 79270454 00000001 00000000 00000000 mscorwks!CMessage::Dispatch+0x162
27e1e63c 7977c16e 00000001 00000000 075d788c mscorlib_ni+0x1b0454
27e1e658 6744d146 0752309c 0f5b19d8 0752304c mscorlib_ni+0x6bc16e
27e1e690 197cb7c7 0d0e3640 79e7a6b8 27e1ea80 System_EnterpriseServices_ni+0x5d146
27e1e70c 197f3d04 27e1eae4 0071f13b 36c6b460 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2b7c7)
27e1e74c 77720df0 27e1eae4 35ccbed0 00000000 System_EnterpriseServices_Wrapper_197f0000!System::EnterpriseServices::Thunk::FilteringCallbackFunction+0x44
27e1e798 7772189c 00000000 35ccbed0 197f3cc0 ole32!EnterForCallback+0xc4
27e1e8f8 776f0418 27e1e7d0 197f3cc0 27e1eae4 ole32!SwitchForCallback+0x1a3
27e1e924 7769c194 35ccbed0 197f3cc0 27e1eae4 ole32!PerformCallback+0x54
27e1e9bc 776e316c 36c6b460 197f3cc0 27e1eae4 ole32!CObjectContext::InternalContextCallback+0x159
27e1ea0c 79e71d8b 36c6b470 197f3cc0 27e1eae4 ole32!CObjectContext::ContextCallback+0x85
27e1ea68 197cbc82 776e30e7 0f5b1a28 072323f0 mscorwks!PInvokeCalliReturnFromCall
27e1ea88 197cbc82 0010c8f0 00000000 00000008 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bc82)
27e1eab4 197cbb51 07231a74 27e1eb2c 79282f85 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bc82)
27e1eac0 79282f85 27e1eb44 00000000 00000000 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bb51)
27e1eb2c 797f5a0d 03100788 072323f0 075d7830 mscorlib_ni+0x1c2f85
00000000 00000000 00000000 00000000 00000000 mscorlib_ni+0x735a0d
And all other threads waiting for thread 142 have same stack trace.
I don't know what thread 141 is doing. Thread 142 cannot open connection to xls file. And not sure why other threads wait for thread 142.
Maybe you have any ideas?
EDIT:
analyze -v -hang
GetPageUrlData failed, server returned HTTP status 404
URL requested: http://watson.microsoft.com/StageOne/dllhst3g_exe/5_2_3790_3959/unknown/0_0_0_0/00000000.htm?Retriage=1
FAULTING_IP:
+1e02faf0157df58
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 0000008d
BUGCHECK_STR: HANG
PROCESS_NAME: dllhst3g.exe
ERROR_CODE: (NTSTATUS) 0xcfffffff -
EXCEPTION_CODE: (NTSTATUS) 0xcfffffff -
MOD_LIST:
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0xc2c (17)
Current frame:
ChildEBP RetAddr Caller,Callee
1974f450 655f1110 (MethodDesc 0x651b7efc +0x30 System.Data.ProviderBase.WrappedIUnknown.ReleaseHandle())
1974f93c 792e5e4f (MethodDesc 0x79107064 +0xf System.Runtime.InteropServices.SafeHandle.Dispose(Boolean))
1974f944 792e5d6b (MethodDesc 0x79107030 +0x1b System.Runtime.InteropServices.SafeHandle.Finalize())
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
0 1d14.71c Speculated (Triage) -->
17 1d14.c2c Critical Section -->
141 1d14.1b38 Event
WAIT_CHAIN_COMMAND: ~0s;k;;~17s;k;;~141s;k;;
BLOCKING_THREAD: 00001b38
DEFAULT_BUCKET_ID: APPLICATION_HANG_WRONG_SYMBOLS
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_WRONG_SYMBOLS
LAST_CONTROL_TRANSFER: from 7c827b89 to 7c82847c
STACK_TEXT:
2a2efdcc 7c827b89 77e6202c 00000003 2a2efe1c ntdll!KiFastSystemCallRet
2a2efdd0 77e6202c 00000003 2a2efe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc
2a2efe78 7739bbd1 00000003 2a2efea0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a
2a2efed4 7739ce36 00000002 2a2eff74 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141
2a2efef0 4a77cb28 00000002 2a2eff74 00000000 user32!MsgWaitForMultipleObjects+0x1f
2a2eff84 77bcb530 33c23fe8 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9
2a2effb8 77e6482f 37e3b7e8 00000000 00000000 msvcrt!_endthreadex+0xa3
2a2effec 00000000 77bcb4bc 37e3b7e8 00000000 kernel32!BaseThreadStart+0x34
FOLLOWUP_IP:
ntdll!KiFastSystemCallRet+0
7c82847c c3 ret
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: ntdll!KiFastSystemCallRet+0
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: ntdll
IMAGE_NAME: ntdll.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4cc18322
STACK_COMMAND: ~141s ; kb
BUCKET_ID: HANG_ntdll!KiFastSystemCallRet+0
FAILURE_BUCKET_ID: APPLICATION_HANG_WRONG_SYMBOLS_cfffffff_ntdll.dll!KiFastSystemCallRet
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/dllhst3g_exe/5_2_3790_3959/45d69678/unknown/0_0_0_0/bbbbbbb4/cfffffff/00000000.htm?Retriage=1
Followup: MachineOwner
Not sure I understand it. But thread 17 is finalizer thread. So looks like it is blocked? And I thinkg it as again some kind of access to xls?
~17s
kb
ChildEBP RetAddr Args to Child
1974f08c 7c827b99 7c83d09c 00003a98 00000000 ntdll!KiFastSystemCallRet
1974f090 7c83d09c 00003a98 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc
1974f0cc 7c83d0e7 00003a98 00000004 00000001 ntdll!RtlpWaitOnCriticalSection+0x1a3
1974f0ec 2ae1fa94 29202124 32787727 00000000 ntdll!RtlEnterCriticalSection+0xa8
WARNING: Stack unwind information not available. Following frames may be wrong.
1974f160 4c856487 2263eff4 00000000 00000005 ACEOLEDB!DllGetClassObject+0xdc2d
1974f174 4c862d06 2263eff4 4c8633c4 2263efe8 oledb32!CACMDynamic::CmFinalRelease+0x50
1974f17c 4c8633c4 2263efe8 2263efe8 1974f1e4 oledb32!CDCM::FinalRelease+0x1b
1974f198 4c8633f6 00000000 2263efe8 1974f1b8 oledb32!ATL::CComPolyObject::~CComPolyObject+0x2a
1974f1a8 4c88d5ad 2263efe8 2370a39c 1974f1f4 oledb32!ATL::CComPolyObject::Release+0x25
1974f1b8 4a757681 1a271fa0 2263efe8 2370a39c oledb32!CDCMCreator::DestroyResource+0xf
1974f1f4 4a75793e 2370a39c 2263efe8 00000000 comsvcs!CHolder::SafeDispenserDriver::DestroyResource+0x20
1974f20c 4a758a35 23151cd0 00000000 2fa44808 comsvcs!CHolder::ProcessDestroyList+0x2e
1974f238 4c88d30e 23151cd0 2263efe8 2263efe8 comsvcs!CHolder::FreeResource+0x7f
1974f268 4c878a3a 2263efe8 00000000 271aa824 oledb32!CDCMCreator::ReleaseResource+0x31
1974f288 4c8545e7 271aa824 00000000 4c85456a oledb32!CDPO::ReturnDCMToPool+0x89
1974f294 4c85456a 271aa818 271aa818 1974f320 oledb32!CDPO::FinalRelease+0xb
1974f2b0 4c88e78a 271aa818 1974f2d0 4c85218c oledb32!ATL::CComPolyObject::~CComPolyObject+0x2a
1974f2bc 4c85218c 00000001 00000000 000eaaf8 oledb32!ATL::CComPolyObject::`scalar deleting destructor'+0xd
1974f2d0 4c8521aa 271aa818 1974f330 79f63ff9 oledb32!ATL::CComPolyObject::Release+0x27
1974f2dc 79f63ff9 271aa824 3b6a3777 000eaaf8 oledb32!ATL::CComContainedObject::Release+0x11
1974f330 79f640ac 271aa828 3b6a3733 000eaaf8 mscorwks!ReleaseTransitionHelper+0x5f
1974f374 79f64110 271aa828 00000000 3b6a37ef mscorwks!SafeReleaseHelper+0x8c
1974f3a8 79f53b5d 271aa828 00000000 3b6a3017 mscorwks!SafeRelease+0x2f
1974f450 655f1110 000eaaf8 00000000 1974f470 mscorwks!MarshalNative::Release+0xb0
1974f460 79e71b4c 1974f4dc 000c5958 1974f4f0 System_Data_ni!_bidW103 (System_Data_ni+0x4a1110)
1974f470 79e821b9 1974f540 00000000 1974f510 mscorwks!CallDescrWorker+0x33
1974f4f0 79e96531 1974f540 00000000 1974f510 mscorwks!CallDescrWorkerWithHandler+0xa3
1974f634 79e96564 655f10e0 1974f6bc 1974f6a4 mscorwks!MethodDesc::CallDescr+0x19c
1974f650 79e96c4c 655f10e0 1974f6bc 1974f6a4 mscorwks!MethodDesc::CallTargetWorker+0x1f
1974f670 79eccd2a 1974f6a4 0b402780 00000004 mscorwks!MethodDescCallSite::Call_RetObjPtr+0x1c
1974f770 79eccd5e 0b402780 000eaaf8 000eaaf8 mscorwks!SafeHandle::RunReleaseMethod+0x89
1974f870 79eccbf4 00000001 0d0e3640 79e7a1c8 mscorwks!SafeHandle::Release+0x11b
1974f89c 79f83999 3b6a3d7b 00000000 000eaaf8 mscorwks!SafeHandle::Dispose+0x23
1974f93c 792e5e4f 1974f970 792e5d6b 0b3ac100 mscorwks!SafeHandle::Finalize+0xab
1974f944 792e5d6b 0b3ac100 ffffffff 00000000 mscorlib_ni+0x225e4f
1974f970 79f7169a 1974f9c4 79ef465c 1974fc1c mscorlib_ni+0x225d6b
1974f9d4 79f7175b 0b402780 792e5d50 08000000 mscorwks!MethodTable::GetObjCreateDelegate+0xaf
1974f9f4 79f71609 0b402780 0b402780 00000000 mscorwks!MethodTable::CallFinalizer+0xa0
1974fa08 79fd46a6 0b402780 3b6a3e1f 00000000 mscorwks!SVR::CallFinalizer+0xa6
1974fa58 79fd45d7 000eaaf8 00000415 1974fadf mscorwks!SVR::GCHeap::TraceGCSegments+0x1b0
1974fae0 79f5832f 0740eeb4 00000000 1974feb0 mscorwks!SVR::GCHeap::TraceGCSegments+0x2f6
1974faf4 79e9848f 1974fd04 000eaaf8 00000000 mscorwks!SVR::ProfScanRootsHelper+0x69
1974fb08 79e9842b 1974feb0 1974fb90 79fa6a6b mscorwks!Thread::DoADCallBack+0x32a
1974fb9c 79e98351 1974feb0 3b6a3f9f 000eaaf8 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
1974fbd8 79ec4322 1974feb0 00000000 1974fc98 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
1974fbe8 79f581f3 1974feb0 1974fc8c 79fa6a6b mscorwks!Thread::RaiseCrossContextException+0x434
1974fc98 79f58279 1a9cb678 79ec430e 1974feb0 mscorwks!Thread::DoADCallBack+0xcd
1974fcb8 79f58265 1974feb0 1974fd20 79f582e4 mscorwks!Thread::DoADCallBack+0x322
1974fcc4 79f582e4 1a9cb678 79f58316 1974fd04 mscorwks!ManagedThreadBase::FinalizerAppDomain+0x25
1974fd20 79fd45d7 000eaaf8 00000000 1974fda7 mscorwks!SVR::GCHeap::TraceGCSegments+0x251
1974fda8 79fd48a8 00000000 00000000 1974feb0 mscorwks!SVR::GCHeap::TraceGCSegments+0x2f6
1974fdc0 79e9848f 1974feb0 00000000 00000000 mscorwks!SVR::GCHeap::FinalizerThreadWorker+0xb7
1974fdd4 79e9842b 1974feb0 1974fe5c 79fa6a6b mscorwks!Thread::DoADCallBack+0x32a
1974fe68 79e98351 1974feb0 3b6a3ae3 00000000 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3
1974fea4 79f074d4 1974feb0 00000000 1a9cb678 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a
1974fecc 79f074e5 79fd4809 00000008 1974ff14 mscorwks!ManagedThreadBase_NoADTransition+0x32
1974fedc 79f090b3 79fd4809 3b6a3b53 00000000 mscorwks!ManagedThreadBase::FinalizerBase+0xd
1974ff14 79f75715 00000000 00000007 ffffffff mscorwks!SVR::GCHeap::FinalizerThreadStart+0xbb
1974ffb8 77e6482f 000eb528 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49
1974ffec 00000000 79f756cf 000eb528 00000000 kernel32!BaseThreadStart+0x34
You need to find who owns the following critical section that your thread is waiting on:
1974f0ec 2ae1fa94 29202124 32787727 00000000 ntdll!RtlEnterCriticalSection+0xa8
You can do an automated critical section analysis with call stacks:
!locks -v
This will dump all critical section locks that are in a locked state and the call stacks of the threads, you then need to scan each call stack for each lock to see if say Thread A is waiting on lock 1 which is owned by Thread B, Thread B is waiting on lock 2 which is owned by Thread A.
Hope this helps
I installed ActivePerl 5.10.1.1007 on my Ubuntu 10.04 machine.
I have a very simple Perl script with the following lines:
use lib "/opt/ActivePerl-5.10/lib";
use Tcl;
my $Interpreter = new Tcl;
$Interpreter->Eval('puts "Hello world"');
$Interpreter->Eval('set ::env(TESTVAR) 55')
The output is the following:
$ /opt/ActivePerl-5.10/bin/perl5.10.1 simple.pl
Hello world
*** glibc detected *** /opt/ActivePerl-5.10/bin/perl5.10.1: munmap_chunk(): invalid pointer: 0x09b5e0d8 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(+0x6b591)[0xb7696591]
/lib/tls/i686/cmov/libc.so.6(+0x6c80e)[0xb769780e]
/opt/ActivePerl-5.10/bin/perl5.10.1(perl_destruct+0xda6)[0x8071716]
/opt/ActivePerl-5.10/bin/perl5.10.1(main+0xb0)[0x8060a30]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb7641bd6]
/opt/ActivePerl-5.10/bin/perl5.10.1(chroot+0x31)[0x80608e1]
======= Memory map: ========
08048000-0815e000 r-xp 00000000 08:11 5154055 /opt/ActivePerl-5.10/bin/perl-static
0815e000-08160000 rwxp 00116000 08:11 5154055 /opt/ActivePerl-5.10/bin/perl-static
09aae000-09bf1000 rwxp 00000000 00:00 0 [heap]
b6ad8000-b6af5000 r-xp 00000000 08:11 3399754 /lib/libgcc_s.so.1
b6af5000-b6af6000 r-xp 0001c000 08:11 3399754 /lib/libgcc_s.so.1
b6af6000-b6af7000 rwxp 0001d000 08:11 3399754 /lib/libgcc_s.so.1
b6af7000-b6b01000 r-xp 00000000 08:11 3424653 /lib/tls/i686/cmov/libnss_files-2.11.1.so
b6b01000-b6b02000 r-xp 00009000 08:11 3424653 /lib/tls/i686/cmov/libnss_files-2.11.1.so
b6b02000-b6b03000 rwxp 0000a000 08:11 3424653 /lib/tls/i686/cmov/libnss_files-2.11.1.so
b6b03000-b6b0b000 r-xp 00000000 08:11 3424655 /lib/tls/i686/cmov/libnss_nis-2.11.1.so
b6b0b000-b6b0c000 r-xp 00007000 08:11 3424655 /lib/tls/i686/cmov/libnss_nis-2.11.1.so
b6b0c000-b6b0d000 rwxp 00008000 08:11 3424655 /lib/tls/i686/cmov/libnss_nis-2.11.1.so
b6b0d000-b6b13000 r-xp 00000000 08:11 3424651 /lib/tls/i686/cmov/libnss_compat-2.11.1.so
b6b13000-b6b14000 r-xp 00006000 08:11 3424651 /lib/tls/i686/cmov/libnss_compat-2.11.1.so
b6b14000-b6b15000 rwxp 00007000 08:11 3424651 /lib/tls/i686/cmov/libnss_compat-2.11.1.so
b6b2a000-b6b2b000 ---p 00000000 00:00 0
b6b2b000-b732b000 rwxp 00000000 00:00 0
b74c1000-b74cb000 r-xp 00000000 08:11 5162486 /opt/ActivePerl-5.10/lib/auto/Tcl/Tcl.so
b74cb000-b74cc000 rwxp 00009000 08:11 5162486 /opt/ActivePerl-5.10/lib/auto/Tcl/Tcl.so
b74cc000-b750b000 r-xp 00000000 08:11 5712734 /usr/lib/locale/en_US.utf8/LC_CTYPE
b750b000-b7629000 r-xp 00000000 08:11 5712735 /usr/lib/locale/en_US.utf8/LC_COLLATE
b7629000-b762b000 rwxp 00000000 00:00 0
b762b000-b777e000 r-xp 00000000 08:11 3424483 /lib/tls/i686/cmov/libc-2.11.1.so
b777e000-b777f000 ---p 00153000 08:11 3424483 /lib/tls/i686/cmov/libc-2.11.1.so
b777f000-b7781000 r-xp 00153000 08:11 3424483 /lib/tls/i686/cmov/libc-2.11.1.so
b7781000-b7782000 rwxp 00155000 08:11 3424483 /lib/tls/i686/cmov/libc-2.11.1.so
b7782000-b7785000 rwxp 00000000 00:00 0
b7785000-b779a000 r-xp 00000000 08:11 3424658 /lib/tls/i686/cmov/libpthread-2.11.1.so
b779a000-b779b000 r-xp 00014000 08:11 3424658 /lib/tls/i686/cmov/libpthread-2.11.1.so
b779b000-b779c000 rwxp 00015000 08:11 3424658 /lib/tls/i686/cmov/libpthread-2.11.1.so
b779c000-b779e000 rwxp 00000000 00:00 0
b779e000-b77a0000 r-xp 00000000 08:11 3424663 /lib/tls/i686/cmov/libutil-2.11.1.so
b77a0000-b77a1000 r-xp 00001000 08:11 3424663 /lib/tls/i686/cmov/libutil-2.11.1.so
b77a1000-b77a2000 rwxp 00002000 08:11 3424663 /lib/tls/i686/cmov/libutil-2.11.1.so
b77a2000-b77ab000 r-xp 00000000 08:11 3424646 /lib/tls/i686/cmov/libcrypt-2.11.1.so
b77ab000-b77ac000 r-xp 00008000 08:11 3424646 /lib/tls/i686/cmov/libcrypt-2.11.1.so
b77ac000-b77ad000 rwxp 00009000 08:11 3424646 /lib/tls/i686/cmov/libcrypt-2.11.1.so
b77ad000-b77d4000 rwxp 00000000 00:00 0
b77d4000-b77f8000 r-xp 00000000 08:11 3424648 /lib/tls/i686/cmov/libm-2.11.1.so
b77f8000-b77f9000 r-xp 00023000 08:11 3424648 /lib/tls/i686/cmov/libm-2.11.1.so
b77f9000-b77fa000 rwxp 00024000 08:11 3424648 /lib/tls/i686/cmov/libm-2.11.1.so
b77fa000-b77fb000 rwxp 00000000 00:00 0
b77fb000-b77fd000 r-xp 00000000 08:11 3424647 /lib/tls/i686/cmov/libdl-2.11.1.so
b77fd000-b77fe000 r-xp 00001000 08:11 3424647 /lib/tls/i686/cmov/libdl-2.11.1.so
b77fe000-b77ff000 rwxp 00002000 08:11 3424647 /lib/tls/i686/cmov/libdl-2.11.1.so
b77ff000-b7812000 r-xp 00000000 08:11 3424650 /lib/tls/i686/cmov/libnsl-2.11.1.so
b7812000-b7813000 r-xp 00012000 08:11 3424650 /lib/tls/i686/cmov/libnsl-2.11.1.so
b7813000-b7814000 rwxp 00013000 08:11 3424650 /lib/tls/i686/cmov/libnsl-2.11.1.so
b7814000-b7816000 rwxp 00000000 00:00 0
b781a000-b781b000 r-xp 00000000 08:11 1859586 /usr/lib/locale/en_US.utf8/LC_NUMERIC
b781b000-b781c000 r-xp 00000000 08:11 5712754 /usr/lib/locale/en_US.utf8/LC_TIME
b781c000-b781d000 r-xp 00000000 08:11 5712755 /usr/lib/locale/en_US.utf8/LC_MONETARY
b781d000-b781e000 r-xp 00000000 08:11 5712756 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
b781e000-b781f000 r-xp 00000000 08:11 1859591 /usr/lib/locale/en_US.utf8/LC_PAPER
b781f000-b7820000 r-xp 00000000 08:11 1859592 /usr/lib/locale/en_US.utf8/LC_NAME
b7820000-b7821000 r-xp 00000000 08:11 5712757 /usr/lib/locale/en_US.utf8/LC_ADDRESS
b7821000-b7822000 r-xp 00000000 08:11 5712758 /usr/lib/locale/en_US.utf8/LC_TELEPHONE
b7822000-b7823000 r-xp 00000000 08:11 1859595 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT
b7823000-b782a000 r-xs 00000000 08:11 5711192 /usr/lib/gconv/gconv-modules.cache
b782a000-b782b000 r-xp 00000000 08:11 5712759 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION
b782b000-b782d000 rwxp 00000000 00:00 0
b782d000-b782e000 r-xp 00000000 00:00 0 [vdso]
b782e000-b7849000 r-xp 00000000 08:11 3401485 /lib/ld-2.11.1.so
b7849000-b784a000 r-xp 0001a000 08:11 3401485 /lib/ld-2.11.1.so
b784a000-b784b000 rwxp 0001b000 08:11 3401485 /lib/ld-2.11.1.so
bfd36000-bfd4b000 rwxp 00000000 00:00 0 [stack]
Aborted
Please help. I can't find a way to make this work. Setting a non-environment variable does not cause the crash. For example:
$Interpreter->Eval('set localvar 55')
THANKS!!!
First stage analysis: memory corruption on exit. I'd say that there's warring over who should free a block and how. Now we dig deeper…
It's possible that you're seeing the consequences of the fact that both Perl and Tcl are independently trying to work around the bugs misfeatures in the system implementation of environment variables. In particular, the C library's setenv() has a number of problems with memory leaks which make it rather more impractical than you might naïvely expect; I'd quite expect Perl to work around this, and I know that Tcl does too, but it does mean that only one language should set environment variables, and that almost certainly needs to be Perl in this case (on the basis of evidence you've presented).
Do you really need to set environment variables from your Tcl code? If you can just not do it, that'd be the easiest way forward. :-) Otherwise, we're talking about having a different build of Tcl (so that it asks the Perl side of things to do the memory management) or doing some magic to disconnect the Tcl side from the real environment. You might also want to report the problem upstream to ActiveState.
I'll keep digging and if I find the solution I'll let you know.
I reported this to ActiveState. Inline is my email and their response, but to sum up:
1. They didn't develop "Inline Tcl" for Perl nor did they test it.
2. It was developed in 2001 and no additions were made to it since then.
3. When it was developed (in 2001), it was marked as "alpha" and no additions were made since then.
Hi Mircea,
Inline::Tcl is not part of ActivePerl. It's not tested with ActivePerl, and it
does not build on our PPM build farm against ActivePerl:
http://ppm4.activestate.com/i686-linux/5.10/1000/R/RR/RRS/Inline-Tcl-0.09.d/log-20090614T112213.txt
Offhand, the error message looks like a compiler mismatch problem, but it could be
almost anything since Inline::Tcl hasn't been touched by the author since 2001,
and even then the release notes describe it as "alpha software". It doesn't seem
to like Linux very much any more:
http://matrix.cpantesters.org/?dist=Inline-Tcl%200.09;maxver=1
Hmm.. The author's home page is 404. You should probably find another way to do it.
You should not need this line: use lib "/opt/ActivePerl-5.10/lib"; unless you have a misconfiguration. You might want to nuke the install from orbit and start over.
Is there any particular reason why you are using ActivePerl? You should be able to simply install perl from source -- you're using a real OS, so I'd take advantage of it :)