Matlab crashes on library initialize when called from Java - matlab

Question

The setup I have is I'm using a Java application to call native C-code with JNI, which in turn starts up the MATLAB runtime and calls functions on it (I know there are other solutions to call MATLAB methods from Java).
The problem is that the MATLAB engine crashes at some point during the initialization and I don't know what's causing it exactly. The crash causes my jvm to terminate, I assume it's some kind of memory corruption.
The C++ code calling MATLAB functions that is actually crashing is
JNIEXPORT void JNICALL
some_jni_vodoo_initializeLibrary(JNIEnv* env, jclass thisClass) {
try {
if (!mclInitializeApplication(NULL, 0)) {
THROW_EXCEPTION(env, "Could not initialize the application properly.");
return;
}
if (!<library>Initialize()) {
THROW_EXCEPTION(env, "Could not initialize the library.");
return;
}
} ...
The function <library>Initialize() crashes here, the Java error log reads
Stack Trace:
[0] jmi.dll:0x793f4175(0x7934cdca, 1, 0x7937e67c "à;.y`E.y#E.ychar in C:\BUILD_ARE..", 0x792d6a32)
[1] jvm.dll:0x792df9a5(0xc0000005, 0x79356791, 0x4961b400 "Ð\8y", 0x6d8b29de)
[2] jvm.dll:0x792e0431(0x8b515008, 0x70f0e8ce, 0x8b5ffffa, 0xc25d5ec6)
------------------------------------------------------------------------
Fatal Java Exception detected at Fri Apr 30 11:08:08 2010
------------------------------------------------------------------------
Configuration:
MATLAB Version: 7.8.0.347 (R2009a)
MATLAB License: unknown
Operating System: Microsoft Windows Vista
Window System: Version 6.0 (Build 6002: Service Pack 2)
Processor ID: x86 Family 6 Model 10 Stepping 5, GenuineIntel
Virtual Machine: Java is not enabled
Default Encoding: windows-1252
Java is not enabled
I really have no idea what could be wrong. Is there not enough memory from the jvm? I guess the problem is somehow related to Java, since calling the JNI functions from a simple test C++ program works fine...
Thanks

Answer 1

Initial Ideas
The only similar thing I've ever seen is a silent exit from MATLAB due to an invalid license.
Another possibility is the nested Java environments. Since MATLAB expects its own Java environment, perhaps there is some problem here. Edit: See below!
Are you running the MATLAB Engine? Or are you calling separate MATLAB commands using some "system" or "exec" call? This might give a good clue.
Answer
The answer appears below in the comments. It is copied here for readability:
When nesting an invocation of MATLAB's engine in a process that has its own JVM, you must either tell MATLAB to run without a JVM by passing -nojvm to mclInitializeApplication or tell it which JVM to use by setting the global MATLAB_JAVA to the location of the JVM you wish to use, C:\Program Files\Java\jre6 for example.
The first option will disable any functionality in MATLAB that requires the JVM, so the second option is preferable.