JVM Crash due to `EXCEPTION_ACCESS_VIOLATION` in org.infinispan.util.concurrent.jdk8backported.LongAdder - jboss

Question

This is happening on JBoss EAP 6.1, for single user it works fine, But as soon as 2 or 3 concurrent user start interacting with the application, JVM gets crashed.
The JVM is crashing with EXCEPTION_ACCESS_VIOLATION; the full trace is below:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x024c8266, pid=5136, tid=8736
#
# JRE version: 6.0_45-b06
# Java VM: Java HotSpot(TM) Client VM (20.45-b01 mixed mode windows-x86 )
# Problematic frame:
# J org.infinispan.util.concurrent.jdk8backported.LongAdder.add(J)V
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x6602dc00): JavaThread "http-ds-7071bc90200f..corp.in/10.112.70.75:8080-6" daemon [_thread_in_Java, id=8736, stack(0x663a0000,0x663f0000)]
siginfo: ExceptionCode=0xc0000005, reading address 0x8ddb6987
Registers:
EAX=0x00000002, EBX=0xffffffff, ECX=0x00000001, EDX=0x0ddb6988
ESP=0x663ee3c0, EBP=0x663eea58, ESI=0x0ddb60a0, EDI=0xffffffff
EIP=0x024c8266, EFLAGS=0x00010293
Top of Stack: (sp=0x663ee3c0)
0x663ee3c0: 0ddb4830 00000000 663eea58 024d2b44
0x663ee3d0: 0ddb1610 0ddb1610 663eea58 024d7368
0x663ee3e0: 0ddb4830 00000000 0ddb1610 0ddb6be0
0x663ee3f0: 22239910 0ddb1610 2223aff0 22239910
0x663ee400: 0ddb60a0 2223c338 2127c660 24bb6ca8
0x663ee410: 24c1f6e0 0ddb6070 00000002 0ddb6988
0x663ee420: 4ac892b3 0ddb1610 663eea58 024d5bf4
0x663ee430: 00000000 00000000 663eea58 024c5a0c
Instructions: (pc=0x024c8266)
0x024c8246: 0c 00 00 00 b8 01 00 00 00 8b f7 e9 5b 00 00 00
0x024c8256: 8b bc 24 80 00 00 00 8b 9c 24 84 00 00 00 3b 02
0x024c8266: f2 0f 10 82 ff ff ff 7f 66 0f 7e c6 66 0f 73 d0
0x024c8276: 20 66 0f 7e c0 8b d6 8b c8 03 d7 13 cb 89 34 24
Register to memory mapping:
EAX=0x00000002 is an unknown value
EBX=0xffffffff is an unknown value
ECX=0x00000001 is an unknown value
EDX=
[error occurred during error reporting (printing register info), id 0xc0000005]
Stack: [0x663a0000,0x663f0000], sp=0x663ee3c0, free space=312k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J org.infinispan.util.concurrent.jdk8backported.LongAdder.add(J)V
j org.infinispan.CacheImpl.put(Ljava/lang/Object;Ljava/lang/Object;JLjava/util/concurrent/TimeUnit;JLjava/util/concurrent/TimeUnit;Ljava/util/EnumSet;Ljava/lang/ClassLoader;)Ljava/lang/Object;+24
j org.infinispan.CacheImpl.put(Ljava/lang/Object;Ljava/lang/Object;JLjava/util/concurrent/TimeUnit;JLjava/util/concurrent/TimeUnit;)Ljava/lang/Object;+12
j org.infinispan.CacheSupport.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+17
j org.infinispan.AbstractDelegatingCache.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+6
j org.infinispan.spring.provider.SpringCache.put(Ljava/lang/Object;Ljava/lang/Object;)V+6
j com.hmtp.security.server.util.SessionTokenValidator.refreshAccessTime(Ljava/lang/String;)V+25
j com.hmtp.security.server.util.SessionTokenValidator.checkAndRenewToken([Ljava/lang/String;)Lorg/springframework/security/web/authentication/rememberme/PersistentRememberMeToken;+174
j com.hmtp.security.server.auth.MultiTenantRememberMeServices.processAutoLoginCookie([Ljava/lang/String;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)Lorg/springframework/security/core/userdetails/UserDetails;+5
j org.springframework.security.web.authentication.rememberme.AbstractRememberMeServices.autoLogin(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)Lorg/springframework/security/core/Authentication;+64
j com.hmtp.common.security.server.BrownstoneRememberMeAuthenticationFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+20
J org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
j org.springframework.security.web.FilterChainProxy.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+40
j org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(Ljavax/servlet/Filter;Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+5
j org.springframework.web.filter.DelegatingFilterProxy.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+71
J org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
J org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
j org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+200
j org.jboss.as.web.session.ClusteredSessionValve.handleRequest(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;Lorg/jboss/servlet/http/HttpEvent;Z)V+61
j org.jboss.as.web.session.ClusteredSessionValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+26
j org.jboss.as.web.session.JvmRouteValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+33
j org.jboss.as.web.session.LockingValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+43
j org.jboss.as.web.security.SecurityContextAssociationValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+402
J org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
j org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V+42
j org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V+188
j org.apache.coyote.http11.Http11Processor.process(Ljava/net/Socket;)Lorg/apache/tomcat/util/net/JIoEndpoint$Handler$SocketState;+349
j org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Ljava/net/Socket;)Lorg/apache/tomcat/util/net/JIoEndpoint$Handler$SocketState;+65
j org.apache.tomcat.util.net.JIoEndpoint$Worker.run()V+128
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [jvm.dll+0xfb88b]
V [jvm.dll+0x18d551]
V [jvm.dll+0xfba31]
V [jvm.dll+0xfba8b]
V [jvm.dll+0xb5e89]
V [jvm.dll+0x119b74]
V [jvm.dll+0x14217c]
C [msvcr71.dll+0x9565] endthreadex+0xa0
C [kernel32.dll+0x4ee6c] BaseThreadInitThunk+0x12
C [ntdll.dll+0x6399b] RtlInitializeExceptionChain+0xef
C [ntdll.dll+0x6396e] RtlInitializeExceptionChain+0xc2
VM state:synchronizing (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00dd8360] Safepoint_lock - owner thread: 0x01a68c00
[0x00dd83c8] Threads_lock - owner thread: 0x01a68c00
Heap
def new generation total 400384K, used 175418K [0x03ae0000, 0x1ed50000, 0x1ed80000)
eden space 355904K, 47% used [0x03ae0000, 0x0df2c4f0, 0x19670000)
from space 44480K, 16% used [0x1c1e0000, 0x1c8e2540, 0x1ed50000)
to space 44480K, 0% used [0x19670000, 0x19670000, 0x1c1e0000)
tenured generation total 889536K, used 111605K [0x1ed80000, 0x55230000, 0x552e0000)
the space 889536K, 12% used [0x1ed80000, 0x25a7d7f0, 0x25a7d800, 0x55230000)
compacting perm gen total 102144K, used 102123K [0x552e0000, 0x5b6a0000, 0x652e0000)
the space 102144K, 99% used [0x552e0000, 0x5b69ad60, 0x5b69ae00, 0x5b6a0000)
No shared spaces configured.
Code Cache [0x01ae0000, 0x02cb8000, 0x03ae0000)
total_blobs=9369 nmethods=9146 adapters=163 free_code_cache=14876096 largest_free_block=384
VM Arguments:
jvm_args: -Dprogram.name=standalone.bat -Xms1303M -Xmx1303M -XX:MaxPermSize=256M -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n -Dorg.jboss.boot.log.file=C:\jboss-eap-6.1\standalone\log\server.log -Dlogging.configuration=file:C:\jboss-eap-6.1\standalone/configuration/logging.properties
java_command: C:\jboss-eap-6.1\jboss-modules.jar -mp C:\jboss-eap-6.1\modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=C:\jboss-eap-6.1
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=C:\Program Files\Java\jdk1.6.0_45
PATH=C:\ProgramData\Oracle\Java\javapath;C:\Program Files\Java\jdk1.6.0_45\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\apache-maven-3.2.2\bin;C:\Program Files\TortoiseSVN\bin;D:\apache-ant-1.9.4\bin
USERNAME=neeraj.ar
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 10, GenuineIntel
--------------- S Y S T E M ---------------
OS: Windows 7 Build 7601 Service Pack 1
CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 23 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1
Memory: 4k page, physical 3338764k(929876k free), swap 15625020k(11901304k free)
vm_info: Java HotSpot(TM) Client VM (20.45-b01) for windows-x86 JRE (1.6.0_45-b06), built on Mar 26 2013 13:40:03 by "java_re" with MS VC++ 7.1 (VS2003)
time: Thu Jul 23 18:41:10 2015
elapsed time: 256 seconds

Answer 1

The crash happens in C1-compiled code on the instruction
f20f1082ffffff7f movsd xmm0, qword ptr [edx+0x7fffffff]
which represents a volatile long load that has not been patched.
I believe this is a JVM bug JDK-6965570 or its duplicate JDK-7004258.
It has been fixed a long ago in JDK 6u60, but you seem to use very old version of JDK.