Tomcat Service not starting with elastic APM java agent - tomcat9

Question

I am trying to start tomcat service in linux with elastic APM agent (java). But every time I start the service, getting these errors
SEVERE: ContainerBase.addChild: start:
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/mis-all-1.0.0]]
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1007)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:983)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:639)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1296)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2037)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.VerifyError: (class: javax/servlet/GenericServlet, method: init signature: (Ljavax/servlet/ServletConfig;)V) invokedynamic bytecode is not supported in this class file version
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2509)
at java.lang.Class.getDeclaredFields(Class.java:1819)
at org.apache.catalina.util.Introspection.getDeclaredFields(Introspection.java:106)
at org.apache.catalina.startup.WebAnnotationSet.loadFieldsAnnotation(WebAnnotationSet.java:270)
at org.apache.catalina.startup.WebAnnotationSet.loadApplicationServletAnnotations(WebAnnotationSet.java:138)
at org.apache.catalina.startup.WebAnnotationSet.loadApplicationAnnotations(WebAnnotationSet.java:70)
at org.apache.catalina.startup.ContextConfig.applicationAnnotationsConfig(ContextConfig.java:417)
at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:891)
at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:388)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5536)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
... 10 more
May 20, 2021 5:48:15 AM org.apache.catalina.startup.HostConfig deployDirectory
SEVERE: Error deploying web application directory []
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/mis-all-1.0.0]]
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1011)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:983)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:639)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1296)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2037)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:745)
I am using below commands to set Environmental variables for APM agent
export CATALINA_OPTS="$CATALINA_OPTS -javaagent:/app/elastic-apm-agent-1.23.0.jar"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.service_name=Stage-pricing"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.application_packages=org,com,com.qr.jadu"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.server_url=http://ip:8200"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.global_labels='env=Stage'"
and logback-classic version 1.0.6
Please suggest a solution or any steps I am missing, thanks in advance.
Update by kriegaex, copied from the OP's comment:
Server version: Apache Tomcat/7.0.106 Server
built: Sep 16 2020 08:33:41 UTC
Server number: 7.0.106.0 OS
OS: Linux OS Version: 4.14.35-2047.502.4.1.el7uek.x86_64
Architecture: amd64
JVM Version: 1.7.0_301-b09
JVM Vendor: Oracle Corporation APM
Server version: 7.12.1
APM Agent language and version: java, 1.23. Also apm agent supports 7u60+, 8u40+, 9, 10, 11.

Answer 1

Caused by: java.lang.VerifyError: (
class: javax/servlet/GenericServlet,
method: init
signature: (Ljavax/servlet/ServletConfig;)V)
invokedynamic bytecode is not supported in this class file version
Piotr is right. Your Elastic APM Agent - whatever that thing does - transforms the byte code of void GenericServlet.init(ServletConfig) in such a way as to utilise invokedynamic byte code instructions, not checking or expecting that the code it weaves into is from an ancient Java version. Invokedynamic was introduced in Java 7, therefore your JAR must contain older versions. Either exclude it from transformation, if the agent can be configured that way, or upgrade the JAR, either using a more recent version or recompiling the API sources to a more recent (Java 7+) byte code version. Most agents probably rather expect Java 8+, but I am not sure about yours.