I am running spring boot applications on WebSphere Liberty servers(version 16.0.0.4). There more than on war with springboot application run on single instance of liberty. The spring library version we have are
Spring Core - 4.3.2.RELEASE
org.springframework.cloud - Brixton.SR5
org.springframework.boot - 1.4.0.RELEASE
Below are the stack trace for dead lock
Could you please help us understand what could be potential issue?
Owns Monitor Lock on com/ibm/ws/classloading/internal/ThreadContextClassLoader#0x00000000C8FAA768
Waiting for Monitor Lock on com/ibm/ws/classloading/internal/AppClassLoader#0x00000000C4126BA8
at java/lang/ClassLoader.loadClass(ClassLoader.java:805(Compiled Code))
at com/ibm/ws/classloading/internal/UnifiedClassLoader.findClass(UnifiedClassLoader.java:116(Compiled Code))
at java/lang/ClassLoader.loadClassHelper(ClassLoader.java:846(Compiled Code))
at java/lang/ClassLoader.loadClass(ClassLoader.java:825(Compiled Code))
at com/ibm/ws/classloading/internal/UnifiedClassLoader.loadClass0(UnifiedClassLoader.java:108)
at com/ibm/ws/classloading/internal/UnifiedClassLoader$Delegation.loadClass(UnifiedClassLoader.java:79)
at com/ibm/ws/classloading/internal/UnifiedClassLoader.loadClass(UnifiedClassLoader.java:103)
at com/ibm/ws/classloading/internal/ThreadContextClassLoader.loadClass(ThreadContextClassLoader.java:112)
at java/lang/ClassLoader.loadClass(ClassLoader.java:805(Compiled Code))
at java/util/ResourceBundle$Control.newBundle(ResourceBundle.java:2666)
at java/util/ResourceBundle.loadBundle(ResourceBundle.java:1527(Compiled Code))
at java/util/ResourceBundle.findBundle(ResourceBundle.java:1491(Compiled Code))
at java/util/ResourceBundle.findBundle(ResourceBundle.java:1445(Compiled Code))
at java/util/ResourceBundle.getBundleImpl(ResourceBundle.java:1380(Compiled Code))
at java/util/ResourceBundle.getBundle(ResourceBundle.java:1093)
at org/apache/bval/jsr/DefaultMessageInterpolator.loadBundle(DefaultMessageInterpolator.java:200)
at org/apache/bval/jsr/DefaultMessageInterpolator.getFileBasedResourceBundle(DefaultMessageInterpolator.java:177)
at org/apache/bval/jsr/DefaultMessageInterpolator.findUserResourceBundle(DefaultMessageInterpolator.java:283)
at org/apache/bval/jsr/DefaultMessageInterpolator.(DefaultMessageInterpolator.java:82)
at org/apache/bval/jsr/DefaultMessageInterpolator.(DefaultMessageInterpolator.java:69)
at org/apache/bval/jsr/ConfigurationImpl.(ConfigurationImpl.java:77)
at org/apache/bval/jsr/ApacheValidationProvider.createGenericConfiguration(ApacheValidationProvider.java:63)
at javax/validation/Validation$GenericBootstrapImpl.configure(Validation.java:164)
at org/springframework/boot/autoconfigure/BackgroundPreinitializer$ValidationInitializer.run(BackgroundPreinitializer.java:107)
at org/springframework/boot/autoconfigure/BackgroundPreinitializer$1.runSafely(BackgroundPreinitializer.java:59)
at org/springframework/boot/autoconfigure/BackgroundPreinitializer$1.run(BackgroundPreinitializer.java:52)
at java/lang/Thread.run(Thread.java:785)
------------------------------------------
Owns Monitor Lock on com/ibm/ws/classloading/internal/AppClassLoader#0x00000000C4126BA8
Waiting for Monitor Lock on com/ibm/ws/classloading/internal/ThreadContextClassLoader#0x00000000C8FAA768
at java/lang/ClassLoader.loadClass(ClassLoader.java:824(Compiled Code))
at com/ibm/ws/classloading/internal/UnifiedClassLoader.loadClass0(UnifiedClassLoader.java:108)
at com/ibm/ws/classloading/internal/UnifiedClassLoader$Delegation.loadClass(UnifiedClassLoader.java:79)
at com/ibm/ws/classloading/internal/UnifiedClassLoader.loadClass(UnifiedClassLoader.java:103)
at com/ibm/ws/classloading/internal/ThreadContextClassLoader.loadClass(ThreadContextClassLoader.java:112)
at java/lang/ClassLoader.loadClass(ClassLoader.java:805(Compiled Code))
at org/apache/logging/log4j/util/LoaderUtil.loadClass(LoaderUtil.java:120)
at org/apache/logging/slf4j/Log4jLogger.createConverter(Log4jLogger.java:416)
at org/apache/logging/slf4j/Log4jLogger.(Log4jLogger.java:54)
at org/apache/logging/slf4j/Log4jLoggerFactory.newLogger(Log4jLoggerFactory.java:37)
at org/apache/logging/slf4j/Log4jLoggerFactory.newLogger(Log4jLoggerFactory.java:29)
at org/apache/logging/log4j/spi/AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org/apache/logging/slf4j/Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org/slf4j/LoggerFactory.getLogger(LoggerFactory.java:358)
at org/apache/commons/logging/impl/SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
at org/apache/commons/logging/impl/SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org/apache/commons/logging/LogFactory.getLog(LogFactory.java:685)
at org/springframework/cloud/bootstrap/config/PropertySourceBootstrapConfiguration.(PropertySourceBootstrapConfiguration.java:63)
at java/lang/Class.forNameImpl(Native Method)
at java/lang/Class.forName(Class.java:348(Compiled Code))
at org/springframework/cglib/core/ReflectUtils.defineClass(ReflectUtils.java:415)
at org/springframework/cglib/core/AbstractClassGenerator.generate(AbstractClassGenerator.java:336)
at org/springframework/cglib/proxy/Enhancer.generate(Enhancer.java:492)
at org/springframework/cglib/core/AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:93)
at org/springframework/cglib/core/AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:91)
at org/springframework/cglib/core/internal/LoadingCache$2.call(LoadingCache.java:54)
at java/util/concurrent/FutureTask.run(FutureTask.java:277(Compiled Code))
at org/springframework/cglib/core/internal/LoadingCache.createEntry(LoadingCache.java:61)
at org/springframework/cglib/core/internal/LoadingCache.get(LoadingCache.java:34)
at org/springframework/cglib/core/AbstractClassGenerator$ClassLoaderData.get(AbstractClassGenerator.java:116)
at org/springframework/cglib/core/AbstractClassGenerator.create(AbstractClassGenerator.java:291)
at org/springframework/cglib/proxy/Enhancer.createHelper(Enhancer.java:480)
at org/springframework/cglib/proxy/Enhancer.createClass(Enhancer.java:337)
at org/springframework/context/annotation/ConfigurationClassEnhancer.createClass(ConfigurationClassEnhancer.java:135)
at org/springframework/context/annotation/ConfigurationClassEnhancer.enhance(ConfigurationClassEnhancer.java:107)
at org/springframework/context/annotation/ConfigurationClassPostProcessor.enhanceConfigurationClasses(ConfigurationClassPostProcessor.java:406)
at org/springframework/context/annotation/ConfigurationClassPostProcessor.postProcessBeanFactory(ConfigurationClassPostProcessor.java:263)
at org/springframework/context/support/PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:284)
at org/springframework/context/support/PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:130)
at org/springframework/context/support/AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:681)
at org/springframework/context/support/AbstractApplicationContext.refresh(AbstractApplicationContext.java:523)
at org/springframework/boot/SpringApplication.refresh(SpringApplication.java:759)
at org/springframework/boot/SpringApplication.refreshContext(SpringApplication.java:369)
at org/springframework/boot/SpringApplication.run(SpringApplication.java:313)
at org/springframework/boot/builder/SpringApplicationBuilder.run(SpringApplicationBuilder.java:134)
at org/springframework/cloud/bootstrap/BootstrapApplicationListener.bootstrapServiceContext(BootstrapApplicationListener.java:140)
at org/springframework/cloud/bootstrap/BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:84)
at org/springframework/cloud/bootstrap/BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:62)
at org/springframework/context/event/SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:166)
at org/springframework/context/event/SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:138)
at org/springframework/context/event/SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:121)
at org/springframework/boot/context/event/EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:68)
at org/springframework/boot/SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54)
at org/springframework/boot/SpringApplication.prepareEnvironment(SpringApplication.java:335)
at org/springframework/boot/SpringApplication.run(SpringApplication.java:307)
at org/springframework/boot/web/support/SpringBootServletInitializer.run(SpringBootServletInitializer.java:150)
at org/springframework/boot/web/support/SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:130)
at org/springframework/boot/web/support/SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:86)
at org/springframework/web/SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:169)
at com/ibm/ws/webcontainer/webapp/WebApp.initializeServletContainerInitializers(WebApp.java:2523)
at com/ibm/ws/webcontainer/webapp/WebApp.initialize(WebApp.java:1054)
at com/ibm/ws/webcontainer/webapp/WebApp.initialize(WebApp.java:6562)
at com/ibm/ws/webcontainer/osgi/DynamicVirtualHost.startWebApp(DynamicVirtualHost.java:468)
at com/ibm/ws/webcontainer/osgi/DynamicVirtualHost.startWebApplication(DynamicVirtualHost.java:463)
at com/ibm/ws/webcontainer/osgi/WebContainer.startWebApplication(WebContainer.java:1004)
at com/ibm/ws/webcontainer/osgi/WebContainer.startModule(WebContainer.java:825)
at com/ibm/ws/app/manager/module/internal/ModuleHandlerBase.deployModule(ModuleHandlerBase.java:100)
at com/ibm/ws/app/manager/module/internal/DeployedModuleInfoImpl.installModule(DeployedModuleInfoImpl.java:50)
at com/ibm/ws/app/manager/module/internal/DeployedAppInfoBase.deployModules(DeployedAppInfoBase.java:420)
at com/ibm/ws/app/manager/module/internal/DeployedAppInfoBase.deployApp(DeployedAppInfoBase.java:406)
at com/ibm/ws/app/manager/war/internal/WARApplicationHandlerImpl.install(WARApplicationHandlerImpl.java:66)
at com/ibm/ws/app/manager/internal/statemachine/StartAction.execute(StartAction.java:141)
at com/ibm/ws/app/manager/internal/statemachine/ApplicationStateMachineImpl.enterState(ApplicationStateMachineImpl.java:1191)
at com/ibm/ws/app/manager/internal/statemachine/ApplicationStateMachineImpl.run(ApplicationStateMachineImpl.java:804)
at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java/lang/Thread.run(Thread.java:785)
The two stack traces are very useful info — thanks.
The first stack trace makes sense to me. The ThreadContextClassLoader is delegating to the AppClassLoader, and this is the correct order of operation and therefore the correct order in which the locks should be obtained.
The second stack trace leaves me somewhat confused. I cannot see any entries in that stack that are obviously holding the AppClassLoader monitor. I have conferred with a colleague, and the most likely scenario is that the Class.forName() is being passed the AppClassLoader and that the native Class.forNameImpl() method is holding the monitor for that loader. This only becomes a problem when the class being initialised (PropertySourceBootstrapConfiguration) reaches out through apache logging to initialise another class, which apache logging loads using the loader obtained from the thread context.
If Springboot were to use the class loader from the thread context instead of the loader obtained from Class.getClassLoader() this might prevent this problem from happening. Springboot can't assume it is running in a container, but it could at least look for a thread context class loader first, before trying Class.getClassLoader().
One possible way to avoid this deadlock would be to separate out the Springboot classes into their own library, and reference this as a common library from the application configuration. Take a look at steps 1–4 in this Knowledge Center article. This should mean that the second thread's stack locks the library class loader rather than the application class loader. This will not guarantee avoiding the deadlock because the ordering is still unexpected, but it might make it unlikely to occur (depending on what the first thread in this scenario is loading and why it coincides with the other thread).
Related
On an AUTOSAR realtime Operating System (OS), the software architecture is layered separately (User space, systemcall interface, kernel space). Also, the switching between user context and kernel context is handled by hardware-specific infrastructure and typically the context switching handler is written in assembly code.
IBM® Rational® Test RealTime v8.0.1 (RTRT) currently treats embedded-assembly-code as mentioned in the below Q&A.
https://www.ibm.com/support/pages/how-treat-embedded-assembly-code ( ** )
RTRT tool using code insertion technololy (technically known as instrumentation process) to insert its own code to measure code coverage of the system under test.
In my case, OS with full pre-emptive design doesn't have the termination points. As the result, OS always runs unless loss of power supply. If there's no work, OS shall be in sleep (normally an idle state and do nothing). If any unexpected errors or exceptions occurs, OS shall be shutdown and run into an infinite loop. These indicated that OS is always running.
I learnt from ( ** ) and ensure context switching working correctly.
But I don't know how to teach RTRT to finish its postprocessing (consisting of attolcov and attolpostpro) in a right way. Note that OS has worked correctly throughout all my tasks already and was confirmed by debugger. SHUTDOWN OS procedure has been executed correctly and OS has been in INFINITE loop (such as while(1){};)
After RTRT ends all its processes, the coverage report of OS module is still empty.
Based on IBM guideline for RTRT
https://www.ibm.com/developerworks/community/forums/atom/download/attachment_14076432_RTRT_User_Guide.pdf?nodeId=de3b0048-968c-4111-897e-b73654af32af
RTRT provides two breakpoints to mark the logging point (priv_writeln) and termination point (priv_close) of its process.
I already tried to drive from INFINITE (my OS) to priv_close (RTRT) by interacting PC register and all Context Switching registers with the Lauterbach debugger but RTRT coverage report was empty even thougth none of errors happened. No error meant that the context switch from kernel space to user space is able to work well and main() function returned correctly.
Solved the problem.
It definitely came from context switching process of Operating System.
In my case, I did a RAM dump to see how user context memory (befor Starting OS) look like.
After that, I will backup all context areas and retore them from Sleep or Infiniteloop in the exact order.
Hereby, RTRT can know the return point and reach its own main() function's _exit to finish report generation progress.
Finally, the code coverage report has been generated.
When my connection is open, the application won't exit, this causes some nasty problems for me (highly concurrent and nested using a shared sesssion, don't know when each part is finished) - is there a way to make sure that the cluster doesn't "hang" the application?
For example here:
object ZombieTest extends App {
val session= Cluster.builder().addContactPoint("localhost").build().connect()
// app doesn't exit unless doing:
session.getCluster.close() // won't exit unless this is called
}
In a slightly biased answer, you could look at https://github.com/outworkers/phantom instead of using the standard java driver.
You get scala.concurrent.Future, monix.eval.Task or even com.twitter.util.Future from a query automatically. You can choose between all three.
DB connection pools are better isolated inside ContactPoint and Database abstraction layers, which have shutdown methods you can easily wire in to your app lifecycle.
It's far faster than the Java driver, as the serialization an de-serialisation of types is wired in compile time via more advanced macro mechanisms.
The short answer is that you need to have a lifecycle way of calling session.close or session.closeAsync when you shut down everything else, it's how it's designed to work.
Quoting the following paragraph from Operating Systems: Three Easy Pieces,
Note that there are two types of register saves/restores that happen
during this protocol. The first is when the timer interrupt occurs; in
this case, the user registers of the running process are implicitly
saved by the hardware, using the kernel stack of that process. The
second is when the OS decides to switch from A to B; in this case, the
kernel registers are explicitly saved by the software (i.e., the OS),
but this time into memory in the process structure of the process.
Reading other literature on context switch I understand that timer interrupt throws the cpu into kernel mode which then saves the process context into kernel stack.
Why is the author talking about a multiple context save emphasising on hardware/software?
The author emphasizes on hardware/software part of it because basically its context saving that is being done , sometimes by hardware and sometimes by software.
When a timer interrupt occurs, the user registers are saved by hardware(meaning saved by the CPU itself) on the kernel stack of that process. When the interrupt handler code is finished, the user registers will be restored using the kernel stack of that process,thereby restoring user stack and process successfully returns to user mode from kernel mode.
In case of a context switch from process A to process B, the very kernel stacks of the two processes A and B are switched,inside the kernel, which indirectly means saving and restoring of kernel registers. The term Software is used because the scheduler process after selecting which process to run next calls a function(thats why software), that does the switching of kernel stacks. The context switch code need not worry about user register values - those are already safely saved away in the kernel stack by that point.
The first is when the timer interrupt occurs; in this case, the user registers of the running process are implicitly saved by the hardware, using the kernel stack of that process.
Often, only SOME of the registers are saved and this is usually to an interrupt stack.
The second is when the OS decides to switch from A to B; in this case, the kernel registers are explicitly saved by the software (i.e., the OS), but this time into memory in the process structure of the process.
Usually this switch occurs in HARDWARE through a special instruction. Maybe they are referring to the fact that the switch is triggered through software as opposed to an interrupt that is triggered by hardware.
Also thanks for that reference. I have just started to go through it. It MUCH better than most of the OS books that only serve to confuse.
Considering any RTOS source code there are couple of files that are specific to 1-Target 2-Compiler. There can be a minimal dependency of the word-size of target controller, interrupt calling the kernel tick function etc. Are there any other dependencies...? Thanks in advance
While working with Micrium and FreeRTOS, I found that since context switching is performed many times between tasks and ISR, it is a general practice to write the best optimized code for storing CPU registers onto task's stack while switching context. To get the best optimization, the CPU register handling is generally written in assembly language. And this is one reason for compiler (toolchain) dependency of RTOS.
I'm currently researching threads in the context of the operating system and I'm unsure if a thread is a set sequence of instructions that can be repeatedly executed or if it is filled and replaced with new instructions by the user or the operating system.
Thanks a bundle!
-Tom
I'm not quite sure what you mean - the compiled instructions for a program are stored in memory and are not changed at runtime (at least for languages which are not JIT-compiled).
A thread is an entirely separate concept from the code itself. A thread gives you the ability to be running at "two places at once" in the code. At a conceptual level, a thread is simply a container for the context that you need at any point in the execution of some code. This means that each thread has a call stack and a set of registers (which are either actually stored in the registers of a processor if the thread is running, or elsewhere if the thread is paused).
Almost all thread libraries work such that a new thread will execute some user-defined function and will then exit. This function can be long-running, just like main() (which is the function executed by the first thread in your process).
If the threads are supported by the OS (ie they are not "green threads"/"fibers") they will exit by calling an OS API which tells the OS it can deallocate any data it has which is associated with that thread.
Sometimes, abstractions are built on top of this mechanism such that a thread or pool of threads will execute a function which simply loops over a queue of tasks to run, but the fundamental mechanism is the same. However, these abstractions are provided by user libraries built on top of the OS threading mechanisms, not by the OS itself.