First of all, i have two environments strictly identically configured (exept IP) with two vm each. One in pre-production and one in production (currently in configuration phase). There is one vm with a liferay 7.0.6 tomcat bundle (build from 7.0.6-cumulative patch of the community-security-team) and an other with postgresql 9.4.26.
Everything works fine on pre-production environment.
On production environment, a few hours after beginning to create users in liferay i ran into this error (full stack at the end) :
Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 - Connection is not available, request timed out after 937980ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:591)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:194)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy7.prepareStatement(Unknown Source)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:452)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1700)
at org.hibernate.loader.Loader.doQuery(Loader.java:801)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.loadEntity(Loader.java:2037)
... 50 more
So i checked if there is differences between my liferay configuration on pre-production and the one in production using comparison software, and exept IP i found nothing. Idem with postgresql configurations on both environment.
I also checked time synchronization between vm and they are both synchronize via ntp on debian pool server.
Database vm :
n$ ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
0.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.002
1.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.002
2.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.002
3.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.002
+mail.klausen.dk 193.67.79.202 2 u 116 1024 377 14.435 -0.214 0.358
+any.time.nl 85.199.214.99 2 u 871 1024 377 1.666 -0.183 0.258
-rag.9t4.net 131.188.3.221 2 u 102 1024 377 16.491 -3.769 0.571
*ntp1.m-online.n 212.18.1.106 2 u 318 1024 377 16.608 -0.263 0.240
-tor-relais1.lin 131.188.3.223 2 u 199 1024 377 14.149 0.272 0.661
-www.kashra.com .DCFp. 1 u 150 1024 377 22.623 1.126 0.816
and Liferay vm :
$ ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
0.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.002
1.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.002
2.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.002
3.debian.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.002
+stratum2-4.ntp. 129.70.137.82 2 u 200 1024 377 30.471 1.069 2.414
+138.201.16.225 131.188.3.221 2 u 696 1024 377 16.613 1.357 0.397
-kuehlich.com 131.188.3.221 2 u 373 1024 377 22.656 2.025 0.885
-time.cloudflare 10.21.8.19 3 u 566 1024 377 8.123 -0.640 0.277
*a.chl.la 131.188.3.222 2 u 167 1024 377 14.472 1.033 2.448
+195.50.171.101 145.253.3.52 2 u 266 1024 377 10.804 0.928 0.395
I also notice an error line in postgresql log about a request of process cancelation on unknown PID happening exactly the amout (937980ms) of milliseconds before the timeout error in Liferay:
LOG: PID 1767 in the cancel request does not match any process
I have tried re-installing Liferay from scratch but nothing change.
It should exist a difference between pre-production and production because it works fine on pre-production but i can't find it.
HikariCP configuration in liferay is default on both environment
jdbc.default.connectionTimeout=30000
jdbc.default.driverClassName=org.postgresql.Driver
jdbc.default.idleConnectionTestPeriod=60
jdbc.default.idleTimeout=600000
jdbc.default.initialPoolSize=10
jdbc.default.liferay.pool.provider=hikaricp
jdbc.default.maxActive=100
jdbc.default.maxIdleTime=3600
jdbc.default.maxLifetime=0
jdbc.default.maxPoolSize=100
jdbc.default.maximumPoolSize=100
jdbc.default.minIdle=10
jdbc.default.minPoolSize=10
jdbc.default.minimumIdle=10
, and same for postgresql :
max_connections = 100
HikariPool full satck from liferay :
2021-06-30 00:12:32.397 ERROR [liferay/scheduler_dispatch-6][JDBCExceptionReporter:234] HikariPool-2 - Connection is not available, request timed out after 937980ms.
2021-06-30 00:12:32.401 ERROR [liferay/scheduler_dispatch-6][BasePersistenceImpl:264] Caught unexpected exception
com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.GenericJDBCException: could not load an entity: [com.liferay.counter.model.impl.CounterImpl#com.liferay.counter.kernel.model.Counter]
at com.liferay.portal.kernel.service.persistence.impl.BasePersistenceImpl.processException(BasePersistenceImpl.java:270)
at com.liferay.counter.service.persistence.impl.CounterFinderImpl._obtainIncrement(CounterFinderImpl.java:391)
at com.liferay.counter.service.persistence.impl.CounterFinderImpl._competeIncrement(CounterFinderImpl.java:339)
at com.liferay.counter.service.persistence.impl.CounterFinderImpl._competeIncrement(CounterFinderImpl.java:325)
at com.liferay.counter.service.persistence.impl.CounterFinderImpl.increment(CounterFinderImpl.java:111)
at com.liferay.counter.service.persistence.impl.CounterFinderImpl.increment(CounterFinderImpl.java:100)
at com.liferay.counter.service.persistence.impl.CounterFinderImpl.increment(CounterFinderImpl.java:95)
at com.liferay.counter.service.impl.CounterLocalServiceImpl.increment(CounterLocalServiceImpl.java:42)
at sun.reflect.GeneratedMethodAccessor638.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:163)
at com.liferay.portal.spring.transaction.CounterTransactionExecutor.execute(CounterTransactionExecutor.java:50)
at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:58)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:137)
at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:169)
at com.sun.proxy.$Proxy78.increment(Unknown Source)
at com.liferay.counter.kernel.service.CounterLocalServiceUtil.increment(CounterLocalServiceUtil.java:238)
at com.liferay.portal.kernel.systemevent.SystemEventHierarchyEntryThreadLocal.push(SystemEventHierarchyEntryThreadLocal.java:134)
at com.liferay.portal.kernel.systemevent.SystemEventHierarchyEntryThreadLocal.push(SystemEventHierarchyEntryThreadLocal.java:96)
at com.liferay.portal.repository.capabilities.TemporaryFileEntriesCapabilityImpl._runWithoutSystemEvents(TemporaryFileEntriesCapabilityImpl.java:313)
at com.liferay.portal.repository.capabilities.TemporaryFileEntriesCapabilityImpl.deleteExpiredTemporaryFileEntries(TemporaryFileEntriesCapabilityImpl.java:113)
at com.liferay.document.library.web.internal.messaging.TempFileEntriesMessageListener.deleteExpiredTemporaryFileEntries(TempFileEntriesMessageListener.java:111)
at com.liferay.document.library.web.internal.messaging.TempFileEntriesMessageListener$1.performAction(TempFileEntriesMessageListener.java:134)
at com.liferay.document.library.web.internal.messaging.TempFileEntriesMessageListener$1.performAction(TempFileEntriesMessageListener.java:130)
at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery.performAction(DefaultActionableDynamicQuery.java:405)
at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery$1.call(DefaultActionableDynamicQuery.java:315)
at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery$1.call(DefaultActionableDynamicQuery.java:277)
at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery.doPerformActions(DefaultActionableDynamicQuery.java:335)
at com.liferay.portal.kernel.dao.orm.DefaultActionableDynamicQuery.performActions(DefaultActionableDynamicQuery.java:86)
at com.liferay.document.library.web.internal.messaging.TempFileEntriesMessageListener.doReceive(TempFileEntriesMessageListener.java:139)
at com.liferay.portal.kernel.messaging.BaseMessageListener.receive(BaseMessageListener.java:26)
at com.liferay.portal.kernel.scheduler.messaging.SchedulerEventMessageListenerWrapper.receive(SchedulerEventMessageListenerWrapper.java:66)
at com.liferay.portal.kernel.messaging.InvokerMessageListener.receive(InvokerMessageListener.java:74)
at com.liferay.portal.kernel.messaging.ParallelDestination$1.run(ParallelDestination.java:52)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:756)
at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:667)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.GenericJDBCException: could not load an entity: [com.liferay.counter.model.impl.CounterImpl#com.liferay.counter.kernel.model.Counter]
at com.liferay.portal.dao.orm.hibernate.ExceptionTranslator.translate(ExceptionTranslator.java:34)
at com.liferay.portal.dao.orm.hibernate.SessionImpl.get(SessionImpl.java:205)
at com.liferay.portal.kernel.dao.orm.ClassLoaderSession.get(ClassLoaderSession.java:326)
at com.liferay.counter.service.persistence.impl.CounterFinderImpl._obtainIncrement(CounterFinderImpl.java:369)
... 36 more
Caused by: org.hibernate.exception.GenericJDBCException: could not load an entity: [com.liferay.counter.model.impl.CounterImpl#com.liferay.counter.kernel.model.Counter]
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.loadEntity(Loader.java:2041)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:86)
at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:76)
at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3294)
at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:496)
at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:477)
at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:227)
at org.hibernate.event.def.DefaultLoadEventListener.lockAndLoad(DefaultLoadEventListener.java:403)
at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:155)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:1090)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:1075)
at org.hibernate.impl.SessionImpl.get(SessionImpl.java:1066)
at com.liferay.portal.dao.orm.hibernate.SessionImpl.get(SessionImpl.java:201)
... 38 more
Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 - Connection is not available, request timed out after 937980ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:591)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:194)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy7.prepareStatement(Unknown Source)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:452)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1700)
at org.hibernate.loader.Loader.doQuery(Loader.java:801)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.loadEntity(Loader.java:2037)
... 50 more
I'm actually out of other idea to test, any help would be thankfull.
Finally we found the solution.
On pre-production environment the two vms are on the same VLAN, which was not the case on production.
Solution: putting the vms on the same VLAN solve the problem.
I have updated scala and sbt version to 2.12.0 and 0.13.8 respectively. Now every build fails with errors:
java.lang.VerifyError: Uninitialized object exists on backward branch 209
Exception Details:
Location:
scala/collection/immutable/HashMap$HashTrieMap.split()Lscala/collection/immutable/Seq; #249: goto
Reason:
Error exists in the bytecode
Bytecode:
0000000: 2ab6 0057 04a0 001e b200 afb2 00b4 04bd
0000010: 0002 5903 2a53 c000 b6b6 00ba b600 bec0
0000020: 00c0 b02a b600 31b8 003b 3c1b 04a4 015e
0000030: 1b05 6c3d 2a1b 056c 2ab6 0031 b700 c23e
0000040: 2ab6 0031 021d 787e 3604 2ab6 0031 0210
0000050: 201d 647c 7e36 05bb 0014 59b2 00b4 2ab6
0000060: 0033 c000 b6b6 00c6 b700 c91c b600 cd3a
0000070: 0619 06c6 001a 1906 b600 d1c0 007d 3a07
0000080: 1906 b600 d4c0 007d 3a08 a700 0dbb 00d6
0000090: 5919 06b7 00d9 bf19 073a 0919 083a 0abb
00000a0: 0002 5915 0419 09bb 0014 59b2 00b4 1909
00000b0: c000 b6b6 00c6 b700 c903 b800 df3a 0e3a
00000c0: 0d03 190d b900 e301 0019 0e3a 1136 1036
00000d0: 0f15 0f15 109f 0027 150f 0460 1510 190d
00000e0: 150f b900 e602 00c0 0005 3a17 1911 1917
00000f0: b800 ea3a 1136 1036 0fa7 ffd8 1911 b800
0000100: eeb7 005c 3a0b bb00 0259 1505 190a bb00
0000110: 1459 b200 b419 0ac0 00b6 b600 c6b7 00c9
0000120: 03b8 00df 3a13 3a12 0319 12b9 00e3 0100
0000130: 1913 3a16 3615 3614 1514 1515 9f00 2715
0000140: 1404 6015 1519 1215 14b9 00e6 0200 c000
0000150: 053a 1819 1619 18b8 00f1 3a16 3615 3614
0000160: a7ff d819 16b8 00ee b700 5c3a 0cb2 00f6
0000170: b200 b405 bd00 0259 0319 0b53 5904 190c
0000180: 53c0 00b6 b600 bab6 00f9 b02a b600 3303
0000190: 32b6 00fb b0
Stackmap Table:
same_frame(#35)
full_frame(#141,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#105]},{})
append_frame(#151,Object[#125],Object[#125])
full_frame(#209,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#105],Object[#125],Object[#125],Object[#125],Object[#125],Top,Top,Object[#20],Object[#55],Integer,Integer,Object[#103]},{Uninitialized[#159],Uninitialized[#159],Integer,Object[#125]})
full_frame(#252,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#105],Object[#125],Object[#125],Object[#125],Object[#125],Top,Top,Object[#20],Object[#55],Integer,Integer,Object[#103]},{Uninitialized[#159],Uninitialized[#159],Integer,Object[#125]})
full_frame(#312,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#105],Object[#125],Object[#125],Object[#125],Object[#125],Object[#2],Top,Object[#20],Object[#55],Integer,Integer,Object[#103],Object[#20],Object[#55],Integer,Integer,Object[#103]},{Uninitialized[#262],Uninitialized[#262],Integer,Object[#125]})
full_frame(#355,{Object[#2],Integer,Integer,Integer,Integer,Integer,Object[#105],Object[#125],Object[#125],Object[#125],Object[#125],Object[#2],Top,Object[#20],Object[#55],Integer,Integer,Object[#103],Object[#20],Object[#55],Integer,Integer,Object[#103]},{Uninitialized[#262],Uninitialized[#262],Integer,Object[#125]})
full_frame(#395,{Object[#2],Integer},{})
at scala.collection.immutable.HashMap$.scala$collection$immutable$HashMap$$makeHashTrieMap(HashMap.scala:179)
at scala.collection.immutable.HashMap$HashMap1.updated0(HashMap.scala:211)
at scala.collection.immutable.HashMap.$plus(HashMap.scala:59)
at scala.collection.immutable.HashMap.$plus(HashMap.scala:62)
at scala.collection.immutable.Map$Map4.updated(Map.scala:201)
at scala.collection.immutable.Map$Map4.$plus(Map.scala:202)
at scala.collection.immutable.Map$Map4.$plus(Map.scala:180)
at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:29)
at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:25)
at scala.collection.generic.Growable.$anonfun$$plus$plus$eq$1(Growable.scala:59)
at scala.collection.generic.Growable$$Lambda$19/924699145.apply(Unknown Source)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:59)
at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:50)
at scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:25)
at scala.collection.generic.GenMapFactory.apply(GenMapFactory.scala:48)
at scala.sys.package$.env(package.scala:61)
at scala.tools.nsc.settings.ScalaSettings.defaultClasspath(ScalaSettings.scala:30)
at scala.tools.nsc.settings.ScalaSettings.defaultClasspath$(ScalaSettings.scala:30)
at scala.tools.nsc.settings.MutableSettings.defaultClasspath(MutableSettings.scala:19)
at scala.tools.nsc.settings.ScalaSettings.$init$(ScalaSettings.scala:60)
at scala.tools.nsc.settings.MutableSettings.<init>(MutableSettings.scala:20)
at scala.tools.nsc.Settings.<init>(Settings.scala:12)
at scala.tools.nsc.Driver.process(Driver.scala:41)
at scala.tools.nsc.Main.process(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at sbt.compiler.RawCompiler.apply(RawCompiler.scala:26)
at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1$$anonfun$apply$2.apply(AnalyzingCompiler.scala:146)
at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1$$anonfun$apply$2.apply(AnalyzingCompiler.scala:142)
at sbt.IO$.withTemporaryDirectory(IO.scala:291)
at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1.apply(AnalyzingCompiler.scala:142)
at sbt.compiler.AnalyzingCompiler$$anonfun$compileSources$1.apply(AnalyzingCompiler.scala:139)
at sbt.IO$.withTemporaryDirectory(IO.scala:291)
at sbt.compiler.AnalyzingCompiler$.compileSources(AnalyzingCompiler.scala:139)
at sbt.compiler.ComponentCompiler$$anonfun$compileAndInstall$1.apply(ComponentCompiler.scala:63)
at sbt.compiler.ComponentCompiler$$anonfun$compileAndInstall$1.apply(ComponentCompiler.scala:60)
at sbt.IO$.withTemporaryDirectory(IO.scala:291)
at sbt.compiler.ComponentCompiler.compileAndInstall(ComponentCompiler.scala:60)
at sbt.compiler.ComponentCompiler$$anonfun$getLocallyCompiled$1.apply$mcV$sp(ComponentCompiler.scala:50)
at sbt.IfMissing$Define.apply(ComponentManager.scala:75)
at sbt.ComponentManager.sbt$ComponentManager$$createAndCache$1(ComponentManager.scala:39)
at sbt.ComponentManager$$anonfun$sbt$ComponentManager$$fromGlobal$1$1.apply(ComponentManager.scala:27)
at sbt.ComponentManager$$anonfun$sbt$ComponentManager$$fromGlobal$1$1.apply(ComponentManager.scala:26)
at sbt.ComponentManager$$anon$1.call(ComponentManager.scala:50)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
at xsbt.boot.Using$.withResource(Using.scala:10)
at xsbt.boot.Using$.apply(Using.scala:9)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.ComponentManager.lock(ComponentManager.scala:50)
at sbt.ComponentManager.lockGlobalCache(ComponentManager.scala:49)
at sbt.ComponentManager.sbt$ComponentManager$$fromGlobal$1(ComponentManager.scala:25)
at sbt.ComponentManager$$anonfun$files$1$$anonfun$apply$2.apply(ComponentManager.scala:44)
at sbt.ComponentManager$$anonfun$files$1$$anonfun$apply$2.apply(ComponentManager.scala:44)
at sbt.ComponentManager.sbt$ComponentManager$$getOrElse$1(ComponentManager.scala:32)
at sbt.ComponentManager$$anonfun$files$1.apply(ComponentManager.scala:44)
at sbt.ComponentManager$$anonfun$files$1.apply(ComponentManager.scala:44)
at sbt.ComponentManager$$anon$1.call(ComponentManager.scala:50)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
at xsbt.boot.Using$.withResource(Using.scala:10)
at xsbt.boot.Using$.apply(Using.scala:9)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.ComponentManager.lock(ComponentManager.scala:50)
at sbt.ComponentManager.lockLocalCache(ComponentManager.scala:47)
at sbt.ComponentManager.files(ComponentManager.scala:44)
at sbt.ComponentManager.file(ComponentManager.scala:53)
at sbt.compiler.ComponentCompiler.getLocallyCompiled(ComponentCompiler.scala:50)
at sbt.compiler.ComponentCompiler.apply(ComponentCompiler.scala:36)
at sbt.compiler.ComponentCompiler$$anon$1.apply(ComponentCompiler.scala:23)
at sbt.compiler.AnalyzingCompiler.loader(AnalyzingCompiler.scala:112)
at sbt.compiler.AnalyzingCompiler.getInterfaceClass(AnalyzingCompiler.scala:117)
at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:98)
at sbt.compiler.AnalyzingCompiler.newCachedCompiler(AnalyzingCompiler.scala:56)
at sbt.compiler.AnalyzingCompiler.newCachedCompiler(AnalyzingCompiler.scala:51)
at sbt.compiler.CompilerCache$$anon$2.apply(CompilerCache.scala:47)
at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:39)
at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileScala$1$1.apply$mcV$sp(MixedAnalyzingCompiler.scala:51)
at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileScala$1$1.apply(MixedAnalyzingCompiler.scala:51)
at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileScala$1$1.apply(MixedAnalyzingCompiler.scala:51)
at sbt.compiler.MixedAnalyzingCompiler.timed(MixedAnalyzingCompiler.scala:75)
at sbt.compiler.MixedAnalyzingCompiler.compileScala$1(MixedAnalyzingCompiler.scala:50)
at sbt.compiler.MixedAnalyzingCompiler.compile(MixedAnalyzingCompiler.scala:65)
at sbt.compiler.IC$$anonfun$compileInternal$1.apply(IncrementalCompiler.scala:160)
at sbt.compiler.IC$$anonfun$compileInternal$1.apply(IncrementalCompiler.scala:160)
at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:66)
at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:64)
at sbt.inc.IncrementalCommon.cycle(IncrementalCommon.scala:31)
at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:62)
at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:61)
at sbt.inc.Incremental$.manageClassfiles(Incremental.scala:89)
at sbt.inc.Incremental$.compile(Incremental.scala:61)
at sbt.inc.IncrementalCompile$.apply(Compile.scala:54)
at sbt.compiler.IC$.compileInternal(IncrementalCompiler.scala:160)
at sbt.compiler.IC$.incrementalCompile(IncrementalCompiler.scala:138)
at sbt.Compiler$.compile(Compiler.scala:128)
at sbt.Compiler$.compile(Compiler.scala:114)
at sbt.Defaults$.sbt$Defaults$$compileIncrementalTaskImpl(Defaults.scala:814)
at sbt.Defaults$$anonfun$compileIncrementalTask$1.apply(Defaults.scala:805)
at sbt.Defaults$$anonfun$compileIncrementalTask$1.apply(Defaults.scala:803)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I am using jdk 1.8.0, OS X 10.11.6. Here code example:
object Main {
def main(args: Array[String]) {
println("test")
}
}
The same code builded fine with scala 2.10
UPD: build.sbt for 2.12 version
name := "hello_world_3"
version := "1.0"
scalaVersion := "2.12.0"
Problem was in minor JDK version. Scala 2.12 require's newer version of JDK then 1.8.0_111. So after JDK upgrade everything work. Thanks to #rumoku
I am a Beginner of spark, and I am trying to use docker to run spark example NetworkWordCount. But no output when I run the example:
start a new terminal by docker exec -it container_id bash,
root#sandbox:/usr/local/spark# nc -lk 9999
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
a ba c d e f g
then start another terminal:
root#sandbox:/usr/local/spark# bin/run-example streaming.NetworkWordCount localhost 9999
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/05/12 02:55:57 INFO StreamingExamples: Setting log level to [WARN] for streaming example. To override add a custom log4j.properties to the classpath.
16/05/12 02:55:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
...
Anyone can help me?
Executing using below command worked for me.
[root#quickstart spark]# ./bin/run-example streaming.NetworkWordCount localhost 9999 --master "local[2]"
I have some data that is slightly noisy. I want to clean it as well as the derivative. A very simple "filter" such as Y(n)=0.2*X(n)+0.8*Y(n-1) suits me well: in other words, the output is smooth enough. Here is an example with both data and its derivative:
Nevertheless, as it's real-time, I cannot accept a big lag in data. For the derivative, the lag is around 10 iterations: 5 iterations (to get the filtered data) and then another 5 iterations (to get the filtered derivative data).
How can I implement an algorithm in Matlab - in fine, I need a c implementation - that produces a similar result but with very little lag, such as a maximum of 3 iterations EVEN and ESPECIALLY for the derivative.
Here is the Matlab code:
%Here is data input;
data = [5554 5767 5737 5828 6011 6011 6103 6225 6408 6286 6347 6439 6591 6683 6530 6652 6805 7019 7080 7019 6988 6988 6958 6958 6927 6896 6866 6805 6835 6713 6591 6713 6713 6652 6713 6713 6835 6896 6744 6622 6683 6561 6591 6591 6622 6561 6500 6652 6774 6805 6958 7049 7171 7232 7293 7293 7415 7476 7690 8087 8056 7965 8178 8270 8148 8178 8392 8239 8056 8026 7873 7873 7476 7446 7476 7293 7171 7141 7263 7019 7141 6774 6652 6530 6530 6469 6561 6622 6652 6713 6866 6958 6958 7110 6958 7049 6805 6866 6866 6744 7049 6805 6652 6530 6622 6744 6408 6652 6683 6713 6713 6622 6744 6805 6713 6683 6805 6835 6927 6866 6805 6744 7049 7232 7171 7141 7202 7354 7324 7415 7415 7415 7965 8178 8178 8178 8178 7995 7995 7965 7843 7904 7904 7812 7812 7720 7690 7843 7873 7904 8056 7904 7843 7934 7873 7873 7781 7781 7873 7873 7873 7690 7659 7690 7720 7751 7659 7598 7629 7659 7568 7537 7629 7537 7659 7568 7568 7476 7568 7598 7354 7324 7476 7354 7385 7202 7293 7293 6958 6866 6866 6683 6927 6927 6927 7019 7019 7049 7141 7202 7293 7385 7568 7446 7446 7415 7354 7263 7293 7202 7202 7202 7232 7202 7263 7141 7171 7019 7141 7232 7080 7232 7141 7263 6958 7019 7019 6866 6896 6744 6774 6805 6835 6866 6774 6744 6774 6744 6744 6866 6958 6958 7019 6988 6988 6896 6958 7080 7080 7171 7232 7202 7354 7385 7385 7476 7720 7690 7598 7659 7659 7690 7720 7720 7690 7690 7781 7751 7659 7812 7873 7659 7629 7446 7446 7629 7354 7415 7476 7568 7415 7385 7385 7324 7202 7324 7110 6958 7019 7019 7049 6774 7202 6561 6408 6195 5920 5920 5920 5859 5615 5645 5584 5676 5432 5401 5126 5096 5004 5004 4791 4852 4791 4791 4730 4760 4730 4730 4699 4760 4669 4730 4760 4760 4791 4882 4882 4882 4974 4943 4974 5065 5096 5157 5218 5279 5279]';
%Define smooth coefficient
c=0.2;
%Filter data
data_filtered=zeros(size(data),1);
for n=drange(2:size(data))
data_filtered(n,1)=c*data(n)+(1-c)*data_filtered(n-1,1);
end
%Graph both data and filtered data
subplot(2,1,1);
plot([1:size(data)], data, [1:size(data)], data_filtered);
%Calculate derivative data. Delta time is constant.
for n=drange(2:size(data_filtered))
data_derivative(n,1)=data(n)-data(n-1,1);
end
%Calculate derivative based on filtered data. Delta time is constant.
for n=drange(2:size(data_filtered))
data_filtered_derivative(n,1)=data_filtered(n)-data_filtered(n-1,1);
end
%Filter derivative (which is based on filtered data)
data_filtered_derivative_filtered=data_filtered_derivative;
for n=drange(2:size(data_filtered_derivative))
data_filtered_derivative_filtered(n,1)=c*data_filtered_derivative(n)+(1-c)*data_filtered_derivative_filtered(n-1,1);
end
%Graph both derivative data and filtered derivative data
subplot(2,1,2);
plot([1:size(data_derivative)], data_derivative, [1:size(data_derivative)], data_filtered_derivative_filtered);
If you want an excellent, low-lag filter it seems something like Jurik JMA might be for you. It is probably pretty expensive (and definitely not free) but the paper might give you ideas on other (possibly free) algorithms.
I have not tested that algorithm, I have just found it and liked the pictures.
The Activity creates a new Thread.
The run() method gets an object (vector) via input stream.
Now I want to display the contents of this Vector on a ListView called direc.
That's all...but its not working and throws an Exception.
The code:
public class FileExplorerActivity extends Activity implements Runnable
{
public static final String TAG="ricky";
Vector<String> dirs = new Vector<String>();
ListView direc;
static ArrayAdapter<String> arrayAdapter;
Handler handler;
#Override
protected void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
setContentView(R.layout.fileexplorer);
direc = (ListView) findViewById(R.id.directoriesss);
handler = new Handler();
Thread thread = new Thread(this);
thread.start();
}
public FileExplorerActivity()
{
}
public void run()
{
Log.d(TAG, "FileExplorer: Thread started");
try
{
Log.d(TAG, "FileExplorer: reading data");
dirs = (Vector<String>) ConnectionThread.din.readObject();
Log.d(TAG, "FileExplorer: read data");
arrayAdapter = new ArrayAdapter<String>(this,
android.R.layout.activity_list_item,
android.R.id.text2 , dirs);
Log.d(TAG, "FileExplorer: ArrayAdapter created");
handler.post( new Runnable() {
public void run()
{
Log.d(TAG, "FileExplorer: In handler Thread");
direc.setAdapter(arrayAdapter);
Log.d(TAG, "FileExplorer: Updated UI");
}
});
Log.d(TAG, "FileExplorer: Directory set");
}catch (Exception e) {
Log.d(TAG, "FileExplorerActivity: Exception:"+e.getMessage());
}
}
}
Here is what Logcat has to say:
> 1. 05-24 20:14:46.601 7759 7834 D ricky : FileExplorer: Thread started
> 2. 05-24 20:14:46.601 7759 7834 D ricky : FileExplorer: reading data
> 3. 05-24 20:14:46.648 7759 7834 D ricky : FileExplorer: read data
> 4. 05-24 20:14:46.648 7759 7834 D ricky : FileExplorer: ArrayAdapter created
> 5. 05-24 20:14:46.648 7759 7834 D ricky : FileExplorer: Directory set
> 6. 05-24 20:14:46.656 7759 7759 D ricky : FileExplorer: In handler Thread
> 7. 05-24 20:14:46.656 7759 7759 D ricky : FileExplorer: Updated UI
> 8. 05-24 20:14:46.664 7759 7759 D AndroidRuntime: Shutting down VM
> 9. 05-24 20:14:46.664 7759 7759 W dalvikvm: threadid=1: thread exiting with uncaught 10. exception (group=0x4001e560)
> 11. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: FATAL EXCEPTION: main
> 12. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: java.lang.NullPointerException
> 13. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.ArrayAdapter.createViewFromResource(ArrayAdapter.java:353)
> 14. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.ArrayAdapter.getView(ArrayAdapter.java:323)
> 15. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.AbsListView.obtainView(AbsListView.java:1456)
> 16. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.ListView.measureHeightOfChildren(ListView.java :1291)
> 17. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.ListView.onMeasure(ListView.java:1202)
> 18. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.view.View.measure(View.java:8355)
> 19. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java :3279)
> 20. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.LinearLayout.measureChildBeforeLayout(LinearLayout.java:1017)
> 21. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.LinearLayout.measureHorizontal(LinearLayout.ja va:701)
> 22. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.LinearLayout.onMeasure(LinearLayout.java:311)
> 23. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.view.View.measure(View.java:8355)
> 24. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java :3279)
> 25. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.FrameLayout.onMeasure(FrameLayout.java:250)
> 26. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.view.View.measure(View.java:8355)
> 27. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.LinearLayout.measureVertical(LinearLayout.java :531)
> 28. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.LinearLayout.onMeasure(LinearLayout.java:309)
> 29. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.view.View.measure(View.java:8355)
> 30. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java :3279)
> 31. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.widget.FrameLayout.onMeasure(FrameLayout.java:250)
> 32. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.view.View.measure(View.java:8355)
> 33. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.view.ViewRoot.performTraversals(ViewRoot.java:882)
> 34. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.view.ViewRoot.handleMessage(ViewRoot.java:1917)
> 35. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.os.Handler.dispatchMessage(Handler.java:99)
> 36. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.os.Looper.loop(Looper.java:130)
> 37. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at android.app.ActivityThread.main(ActivityThread.java:3859)
> 38. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at java.lang.reflect.Method.invokeNative(Native Method)
> 39. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at java.lang.reflect.Method.invoke(Method.java:507)
> 40. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:839)
> 41. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:597)
> 42. 05-24 20:14:46.687 7759 7759 E AndroidRuntime: at dalvik.system.NativeStart.main(Native Method)
Note the line at which the NullPointerException occurs:
android.widget.ArrayAdapter.createViewFromResource(ArrayAdapter.java:353)
This is becauseandroid.R.layout.activity_list_item does not contain android.R.id.text2.
You can use android.R.id.text1 instead or replace the row layout with simple_list_item_2.
I would suggest creating your own row layout to avoid these situations in the first place.
Move this
arrayAdapter = new ArrayAdapter<String>(this, android.R.layout.activity_list_item,
android.R.id.text2 , dirs);
inside your run() method so that it runs on the UI thread. You shouldn't create an ArrayAdapter outside of the UI thread.