Deserialization Exception in twitter/chill with Play2 and Scala 2.11.1 - scala

i am using Scala 2.11.1 and Hazelcast 3.5 and twitter-chill for serialization
i have a class User and serializer UserSerializer both on Server(Sbt project) and client side (Play2 project)
i am storing User object in Hazelcast ,
my serializer is working fine on server but the read method of the same serializer gives me exception
here is my serializer
// takes the bytes and converts into User Object (Deserializer)
#throws(classOf[IOException])
override def read(in : ObjectDataInput) : User = {
log.info("*********** Deserializer from Client called ")
val bytes = ByteArrayConverter.getByteArray(in).toByteArray();
log.info("Bytes size: "+bytes.length)
val tryDecode: scala.util.Try[Object] = KryoInjection.invert(bytes)
tryDecode match {
case Success(obj) => log.info(" ------ Success case executed ")
obj.asInstanceOf[User]
case Failure(e) => log.info("------ Failure case executed ")
log.error("deserializer from Client : {}" ,e.printStackTrace())
null
}
}
the problem is at this line
at UserSerializer.read(UserSerializer.scala:44)
which is this line from the above method
val tryDecode: scala.util.Try[Object] = KryoInjection.invert(bytes)
i dont know whats the problem because same code works fine on Hazelcast server but gives problem on client side
here are the complete stack traces from client side (Play2)
17:24:48.822 276580 [play-akka.actor.default-dispatcher-7]
UserSerializer INFO - *********** Deserializer from Client
called 17:24:48.839 276597 [play-akka.actor.default-dispatcher-7]
ByteArrayConverter$ INFO - ---- object to byte array converter
17:24:48.841 276599 [play-akka.actor.default-dispatcher-7]
ByteArrayConverter$ INFO - ---- conversion done now returning the byte
array 17:24:48.841 276599 [play-akka.actor.default-dispatcher-7]
UserSerializer INFO - Bytes size: 596 17:24:48.917 276675
[play-akka.actor.default-dispatcher-7] UserSerializer INFO - ------
Failure case executed com.twitter.bijection.InversionFailure: Failed
to invert: [B#1296562 at
com.twitter.bijection.InversionFailure$$anonfun$partialFailure$1.applyOrElse(InversionFailure.scala:43)
at
com.twitter.bijection.InversionFailure$$anonfun$partialFailure$1.applyOrElse(InversionFailure.scala:42)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at scala.util.Failure.recoverWith(Try.scala:203) at
com.twitter.bijection.Inversion$.attempt(Inversion.scala:30) at
com.twitter.chill.KryoInjection$.invert(KryoInjection.scala:29) at
UserSerializer.read(UserSerializer.scala:44)
at
UserSerializer.read(UserSerializer.scala:22)
at
com.hazelcast.nio.serialization.StreamSerializerAdapter.read(StreamSerializerAdapter.java:41)
at
com.hazelcast.nio.serialization.SerializationServiceImpl.toObject(SerializationServiceImpl.java:276)
at
com.hazelcast.client.spi.ClientProxy.toObject(ClientProxy.java:173)
at com.hazelcast.client.spi.ClientProxy.invoke(ClientProxy.java:131)
at
com.hazelcast.client.proxy.ClientMapProxy.get(ClientMapProxy.java:200)
at
play.api.mvc.ActionBuilder$$anonfun$apply$17.apply(Action.scala:464)
at
play.api.mvc.ActionBuilder$$anonfun$apply$17.apply(Action.scala:464)
at
play.api.mvc.ActionBuilder$$anonfun$apply$16.apply(Action.scala:433)
at
play.api.mvc.ActionBuilder$$anonfun$apply$16.apply(Action.scala:432)
at play.api.mvc.Action$.invokeBlock(Action.scala:556) at
play.api.mvc.Action$.invokeBlock(Action.scala:555) at
play.api.mvc.ActionBuilder$$anon$1.apply(Action.scala:518) at
play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130)
at
play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130)
at play.utils.Threads$.withContextClassLoader(Threads.scala:21) at
play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:129)
at
play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:128)
at scala.Option.map(Option.scala:146) at
play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:128) at
play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:121) at
play.api.libs.iteratee.DoneIteratee$$anonfun$mapM$2.apply(Iteratee.scala:705)
at
play.api.libs.iteratee.DoneIteratee$$anonfun$mapM$2.apply(Iteratee.scala:705)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: com.esotericsoftware.kryo.KryoException: Unable to find
class: User at
com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
at
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610) at
com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721) at
com.twitter.chill.SerDeState.readClassAndObject(SerDeState.java:61)
at com.twitter.chill.KryoPool.fromBytes(KryoPool.java:94) at
com.twitter.chill.KryoInjection$$anonfun$invert$1.apply(KryoInjection.scala:30)
at
com.twitter.chill.KryoInjection$$anonfun$invert$1.apply(KryoInjection.scala:30)
at
com.twitter.bijection.Inversion$$anonfun$attempt$1.apply(Inversion.scala:30)
at scala.util.Try$.apply(Try.scala:192) ... 36 more Caused by:
java.lang.ClassNotFoundException: User at
java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:348) at
com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
... 45 more 17:24:48.942 276700
[play-akka.actor.default-dispatcher-7] UserSerializer ERROR -
deserializer from Client : ()

Related

App can't run from eclipse which created by 'jhipster --skipClient'

java.lang.IllegalStateException: Error processing condition on org.springframework.boot.autoconfigure.context.MessageSourceAutoConfiguration
at org.springframework.boot.autoconfigure.condition.SpringBootCondition.matches(SpringBootCondition.java:64)
at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108)
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader$TrackedConditionEvaluator.shouldSkip(ConfigurationClassBeanDefinitionReader.java:441)
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:129)
at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:118)
at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:328)
at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:233)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:271)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:91)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:692)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:530)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:386)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
at com.einfochips.demo.TestApp.main(TestApp.java:63)
Caused by: java.lang.IllegalStateException: Failed to introspect Class [org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration] from ClassLoader [sun.misc.Launcher$AppClassLoader#659e0bfd]

Play Scala Oauth Provider Error

I am building a Play Scala project and need an authentication for the apps.
I am using this one : https://github.com/nulab/scala-oauth2-provider as the Oauth Provider libs.
I have implement everything based on the docs. But when implement the Auth on the controller below :
object Token extends MainController with OAuth2Provider {
def accessToken = Action.async { implicit request =>
issueAccessToken(new MyDataHandler()
}
And here my router :
GET /test controllers.Token.test
POST /token controllers.Token.accessToken
When I access localhost/token, i got this error :
And here the error from console :
play.api.Application$$anon$1: Execution exception[[RuntimeException: java.lang.IncompatibleClassChangeError: Found interface play.api.mvc.Headers, but class was expected]]
at play.api.Application$class.handleError(Application.scala:296) ~[play_2.10-2.3.7.jar:2.3.7]
at play.api.DefaultApplication.handleError(Application.scala:402) [play_2.10-2.3.7.jar:2.3.7]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [play_2.10-2.3.7.jar:2.3.7]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [play_2.10-2.3.7.jar:2.3.7]
at scala.Option.map(Option.scala:145) [scala-library-2.10.5.jar:na]
Caused by: java.lang.RuntimeException: java.lang.IncompatibleClassChangeError: Found interface play.api.mvc.Headers, but class was expected
at play.api.mvc.ActionBuilder$$anon$1.apply(Action.scala:523) ~[play_2.10-2.3.7.jar:2.3.7]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) ~[play_2.10-2.3.7.jar:2.3.7]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) ~[play_2.10-2.3.7.jar:2.3.7]
at play.utils.Threads$.withContextClassLoader(Threads.scala:21) ~[play_2.10-2.3.7.jar:2.3.7]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:129) ~[play_2.10-2.3.7.jar:2.3.7]
Caused by: java.lang.IncompatibleClassChangeError: Found interface play.api.mvc.Headers, but class was expected
at scalaoauth2.provider.OAuth2BaseProvider$class.play2oauthRequest(OAuth2Provider.scala:24) ~[play2-oauth2-provider_2.10-0.15.1.jar:0.15.1]
at controllers.Token$.play2oauthRequest(Token.scala:14) ~[na:na]
at scalaoauth2.provider.OAuth2Provider$class.issueAccessToken(OAuth2Provider.scala:138) ~[play2-oauth2-provider_2.10-0.15.1.jar:0.15.1]
at controllers.Token$.issueAccessToken(Token.scala:14) ~[na:na]
at controllers.Token$$anonfun$accessToken$1.apply(Token.scala:25) ~[na:na]
I have implement everything what does docs said. Sorry I am actually new on Scala and on the Play it self.
What do you use Play and scala-oauth2-provider version?
The error looks Play version is different.
If you use Play 2.3, you should use 0.14.0 in scala-oauth2-provider.
See readme.

Exception thrown while deploying spring batch job in SpringXD Singlenode

Exception thrown while deploying spring batch job in SpringXD Singlenode (1.2.0 Distribution)
Please let me know, how this exception can be fixed (or) any further details needed
org.springframework.beans.factory.BeanDefinitionStoreException: Invalid
bean definition with name 'registrar' defined in class path resource
[META-INF/spring-xd/plugins/job/job-module-beans.xml]:
Could not resolve placeholder 'xd.job.name' in string value "${xd.job.name}";
nested exception is java.lang.IllegalArgumentException: Could not
resolve placeholder 'xd.job.name' in string value "${xd.job.name}"
Following are the trace
2015-07-24T15:29:11-0700 1.2.0.RELEASE INFO DeploymentsPathChildrenCache-0 container.DeploymentListener - Deploying module [ModuleDescriptor#6435a6c5 moduleName = 'category', moduleLabel = 'category', group = 'testcategory', sourceChannelName = [null], sinkChannelName = [null], index = 0, type = job, parameters = map['dateFormat' -> 'yyyy-MM-dd', 'makeUnique' -> 'true'], children = list[[empty]]]
2015-07-24T15:29:12-0700 1.2.0.RELEASE WARN DeploymentsPathChildrenCache-0 annotation.AnnotationConfigApplicationContext - Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'registrar' defined in class path resource [META-INF/spring-xd/plugins/job/job-module-beans.xml]: Could not resolve placeholder 'xd.job.name' in string value "${xd.job.name}"; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'xd.job.name' in string value "${xd.job.name}"
at org.springframework.beans.factory.config.PlaceholderConfigurerSupport.doProcessProperties(PlaceholderConfigurerSupport.java:211) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:222) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:86) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:265) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:162) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:606) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:462) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:686) [spring-boot-1.2.3.RELEASE.jar:1.2.3.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:320) [spring-boot-1.2.3.RELEASE.jar:1.2.3.RELEASE]
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:139) [spring-boot-1.2.3.RELEASE.jar:1.2.3.RELEASE]
at org.springframework.xd.module.core.SimpleModule.initialize(SimpleModule.java:213) [spring-xd-module-1.2.0.RELEASE.jar:1.2.0.RELEASE]
at org.springframework.xd.dirt.module.ModuleDeployer.doDeploy(ModuleDeployer.java:217) [spring-xd-dirt-1.2.0.RELEASE.jar:1.2.0.RELEASE]
at org.springframework.xd.dirt.module.ModuleDeployer.deploy(ModuleDeployer.java:200) [spring-xd-dirt-1.2.0.RELEASE.jar:1.2.0.RELEASE]
at org.springframework.xd.dirt.server.container.DeploymentListener.deployModule(DeploymentListener.java:365) [spring-xd-dirt-1.2.0.RELEASE.jar:1.2.0.RELEASE]
at org.springframework.xd.dirt.server.container.DeploymentListener.deployJobModule(DeploymentListener.java:291) [spring-xd-dirt-1.2.0.RELEASE.jar:1.2.0.RELEASE]
at org.springframework.xd.dirt.server.container.DeploymentListener.onChildAdded(DeploymentListener.java:181) [spring-xd-dirt-1.2.0.RELEASE.jar:1.2.0.RELEASE]
at org.springframework.xd.dirt.server.container.DeploymentListener.childEvent(DeploymentListener.java:149) [spring-xd-dirt-1.2.0.RELEASE.jar:1.2.0.RELEASE]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:509) [curator-recipes-2.6.0.jar:na]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:503) [curator-recipes-2.6.0.jar:na]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.6.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:na]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.6.0.jar:na]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:500) [curator-recipes-2.6.0.jar:na]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.6.0.jar:na]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$10.run(PathChildrenCache.java:762) [curator-recipes-2.6.0.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_31]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_31]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_31]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_31]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'xd.job.name' in string value "${xd.job.name}"
When creating your job, you will usually pass a name and definition argument. The definition will be put in xd.job.name that serves as a placeholder for the current job's name. It will be filled in, for example in job-module-beans.xml, when xd.job.name is referenced.
Check out this example:
xd:> job create --name helloWord --definition "hellojob" --deploy
xd:> job launch helloWord --params {"someStringParam":"foobar","-someOtherParam(long)":"123456"}
Make sure you pass the correct definition when creating your job during deployment.

Usage of mapFuture in akka streams

I'm playing with Akka Streams and have been trying to do some enrichment and processing of events polled from a MongoDB collection. I'm having some doubts, however, about the best approach to implement my event enrichers, which may need to connect to an external data source.
The mapFuture seems to be a good fit, but I'm having some issues:
class EventEnricherActor extends Actor with ActorLogging {
// ...
def receive = {
case e: Event =>
sender ! augmentEvent(e)
}
}
And my app:
val enricherActor = actorSystem.actorOf(Props[EventEnricherActor])
Flow(mongodbConsumer).
mapFuture(msg => enricherActor ? msg).
onComplete(materializer) { _ => actorSystem.shutdown()}
However, I'm stuck at this error:
java.lang.ClassCastException: messages.Event cannot be cast to scala.runtime.Nothing$
on the call to mapFuture.
What am I missing?
Any better ideas to deal with these enrichments?
Update
Stack trace:
[ERROR] [08/12/2014 11:48:07.314] [actor-system-akka.actor.default-dispatcher-2] [akka://actor-system/user/flow-1-2-transform] failure during processing
java.lang.ClassCastException: messages.Event cannot be cast to scala.runtime.Nothing$
at apps.Consume$$anonfun$1$$anonfun$apply$1.apply(Consume.scala:47)
at akka.stream.impl.MapFutureProcessorImpl$$anonfun$1.apply$mcV$sp(MapFutureProcessorImpl.scala:125)
at akka.stream.impl.Pump$$anonfun$pump$1.apply$mcV$sp(Transfer.scala:163)
at akka.stream.impl.Pump$$anonfun$pump$1.apply(Transfer.scala:163)
at akka.stream.impl.Pump$$anonfun$pump$1.apply(Transfer.scala:163)
at akka.stream.impl.ActorBasedFlowMaterializer$.withCtx(ActorBasedFlowMaterializer.scala:133)
at akka.stream.impl.Pump$class.pump(Transfer.scala:163)
at akka.stream.impl.ActorProcessorImpl.pump(ActorProcessor.scala:238)
at akka.stream.impl.BatchingInputBuffer.enqueueInputElement(ActorProcessor.scala:93)
at akka.stream.impl.BatchingInputBuffer$$anonfun$upstreamRunning$1.applyOrElse(ActorProcessor.scala:140)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at akka.stream.impl.SubReceive.apply(Transfer.scala:18)
at akka.stream.impl.SubReceive.apply(Transfer.scala:14)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at akka.stream.impl.SubReceive.applyOrElse(Transfer.scala:14)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:165)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:166)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at akka.stream.impl.ActorProcessorImpl.aroundReceive(ActorProcessor.scala:238)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Thanks
My mistake, mongodbConsumer (an ActorProducer) was untyped!
So this:
val mongodbConsumer = ActorProducer[Event](system.actorOf(MongodbConsumerActor.props(db)))
Instead of:
val mongodbConsumer = ActorProducer(system.actorOf(MongodbConsumerActor.props(db)))

Spark Streaming Stateful Network Word Count

This is the example code that came with Spark. I copied the code here and this is the link to it: https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala. However, when I was trying to run the program using command "bin/run-example org.apache.spark.examples.streaming.StatefulNetworkWordCount localhost 9999", I was given the following error:
14/07/20 11:52:57 ERROR ActorSystemImpl: Uncaught fatal error from thread [spark-akka.actor.default-dispatcher-4] shutting down ActorSystem [spark]
java.lang.NoSuchMethodError: java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
at org.apache.spark.streaming.scheduler.JobScheduler.getPendingTimes(JobScheduler.scala:114)
at org.apache.spark.streaming.Checkpoint.<init>(Checkpoint.scala:43)
at org.apache.spark.streaming.scheduler.JobGenerator.doCheckpoint(JobGenerator.scala:259)
at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:167)
at org.apache.spark.streaming.scheduler.JobGenerator$$anonfun$start$1$$anon$1$$anonfun$receive$1.applyOrElse(JobGenerator.scala:76)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Exception in thread "Thread-37" org.apache.spark.SparkException: Job cancelled because SparkContext was shut down
at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:639)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:638)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:638)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.postStop(DAGScheduler.scala:1215)
at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:201)
at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:163)
at akka.actor.ActorCell.terminate(ActorCell.scala:338)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:431)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:240)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
14/07/20 11:53:00 ERROR Executor: Exception in task ID 0
java.lang.IllegalStateException: cannot create children while terminating or terminated
at akka.actor.dungeon.Children$class.makeChild(Children.scala:184)
at akka.actor.dungeon.Children$class.attachChild(Children.scala:42)
at akka.actor.ActorCell.attachChild(ActorCell.scala:338)
at akka.actor.ActorSystemImpl.actorOf(ActorSystem.scala:518)
at org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.<init>(ReceiverSupervisorImpl.scala:67)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:263)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$9.apply(ReceiverTracker.scala:257)
at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1080)
at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1080)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
at org.apache.spark.scheduler.Task.run(Task.scala:51)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
14/07/20 11:53:06 ERROR ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[spark-akka.actor.default-dispatcher-13,5,main]
org.apache.spark.SparkException: Error sending message to BlockManagerMaster [message = HeartBeat(BlockManagerId(<driver>, x-131-212-225-148.uofm-secure.wireless.umn.edu, 47668, 0))]
at org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:251)
at org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:51)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:113)
at org.apache.spark.storage.BlockManager$$anonfun$initialize$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(BlockManager.scala:158)
at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:790)
at org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:158)
at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
at akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.run(Scheduler.scala:464)
at akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:281)
at akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:280)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at akka.actor.LightArrayRevolverScheduler.close(Scheduler.scala:279)
at akka.actor.ActorSystemImpl.stopScheduler(ActorSystem.scala:630)
at akka.actor.ActorSystemImpl$$anonfun$_start$1.apply$mcV$sp(ActorSystem.scala:582)
at akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
at akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:596)
at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.runNext$1(ActorSystem.scala:750)
at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply$mcV$sp(ActorSystem.scala:753)
at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
at akka.util.ReentrantGuard.withGuard(LockUtil.scala:15)
at akka.actor.ActorSystemImpl$TerminationCallbacks.run(ActorSystem.scala:746)
at akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
at akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://spark/user/BlockManagerMaster#1887396223]] had already been terminated.
at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
at org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:236)
****************CODE********************
object StatefulNetworkWordCount {
def main(args: Array[String]) {
if (args.length < 2) {
System.err.println("Usage: StatefulNetworkWordCount <hostname> <port>")
System.exit(1)
}
StreamingExamples.setStreamingLogLevels()
val updateFunc = (values: Seq[Int], state: Option[Int]) => {
val currentCount = values.foldLeft(0)(_ + _)
val previousCount = state.getOrElse(0)
Some(currentCount + previousCount)
}
val sparkConf = new SparkConf().setAppName("StatefulNetworkWordCount")
// Create the context with a 1 second batch size
val ssc = new StreamingContext(sparkConf, Seconds(1))
ssc.checkpoint(".")
// Create a NetworkInputDStream on target ip:port and count the
// words in input stream of \n delimited test (eg. generated by 'nc')
val lines = ssc.socketTextStream(args(0), args(1).toInt)
val words = lines.flatMap(_.split(" "))
val wordDstream = words.map(x => (x, 1))
// Update the cumulative count using updateStateByKey
// This will give a Dstream made of state (which is the cumulative count of the words)
val stateDstream = wordDstream.updateStateByKey[Int](updateFunc)
stateDstream.print()
ssc.start()
ssc.awaitTermination()
}
}
I wonder if it is because, it is trying to set up the checkpoint at my local file system by doing commands "ssc.checkpoint(".")", while the file is not a file compatible with hadoop? (the file must be compatible with hadoop in order to set the checkpoint) If it is, how could I fix it? Thanks!
what's your JRE run time release, 1.7 or 1.8 , I have the similar issue that, I compile spark-source-code in 1.8, but if I use 1.7 to run the code, this issue will happen, change back to 1.8 as run time issue will solve.
jdk 1.8, concurrenthashmap.java(line 812):
// views
private transient KeySetView<K,V> keySet;
private transient ValuesView<K,V> values;
private transient EntrySetView<K,V> entrySet;
jdk 1.7 don't have above code
Hope it will help ^_^