From a remote scala program, using Spark 1.3, how do I initialize the sparkContext so that I can connect to Spark running on YARN? i.e. where do I put the address of the YARN node(s)?
Currently my program contains:
val conf = new SparkConf().setMaster("yarn-client").setAppName("MyApp")
val sc = new SparkContext(conf)
and it yields
[error] (run-main-0) java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1959)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:104)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:179)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:310)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:163)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:269)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:272)
at SparkExampleLocalDriver$.main(SparkExample.scala:9)
at SparkExampleLocalDriver.main(SparkExample.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: org.apache.spark.SparkException: Unable to load YARN support
at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:217)
at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:212)
at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1959)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:104)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:179)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:310)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:163)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:269)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:272)
at SparkExampleLocalDriver$.main(SparkExample.scala:9)
at SparkExampleLocalDriver.main(SparkExample.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:195)
at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:213)
at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:212)
at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1959)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:104)
at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:179)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:310)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:163)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:269)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:272)
at SparkExampleLocalDriver$.main(SparkExample.scala:9)
at SparkExampleLocalDriver.main(SparkExample.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
Your Spark binary doesn't contain YARN related classes
Use pre-built binary for hadoop
http://www.apache.org/dyn/closer.cgi/spark/spark-1.3.1/spark-1.3.1-bin-hadoop2.4.tgz
If you are compiling source, include yarn and hadoop profiles
./make-distribution.sh --tgz --skip-java-test -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0
Related
I run file jar Word Count but have this error
NoClassDefFoundError: org/Apache/commons/lang/StringUtils ..
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
at com.mongodb.hadoop.util.MongoConfigUtil.getMongoURIs(MongoConfigUtil.java:409)
at com.mongodb.hadoop.util.MongoConfigUtil.getOutputURIs(MongoConfigUtil.java:598)
at com.mongodb.hadoop.MongoOutputFormat.checkOutputSpecs(MongoOutputFormat.java:32)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:277)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1571)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1568)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1568)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1589)
at WordCount.main(WordCount.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
I have generated stubs of contract tests via producer.
I have installed Spring Cloud CLI (version Spring Cloud CLI v2.2.0.BUILD-SNAPSHOT).
I have created file stubrunner.yml in path C:\Work with next properties:
stubrunner:
stubsMode: LOCAL
ids: com.project:users:+:stubs:8555
repositoryRoot: classpath:m2repo/repository/
In command line I moved to whis path C:\Workand then type spring cloud stubrunner and as resut I got 2 exception messages
1-st exception:
o.s.c.l.deployer.DeployerApplication : Starting DeployerApplication v2.2.0.BUILD-SNAPSHOT on User with PID 3144 (C:\Users\User\.m2\repository\org\springframework\cloud\launcher\spring-cloud-launcher-deployer\2.2.0.BUILD-SNAPSHOT\spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar started by User in C:\Work)
o.s.c.l.deployer.DeployerApplication : No active profile set, falling back to default profiles: default
o.s.c.l.deployer.DeployerApplication : Started DeployerApplication in 0.776624926 seconds (JVM running for 3.148)
o.s.cloud.launcher.deployer.Deployer : Loaded YAML properties from: file [C:\Work\.\stubrunner.yml]
o.s.boot.loader.thin.PathResolver : Extracting dependencies from: file:/C:/Users/User/.m2/repository/org/springframework/cloud/launcher/spring-cloud-launcher-stubrunner/2.2.0.BUILD-SNAPSHOT/spring-cloud-launcher-stubrunner-2.2.0.BUILD-SNAPSHOT.jar, with profiles []
o.s.boot.loader.thin.PathResolver : Searching for properties in: file:/C:/Users/User/.m2/repository/org/springframework/cloud/launcher/spring-cloud-launcher-stubrunner/2.2.0.BUILD-SNAPSHOT/spring-cloud-launcher-stubrunner-2.2.0.BUILD-SNAPSHOT.jar
o.s.boot.loader.thin.PathResolver : Extracting dependencies from: class path resource [META-INF/thin/empty-pom.xml], with profiles []
o.s.b.loader.thin.MavenSettingsReader : Reading settings from: C:\Users\User\.m2\settings.xml
o.s.boot.loader.thin.DependencyResolver : Computing dependencies from pom and properties
o.s.c.deployer.thin.ThinJarAppWrapper : Cannot deploy org.springframework.cloud.launcher:spring-cloud-launcher-stubrunner:jar:2.2.0.BUILD-SNAPSHOT
java.lang.IllegalStateException: Failed to introspect Class [org.springframework.cloud.deployer.thin.ContextRunner] from ClassLoader [java.net.URLClassLoader#430df350]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:476) ~[spring-core-5.2.0.BUILD-SNAPSHOT.jar:5.2.0.BUILD-SNAPSHOT]
at org.springframework.util.ReflectionUtils.findMethod(ReflectionUtils.java:239) ~[spring-core-5.2.0.BUILD-SNAPSHOT.jar:5.2.0.BUILD-SNAPSHOT]
at org.springframework.cloud.deployer.thin.ThinJarAppWrapper.runContext(ThinJarAppWrapper.java:132) ~[spring-cloud-deployer-thin-1.0.22.RELEASE.jar:na]
at org.springframework.cloud.deployer.thin.ThinJarAppWrapper.run(ThinJarAppWrapper.java:90) ~[spring-cloud-deployer-thin-1.0.22.RELEASE.jar:na]
at org.springframework.cloud.deployer.thin.AbstractThinJarSupport.deploy(AbstractThinJarSupport.java:60) [spring-cloud-deployer-thin-1.0.22.RELEASE.jar:na]
at org.springframework.cloud.deployer.thin.ThinJarAppDeployer.deploy(ThinJarAppDeployer.java:63) [spring-cloud-deployer-thin-1.0.22.RELEASE.jar:na]
at org.springframework.cloud.launcher.deployer.Deployer.deployInternal(Deployer.java:203) [spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.cloud.launcher.deployer.Deployer.deploy(Deployer.java:108) [spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.cloud.launcher.deployer.DeployerApplication.launch(DeployerApplication.java:121) [spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.cloud.launcher.deployer.DeployerApplication.run(DeployerApplication.java:64) [spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.cloud.launcher.deployer.DeployerApplication.main(DeployerApplication.java:54) [spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [spring-boot-thin-launcher-1.0.22.RELEASE-exec.jar:1.0.22.RELEASE]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [spring-boot-thin-launcher-1.0.22.RELEASE-exec.jar:1.0.22.RELEASE]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [spring-boot-thin-launcher-1.0.22.RELEASE-exec.jar:1.0.22.RELEASE]
at org.springframework.boot.loader.thin.ThinJarLauncher.launch(ThinJarLauncher.java:193) [spring-boot-thin-launcher-1.0.22.RELEASE-exec.jar:1.0.22.RELEASE]
at org.springframework.boot.loader.thin.ThinJarLauncher.main(ThinJarLauncher.java:140) [spring-boot-thin-launcher-1.0.22.RELEASE-exec.jar:1.0.22.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
at org.springframework.boot.loader.wrapper.ThinJarWrapper.launch(ThinJarWrapper.java:140) [spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.boot.loader.wrapper.ThinJarWrapper.main(ThinJarWrapper.java:107) [spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
at org.springframework.cloud.launcher.cli.LauncherCommand$LauncherOptionHandler.run(LauncherCommand.java:115) [spring-cloud-launcher-cli-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.boot.cli.command.options.OptionHandler.run(OptionHandler.java:85) [classes!/:2.1.7.RELEASE]
at org.springframework.boot.cli.command.OptionParsingCommand.run(OptionParsingCommand.java:54) [classes!/:2.1.7.RELEASE]
at org.springframework.boot.cli.command.CommandRunner.run(CommandRunner.java:219) [classes!/:2.1.7.RELEASE]
at org.springframework.boot.cli.command.CommandRunner.runAndHandleErrors(CommandRunner.java:171) [classes!/:2.1.7.RELEASE]
at org.springframework.boot.cli.SpringCli.main(SpringCli.java:64) [classes!/:2.1.7.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [spring-boot-thin-launcher-1.0.22.RELEASE-exec.jar:1.0.22.RELEASE]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [spring-boot-thin-launcher-1.0.22.RELEASE-exec.jar:1.0.22.RELEASE]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) [spring-boot-thin-launcher-1.0.22.RELEASE-exec.jar:1.0.22.RELEASE]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52) [spring-boot-cli-2.1.7.RELEASE.jar:2.1.7.RELEASE]
Caused by: java.lang.NoClassDefFoundError: org/springframework/context/ConfigurableApplicationContext
at java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.8.0_211]
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) ~[na:1.8.0_211]
at java.lang.Class.getDeclaredMethods(Class.java:1975) ~[na:1.8.0_211]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:458) ~[spring-core-5.2.0.BUILD-SNAPSHOT.jar:5.2.0.BUILD-SNAPSHOT]
... 43 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.springframework.context.ConfigurableApplicationContext
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[na:1.8.0_211]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_211]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_211]
... 47 common frames omitted
And 2-nd exception:
o.s.cloud.launcher.cli.LauncherCommand : Error running spring cloud
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
at org.springframework.cloud.launcher.cli.LauncherCommand$LauncherOptionHandler.run(LauncherCommand.java:115) ~[na:na]
at org.springframework.boot.cli.command.options.OptionHandler.run(OptionHandler.java:85) [classes!/:2.1.7.RELEASE]
at org.springframework.boot.cli.command.OptionParsingCommand.run(OptionParsingCommand.java:54) [classes!/:2.1.7.RELEASE]
at org.springframework.boot.cli.command.CommandRunner.run(CommandRunner.java:219) [classes!/:2.1.7.RELEASE]
at org.springframework.boot.cli.command.CommandRunner.runAndHandleErrors(CommandRunner.java:171) [classes!/:2.1.7.RELEASE]
at org.springframework.boot.cli.SpringCli.main(SpringCli.java:64) [classes!/:2.1.7.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [spring-boot-cli-2.1.7.RELEASE.jar:2.1.7.RELEASE]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [spring-boot-cli-2.1.7.RELEASE.jar:2.1.7.RELEASE]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) [spring-boot-cli-2.1.7.RELEASE.jar:2.1.7.RELEASE]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52) [spring-boot-cli-2.1.7.RELEASE.jar:2.1.7.RELEASE]
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
at org.springframework.boot.loader.wrapper.ThinJarWrapper.launch(ThinJarWrapper.java:140) ~[spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.boot.loader.wrapper.ThinJarWrapper.main(ThinJarWrapper.java:107) ~[spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
... 18 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [spring-boot-cli-2.1.7.RELEASE.jar:2.1.7.RELEASE]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [spring-boot-cli-2.1.7.RELEASE.jar:2.1.7.RELEASE]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [spring-boot-cli-2.1.7.RELEASE.jar:2.1.7.RELEASE]
at org.springframework.boot.loader.thin.ThinJarLauncher.launch(ThinJarLauncher.java:193) ~[spring-boot-thin-launcher-1.0.22.RELEASE.jar:1.0.22.RELEASE]
at org.springframework.boot.loader.thin.ThinJarLauncher.main(ThinJarLauncher.java:140) ~[spring-boot-thin-launcher-1.0.22.RELEASE.jar:1.0.22.RELEASE]
... 24 common frames omitted
Caused by: java.lang.IllegalStateException: Failed to introspect Class [org.springframework.cloud.deployer.thin.ContextRunner] from ClassLoader [java.net.URLClassLoader#430df350]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:476) ~[spring-core-5.1.9.RELEASE.jar!/:5.1.9.RELEASE]
at org.springframework.util.ReflectionUtils.findMethod(ReflectionUtils.java:239) ~[spring-core-5.1.9.RELEASE.jar!/:5.1.9.RELEASE]
at org.springframework.util.ReflectionUtils.findMethod(ReflectionUtils.java:218) ~[spring-core-5.1.9.RELEASE.jar!/:5.1.9.RELEASE]
at org.springframework.cloud.deployer.thin.ThinJarAppWrapper.isRunning(ThinJarAppWrapper.java:118) ~[spring-cloud-deployer-thin-1.0.22.RELEASE.jar:na]
at org.springframework.cloud.deployer.thin.ThinJarAppWrapper.getState(ThinJarAppWrapper.java:218) ~[spring-cloud-deployer-thin-1.0.22.RELEASE.jar:na]
at org.springframework.cloud.deployer.thin.InMemoryAppInstanceStatus.getState(ThinJarAppDeployer.java:206) ~[spring-cloud-deployer-thin-1.0.22.RELEASE.jar:na]
at org.springframework.cloud.deployer.spi.app.AppStatus.getState(AppStatus.java:92) ~[spring-cloud-deployer-spi-1.1.4.RELEASE.jar:1.1.4.RELEASE]
at org.springframework.cloud.launcher.deployer.Deployer.getAppStatus(Deployer.java:251) ~[spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.cloud.launcher.deployer.Deployer.deployInternal(Deployer.java:204) ~[spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.cloud.launcher.deployer.Deployer.deploy(Deployer.java:108) ~[spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.cloud.launcher.deployer.DeployerApplication.launch(DeployerApplication.java:121) ~[spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.cloud.launcher.deployer.DeployerApplication.run(DeployerApplication.java:64) ~[spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
at org.springframework.cloud.launcher.deployer.DeployerApplication.main(DeployerApplication.java:54) ~[spring-cloud-launcher-deployer-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
... 33 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/springframework/context/ConfigurableApplicationContext
at java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.8.0_211]
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) ~[na:1.8.0_211]
at java.lang.Class.getDeclaredMethods(Class.java:1975) ~[na:1.8.0_211]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:458) ~[spring-core-5.1.9.RELEASE.jar!/:5.1.9.RELEASE]
... 45 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.springframework.context.ConfigurableApplicationContext
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[na:1.8.0_211]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_211]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_211]
... 49 common frames omitted
in m2/repository settings.xml file has next text:
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<pluginGroups>
</pluginGroups>
<proxies>
</proxies>
<mirrors>
</mirrors>
<profiles>
</profiles>
</settings>
What I need to do for starting stubrunner server voa Spring Cloud CLI?
Spring cloud cli 2.2.0 does not work with 2.3.1 spring boot CLI, try:
Spring boot CLI 2.3.1.RELEASE
Spring cloud CLI 2.2.1.RELEASE
I have a dse graph in validation/prod environement.
The problem occurs when I try to launch a DSEGraphFrame query using Spark in Scala.
val graph = spark.dseGraph("my_graph")
generates the following exception:
Exception in thread "main"
com.datastax.driver.core.exceptions.InvalidQueryException: The method
DseGraphRpc.getSchemaBlob does not exist. Make sure that the required
component for that method is active/enabled
at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:40)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:26)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:284)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.datastax.spark.connector.cql.SessionProxy.invoke(SessionProxy.scala:37)
at com.sun.proxy.$Proxy27.execute(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.datastax.spark.connector.cql.SessionProxy.invoke(SessionProxy.scala:37)
at com.sun.proxy.$Proxy28.execute(Unknown Source)
at com.datastax.bdp.util.rpc.RpcUtil.callInternal(RpcUtil.java:57)
at com.datastax.bdp.util.rpc.RpcUtil.call(RpcUtil.java:40)
at com.datastax.bdp.graph.spark.DseGraphRpc.callGetSchema(DseGraphRpc.java:45)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrame$$anonfun$getSchemaFromServer$1.apply(DseGraphFrame.scala:586)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrame$$anonfun$getSchemaFromServer$1.apply(DseGraphFrame.scala:586)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:115)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$withSessionDo$1.apply(CassandraConnector.scala:114)
at com.datastax.spark.connector.cql.CassandraConnector.closeResourceAfterUse(CassandraConnector.scala:158)
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:114)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrame$.getSchemaFromServer(DseGraphFrame.scala:586)
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.apply(DseGraphFrameBuilder.scala:257)
at com.datastax.bdp.graph.spark.graphframe.SparkSessionFunctions.dseGraph(SparkSessionFunctions.scala:20)
What could I do to run DSEGraphFrame properly?
The problem comes from a node in dse cluster wich the graph isn't activated
I'm trying to connect to hbase from Pyspark using SHC API by referring the below link.
https://community.hortonworks.com/questions/143802/read-hbase-with-pyspark-from-jupyter-notebook.html
Sample code:
spark = SparkSession.builder.appName("Hbase Read").getOrCreate()
data_source_format = 'org.apache.spark.sql.execution.datasources.hbase'
catalog = ''.join("""{
"table":{"namespace":"default", "name":"table"},
"rowkey":"key",
"columns":{
"firstcol":{"cf":"rowkey", "col":"key", "type":"string"},
"secondcol":{"cf":"cf", "col":"col1", "type":"int"}
}
}""".split())
df = spark.read \
.options(catalog=catalog) \
.format(data_source_format) \
.load()
df.show()
Spark-submit:
spark-submit --packages com.hortonworks:shc:1.1.1-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/ TestHbaseRead.py
i'm getting this error.
Error log:
: java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.execution.datasources.hbase. Please find packages at http://spark.apache.org/third-party-projects.html
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:190)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.execution.datasources.hbase.DefaultSource
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23$$anonfun$apply$15.apply(DataSource.scala:618)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23$$anonfun$apply$15.apply(DataSource.scala:618)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23.apply(DataSource.scala:618)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23.apply(DataSource.scala:618)
at scala.util.Try.orElse(Try.scala:84)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:618)
... 13 more
I am using a fastutill ObjectArrayList[object] instead of ArrayBuffer in scala in my spark streamign applicatiuon (inside a mapwithstae function)
I use the lates version..using below sbt
libraryDependencies += "it.unimi.dsi" % "fastutil" % "7.0.13"
However, it throws an exception
yarn.ApplicationMaster: User class threw exception: java.lang.NoClassDefFoundError: it/unimi/dsi/fastutil/objects/ObjectArrayList
java.lang.NoClassDefFoundError: it/unimi/dsi/fastutil/objects/ObjectArrayList
at MapRRecoverableConsumer$$anonfun$5.apply(MapRRecoverableConsumer.scala:756)
at MapRRecoverableConsumer$$anonfun$5.apply(MapRRecoverableConsumer.scala:293)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:864)
at MapRRecoverableConsumer$.main(MapRRecoverableConsumer.scala:863)
at MapRRecoverableConsumer.main(MapRRecoverableConsumer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
Caused by: java.lang.ClassNotFoundException: it.unimi.dsi.fastutil.objects.ObjectArrayList
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Ok..just added as jars parameter in spark submit and it resolved that..