I started new project using sbt new vert-x3/vertx-scala.g8 command. In sbt console entered following command :
vertx.deployVerticle(nameForVerticle[HttpVerticle])
The following error is reported:
vertx.deployVerticle(nameForVerticle[HttpVerticle])
<console>:12: error: not found: value vertx
vertx.deployVerticle(nameForVerticle[HttpVerticle])
^
<console>:12: error: not found: value nameForVerticle
vertx.deployVerticle(nameForVerticle[HttpVerticle])
^
<console>:12: error: not found: type HttpVerticle
Followed the steps specified on this page: https://github.com/vert-x3/vertx-sbt-starter
How to get sample project running?
I think g8 template is a little bit broken. I made it work using below tricks:
Use latest SBT version 1.2.8 in file: project/build.properties
When you run console, import HttpVerticle class manually. In my case, I have test.HttpVerticle as class name. Because I used package name as 'test', when I was running SBT new command to initialise the project
scala> import test.HttpVerticle
scala> vertx.deployVerticle(nameForVerticle[HttpVerticle])
// This stuff is going to printed in a second:
scala> Thread Thread[vert.x-eventloop-thread-0,5,run-main-group-0] has been blocked for 2377 ms, time limit is 2000
Thread Thread[vert.x-eventloop-thread-0,5,run-main-group-0] has been blocked for 3378 ms, time limit is 2000
Thread Thread[vert.x-eventloop-thread-0,5,run-main-group-0] has been blocked for 4384 ms, time limit is 2000
And then try to trigger the server:
curl http://localhost:8666/hello
It should reply with "world".
Again, as for class name. If you didn't use any package name when running sbt new initialisation process, then try just import class like this: import HttpVerticle
Related
I am using hadoop 2.7.2 , hbase 1.4.9, spark 2.2.0, scala 2.11.8 and java 1.8 .
I run this command whithout having any error:
val Patterns_fromHbase = mimic_PatternsFromHbase.mapPartitions(f=> f.map(row1 => (Bytes.toString(row1._2.getRow), Bytes.toString(row1._2.getValue(Bytes.toBytes("sepsiscategories"),Bytes.toBytes("subject_id")))))).toDF("id","subject_id")
Then I run this command :
mimic_PatternsFromHbase.createOrReplaceTempView("subject_id_table")
and I just have this error:
:57: error: value createOrReplaceTempView is not a member of
org.apache.spark.rdd.RDD[(org.apache.hadoop.hbase.io.ImmutableBytesWritable,
org.apache.hadoop.hbase.client.Result)]
mimic_PatternsFromHbase.createOrReplaceTempView("subject_id_table")
what is the cause of this error and how to fix it please
I found my fault, it is a fault of bad attention.
in the place to call createOrReplaceView method by calling them by Patterns_fromHbase I called her by mimic_PatternsFromHbase
Running basic df.show() post spark notebook installation
I am getting the following error when running scala - spark code on spark-notebook. Any idea when this occurs and how to avoid?
[org.apache.spark.repl.ExecutorClassLoader] Failed to check existence of class org.apache.spark.sql.catalyst.expressions.Object on REPL class server at spark://192.168.10.194:50935/classes
[org.apache.spark.util.Utils] Aborting task
[org.apache.spark.repl.ExecutorClassLoader] Failed to check existence of class org on REPL class server at spark://192.168.10.194:50935/classes
[org.apache.spark.util.Utils] Aborting task
[org.apache.spark.repl.ExecutorClassLoader] Failed to check existence of class
I installed the spark on local, and when I was using following code it was giving me the same error.
spark.read.format("json").load("Downloads/test.json")
I think the issue was, it was trying to find some master node and taking some random or default IP. I specified the mode and then provided the IP as 127.0.0.1 and it resolved my issue.
Solution
Run the spark using local master
usr/local/bin/spark-shell --master "local[4]" --conf spark.driver.host=127.0.0.1'
The Spark I connected to, is not built on my local computer but a remote one. Everytime when I connect to it http://xx.xxx.xxx.xxx:10000/, the error says:
[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /usr/local/spark/python/pyspark/shell.py:
18/03/07 08:52:53 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Anyways, I still keep trying to run it on Jupyter notebook:
from pyspark.conf import SparkConf
SparkSession.builder.config(conf=SparkConf())
dir(spark)
When I ran it yesterday, it shows directory. when I did it today, it says:
NameError: name 'spark' is not defined
Any suggestion is appreciated!
you re missing the spark variable
from pyspark.conf import SparkConf
spark=SparkSession.builder.config(conf=SparkConf())
dir(spark)
Sbt seems to be using different classloaders, making some tests failing when run more than once in an sbt session, with the following error:
[info] java.lang.ClassCastException: net.i2p.crypto.eddsa.EdDSAPublicKey cannot be cast to net.i2p.crypto.eddsa.EdDSAPublicKey
[info] at com.advancedtelematic.libtuf.crypt.EdcKeyPair$.generate(RsaKeyPair.scala:120)
I tried equivalent code using pattern matching instead of asInstanceOf and I get the same result.
How can I make sure sbt uses the same class loader for all test executions in the same session?
I think it's related to this: Do security providers cause ClassLoader leaks in Java?. Basically Security is re-using providers from old class-loaders. So this could happen in any multi-classpath environment (like OSGi), not just SBT.
Fix for your build.sbt (without forking):
testOptions in Test += Tests.Cleanup(() =>
java.security.Security.removeProvider("BC"))
Experiment:
sbt-classloader-issue$ sbt
> test
[success] Total time: 1 s, completed Jul 6, 2017 11:43:53 PM
> test
[success] Total time: 0 s, completed Jul 6, 2017 11:43:55 PM
Explanation:
As I can see from your code (published here):
Security.addProvider(new BouncyCastleProvider)
you're reusing the same BouncyCastleProvider provider every-time you run a test, as your Security.addProvider works only first time. As sbt creates new class-loader for every "test" run, but re-uses the same JVM - Security is kind-of JVM-scoped singleton as it was loaded by JVM-bootstrap, so classOf[java.security.Security].getClassLoader() == null and sbt cannot reload/reinitialize this class.
And you can easily check that
classOf[org.bouncycastle.jce.spec.ECParameterSpec].getClassLoader()
res30: ClassLoader = URLClassLoader with NativeCopyLoader with RawResources
org.bouncycastle classes are loaded with custom classloader (from sbt) which changes every-time you run test.
So this code:
val generator = KeyPairGenerator.getInstance("ECDSA", "BC")
gets instance of class loaded from old classloader (the one used for first "test" run) and you're trying to initialize it with spec from new classloader:
generator.initialize(ecSpec)
That's why you're getting "parameter object not a ECParameterSpec" exception. The reasoning around "net.i2p.crypto.eddsa.EdDSAPublicKey cannot be cast to net.i2p.crypto.eddsa.EdDSAPublicKey" is basically same.
Using sbt package I have the following error
Spark Scala error while loading BytesWritable, invalid LOC header (bad signature)
My code is
....
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
......
object Test{
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Test")
val sc = new SparkContext(conf) // the error is due by this
......
}
}
Pls re-load your JARs and / or library dependencies as they might be corrupted while building jar through sbt - could be issue with one of their update. Second alternative is that you have too many temp files open, check your 4040-9 ports on master if there are any jobs hanging and kill them if so, you can also check how increase open files you have on linux:/etc/security/limits.conf where hard nofile ***** and soft nofile ***** then reboot and ulimit -n ****
I was using spark-mllib_2.11 and it gave me the same error. I had to use version 2.10 of Spark MLIB to get rid of it.
Using Maven:
<artifactId>spark-mllib_2.10</artifactId>