I have been using spark code inside scala future.
I am facing string error like Assertion Failed <None>.
This comes encapsulated on Boxed error.
Each time stack trace starts with sqlContext.udf.register line.
When I put synchonized block around udf register statement, Error goes away.
Scala version - 2.10.8
Spark - 1.6.0
Related
I am using hadoop 2.7.2 , hbase 1.4.9, spark 2.2.0, scala 2.11.8 and java 1.8 .
I run this command whithout having any error:
val Patterns_fromHbase = mimic_PatternsFromHbase.mapPartitions(f=> f.map(row1 => (Bytes.toString(row1._2.getRow), Bytes.toString(row1._2.getValue(Bytes.toBytes("sepsiscategories"),Bytes.toBytes("subject_id")))))).toDF("id","subject_id")
Then I run this command :
mimic_PatternsFromHbase.createOrReplaceTempView("subject_id_table")
and I just have this error:
:57: error: value createOrReplaceTempView is not a member of
org.apache.spark.rdd.RDD[(org.apache.hadoop.hbase.io.ImmutableBytesWritable,
org.apache.hadoop.hbase.client.Result)]
mimic_PatternsFromHbase.createOrReplaceTempView("subject_id_table")
what is the cause of this error and how to fix it please
I found my fault, it is a fault of bad attention.
in the place to call createOrReplaceView method by calling them by Patterns_fromHbase I called her by mimic_PatternsFromHbase
I am getting the error "Uncaught Exception java.lang.stack overflow error null in Jmeter while performing only one thread. Initially, all requests passed, when I try to click on 11th request to see the HTML response type it starts throwing errors. Since that page contains more images with search functionality
You can run the Java VM with a larger stack size by applying a flag like -Xss1024k. From what I found, the default is 1024k (1 MB).
For example:
$ java -Xss2048k -jar myfile.jar
or
$ java -Xss256m -jar myfile.jar
The alternative is to optimize your code. However,
Premature optimization is the root of all evil.
- Donald Knuth
I am trying to write a program that enables me to run predefined KSQL operations on Kafka topics in Scala, but I don't want to open the KSQL Cli everytime. Therefore I want to start the KSQL "Server" from within my Scala program. If I understand the KSQL source code right, I have to build and start a KsqlRestApplication:
def restServer = KsqlRestApplication.buildApplication(new
KsqlRestConfig(defaultServerProperties), true, new VersionCheckerAgent
{override def start(ksqlModuleType: KsqlModuleType, properties:
Properties): Unit = ???})
But when I try doing that, I get the following error:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.kafka.streams.StreamsConfig.getConsumerConfigs(Ljava/lang/String;Ljava/lang/String;)Ljava/util/Map;
at io.confluent.ksql.rest.server.BrokerCompatibilityCheck.create(BrokerCompatibilityCheck.java:62)
at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:241)
I looked into the function call in BrokerCompatibilityCheck and in the create function it calls the StreamsConfig.getConsumerConfigs() with 2 Strings as parameters instead of the parameters defined in
https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/StreamsConfig.html#getConsumerConfigs(StreamThread,%20java.lang.String,%20java.lang.String).
Are my KSQL and Kafka version simply not compatible or am I doing something wrong?
I am using KSQL version 4.1.0-SNAPSHOT and Kafka version 1.0.0.
Yes, NoSuchMethodError typically indicates a version incompatibility between libraries.
The link you posted is to javadoc for kafka 0.10.2. The method hasn't changed in 1.0 but indeed in the upcoming 1.1 it only takes 2 Strings:
https://kafka.apache.org/11/javadoc/org/apache/kafka/streams/StreamsConfig.html#getConsumerConfigs(java.lang.String,%20java.lang.String)
. That suggests the version of KSQL you're using (4.1.0-SNAPSHOT) depends on version 1.1 of kafka streams, which is currently in the release candidate phase and I believe and should be out soon:
https://lists.apache.org/thread.html/780c4458b16590e99261b69d7b41b9ec374a3226d72c8d38885a008a#%3Cusers.kafka.apache.org%3E
As per that email you can find the latest (1.1.0-rc2) artifacts in the apache staging repo:
https://repository.apache.org/content/groups/staging/
I have this issue
The initialization of the DataSource's outputs caused an error: The UDF class is not a proper subclass of org.apache.flink.api.common.functions.MapFunction
generated by this code:
val probes: DataSet[Probe] = env.createInput[InputProbe](new ProbesInputFormat).map { i =>
new Probe(
i.rssi,
0,
i.macHash,
i.deviceId,
0,
i.timeStamp)
}
I'm using scala 2.11 on flink 1.4.0 with IDEA.
On Dev machine i have no issue and the job runs properly, while on a Flink Standalone Cluster of 3 nodes i encountered the above error.
Can you help me please ;(
UPDATE:
I resolved implementing a class that extends from RichMapFunction, i don't know why but seems that lambda function => are not supported properly.
Now i have a new issue:
java.lang.ClassCastException: hk.huko.aps2.entities.Registry cannot be cast to scala.Product
Should i open a new POST?
I resolved the issue. It happened because flink load my job JAR many times (classloader) and in somehow it produced that error.
The solution is to not creare a JAR including all external JARs dependencies, but to copy into flink/lib folder those libraries plus your job JAR.
Was trying to implement the triggers in Cassandra.
Have been trying to use the available Cassandra help: https://github.com/hmsonline/cassandra-triggers
Porting it on the latest version 1.2.3, and following the GettingStarted instruction.
when i tried to set the value for the trigger, inserting the data to fire the log, and checked the logs, following is the error which i received
java.lang.AssertionError
at org.apache.cassandra.thrift.ThriftSessionManager.currentSession(ThriftSessionManager.java:51)
at org.apache.cassandra.thrift.CassandraServer.state(CassandraServer.java:88)
at org.apache.cassandra.thrift.CassandraServer.validateLogin(CassandraServer.java:881)
at org.apache.cassandra.thrift.CassandraServer.set_keyspace(CassandraServer.java:1492)
at com.hmsonline.cassandra.triggers.dao.CassandraStore.getConnection(CassandraStore.java:42)
at com.hmsonline.cassandra.triggers.dao.ConfigurationStore.getConfiguration(ConfigurationStore.java:76)
at com.hmsonline.cassandra.triggers.dao.ConfigurationStore.isCommitLogEnabled(ConfigurationStore.java:44)
at com.hmsonline.cassandra.triggers.TriggerTask.run(TriggerTask.java:47)
at java.lang.Thread.run(Thread.java:636)
But we test it works fine at older version(ex: 1.1.2)
So is this a configuration problem, or the Thrift API implementation has been changed?
thanks