Is there any specific Spark Job Server version matching with Spark 1.6.0 ?
As per the version information in https://github.com/spark-jobserver/spark-jobserver, I see SJS is available only for 1.6.1 not for 1.6.0.
Our CloudEra hosted Spark is running on 1.6.0
I deployed SJS by configuring the spark home to 1.6.1. When I submitted jobs, I see job ids are getting generated but I can't see the job result.
Any inputs?
No, there is no SJS version tied to spark 1.6.0. But it should be easy for you to compile against 1.6.0. May be you could modify this https://github.com/spark-jobserver/spark-jobserver/blob/master/project/Versions.scala#L10 and try.
Related
Spark-Cassandra experts: Will Apache Spark 1.4 work with Apache Cassandra 3.0 in Datastax installations?. We are considering several options for migrating DSE 4.8 (Spark 1.4 and Cassandra 2.1) to DSE 5.0 (Spark 1.6 and Cassandra 3.0). One option is to update Cassandra Cluster to DSE 5.0 and leave Spark cluster on DSE 4.8. This means we have to make Apache Spark 1.4 work with Apache Cassandra 3.0. We use https://github.com/datastax/spark-cassandra-connector versions 1.4 (DSE 4.8) and 1.6(DSE 5.0). Has someone tried using Spark 1.4 (DSE 4.8) with Cassandra 3.0 (DSE 5.0) ?.
As I can see from the Maven Central, Spark Cassandra Connector 1.4.5 did use the version 2.1.7 of the Java driver. According the compatibility matrix in official documentation, the driver 2.1.x won't work with Cassandra 3.0... You can of course test it, but I doubt that it will work - driver is usually backward compatible, but not forward compatible...
I recommend to perform migration to DSE 5.0, and then move to 5.1 fast enough, as 5.0 could be EOL soon.
P.S. If you have more questions, I recommend to join the DataStax Academy Slack - there is a separate channel about spark cassandra connector there.
I have Apache Spark installed on a cluster. I can run spark-shell on the cluster master node. So, it means there is the scala installed to this machine. However, I cannot start neither sbt nor scalac. Is it possible to obtain spark's scala and how to do it?
No, Its not. You have to install manually.
Go through these links.
https://www.scala-lang.org/download/
https://www.scala-sbt.org/1.0/docs/Installing-sbt-on-Linux.html
I am using the KafkaConsumer82 and connector jar version is 0.10.2 and kafka version is 0.9.1 and flink version is 1.0.0.
The Java consumer works fine when I run it from with in an IDE as a standalone main program. But when I run it from flink run, then I don't see the messages being consumed and don't see any log in stdout of the JobManager in 1ocalhost:8081. Please let me know what might be the issue.
As a first step I would suggest getting the versions in sync. If you're using Kafka 0.9 and Flink 1.0.0 I would suggest using flink-connector-kafka-0.9 version 1.0.0 which contains FlinkKafkaConsumer09.
I am using the Hortonworks release which uses Spark 1.3.
When I ran my app using Spark API 1.5 it failed, but switching to 1.3 worked.
There are new features in 1.5 that I want to use on the server but apparently this is not possible without upgrading it.
How do I upgrade Spark from 1.3 to 1.5 on Ambari?
While launching standalone cluster on spark streaming, I am not able to find ./bin/spark-class command.
Please let me know, if I need to do any additional configurations for getting "spark-class".
Which version of Spark are you using? Starting with Spark 0.9.0, spark-class is located in the bin folder, but in earlier versions it was at the root of SPARK_HOME.
Perhaps you're following instructions for Spark 0.9.0+ even though you've installed an earlier version of Spark? You can find documentation for older releases of Spark on the Spark documentation overview page.