I am new to Scala.
Can any one suggest me what are the jar files required for running Apache Spark with Scala in Linux environment.Below code was a piece of original code. I am getting exceptions like java.lang.NoSuchMethodError: org.jboss.netty.channel.socket.nio.NioWorkerPool.(Ljava/util/concurrent/Executor;I)V
java -cp ".:/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p1876.1944/jars/:./"
TestAll.scala
import org.apache.spark.SparkContext._
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.SQLContext
import java.io._
import java.sql.{Connection,DriverManager}
import scala.collection._
import scala.collection.mutable.MutableList
object TestAll {
def main(args: Array[String]) {
val conf =new SparkConf().setAppName("Testing App").setMaster("local")
val sc=new SparkContext(conf)
println("Hello, world!")
}
}
You need to download Spark from here. Choose the "Pre-built with Hadoop" option. Then you can follow the directions of the Quick Start. This will get you through the Hello World. I am not sure which IDE you are using, but the most friendly for Scala is Intellij IDEA
Related
I'm working on a scala source code, but when compiling, it doesn't find this import:
I'm new to scala, how can I bring this function to my scala (ExporterUtil)?
import org.joda.time.DateTime
import org.joda.time.format.DateTimeFormat
import scala.collection.mutable.ArrayBuffer
import br.net.empresa.digital.exporter.ExporterUtil
When compiling, it doesn't find this directory (br.net.empresa.digital.exporter.ExporterUtil), how can I bring this dir to my folder?
I am trying to write one simple program in Scala but when I use SparkContext in Intellij this is throwing an error. Can someone give me any solution?
Scala 3.1.1
Spark version 3.2.1
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
object Wordcount extends App {
val sc = new SparkContext("Local[*]","wordcount")
}
I am trying to use apache spark in Scala in IntelliJ. I am importing spark like this
import org.apache.spark._
import org.apache.spark.SparkContext._
import org.apache.spark.sql._
import org.apache.log4j._
however when I build my project I receive the error object apache is not a member of package org import org.apache.spark._
how can I fix this error?
I have started to build a small scala code.
It will be expanded to open SparkContext and do some data processing.
As of now I have only imported 3 spark libraries to run a simple code.
I am getting error "error: object apache is not a member of package org".
Question is: How can I compile manually using scalac so that the compilation process can include the Spark libraries - without Maven or sbt?
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
object HelloWorld {
def main(args: Array[String]) {
println("Hello, world!")
}
}
my SPARK version is spark--version : 2.3.2.
while importing
import sqlContext.implicits
i am getting error :
cannot resolve symbol sqlcontext
i am using Intellij and scala
Scala version 2.11.8
Kindly share your thoughts.
The import you're trying will not work because the object is defined within the class SQLContext
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
Take a look at Why import implicit SqlContext.implicits._ after initializing SQLContext in a scala spark application
Hope this helps!
You have to create the sqlContext object first from the SQLContext, see https://spark.apache.org/docs/2.3.0/api/java/org/apache/spark/sql/SQLContext.implicits$.html. Otherwise, as it says, it doesn't know about such an objectl