I have the following code:
package db
import com.mongodb.casbah.{ MongoCollection, MongoConnection, MongoDB }
import com.mongodb.casbah.commons.{ MongoDBObject }
object Test {
def main(argc: Array[String]) {
val db = MongoDB(MongoConnection(), "test")
val objText = MongoDBObject("user" -> "This the text is test number the two")
val coll = db.getCollection("coll_tet")
coll.insert(objText)
print(coll.find())
}
}
And I do scalac -classpath ... test.scala
When I tried to run this: scala db.Test -classpath ... I'm get an error:
Exception in thread "main" java.lang.NoClassDefFoundError: com/mongodb/Mongo
Why is this happening?
Why don't you just add the Imports object?
import com.mongodb.casbah.Imports._
And it should take care of all your import problems.
Related
When I build my sbt project in Intellij I get the following error:
Error:(7, 8) object DfUtils is not a member of package com.naturalint.xspark.common.sparkutils
import com.naturalint.xspark.common.sparkutils.DfUtils
The object is a part of the package and when I ran sbt compile or package or assembly in terminal everything clears fine. When I move the object to a different location in the project it sometimes work for a day and then again gives the error
This is the content of the file:
package com.naturalint.xspark.common.sparkutils
import org.apache.spark.sql._
import scala.reflect._
object DfUtils {
def removeColumnThatAreNotInEncoder[T <: Product : ClassTag: Encoder](dataEntity: Dataset[T]): Dataset[T] = {
val declaredFields: Array[String] = classTag[T].runtimeClass.getDeclaredFields.map(x => x.getName)
val columns: Array[Column] = declaredFields.map(x => new Column(x))
dataEntity.select(columns: _*).as[T]
}
}
Any thoughts?
Thanks
Nir
I have a small Scala code which works properly on Spark-Shell but not in Eclipse with Scala plugin. I can access hdfs using plugin tried writing another file and it worked..
FirstSpark.scala
package bigdata.spark
import org.apache.spark.SparkConf
import java. io. _
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
object FirstSpark {
def main(args: Array[String])={
val conf = new SparkConf().setMaster("local").setAppName("FirstSparkProgram")
val sparkcontext = new SparkContext(conf)
val textFile =sparkcontext.textFile("hdfs://pranay:8020/spark/linkage")
val m = new Methods()
val q =textFile.filter(x => !m.isHeader(x)).map(x=> m.parse(x))
q.saveAsTextFile("hdfs://pranay:8020/output") }
}
Methods.scala
package bigdata.spark
import java.util.function.ToDoubleFunction
class Methods {
def isHeader(s:String):Boolean={
s.contains("id_1")
}
def parse(line:String) ={
val pieces = line.split(',')
val id1=pieces(0).toInt
val id2=pieces(1).toInt
val matches=pieces(11).toBoolean
val mapArray=pieces.slice(2, 11).map(toDouble)
MatchData(id1,id2,mapArray,matches)
}
def toDouble(s: String) = {
if ("?".equals(s)) Double.NaN else s.toDouble
}
}
case class MatchData(id1: Int, id2: Int,
scores: Array[Double], matched: Boolean)
Error Message:
Exception in thread "main" org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2032)
at org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:335)
at org.apache.spark.rdd.RDD$$anonfun$filter$1.apply(RDD.scala:334)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
Can anyone please help me with this
Try changing class Methods { .. } to object Methods { .. }.
I think the problem is at val q =textFile.filter(x => !m.isHeader(x)).map(x=> m.parse(x)). When Spark sees the filter and map functions it tries to serialize the functions passed to them (x => !m.isHeader(x) and x=> m.parse(x)) so that it can dispatch the work of executing them to all of the executors (this is the Task referred to). However, to do this, it needs to serialize m, since this object is referenced inside the function (it is in the closure of the two anonymous methods) - but it cannot do this since Methods is not serializable. You could add extends Serializable to the Methods class, but in this case an object is more appropriate (and is already Serializable).
In a Play app, using Salat and Casbah, I am trying to de-serialize a DBObject into an object of type Task, but I am getting this error when calling .asObject:
No implicit view available from com.mongodb.casbah.Imports.DBObject =>
com.mongodb.casbah.Imports.MongoDBObject. Error occurred in an
application involving default arguments.
The object is serialized correctly with .asDBObject, and written to the database as expected.
What is causing this behaviour, and what can be done to solve it? Here's the model involved:
package models
import db.{MongoFactory, MongoConnection}
import com.novus.salat._
import com.novus.salat.global._
import com.novus.salat.annotations._
import com.mongodb.casbah.Imports._
import com.mongodb.casbah.commons.Imports._
import play.api.Play
case class Task(label: String, _id: ObjectId=new ObjectId)
object Task {
implicit val ctx = new Context {
val name = "Custom_Classloader"
}
ctx.registerClassLoader(Play.classloader(Play.current))
val taskCollection = MongoFactory.database("tasks")
def create(label: String): Task = {
val task = new Task(label)
val dbObject = grater[Task].asDBObject(task)
taskCollection.save(dbObject)
grater[Task].asObject(dbObject)
}
def all(): List[Task] = {
val results = taskCollection.find()
val tasks = for (item <- results) yield grater[Task].asObject(item)
tasks.toList
}
}
Versions
casbah: "2.8.1"
scala: "2.11.6"
salat: "1.9.9"
Instructions on creating a custom context:
First, define a custom context as
implicit val ctx = new Context { /* custom behaviour */ }
in a package object
Stop importing com.novus.salat.global._
Import your own custom context everywhere instead.
Source: https://github.com/novus/salat/wiki/CustomContext
I'm trying to write a little REST api using Scala, Spray.io, Elastic4s and ElasticSearch.
My ES instance is running with default parameters, I just changed the parameter network.host to 127.0.0.1.
Here is my spray routing definition
package com.example
import akka.actor.Actor
import spray.routing._
import com.example.core.control.CrudController
class ServiceActor extends Actor with Service {
def actorRefFactory = context
def receive = runRoute(routes)
}
trait Service extends HttpService {
val crudController = new CrudController()
val routes = {
path("ads" / IntNumber) {
id =>
get {
ctx =>
ctx.complete(
crudController.getFromElasticSearch
)
}
}
}
}
My crudController :
package com.example.core.control
import com.example._
import org.elasticsearch.action.search.SearchResponse
import scala.concurrent._
import scala.util.{Success, Failure}
import ExecutionContext.Implicits.global
class CrudController extends elastic4s
{
def getFromElasticSearch : String = {
val something: Future[SearchResponse] = get
something onComplete {
case Success(p) => println(p)
case Failure(t) => println("An error has occured: " + t)
}
"GET received \n"
}
}
And a trait elastic4s who is encapsulating the call to elastic4s
package com.example
import com.sksamuel.elastic4s.ElasticClient
import com.sksamuel.elastic4s.ElasticDsl._
import scala.concurrent._
import org.elasticsearch.action.search.SearchResponse
trait elastic4s {
def get: Future[SearchResponse] = {
val client = ElasticClient.remote("127.0.0.1", 9300)
client execute { search in "ads"->"categories" }
}
}
This code runs well, and gives me this output :
[INFO] [03/26/2014 11:41:50.957] [on-spray-can-akka.actor.default-dispatcher-4] [akka://on-spray-can/user/IO-HTTP/listener-0] Bound to localhost/127.0.0.1:8080
But when a try to access to the route "localhost/ads/8" with my browser, the case Failure is always triggered and I got this error output on my intellij console :
An error has occured: org.elasticsearch.transport.RemoteTransportException: [Skinhead][inet[/127.0.0.1:9300]][search]
(No console output with elasticSearch running on my terminal)
Is this exception related to ElasticSearch, or am I doing wrong with my Future declaration ?
I suppose you should use ElasticClient.local in this case, as specified in elastic4s docs:
https://github.com/sksamuel/elastic4s
To specify settings for the local node you can pass in a settings object like this:
val settings = ImmutableSettings.settingsBuilder()
.put("http.enabled", false)
.put("path.home", "/var/elastic/")
val client = ElasticClient.local(settings.build)
I've defined a custom input and output format over in XMLIO.scala:
import scala.xml.Node
import org.apache.hadoop.lib.input.FileInputFormat
import org.apache.hadoop.lib.output.FileOutputFormat
import org.apache.hadoop.mapreduce.{ RecordReader, RecordWriter }
// ...
object XMLIO {
class XMLInputFormat extends FileInputFormat[LongWritable, Node] { /*...*/ }
class XMLRecordReader extends RecordReader[LongWritable, Node] { /*...*/ }
class XMLOutputFormat extends FileOutputFormat[LongWritable, Node] { /*...*/ }
class XMLRecordWriter extends RecordWriter[LongWritable, Node] { /*...*/ }
}
Which I'm trying to use for a job I'm defining over in Example.scala:
import XMLIO._
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.mapreduce.Job
object Example {
#throws(classOf[Exception])
def main( args : Array[String] ) {
val job = new Job(new Configuration(), "")
job setInputFormatClass classOf[XMLInputFormat]
}
}
However, this is giving me a compiler error:
[ERROR] /path/to/Example.scala:8: error: type mismatch;
[INFO] found : java.lang.Class[XMLInputFormat](classOf[XMLInputFormat])
[INFO] required: java.lang.Class[_ <: org.apache.hadoop.mapreduce.InputFormat]
[INFO] job setInputFormatClass classOf[XMLInputFormat]
[INFO] ^
Which seemed odd to me, given that XMLInputFormat is a subclass of FileInputFormat, which is a subclass of InputFormat.
Playing around a bit in the REPL, I found a weird workaround. If I create an instance of XMLInputFormat before I set the input format class, there's no compiler error. That is, the following compiles fine:
import XMLIO._
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.mapreduce.Job
object Example {
#throws(classOf[Exception])
def main( args : Array[String] ) {
val x = new XMLInputFormat()
val job = new Job(new Configuration(), "")
job setInputFormatClass classOf[XMLInputFormat]
}
}
What's going on here? Is there a fix for this that doesn't seem like so much of a hack?
Looks like this was a bug with scala 2.9.0 (which is what I was using). When I upgraded to scala 2.9.1, the problem went away.