MongoDB Scala Driver doesn't allow use Map collection in case class - mongodb

I just started using MongoDB and I'm trying to write a small application to test Mongo with Scala. I created the following case class in order to cast the Documents to a Scala class:
case class User(
_id: ObjectId,
userId: String,
items: Map[String, Int]
)
object User {
def apply(userId: String , items: Map[String, Int]): User =
new User(new ObjectId, userId, items)
implicit val codecRegistry: CodecRegistry =
fromRegistries(fromProviders(classOf[User]), DEFAULT_CODEC_REGISTRY)
}
I get the following error but I don't know why since the Map keys are in fact strings.
[ERROR] error: Maps must contain string types for keys
[INFO] implicit val codecRegistry: CodecRegistry = fromRegistries (fromProviders (classOf [User]), DEFAULT_CODEC_REGISTRY)
[INFO] ^
[ERROR] one error found
I'm also applying the codecRegistry to the MongoDatabase.
Thank you very much.

The problem was that I was using a version of the driver that is compiled for Scala 2.11 and not 2.12. By changing the Maven dependency from
<dependency>
<groupId>org.mongodb.scala</groupId>
<artifactId>mongo-scala-driver_2.11</artifactId>
<version>2.2.1</version>
</dependency>
to
<dependency>
<groupId>org.mongodb.scala</groupId>
<artifactId>mongo-scala-driver_2.12</artifactId>
<version>2.2.1</version>
</dependency>
solved the problem.

Related

java.lang.NoSuchFieldException: handle Embedded MongoDB with play framework

I'm trying to enable in-memory mongodb in testing, I am using simplyscala to do the work.
class controllerSpec extends PlaySpec with GuiceOneAppPerTest with Injecting with BeforeAndAfterAll with MongoEmbedDatabase{
//declares a variable which will hold the reference to running mongoDB Instance
var mongoInstance: MongodProps = null
// Start In-memory Mongo instance in before statement
override def beforeAll(): Unit =
try{
val rnd = new scala.util.Random
val range = 12000 to 36000
val portNum = range(rnd.nextInt(range length))
mongoInstance = mongoStart(portNum) } //Try starting mongo on random port number
catch { case ex:Exception => } // Handle exception In case local mongo is running//code to run before all tests starts
override def afterAll(): Unit = mongoStop(mongoInstance)
However I have error :
[info] controllerSpec:
java.lang.NoSuchFieldException: handle
at java.base/java.lang.Class.getDeclaredField(Class.java:2411)
at de.flapdoodle.embed.process.runtime.Processes.windowsProcessId(Processes.java:109)
at de.flapdoodle.embed.process.runtime.Processes.access$200(Processes.java:51)
at de.flapdoodle.embed.process.runtime.Processes$PidHelper$2.getPid(Processes.java:209)
| => rat de.flapdoodle.embed.process.runtime.Processes.processId(Processes.java:72)
at de.flapdoodle.embed.process.runtime.ProcessControl.<init>(ProcessControl.java:64)
at de.flapdoodle.embed.process.runtime.ProcessControl.start(ProcessControl.java:205)
at de.flapdoodle.embed.process.runtime.AbstractProcess.<init>(AbstractProcess.java:98)
at de.flapdoodle.embed.mongo.AbstractMongoProcess.<init>(AbstractMongoProcess.java:53)
at de.flapdoodle.embed.mongo.MongodProcess.<init>(MongodProcess.java:50)
at de.flapdoodle.embed.mongo.MongodExecutable.start(MongodExecutable.java:44)
at de.flapdoodle.embed.mongo.MongodExecutable.start(MongodExecutable.java:34)
at de.flapdoodle.embed.process.runtime.Executable.start(Executable.java:101)
at com.github.simplyscala.MongoEmbedDatabase.mongoStart(MongoEmbedDatabase.scala:26)
at com.github.simplyscala.MongoEmbedDatabase.mongoStart$(MongoEmbedDatabase.scala:22)
at controllersSpec.ssmServiceSpec.ssmControllerSpec.mongoStart(ssmControllerSpec.scala:22)
at controllersSpec.ssmServiceSpec.ssmControllerSpec.beforeAll(ssmControllerSpec.scala:32)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
at controllersSpec.ssmServiceSpec.ssmControllerSpec.run(ssmControllerSpec.scala:22)
at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:317)
at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:510)
at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:304)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Does it have something to do with the BeforeAndAfterAll? Please tell me if I did anything wrong thank you.
Upgrade the version of the Embedded MongoDB dependency test driver to 3.4.3
Gradle:
testImplementation 'de.flapdoodle.embed:de.flapdoodle.embed.mongo:3.4.3'
pom.xml
<dependency>
<groupId>de.flapdoodle.embed</groupId>
<artifactId>de.flapdoodle.embed.mongo</artifactId>
<version>3.4.3</version> <!-- make this version a property instead-->
<scope>test</scope>
</dependency>

custom timestamp extractor in kafka streams

I'm trying to use kafka streams to process some data from a kafka topic. The data comes from kafka topic that was written to by kafka 0.11.0 something which doesn't have the embedded timestamp. After some reading on the internet, I came to understand that I can solve this problem by extending TimestampExtractor class in a custom class and passing it in the StreamsConfig.
I did so like this --
class MyEventTimestampExtractor extends TimestampExtractor {
override def extract(record: ConsumerRecord[AnyRef, AnyRef], prev: Long) = {
record.value() match {
case w: String => 1000L
case _ => throw new RuntimeException(s"Called for $record")
}
}
}
I based it off of this code on github
But, I get this error when I do an sbt run
[error] /home/someuser/app/blahblah/src/main/scala/main.scala:34: class MyEventTimestampExtractor needs to be abstract, since method extract in trait TimestampExtractor of type (x$1: org.apache.kafka.clients.consumer.ConsumerRecord[Object,Object], x$2: Long)Long is not defined
[error] (Note that Long does not match Long)
[error] class MyEventTimestampExtractor extends TimestampExtractor {
[error] ^
[error] /home/someuser/app/blahblah/src/main/scala/main.scala:35: method extract overrides nothing.
[error] Note: the super classes of class MyEventTimestampExtractor contain the following, non final members named extract:
[error] def extract(x$1: org.apache.kafka.clients.consumer.ConsumerRecord[Object,Object],x$2: Long): Long
[error] override def extract(record: ConsumerRecord[AnyRef, AnyRef], prev: Long): Long = {
[error] ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed
My build.sbt file is this --
name := "kafka streams experiment"
version := "1.0"
scalaVersion := "2.12.4"
libraryDependencies ++= Seq(
"org.apache.kafka" % "kafka-streams" % "1.0.0"
)
I don't really understand the error. Specifically the part around Note that Long does not match Long. What could I be doing wrong? thanks!
You need java.Long since this API is defined in Java whereas you are using Scala Long
Try with (watch the type of the prev function argument:
override def extract(record: ConsumerRecord[AnyRef, AnyRef], prev: java.lang.Long) = {

Codec for ADT do not compile

I'm using the scala driver to make IO operations with mongodb. My scala version is 2.11.11 and the mongo db driver is 2.2.0.
I take the example in documentation about ADT :
sealed class Tree
case class Branch(b1: Tree, b2: Tree, value: Int) extends Tree
case class Leaf(value: Int) extends Tree
val codecRegistry = fromRegistries( fromProviders(classOf[Tree]), DEFAULT_CODEC_REGISTRY )
This code didn't compile.
No known subclasses of the sealed class
[error] val codecRegistry = fromRegistries( fromProviders(classOf[Tree]), DEFAULT_CODEC_REGISTRY )
[error] ^
[error] knownDirectSubclasses of Tree observed before subclass Branch registered
[error] knownDirectSubclasses of Tree observed before subclass Leaf registered
Did I miss something ?
Update
Below a complete example of what I'm tring to do.
build.sbt
name := "mongodb-driver-test"
version := "1.0"
scalaVersion := "2.11.11"
libraryDependencies += "org.mongodb.scala" %% "mongo-scala-driver" % "2.2.0"
file Models.scala
import org.mongodb.scala.bson.codecs.{DEFAULT_CODEC_REGISTRY, Macros}
import org.bson.codecs.configuration.CodecRegistries.{fromProviders, fromRegistries}
/**
* Created by alifirat on 02/01/18.
*/
object Models {
sealed class Tree
case class Branch(b1: Tree, b2: Tree, value: Int) extends Tree
case class Leaf(value: Int) extends Tree
val treeCodec = Macros.createCodecProvider[Tree]()
val treeCodecRegistry = fromRegistries( fromProviders(treeCodec), DEFAULT_CODEC_REGISTRY )
}
Then, just do :
sbt compile
You will get :
[error] val treeCodec = Macros.createCodecProvider[Tree]()
[error] ^
[error] knownDirectSubclasses of Tree observed before subclass Branch registered
[error] knownDirectSubclasses of Tree observed before subclass Leaf registered
[error] three errors found
[error] (compile:compileIncremental) Compilation failed
If I change the scala version to 2.12.0, I didn't have any errors at compile time ...
I'm using driver version 2.6.0 and Scala version 2.12.8 and still get the same problem.
My workaround is to remove the keyword sealed in front of that sealed class, compile, put it back, and then compile again. But it's very cumbersome.

use spark sql through Scala IDE

i want to try spark sql , i used at first the bin/spark-shell
inserting this code
val sqlcontext=new org.apache.spark.sql.SQLContext(sc)
val data=sc.textFile("hdfs://localhost:9000/cars.csv")
val mapr=data.map (p => p.split(','))
val MyMatchRDD=mapr.map(p =>MyMatch(p(0).toString(),p(1).toString(),p(2).toString(),
p(3).toString(),p(4).toString(),p(5).toString(),p(6).toString(),p(7).toString(),
p(8).toString()))
import sqlcontext.implicits._
val personDF=MyMatchRDD.toDF()
personDF.registerTempTable("Person")
val res = sqlcontext.sql("SELECT * FROM Person")
res.collect().foreach(println)
i didn't get any issue ,all is good.
But when i used the scala ide
i used in pom file (maven)
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_2.10</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.3.0</version>
</dependency>
and i used the same code
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext._
import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.slf4j.Logger
import org.slf4j.LoggerFactory;
object SparkSQL {
case class MyMatch( col1: String, col2: String,col3: String, col4 :String ,col5: String,
col6: String,col7 :String ,col8: String,
col9: String)
def main(args:Array[String]) {
val sparkConf = new SparkConf().setAppName("HiveFromSpark").setMaster("local")
val sc = new SparkContext(sparkConf)
val sqlcontext=new org.apache.spark.sql.SQLContext(sc)
val data=sc.textFile("hdfs://localhost:9000/cars.csv")
val mapr=data.map (p => p.split(','))
val MyMatchRDD=mapr.map(p =>MyMatch(p(0).toString(),p(1).toString(),p(2).toString(),p(3).toString(),
p(4).toString(),p(5).toString(),p(6).toString(),p(7).toString(),
p(8).toString()) )
import sqlcontext.implicits._
val personDF=MyMatchRDD.toDF()
personDF.registerTempTable("Person")
val res = sqlcontext.sql("SELECT * FROM Person")
res.collect().foreach(println)
}
}
i got this issue
Exception in thread "main"
scala.reflect.internal.MissingRequirementError: class
org.apache.spark.sql.catalyst.ScalaReflection in JavaMirror with
primordial classloader with boot classpath
[D:\scala-SDK-4.4.1-vfinal-2.11-win32.win32.x86_64\eclipse\plugins\org.scala-
Thanks in advance for your help
You are using an wrong Scala version - Spark's compiled with Scala version 2.10. Check your runtime and compiler Scala version.
Why you're using so old dependencies? Spark has version 2.0.2 right now with Scala 2.11
Recommended actions:
(optional) Change <version>1.3.0</version> to <version>2.0.2</version>
In your Scala compiler, change version to 2.11 (if updated to Spark 2) or 2.10 (if you still use Spark 1)
Make sure you have proper Scala version installed on your machine - 2.11 in case of Spark 2, 2.10 in case of Spark 1. You can check Scala version by typing scala -version in console
Make sure your Scala IDE supports Scala version that was choosen

How to parse JSON with lift-json in Scala?

When I am trying parse the json object I am getting the below error.
import net.liftweb.json._
object SarahEmailPluginConfigTest {
implicit val formats = DefaultFormats
case class Mailserver(url: String, username: String, password: String)
val json = parse("""{"url": "imap.yahoo.com", "username": "myusername", "password": "mypassword" }""")
def main(args: Array[String]) {
val m = json.extract[Mailserver]
println(m.url)
println(m.username)
println(m.password)
}
}
I have added "lift-json_2.9.0-1-2.4.jar " to my build path and I am getting following error:
could not find implicit value for parameter formats: net.liftweb.json.Formats
not enough arguments for method extract: (implicit formats: net.liftweb.json.Formats, implicit mf: scala.reflect.Manifest[MailServer])MailServer. Unspecified value parameters formats, mf
Your example works for me on scala 2.11.7 and lift-json-2.6.2. What version of scala are you using? From the name of the jar you gave above you should be using scala 2.9.* which is pretty old. If you're not on scala 2.9.* I guess it is because of binary incompatibilities between the Scala versions. If you are using sbt, try the following as build.sbt:
name := "<name of your project>"
scalaVersion := "2.11.7"
libraryDependencies += "net.liftweb" %% "lift-json" % "2.6.2"
You can then remove the old jar file because sbt takes care of that for you.