When I am trying parse the json object I am getting the below error.
import net.liftweb.json._
object SarahEmailPluginConfigTest {
implicit val formats = DefaultFormats
case class Mailserver(url: String, username: String, password: String)
val json = parse("""{"url": "imap.yahoo.com", "username": "myusername", "password": "mypassword" }""")
def main(args: Array[String]) {
val m = json.extract[Mailserver]
println(m.url)
println(m.username)
println(m.password)
}
}
I have added "lift-json_2.9.0-1-2.4.jar " to my build path and I am getting following error:
could not find implicit value for parameter formats: net.liftweb.json.Formats
not enough arguments for method extract: (implicit formats: net.liftweb.json.Formats, implicit mf: scala.reflect.Manifest[MailServer])MailServer. Unspecified value parameters formats, mf
Your example works for me on scala 2.11.7 and lift-json-2.6.2. What version of scala are you using? From the name of the jar you gave above you should be using scala 2.9.* which is pretty old. If you're not on scala 2.9.* I guess it is because of binary incompatibilities between the Scala versions. If you are using sbt, try the following as build.sbt:
name := "<name of your project>"
scalaVersion := "2.11.7"
libraryDependencies += "net.liftweb" %% "lift-json" % "2.6.2"
You can then remove the old jar file because sbt takes care of that for you.
Related
I want to validate a downloaded Json file from server during build time and failed the build, if there are any errors.
Is it possible to parse/validate Json in build.sbt?
Your build.sbt is scala code so it can do everything you can do with other scala code.
You should be able to add dependencies (e.g. a json parsing library) of your build.sbt code in project/build.sbt since sbt is recursive.
Here is an example to supplement Jasper-M's answer.
For example, add liahoy's requests-scala HTTP client library, and upickle JSON deserialisation library to project/builds.sbt
libraryDependencies ++= List(
"com.lihaoyi" %% "requests" % "0.6.0",
"com.lihaoyi" %% "upickle" % "1.1.0"
)
Then under project/Preconditions.scala add the following object which will contain assertions you want to check before running the build
object Preconditions {
import scala.util.Try
import requests._
import upickle.default._
case class User(login: String, id: Int)
implicit val userRW: ReadWriter[User] = macroRW
def validateUserJson() = {
val result = Try(read[User](get("https://api.github.com/users/lihaoyi").text)).isSuccess
assert(result, "User JSON should be valid")
}
}
Now these facilities will be available to build.sbt under the root project. Lets create a task in build.sbt to run the assertions
lazy val checkPreconditions = taskKey[Unit]("Validate pre-conditions before building")
checkPreconditions := {
Preconditions.validateUserJson()
println("All preconditions passed!")
}
and finally lets make compile task dependant on checkPreconditions task using dependsOn like so
Compile / compile := (Compile / compile).dependsOn(checkPreconditions).value
Now executing sbt compile should check pre-conditions before proceeding with compilation.
Hi I have a file given to by my teacher. It is about Scala and Spark.
When I run the code it gives me this exception:
(run-main-0) scala.ScalaReflectionException: class java.sql.Date in
JavaMirror with ClasspathFilter
The file itself looks like this:
import org.apache.spark.ml.feature.Tokenizer
import org.apache.spark.sql.Dataset
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._
object Main {
type Embedding = (String, List[Double])
type ParsedReview = (Integer, String, Double)
org.apache.log4j.Logger getLogger "org" setLevel
(org.apache.log4j.Level.WARN)
org.apache.log4j.Logger getLogger "akka" setLevel
(org.apache.log4j.Level.WARN)
val spark = SparkSession.builder
.appName ("Sentiment")
.master ("local[9]")
.getOrCreate
import spark.implicits._
val reviewSchema = StructType(Array(
StructField ("reviewText", StringType, nullable=false),
StructField ("overall", DoubleType, nullable=false),
StructField ("summary", StringType, nullable=false)))
// Read file and merge the text abd summary into a single text column
def loadReviews (path: String): Dataset[ParsedReview] =
spark
.read
.schema (reviewSchema)
.json (path)
.rdd
.zipWithUniqueId
.map[(Integer,String,Double)] { case (row,id) => (id.toInt, s"${row getString 2} ${row getString 0}", row getDouble 1) }
.toDS
.withColumnRenamed ("_1", "id" )
.withColumnRenamed ("_2", "text")
.withColumnRenamed ("_3", "overall")
.as[ParsedReview]
// Load the GLoVe embeddings file
def loadGlove (path: String): Dataset[Embedding] =
spark
.read
.text (path)
.map { _ getString 0 split " " }
.map (r => (r.head, r.tail.toList.map (_.toDouble))) // yuck!
.withColumnRenamed ("_1", "word" )
.withColumnRenamed ("_2", "vec")
.as[Embedding]
def main(args: Array[String]) = {
val glove = loadGlove ("Data/glove.6B.50d.txt") // take glove
val reviews = loadReviews ("Data/Electronics_5.json") // FIXME
// replace the following with the project code
glove.show
reviews.show
spark.stop
}
}
I need to keep the line
import org.apache.spark.sql.Dataset
because some code depends on it but it is exactly because of it I have an exception throw.
My build.sbt file looks like this:
name := "Sentiment Analysis Project"
version := "1.1"
scalaVersion := "2.11.12"
scalacOptions ++= Seq("-unchecked", "-deprecation")
initialCommands in console :=
"""
import Main._
"""
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
libraryDependencies += "org.apache.spark" %% "spark-mllib" %
"2.3.0"
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.5"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" %
"test"
The Scala guide recommends you compile with Java8:
We recommend using Java 8 for compiling Scala code. Since the JVM is backward compatible, it is usually safe to use a newer JVM to run your code compiled by the Scala compiler for older JVM versions.
Although it's only a recommendation, I found it to fix the issue you mention.
In order to install Java 8 using Homebrew, it's best to use jenv which will help you handle multiple Java versions should you need to.
brew install jenv
Then run the following to add a tap (repository) of alternative versions of casks, since Java 8 is not in the default tap anymore:
brew tap homebrew/cask-versions
To install Java 8:
brew cask install homebrew/cask-versions/adoptopenjdk8
Run the following to add the previously installed Java version to jenv's list of versions:
jenv add /Library/Java/JavaVirtualMachines/<installed_java_version>/Contents/Home
Finally run
jenv global 1.8
or
jenv local 1.8
to use Java 1.8 globally or locally (in the current folder).
Fore more information, follow the instructions at jenv's website
I'm trying to use pureConfig and configFactory for my spark application configuration.
here is my code:
import pureconfig.{loadConfigOrThrow}
object Source{
def apply(keyName: String, configArguments: Config): Source = {
keyName.toLowerCase match {
case "mysql" =>
val properties = loadConfigOrThrow[DBConnectionProperties](configArguments)
new MysqlSource(None, properties)
case "files" =>
val properties = loadConfigOrThrow[FilesSourceProperties](configArguments)
new Files(properties)
case _ => throw new NoSuchElementException(s"Unknown Source ${keyName.toLowerCase}")
}
}
}
import Source
val config = ConfigFactory.parseString(result.mkString("\n"))
val source = Source("mysql",config.getConfig("source.mysql"))
when I run it from the IDE (intelliJ) or directly from java
(i.e java jar...) it works fine.
But when I run it with spark-submit it fails with the following error:
Exception in thread "main" java.lang.NoSuchMethodError: shapeless.Witness$.mkWitness(Ljava/lang/Object;)Lshapeless/Witness;
From a quick search I found a similar similar to this question.
which suggest the reason for this is due to the fact both spark and pureConfig depends on Shapeless but with different versions,
I tried to shade it as suggested in the answer
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("shapeless.**" -> "shadeshapless.#1")
.inLibrary("com.github.pureconfig" %% "pureconfig" % "0.7.0").inProject
)
but it didn't work as well
can it be from a different reason?
any idea what may work?
Thanks
You also have to shade shapeless inside its own JAR, in addition to pureconfig:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("shapeless.**" -> "shadeshapless.#1")
.inLibrary("com.chuusai" % "shapeless_2.11" % "2.3.2")
.inLibrary("com.github.pureconfig" %% "pureconfig" % "0.7.0")
.inProject
)
Make sure to add the correct shapeless version.
I do a GET HTTP call to a rest service which returns a json. I would like to parse the json to a scala object but here I got stuck. I am using the Akka api and I can't manage to retrieve the content from the Akka's ResponseEntity
Here is my sbt file:
name := "ScalaHttp"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies ++={
val akkaV = "2.4.5"
Seq(
"com.typesafe.akka" %% "akka-http-core" % akkaV,
"com.typesafe.play" %% "play-json" % "2.4.0-M3"
)
}
And here is the app
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.HttpRequest
import akka.stream.ActorMaterializer
import scala.concurrent.ExecutionContext.Implicits.global
object Sender {
def main(args: Array[String]): Unit = {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
Http().singleRequest(HttpRequest(uri = "http://declinators.com/declinator?noun=komunikacja")) foreach {
y => println(y.entity)
println(y.entity.getContentType())
println(y.entity.contentType)
}
}
}
This prints:
HttpEntity.Strict(application/json,{"nominative":"komunikacja","genitive":"komunikacji","dative":"komunikacji","accusative":"komunikację","instrumental":"komunikacją","locative":"komunikacji","vocative":"komunikacjo"})
application/json
application/json
Here come the questions:
1. Why ResponseEntity supplies getContentType() and contentType()? They return the same thing.
2. Getting the contentyType is easy, you have two ways for doing it, but how can I get the content itself, so I can play (i.e. parse it using play) with the json!
You can use entity.data.toString for Binary content type, or the following code piece
data.decodeString(nb.charset.value)
Please follow the HttpEntity.Strict.toString implementation here for detail:
https://github.com/akka/akka/blob/master/akka-http-core/src/main/scala/akka/http/scaladsl/model/HttpEntity.scala#L316
I am trying to use LDAP via unboundid in scala but the compiler keeps crashing.
I just created an object that looks like this:
package utils
import com.unboundid.ldap.sdk._
object LdapHelper {
val ldap = LDAPConnection("ldap.example.com", 389)
}
I added this: "com.unboundid" % "unboundid-ldapsdk" % "2.3.1" to my appDependencies in Build.scala. I use Play 2.1 and Scala Version 2.10.1.
I get a very strange error message (see below):
The error message is so strange that i really dont know where to begin to look for hints.
Not sure if the problem is in unboundid, play, scala, sbt?
How can i successfully integrate unboundid in my scala project?
Thanks
Error in Scala compiler: assertion failed: while compiling: C:\play\todolist\app\utils\LdapHelper.scala during phase: global=typer, atPhase=parser library version: version 2.10.2 compiler version: version 2.10.2 reconstructed args: -classpath C:\play\todolist.target;C:\eclipse\scala-SDK-3.0.1-vfinal-2.10-win32.win32.x86_64\configuration\org.eclipse.
...
last tree to typer: Ident(LDAPConnection)
symbol: (flags: )
symbol definition:
symbol owners:
context owners: value ldap -> object LdapHelper -> package utils
== Enclosing template or block ==
Template( // val : in object LdapHelper
"java.lang.Object" // parents
ValDef(
private
"_"
)
// 3 statements
DefDef( // def : in object LdapHelper
""
[]
List(Nil)
Block(
Apply(
super.""
Nil
)
()
)
)
DefDef( // def x: in object LdapHelper
"x"
[]
Nil
()
)
ValDef( // private[this] val ldap: in object LdapHelper
private
"ldap"
Apply(
"LDAPConnection"
// 2 arguments
"ldap.example.com"
389
)
)
)
There was a warning that turned into an assert in Scala 2.10.2 causing this.
There is a bug open here:
https://issues.scala-lang.org/browse/SI-7014
And a fix staged for 2.10.4:
https://github.com/scala/scala/pull/2829
You can ask Play to use Scala 2.10.4-SNAPSHOT by using the following Build.scala:
import sbt._
import Keys._
import play.Project._
object ApplicationBuild extends Build {
val appName = "AppName"
val appVersion = "1.0-SNAPSHOT"
val mainDeps = Seq(
jdbc,
anorm,
cache
)
lazy val main = play.Project(appName, appVersion, mainDeps).settings(
scalaVersion := "2.10.4-SNAPSHOT"
)
}
If you are using build.sbt the file would look like:
import play.Project._
playScalaSettings
name := "AppName"
version := "1.0-SNAPSHOT"
scalaVersion := "2.10.4-SNAPSHOT"
libraryDependencies ++= Seq(jdbc, anorm, cache)
Note: if building from sbt (instead of play) you may have to add a repository resolver under the scalaVersion line such as:
resolvers += "Typesafe repository" at "http://repo.typesafe.com/typesafe/repo/"
The answer from #jeckhart works.
Firstly I use Scala 2.10.4-RC1 to build the Play 2.3 SNAPSHOT. Then use the output to compile with UnboundID.
Finally everything compiles with no assertion or error.
To build Play 2.3 SNAPSHOT using Scala 2.10.4-RC1, I modified the file framework/project/Build.scala.
Change these two section from
val buildScalaVersion = propOr("scala.version", "2.10.3")
val buildScalaVersionForSbt = propOr("play.sbt.scala.version", "2.10.3")
to
val buildScalaVersion = propOr("scala.version", "2.10.4-RC1")
val buildScalaVersionForSbt = propOr("play.sbt.scala.version", "2.10.4-RC1")