Circe Encoders and Decoders with Http4s - scala

I am trying to use http4s, circe and http4s-circe.
Below I am trying to use the auto derivation feature of circe.
import org.http4s.client.blaze.SimpleHttp1Client
import org.http4s.Status.ResponseClass.Successful
import io.circe.syntax._
import org.http4s._
import org.http4s.headers._
import org.http4s.circe._
import scalaz.concurrent.Task
import io.circe._
final case class Login(username: String, password: String)
final case class Token(token: String)
object JsonHelpers {
import io.circe.generic.auto._
implicit val loginEntityEncoder : EntityEncoder[Login] = jsonEncoderOf[Login]
implicit val loginEntityDecoder : EntityDecoder[Login] = jsonOf[Login]
implicit val tokenEntityEncoder: EntityEncoder[Token] = jsonEncoderOf[Token]
implicit val tokenEntityDecoder : EntityDecoder[Token] = jsonOf[Token]
}
object Http4sTest2 extends App {
import JsonHelpers._
val url = "http://"
val uri = Uri.fromString(url).valueOr(throw _)
val list = List[Header](`Content-Type`(MediaType.`application/json`), `Accept`(MediaType.`application/json`))
val request = Request(uri = uri, method = Method.POST)
.withBody(Login("foo", "bar").asJson)
.map{r => r.replaceAllHeaders(list :_*)}.run
val client = SimpleHttp1Client()
val result = client.fetch[Option[Token]](request){
case Successful(response) => response.as[Token].map(Some(_))
case _ => Task(Option.empty[Token])
}.run
println(result)
}
I get multiple instances of these two compiler errors
Error:scalac: missing or invalid dependency detected while loading class file 'GenericInstances.class'.
Could not access type Secondary in object io.circe.Encoder,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'GenericInstances.class' was compiled against an incompatible version of io.circe.Encoder.
Error:(25, 74) could not find implicit value for parameter encoder: io.circe.Encoder[Login]
implicit val loginEntityEncoder : EntityEncoder[Login] = jsonEncoderOf[Login]

I was able to solve this. I did a search on google on sbt circe dependency and I copy pasted the first search result. that was circe 0.1 and that is why things were not working for me.
I changed my dependencies to
libraryDependencies ++= Seq(
"org.http4s" %% "http4s-core" % http4sVersion,
"org.http4s" %% "http4s-dsl" % http4sVersion,
"org.http4s" %% "http4s-blaze-client" % http4sVersion,
"org.http4s" %% "http4s-circe" % http4sVersion,
"io.circe" %% "circe-core" % "0.7.0",
"io.circe" %% "circe-generic" % "0.7.0"
)
and now automatic derivation works fine and I am able to compile the code below
import org.http4s.client.blaze.SimpleHttp1Client
import org.http4s._
import org.http4s.headers._
import org.http4s.circe._
import scalaz.concurrent.Task
import io.circe.syntax._
import io.circe.generic.auto._
import org.http4s.Status.ResponseClass.Successful
case class Login(username: String, password: String)
case class Token(token: String)
object JsonHelpers {
implicit val loginEntityEncoder : EntityEncoder[Login] = jsonEncoderOf[Login]
implicit val loginEntityDecoder : EntityDecoder[Login] = jsonOf[Login]
implicit val tokenEntityEncoder: EntityEncoder[Token] = jsonEncoderOf[Token]
implicit val tokenEntityDecoder : EntityDecoder[Token] = jsonOf[Token]
}
object Http4sTest2 extends App {
import JsonHelpers._
val url = "http://"
val uri = Uri.fromString(url).valueOr(throw _)
val list = List[Header](`Content-Type`(MediaType.`application/json`), `Accept`(MediaType.`application/json`))
val request = Request(uri = uri, method = Method.POST)
.withBody(Login("foo", "bar").asJson)
.map{r => r.replaceAllHeaders(list :_*)}.run
val client = SimpleHttp1Client()
val result = client.fetch[Option[Token]](request){
case Successful(response) => response.as[Token].map(Some(_))
case _ => Task(Option.empty[Token])
}.run
println(result)
}

Related

Mapping a string to case classes using Scala play

Using the Scala play library I'm attempting to parse the string :
var str = "{\"payload\": \"[{\\\"test\\\":\\\"123\\\",\\\"tester\\\":\\\"456\\\"}," +
"{\\\"test1\\\":\\\"1234\\\",\\\"tester2\\\":\\\"4567\\\"}]\"}";
into a list of Payload classes using code below :
import play.api.libs.json._
object TestParse extends App {
case class Payload(test : String , tester : String)
object Payload {
implicit val jsonFormat: Format[Payload] = Json.format[Payload]
}
var str = "{\"payload\": \"[{\\\"test\\\":\\\"123\\\",\\\"tester\\\":\\\"456\\\"}," +
"{\\\"test1\\\":\\\"1234\\\",\\\"tester2\\\":\\\"4567\\\"}]\"}";
println((Json.parse(str) \ "payload").as[List[Payload]])
}
build.sbt :
name := "akka-streams"
version := "0.1"
scalaVersion := "2.12.8"
lazy val akkaVersion = "2.5.19"
lazy val scalaTestVersion = "3.0.5"
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-stream" % akkaVersion,
"com.typesafe.akka" %% "akka-stream-testkit" % akkaVersion,
"com.typesafe.akka" %% "akka-testkit" % akkaVersion,
"org.scalatest" %% "scalatest" % scalaTestVersion
)
// https://mvnrepository.com/artifact/com.typesafe.play/play-json
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.10.0-RC6"
It fails with exception :
Exception in thread "main" play.api.libs.json.JsResultException: JsResultException(errors:List((,List(JsonValidationError(List("" is not an object),WrappedArray())))))
Is the case class structure incorrect ?
I've updated the code to :
import play.api.libs.json._
object TestParse extends App {
import TestParse.Payload.jsonFormat
object Payload {
implicit val jsonFormat: Format[RootInterface] = Json.format[RootInterface]
}
case class Payload (
test: Option[String],
tester: Option[String]
)
case class RootInterface (
payload: List[Payload]
)
val str = """{"payload": [{"test":"123","tester":"456"},{"test1":"1234","tester2":"4567"}]}"""
println(Json.parse(str).as[RootInterface])
}
which returns error :
No instance of play.api.libs.json.Format is available for scala.collection.immutable.List[TestParse.Payload] in the implicit scope (Hint: if declared in the same file, make sure it's declared before)
implicit val jsonFormat: Format[RootInterface] = Json.format[RootInterface]
This performs the task but there are cleaner solutions :
import akka.actor.ActorSystem
import akka.stream.scaladsl.{Flow, Sink, Source}
import org.scalatest.Assertions._
import spray.json.{JsObject, JsonParser}
import scala.concurrent.Await
import scala.concurrent.duration.DurationInt
object TestStream extends App {
implicit val actorSystem = ActorSystem()
val mapperFlow = Flow[JsObject].map(x => {
x.fields.get("payload").get.toString().replace("{", "")
.replace("}", "")
.replace("[", "")
.replace("]", "")
.replace("\"", "")
.replace("\\", "")
.split(":").map(m => m.split(","))
.toList
.flatten
.grouped(4)
.map(m => Test(m(1), m(3).toDouble))
.toList
})
val str = """{"payload": [{"test":"123","tester":"456"},{"test":"1234","tester":"4567"}]}"""
case class Test(test: String, tester: Double)
val graph = Source.repeat(JsonParser(str).asJsObject())
.take(3)
.via(mapperFlow)
.mapConcat(identity)
.runWith(Sink.seq)
val result = Await.result(graph, 3.seconds)
println(result)
assert(result.length == 6)
assert(result(0).test == "123")
assert(result(0).tester == 456 )
assert(result(1).test == "1234")
assert(result(1).tester == 4567 )
assert(result(2).test == "123")
assert(result(2).tester == 456 )
assert(result(3).test == "1234")
assert(result(3).tester == 4567 )
}
Alternative, ioiomatic Scala answers are welcome.

How to convert JSON to scala shapeless.hlist?

I got json like {"name":"susan","age":25},and a hint to json keyset like "name:String,age:Int",how to create a HList from that json?
Based on the code you added and then deleted, it seems, having a runtime-string hint "name:String,age:Int" and runtime-string json {"name":"susan","age":25}, you want to get an HList with runtime reflection. You can do this as follows
import shapeless.HList
import scala.reflect.runtime
import scala.reflect.runtime.universe._
import scala.tools.reflect.ToolBox
val tb = runtime.currentMirror.mkToolBox()
val jsonStr = """{"name":"susan","age":25}"""
val hint = "name:String,age:Int"
val classType = tb.define(tb.parse(s"case class Test($hint)").asInstanceOf[ImplDef]).asClass.toType
val hlist = tb.eval(q"""
import io.circe.generic.auto._
import io.circe.parser.decode
val classInstance = decode[$classType]($jsonStr)
import shapeless.Generic
Generic[$classType].to(classInstance.toOption.get)
""").asInstanceOf[HList]
println(hlist) // susan :: 25 :: HNil
Please notice that you do everything at runtime so you'll have no access to type String :: Int :: HNil at compile time and hlist has static type just HList (not String :: Int :: HNil) and HList is actually not better than just List[Any].
build.sbt
libraryDependencies ++= Seq(
scalaOrganization.value % "scala-reflect" % scalaVersion.value,
scalaOrganization.value % "scala-compiler" % scalaVersion.value,
"com.chuusai" %% "shapeless" % "2.4.0-M1",
"io.circe" %% "circe-core" % "0.13.0",
"io.circe" %% "circe-parser" % "0.13.0",
"io.circe" %% "circe-generic" % "0.13.0"
)
Actually, I guess we do someting strange. We use highly type-level libraries (shapeless, circe) aimed to static type safety and then run them at runtime with reflection ignoring type safety and getting actually List[Any] (HList).
I guess that if List[Any] (list of field values) is enough for you then you just need to use a more runtime library. For example, with json4s
import org.json4s.{JInt, JObject, JString, JValue}
import org.json4s.jackson.JsonMethods._
val jsonStr: String = """{"name":"susan","age":25}"""
val json: JValue = parse(jsonStr) //JObject(List((name,JString(susan)), (age,JInt(25))))
val l: List[JValue] = json.asInstanceOf[JObject].obj.map(_._2) //List(JString(susan), JInt(25))
val res: List[Any] = l.map {
case JString(s) => s
case JInt(n) => n
} //List(susan, 25)
build.sbt
libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.6.9"
Actually the same can be done with Circe, just with parse instead of decode[A]
import io.circe.{Json, JsonNumber}
import io.circe.parser.parse
val jsonStr: String = """{"name":"susan","age":25}"""
val json: Json = parse(jsonStr).toOption.get //{"name":"susan","age":25}
val l: List[Json] = json.asObject.get.values.toList //List("susan", 25)
val res: List[Any] = l.map(_.fold[Any](null, null, (_: JsonNumber).toInt.get, identity[String], null, null)) //List(susan, 25)
If you need an instance of case class or a tuple rather than HList replace
tb.eval(q"""
import io.circe.generic.auto._
import io.circe.parser.decode
val classInstance = decode[$classType]($jsonStr)
import shapeless.Generic
Generic[$classType].to(classInstance.toOption.get)
""").asInstanceOf[HList] // susan :: 25 :: HNil
with
tb.eval(q"""
import io.circe.generic.auto._
import io.circe.parser.decode
decode[$classType]($json).toOption.get
""").asInstanceOf[Product] //Test(susan,25)
or
tb.eval(q"""
import io.circe.generic.auto._
import io.circe.parser.decode
val classInstance = decode[$classType]($json)
import shapeless.Generic
Generic[$classType].to(classInstance.toOption.get).tupled
""").asInstanceOf[Product] //(susan,25)
correspondingly.

ambiguous implicit values: match expected type cats.derived.MkShow[A]: show cats:kittens

I'm trying to create Show Instance for my custom Config class.
The build.sbt file is -
name := "circe-demo"
version := "0.1"
scalaVersion := "2.11.12"
resolvers += Resolver.bintrayRepo("ovotech", "maven")
libraryDependencies += "io.circe" %% "circe-core" % "0.11.0"
libraryDependencies += "io.circe" %% "circe-parser" % "0.11.0"
libraryDependencies += "io.circe" %% "circe-generic" % "0.11.0"
libraryDependencies += "org.typelevel" %% "kittens" % "1.2.0"
libraryDependencies ++= Seq(
"is.cir" %% "ciris-cats",
"is.cir" %% "ciris-cats-effect",
"is.cir" %% "ciris-core",
"is.cir" %% "ciris-enumeratum",
"is.cir" %% "ciris-refined"
).map(_ % "0.12.1")
Complete code is -
import enumeratum.{Enum, EnumEntry}
sealed abstract class AppEnvironment extends EnumEntry
object AppEnvironment extends Enum[AppEnvironment] {
case object Local extends AppEnvironment
case object Testing extends AppEnvironment
case object Production extends AppEnvironment
override val values: Vector[AppEnvironment] =
findValues.toVector
}
import java.net.InetAddress
import scala.concurrent.duration.Duration
final case class ApiConfig(host: InetAddress, port: Int, apiKey: String, timeout: Duration)
import java.net.InetAddress
import cats.Show
import cats.derived.semi
import ciris.config.loader.AppEnvironment.{Local, Production, Testing}
import enumeratum.EnumEntry
import eu.timepit.refined.auto._
import eu.timepit.refined.types.string.NonEmptyString
import scala.concurrent.duration._
final case class Config(appName: NonEmptyString, environment: AppEnvironment, api: ApiConfig)
object Config {
implicit val showConfig: Show[Config] = {
implicit val showDuration: Show[Duration] =
Show.fromToString
implicit val showInetAddress: Show[InetAddress] =
Show.fromToString
implicit def showEnumEntry[E <: EnumEntry]: Show[E] =
Show.show(_.entryName)
// Show.show[Config](x => s"api = ${x.api} appName = ${x.appName} environment ${x.environment}")
semi.show
}
}
semi.show in the above code throws the below exception -
[error] /Users/rajkumar.natarajan/Documents/Coding/kafka_demo/circe-demo/src/main/scala/ciris/config/loader/Config.scala:32:5: ambiguous implicit values:
[error] both value emptyProductDerivedShow in trait MkShowDerivation of type => cats.derived.MkShow[shapeless.HNil]
[error] and method emptyCoproductDerivedShow in trait MkShowDerivation of type => cats.derived.MkShow[shapeless.CNil]
[error] match expected type cats.derived.MkShow[A]
[error] show
[error] ^
[error] one error found
[error] (Compile / compileIncremental) Compilation failed
[error]
I'm new to functional programming using cats.
How can I resolve this exception.
Unfortunately error reporting when such complicated implicits and macros are involved is far from perfect. The message you see actually means that some required implicits for the real generator (MkShow.genericDerivedShowProduct in this case) have not been found and the search went back to some where basic stuff where there is an ambiguity. And the stuff that is missing is mostly very basic such as an implicit for Show[Int] or Show[String]. The simplest way to get them all is to import cats.implicits._ but that will also bring catsStdShowForDuration which is a Show[Duration]. But since it's implementation is really the same as your custom one, it is easier to remove your custom one. One more thing that is missing is Show[NonEmptyString] and it is easy to create one
implicit def showNonEmptyString: Show[NonEmptyString] = Show.show(nes => nes)
To sum up, when I define your showConfig as
implicit val showConfig: Show[Config] = {
import cats.implicits._
// is already defined in cats.implicits._
//implicit val showDuration: Show[Duration] = Show.fromToString
implicit val showInetAddress: Show[InetAddress] = Show.fromToString
implicit def showEnumEntry[E <: EnumEntry]: Show[E] = Show.show(_.entryName)
implicit def showNonEmptyString: Show[NonEmptyString] = Show.show(nes => nes)
// Show.show[Config](x => s"api = ${x.api} appName = ${x.appName} environment ${x.environment}")
semi.show
}
it compiles for me.
P.S. is there any good reason why you put your AppEnvironment under ciris.* package? I'd say that generally putting your custom code into packages of 3-rd party library is an easy way to mess things up.

Misunderstanding error about ReactiveMongo

I defined the following class which I want to modelize :
case class Record(
recordKey: String,
channels: Map[String, Channel],
)
object Record {
implicit val RecordFormat = Json.format[Record]
}
Now, I want to get one object of this type from reactive mongo like this (in an another class):
import scala.concurrent.duration._
import scala.concurrent.{Future, Await}
import scala.concurrent.ExecutionContext.Implicits.global
import reactivemongo.api._
import reactivemongo.api.collections.bson.BSONCollection
import reactivemongo.bson.BSONDocument
object Test {
val collection = connect()
val timeout = 10.seconds
def connect() : BSONCollection = {
val config = ConfigFactory.load()
val driver = new MongoDriver
val connection = driver.connection(List(config.getString("mongodb.uri")))
val db = connection("/toto")
db.collection("foo")
}
def findRecord(recordKey : String) : Record = {
return Test.collection
.find(BSONDocument("recordKey"->recordKey))
.one[Record]
}
But this code doesn't compile :
could not find implicit value for parameter reader: reactivemongo.bson.BSONDocumentReader[Record]
Could someone explain how to fix this issue ?
I also test that :
def findRecord(recordKey : String) : Record = {
val futureRecord : Future[Option[Record]] =
Test.collection
.find(BSONDocument("recordKey"->recordKey))
.one[Record]
return Await.result(futureRecord, 10.seconds).getOrElse(null)
}
I also added my build.sbt :
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-streaming_2.10" % "1.5.2",
"org.apache.spark" % "spark-streaming-kafka_2.10" % "1.5.2",
"org.slf4j" % "slf4j-api" % "1.7.13",
"org.slf4j" % "slf4j-simple" % "1.7.13",
"com.amazonaws" % "aws-java-sdk" % "1.10.12",
"com.typesafe.play" % "play-json_2.10" % "2.4.6",
"com.typesafe" % "config" % "1.3.0",
"org.scalaj" %% "scalaj-http" % "2.2.1",
"com.typesafe.akka" % "akka-actor_2.10" % "2.3.14",
"org.reactivemongo" %% "reactivemongo" % "0.11.9",
"com.github.nscala-time" %% "nscala-time" % "2.6.0"
)
Note that is it not a Play App.
You need to define a BSONDocumentReader for your case class Record. Here is a link to the Documentation. Very similar to Play JSON Readers and Writers Reactive Mongo needs to understand how to convert back and forth between your domain object and BSONDocument. Similar to Play JSON as well you can write these out in a more manual style(Write a BSONDocumentReader instance and a BSONDocumentWriter instance) and customize every detail and apply transformations etc. Similar in style to Play JSON's format you used above ReactiveMongo does provide helpful macros to generate these classes for you.
For your Record class you would need to add an implicit val like this to your object:
import reactivemongo.bson._
implicit val recordHandler: BSONHandler[BSONDocument, Record] = Macros.handler[Record]
/* Or only one of these [if your only ever writing or reading this data etc:
implicit val recordReader: BSONDocumentReader[Record] = Macros.reader[Record]
implicit val recordWriter: BSONDocumentWriter[Record] = Macros.writer[Record]
*/
I would say try to start with the Macros and see if those meet your needs. If you need more control of the processing/transformation you can define your own BSONDocumentReader and BSONDocumentWriter instances.
updated record class
import play.api.libs.json.Json
import reactivemongo.bson._
case class Channel(label: String,amplitude: Double,position: Option[String])
object Channel {
implicit val ChannelFormat = Json.format[Channel]
implicit val channelHandler: BSONHandler[BSONDocument, Channel] = Macros.handler[Channel]
}
object RecordType extends Enumeration {
type RecordType = Value
val T1 = Value
implicit val enumFormat = new Format[RecordType] {
def reads(json: JsValue) = JsSuccess(RecordType.withName(json.as[String]))
def writes(enum: RecordType) = JsString(enum.toString)
}
implicit object RecordTypeReader extends BSONDocumentReader[RecordType] {
def read(doc: BSONDocument) : RecordType = {
RecordType.withName(doc.getAs[String]("recordType").get)
}
}
implicit object RecordTypeWriter extends BSONDocumentWriter[RecordType] {
def write(recordType: RecordType) : BSONDocument = BSONDocument(
"recordType" -> BSONString(recordType.toString)
)
}
}
case class Record(recordKey: String,recordType: RecordType.Value,channels: Map[String, Channel])
object Record {
implicit val RecordFormat = Json.format[Record]
implicit val recordHandler: BSONHandler[BSONDocument, Record] = Macros.handler[Record]
}

Sending post with json using spray?

Sorry I can't manage to make this work: I need to add some json to a post, so following the documentation: http://spray.io/documentation/1.1-M8/spray-httpx/request-building/ :
import scala.util.{Success, Failure}
import akka.actor.{Props, ActorSystem}
import spray.can.client.DefaultHttpClient
import spray.client.HttpConduit
import spray.httpx.SprayJsonSupport
import spray.http._
import spray.json.JsonParser._
import spray.json._
import HttpMethods._
import HttpHeaders._
import MediaTypes._
import spray.httpx.RequestBuilding._
import scala.concurrent.ExecutionContext.Implicits.global
...
val req = HttpRequest(method = POST, uri = "/api/1.0/users/ping.json", entity = HttpEntity(`application/json`,"""{ "key"="whatever" }"""))
and it never compiles:
overloaded method value apply with alternatives:
[error] (optionalBody: Option[spray.http.HttpBody])spray.http.HttpEntity <and>
[error] (buffer: Array[Byte])spray.http.HttpEntity <and>
[error] (string: String)spray.http.HttpEntity
[error] cannot be applied to (spray.http.MediaType, String)
[error] val req = HttpRequest(method = POST, uri = "/api/1.0/users/ping.json", entity = HttpEntity(`application/json`,"""{ "key"="whatever"}"""))
Had the same problem and found the solution here:
https://github.com/spray/spray-json/blob/master/src/main/scala/spray/json/AdditionalFormats.scala#L30-41
This finally worked for me:
import spray.httpx.SprayJsonSupport
import spray.json.AdditionalFormats
object Client extends SprayJsonSupport with AdditionalFormats {
val email = "..."
val password = "..."
val pipeline = sendReceive
pipeline(Post("http://something.com/login", s"""{
"email": "$email",
"password": "$password"
}""".asJson.asJsObject))
}
Sorry, but you question is a bit cumbersome, for me at least. If you want to make a POST request in Spray with some Json as HttpEntity, then you should try to do it with Spray-Clients pipelining, it's drop dead simple.
You need to create a simple pipe:
val pipe: HttpRequest => Future[HttpResponse] = sendReceive
and then build a request:
import spray.json.SprayJsonSupport._
pipe(Post("/api/1.0/users/ping", """{ "key"="whatever" }""".asJson))
This will return you a Future with HttpResponse, if you want some specific result, let's say, for example, some confirmation code, then add unmarshall step to your pipe:
val pipe: HttpRequest => Future[ConfCode] = sendReceive ~> unmarshal[ConfCode]
That is what i tried too, but it doesn't work it is telling me that I am missing an implicit:
val pipeline = sendReceive(conduit)
val responseF = pipeline(Post("/api/1.0/users/ping.json", """{ "key": "whatever" }""".asJson))
responseF onComplete { ...
but i always get :
could not find implicit value for evidence parameter of type spray.httpx.marshalling.Marshaller[spray.json.JsValue]
[error] val responseF = pipeline(Post("/api/1.0/users/ping.json", """{ "key": "whatever" }""".asJson))
also the import you have in your previous snapshot does not work.... Am i using a bad version of the library? My sbt settings are:
val sprayVersion = "1.1-M7"
val akkaVersion = "2.1.1"
"io.spray" %% "spray-json" % "1.2.5",
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion,
"io.spray" % "spray-client" % sprayVersion,