Get a Stream of Entity from an Http4s Response with Circe - scala

I'm trying to retrieve a Stream[IO, Job] from an http4s Response, but the Scala compiler warns me that it cannot find any suitable Decoder:
Cannot decode into a value of type fs2.Stream[IO,Job], because no EntityDecoder[[+A]IO[A], fs2.Stream[IO,Job]] instance could be found.
[error] retrieved <- response.as[fs2.Stream[IO, Job]]
The code that generates the above error is the following:
import io.circe.generic.auto._
import org.http4s.circe.CirceEntityCodec._
// Many other imports
"should return the stream of all jobs" in {
for {
response <- jobsRoutes.orNotFound.run(
Request(
method = Method.GET,
uri = uri"/jobs",
headers = Headers(Accept(MediaType.`text/event-stream`))
)
)
retrieved <- response.as[fs2.Stream[IO, Job]]
} yield {
response.status shouldBe Status.Ok
}
}
In the build.sbt file, I have the following dependencies:
// Many other omitted dependencies
"org.http4s" %% "http4s-circe" % "0.23.14",
"io.circe" %% "circe-generic" % "0.14.2",
"io.circe" %% "circe-fs2" % "0.14.0",
The definition of the Job entity is:
final case class Job(
id: UUID,
date: Long,
salaryLo: Option[Int],
salaryHi: Option[Int],
currency: Option[String],
location: String,
tags: List[String],
description: String,
localUrl: Option[String],
externalUrl: Option[String]
image: Option[String],
country: Option[String],
title: String,
company: String,
seniority: Option[String],
other: Option[String]
)
I cannot understand what's going on.

Related

ZIO: How to return JSON ? [instead of using case class in ZIO-Http use schema to map?]

I tried directly getting body of JSON in code which I then want to convert to Avro to write to a kafka topic.
Here is my code with case class:
import zhttp.http._
import zio._
import zhttp.http.{Http, Method, Request, Response, Status}
import zhttp.service.Server
import zio.json._
import zio.kafka._
import zio.kafka.serde.Serde
import zio.schema._
case class Experiments(experimentId: String,
variantId: String,
accountId: String,
deviceId: String,
date: Int)
//case class RootInterface (events: Seq[Experiments])
object Experiments {
implicit val encoder: JsonEncoder[Experiments] = DeriveJsonEncoder.gen[Experiments]
implicit val decoder: JsonDecoder[Experiments] = DeriveJsonDecoder.gen[Experiments]
implicit val codec: JsonCodec[Experiments] = DeriveJsonCodec.gen[Experiments]
implicit val schema: Schema[Experiments] = DeriveSchema.gen
}
object HttpService {
def apply(): Http[ExpEnvironment, Throwable, Request, Response] =
Http.collectZIO[Request] {
case req#(Method.POST -> !! / "zioCollector") =>
val c = req.body.asString.map(_.fromJson[Experiments])
for {
u <- req.body.asString.map(_.fromJson[Experiments])
r <- u match {
case Left(e) =>
ZIO.debug(s"Failed to parse the input: $e").as(
Response.text(e).setStatus(Status.BadRequest)
)
case Right(u) =>
println(s"$u + =====")
ExpEnvironment.register(u)
.map(id => Response.text(id))
}
}
yield r
}
}
// val experimentsSerde: Serde[Any, Experiments] = Serde.string.inmapM { string =>
// //desericalization
// ZIO.fromEither(string.fromJson[Experiments].left.map(errorMessage => new RuntimeException(errorMessage)))
// } { theMatch =>
// ZIO.effect(theMatch.toJson)
//
// }
object ZioCollectorMain extends ZIOAppDefault {
def run: ZIO[Environment with ZIOAppArgs with Scope, Any, Any] = {
Server.start(
port = 9001,
http = HttpService()).provide(ZLayerExp.layer)
}
}
I'm looking into Zio-Json but no success yet, any help is appreciated !
We could also schema something to get the avro generic record
here's my json :
{
"experimentId": "abc",
"variantId": "123",
"accountId": "123",
"deviceId": "123",
"date": 1664544365
}
This function works for me in Scala 3 (sorry, I didn't include all the code but it should be enough):
import zio.*
import zio.Console.printLine
import zhttp.http.*
import zhttp.service.Server
import zio.json.*
...
case class Experiments(experimentId: String,
variantId: String,
accountId: String,
deviceId: String,
date: Int)
//case class RootInterface (events: Seq[Experiments])
object Experiments:
implicit val encoder: JsonEncoder[Experiments] = DeriveJsonEncoder.gen[Experiments]
implicit val decoder: JsonDecoder[Experiments] = DeriveJsonDecoder.gen[Experiments]
implicit val codec: JsonCodec[Experiments] = DeriveJsonCodec.gen[Experiments]
val postingExperiment: Http[Any, Throwable, Request, Response] =
Http.collectZIO[Request] {
case req#(Method.POST -> !! / "zioCollector") =>
//val c = req.body.asString.map(_.fromJson[Experiments])
val experimentsZIO = req.body.asString.map(_.fromJson[Experiments])
for {
experimentsOrError <- experimentsZIO
response <- experimentsOrError match {
case Left(e) => ZIO.debug(s"Failed to parse the input: $e").as(
Response.text(e).setStatus(Status.BadRequest)
)
case Right(experiments) => ZIO.succeed(Response.json(experiments.toJson))
}
} yield response
}
I modified your code slightly (you didn't post your ExpEnvironment class), and it returns back the object posted to the url.
and the test code is:
import sttp.client3.{SimpleHttpClient, UriContext, basicRequest}
object TestExperiments:
def main(args: Array[String]): Unit =
val client = SimpleHttpClient()
//post request
val request = basicRequest
.post(uri"http://localhost:9009/zioCollector")
.body("{ \"experimentId\": \"abc\", \"variantId\": \"123\", \"accountId\": \"123\", \"deviceId\": \"123\", \"date\": 1664544365 }")
val response = client.send(request)
println(response.body)
val invalidJsonRequest = basicRequest
.post(uri"http://localhost:9009/zioCollector")
.body("{ \"experimentId\": \"abc\", \"variantId\": \"123\", \"accountId\": \"123\", \"deviceId\": \"123\", \"date\": 1664544365 ") // missing the closing bracket
val invalidJsonResponse = client.send(invalidJsonRequest)
println(invalidJsonResponse.body)
You have to add: "com.softwaremill.sttp.client3" %% "core" % "3.8.3" to your sbt file.
build.sbt:
ThisBuild / scalaVersion := "3.2.0"
ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / organization := "TestSpeed"
ThisBuild / organizationName := "example"
lazy val root = (project in file("."))
.settings(
name := "TestZio",
libraryDependencies ++= Seq(
"dev.zio" %% "zio" % "2.0.2",
"dev.zio" %% "zio-json" % "0.3.0-RC11",
"io.d11" %% "zhttp" % "2.0.0-RC11",
"dev.zio" %% "zio-test" % "2.0.2" % Test,
"com.softwaremill.sttp.client3" %% "core" % "3.8.3" % Test
),
testFrameworks += new TestFramework("zio.test.sbt.ZTestFramework")
)
I didn't include anything related to avro because I am not familiar with it.

Matchers for cats EitherT and effects

I'm new in cats and functional programming and I'm struggling with unit testing functional data types like EitherT. Having example code:
class Library[F[_]]() {
def create(book: Book)(implicit M: Monad[F]): EitherT[F, BookAlreadyExistsError, Book] = ...
}
I'd like to test it using Spec2 but I don't know how to do it properly. Tried something like this but it does not work:
val library = Library[IO]()
test("create book") {
val book = Book("Title 1", 2016, "author 1")
(for (
resultBook <- library.create(book)
) yield resultBook shouldEqual ???
).unsafeRunSync()
}
I'd like to have very simple assertions like this:
resultBook shouldEqual Right(Book("Title 1", 2016, "author 1"))
// or
resultBook shouldEqual Left(BookAlreadyExistsError)
specs2-cats provides IOMatchers trait which enables the following syntax
library.create(book).value must returnValue(Right(book))
where
libraryDependencies += "org.specs2" %% "specs2-core" % "4.8.1" % Test,
libraryDependencies += "org.specs2" %% "specs2-cats" % "4.8.1" % Test,
Here is a working example
import cats.data.EitherT
import cats.effect.IO
import org.specs2.mutable.Specification
import org.specs2.matcher.IOMatchers
class CatsSpec extends Specification with IOMatchers {
case class Book(title: String, year: Int, author: String)
def create(book: Book): EitherT[IO, String, Book] = EitherT(IO(Right(book).withLeft[String]))
val book = Book("Title 1", 2016, "author 1")
"specs2-cats dependency" should {
"provide matcher for IO effect" in {
create(book).value must returnValue(Right(book))
}
}
}

Trouble serializing optional value class instances with json4s

I'm trying to serialize a case class with optional value-class field to JSON using json4s. So far, I'm not able to get the optional value-class field rendered correctly (see the snippet below with examples).
I tried json-native and json-jackson libraries, results are identical.
Here's a short self-contained test
import org.json4s.DefaultFormats
import org.scalatest.FunSuite
import org.json4s.native.Serialization._
class JsonConversionsTest extends FunSuite {
implicit val jsonFormats = DefaultFormats
test("optional value-class instance conversion") {
val json = writePretty(Foo(Option(Id(123)), "foo-name", Option("foo-opt"), Id(321)))
val actual =
"""
|{
| "id":{
| "value":123
| },
| "name":"foo-name",
| "optField":"foo-opt",
| "nonOptId":321
|}
|""".stripMargin.trim
assert(json === actual)
val correct =
"""
|{
| "id": 123,
| "name":"foo-name",
| "optField":"foo-opt",
| "nonOptId":321
|}
|""".stripMargin.trim
assert(json !== correct)
}
}
case class Id(value: Int) extends AnyVal
case class Foo(id: Option[Id], name: String, optField: Option[String], nonOptId: Id)
I'm using scala 2.12 and the latest json4s-native version:
"org.json4s" %% "json4s-native" % "3.6.7"
It looks very similar to this issue, doesn't seem to be fixed or commented on.
A custom serializer would save your day.
object IdSerializer extends CustomSerializer[Id] ( format => (
{ case JInt(a) => Id(a.toInt) },
{ case a: Id => JInt(a.value) }
))
implicit val formats = DefaultFormats + IdSerializer
val json = writePretty(Foo(Option(Id(123)), "foo-name", Option("foo-opt"), Id(321)))
In any case try other options like using of jsoniter-scala instead. It is much safe and efficient than json4s especially in serialization of prettified JSON.
Add/replace dependencies:
libraryDependencies ++= Seq(
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-core" % "0.55.2" % Compile,
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-macros" % "0.55.2" % Provided // required only in compile-time
)
Derive a codec for the top-level data structure and use it to serialize into a string:
import com.github.plokhotnyuk.jsoniter_scala.macros._
import com.github.plokhotnyuk.jsoniter_scala.core._
case class Id(value: Int) extends AnyVal
case class Foo(id: Option[Id], name: String, optField: Option[String], nonOptId: Id)
implicit val codec: JsonValueCodec[Foo] = JsonCodecMaker.make(CodecMakerConfig())
val json = writeToString(Foo(Option(Id(123)), "foo-name", Option("foo-opt"), Id(321)), WriterConfig(indentionStep = 2))
val correct =
"""{
| "id": 123,
| "name": "foo-name",
| "optField": "foo-opt",
| "nonOptId": 321
|}""".stripMargin
assert(json == correct)
There are also more efficient options to store into byte array, java.nio.ByteBuffer or java.io.OutputStream immediately.

What functional technique enables not having to pass configuration through functions

As I'm delving more into FP, I'm curious about the 'best' way to store settings that are loaded from config files. I've just been creating a case class with all the necessary config variables and setting that on app start. I then pass that case class into whatever function requires info from it.
However, it seems quite annoying especially when that settings case class has to propagate through many functions. Is there a better way to do this?
Reader monad provides a way of propagating configuration without having to pass it as a parameter throughout all functions that need it. Contrast the following two implementations:
 Config available from context via Reader[Config, String]
object ConfigFunctional extends App {
case class Config(username: String, password: String, host: String)
def encodeCredentials: Reader[Config, String] = Reader { config =>
Base64.getEncoder.encodeToString(s"${config.username}:${config.password}".getBytes())
}
def basicAuth(credentials: String): Reader[Config, String] = Reader { config =>
Http(s"${config.host}/HTTP/Basic/")
.header("Authorization", s"Basic $credentials")
.asString
.body
}
def validateResponse(body: String): Reader[Config, Either[String, String]] = Reader { _ =>
if (body.contains("Your browser made it"))
Right("Credentials are valid!")
else
Left("Wrong credentials")
}
def program: Reader[Config, Either[String, String]] = for {
credentials <- encodeCredentials
response <- basicAuth(credentials)
validation <- validateResponse(response)
} yield validation
val config = Config("guest", "guest", "https://jigsaw.w3.org")
println(program.run(config))
}
Config passed in as an argument
object ConfigImperative extends App {
case class Config(username: String, password: String, host: String)
def encodeCredentials(config: Config): String = {
Base64.getEncoder.encodeToString(s"${config.username}:${config.password}".getBytes())
}
def basicAuth(credentials: String, config: Config): String = {
Http(s"${config.host}/HTTP/Basic/")
.header("Authorization", s"Basic $credentials")
.asString
.body
}
def validateResponse(body: String): Either[String, String] = {
if (body.contains("Your browser made it"))
Right("Credentials are valid!")
else
Left("Wrong credentials")
}
def program(config: Config): Either[String, String] = {
val credentials = encodeCredentials(config)
val response = basicAuth(credentials, config)
val validation = validateResponse(response)
validation
}
val config = Config("guest", "guest", "https://jigsaw.w3.org")
println(program(config))
}
Both implementations should output Right(Credentials are valid!), however notice how in the first implementation config: Config is not a method parameter, for example, contrast encodeCredentials:
def encodeCredentials: Reader[Config, String]
def encodeCredentials(config: Config): String
Config appears in the return type instead of being a parameter. We can interpret this as meaning
"When encodeCredentials runs in the context that provides a
Config, then it will produce a String result."
The "context" here is represented by Reader monad.
Furthermore, notice how Config is not a parameter even in the main business logic
def program: Reader[Config, Either[String, String]] = for {
credentials <- encodeCredentials
response <- basicAuth(credentials)
validation <- validateResponse(response)
} yield validation
We let the methods evaluate in the context containing Config via run function:
program.run(config)
To run above examples we need the following dependencies
scalacOptions += "-Ypartial-unification",
libraryDependencies ++= Seq(
"org.typelevel" %% "cats-core" % "1.6.0",
"org.scalaj" %% "scalaj-http" % "2.4.1"
)
and imports
import cats.data.Reader
import java.util.Base64
import scalaj.http.Http

playframework scala slick how many attributes are allowed

In my Playframework scala application I have the following model:
case class ProcessTemplatesModel(
id: Option[Int] = None,
title: String,
version: String,
createdat: Option[String],
updatedat: Option[String],
deadline: Option[Date],
status: Option[String],
comment: Option[String],
checked: Option[Boolean],
checkedat: Option[Date],
approved: Option[Boolean],
approvedat: Option[Date],
deleted: Boolean,
approveprocess: Int,
trainingsprocess: Option[Int],
previousVersion: Option[Int],
originTemplate: Option[Int],
client: Int,
approveProcessInstance: Option[Int],
responsible: Option[Seq[UserModel]],
accountable: Option[Seq[UserModel]],
consulted: Option[Seq[UserModel]],
informed: Option[Seq[UserModel]])
object ProcessTemplatesModel {
implicit val processFormat = Json.format[ProcessTemplatesModel]
}
Today I added the approveProcessInstance: Option[Int],
Now I got this error while it compiles: No unapply or unapplySeq function found ... on this line: implicit val processFormat = Json.format[ProcessTemplatesModel]
Why does this fail in this case?
22 values is a max in play JSON, you can use 3rd party libraries to increase the number.
Here the issue thread in the Play source:
https://github.com/playframework/playframework/issues/3174
One of the possible solution:
https://github.com/xdotai/play-json-extensions
Example from my build.sbt
libraryDependencies ++= Seq(
cache,
filters,
ws,
// More than 22 fields in Json
"ai.x" %% "play-json-extensions" % "0.8.0"
)
For Play 2.6 you need to use version 10:
"ai.x" %% "play-json-extensions" % "0.10.0"
Then, in the file with JSON:
import ai.x.play.json.Jsonx
implicit val processFormat = Jsonx.formatCaseClass[ProcessTemplatesModel]
More details: https://github.com/xdotai/play-json-extensions#create-explicit-formatter
One other idea would be to decompose the model into fine grained models and have a composition of it. This way, you do not have to import yet another library!