I'm trying to write a Circe encoder for an object which has a field of scala.collection.immutable.SortedMultiDict. Circe can't find an encoder instance for that, so I need to write one.
import io.circe.{Decoder, Encoder, HCursor}
import io.circe.generic.semiauto._
import io.circe.parser.decode
import scala.collection.immutable.SortedMultiDict
import io.circe.syntax._
implicit val mapEncoder: Encoder[List[(Long, String)]] = deriveEncoder[List[(Long, String)]]
implicit val mapDecoder: Decoder[List[(Long, String)]] = deriveDecoder[List[(Long, String)]]
implicit val oneEncoder: Encoder[SortedMultiDict[Long, String]] = (a: SortedMultiDict[Long, String]) =>
mapEncoder(a.toList)
implicit val oneDecoder: Decoder[SortedMultiDict[Long, String]] = (c: HCursor) =>
mapDecoder.map(SortedMultiDict.from[Long, String])(c)
Sadly, this isn't correct...
val test = SortedMultiDict.from[Long, String](Seq(1666268475626L -> "a5d9f51d-35c7-4fef-b4a3-3d28944eeb2b", 1666268475626L -> "df359396-043c-4b65-bc3 -bf309d433ff5"))
val encodedData = test.asJson.noSpaces
val roundTrip = decode[SortedMultiDict[Long, String]](encodedData)
results in
scala> roundTrip
val res2: Either[io.circe.Error,scala.collection.immutable.SortedMultiDict[Long,String]] = Left(DecodingFailure(Attempt to decode value on failed cursor, List(DownField(head), DownField(::))))
In fact, the derived list encoder doesn't appear to work...
scala> val myList = List((1666268475626L, "a5d9f51d-35c7-4fef-b4a3-3d28944eeb2b"), (1666268475626L, "df359396-043c-4b65-bc3 -bf309d433ff5"))
val myList: List[(Long, String)] = List((1666268475626,a5d9f51d-35c7-4fef-b4a3-3d28944eeb2b), (1666268475626,df359396-043c-4b65-bc3 -bf309d433ff5))
scala> decode[List[(Long, String)]](myList.asJson.noSpaces)
val res0: Either[io.circe.Error,List[(Long, String)]] = Left(DecodingFailure(Attempt to decode value on failed cursor, List(DownField(head), DownField(::))))
Are my expectations of how to do the round trip of encoding/decoding wrong? It's what I'd understood from Circe's codec docs.
EDIT: Well, it works if I change the map codecs to be:
implicit val mapEncoder: Encoder[List[(Long, String)]] = Encoder.encodeList[(Long, String)]
implicit val mapDecoder: Decoder[List[(Long, String)]] = Decoder.decodeList[(Long, String)]
I still don't really understand why the earlier ones don't work, though, explications welcome...
Related
in the below given code snipet i have declared and rdd by parallelizing a List(1,2,3,4) what i wanted do was to append List(1,2,3,4) to each element of the above rdd. i did so by using nested flatMap function scince it can return multiple values for each element of a RDD .The code is as follows
val rand6=sc.parallelize(List(1,2,3,4))
val bv=sc.broadcast(List(5,6,7,8))
rand6.flatMap(s=>{
val c=List(1,2,3,4)
val a=List(s,c)
val b=a.flatMap(r=>r)
b
})
But I am getting the following Error
command-1095314872161512:74: error: type mismatch;
found : Any
required: scala.collection.GenTraversableOnce[?]
val b=a.flatMap(r=>r)
^
is is the problem with the syntax or we are not supposed to use flatMaps in this fashion
it would be very helpful if someone can help me to understand this
Try to add type wherever possible in your scala code
Depending on your question description came up with below solution
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.rdd.RDD
object RandomDF {
def main(args: Array[String]): Unit = {
val spark = Constant.getSparkSess
val sc = spark.sparkContext
val rand6 : RDD[Int] =sc.parallelize(List(1,2,3,4))
val bv: Broadcast[List[Int]] =sc.broadcast(List(5,6,7,8))
val output = rand6.map( (s : Int)=>{
val c : List[Int] =List(1,2,3,4)
val a = s :: c
// val b = a.flatMap(r=>r)
// b
a
}).collect().toList
println(output)
}
}
I have below case class
case class Foo(code: Int, msg: String, headers: Map[String,String] = Map.empty)
Below is the code that I've tried so far -
import scodec._
import scodec.codecs._
implicit val mapCodec: Codec[List[(String, String)]] = sizedList()
implicit val fooCodec : Codec[Foo] = {
("code" | int32) :: ("msg" | cstring) :: ("headers" | mapCodec)
}.as[Foo]
I don't know how to write Codec for Map[String, String]. I checked online documentation but it is still in TODO.
Any idea how can I write the codec for Map[String, String]?
What you would need to do is to define Codec for tuple of strings, which then you will need to use to create codec for List[(String, String)] which can be converted to Map[String, String] and vice versa, hence covert Codec using xmap function.
So the final solution might look like:
import scodec._
import scodec.codecs._
case class Foo(code: Int, msg: String, headers: Map[String,String] = Map.empty)
implicit val tupleCodec : Codec[(String, String)] = cstring.pairedWith(cstring)
implicit val mapCodec: Codec[Map[String, String]] = list(tupleCodec).xmap(_.toMap, _.toList)
implicit val fooCodec : Codec[Foo] = {
("code" | int32) :: ("msg" | cstring) :: ("headers" | mapCodec)
}.as[Foo]
Hope this helps!
I am very new to akka-http, and I would like to stream a csv with an arbitrary number of lines.
For instance, I would like to return :
a,1
b,2
c,3
with the following code
implicit val actorSystem = ActorSystem("system")
implicit val actorMaterializer = ActorMaterializer()
val map = new mutable.HashMap[String, Int]()
map.put("a", 1)
map.put("b", 2)
map.put("c", 3)
val `text/csv` = ContentType(MediaTypes.`text/csv`, `UTF-8`)
val route =
path("test") {
complete {
HttpEntity(`text/csv`, ??? using map)
}
}
Http().bindAndHandle(route,"localhost",8080)
Thanks for your help
EDIT: Thanks to Ramon J Romero y Vigil
package test
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.HttpCharsets.`UTF-8`
import akka.http.scaladsl.model._
import akka.http.scaladsl.server.Directives._
import akka.stream._
import akka.util.ByteString
import scala.collection.mutable
object Test{
def main(args: Array[String]) {
implicit val actorSystem = ActorSystem("system")
implicit val actorMaterializer = ActorMaterializer()
val map = new mutable.HashMap[String, Int]()
map.put("a", 1)
map.put("b", 2)
map.put("c", 3)
val mapStream = Stream.fromIterator(() => map.toIterator)
.map((k: String, v: Int) => s"$k,$v")
.map(ByteString.apply)
val `text/csv` = ContentType(MediaTypes.`text/csv`, `UTF-8`)
val route =
path("test") {
complete {
HttpEntity(`text/csv`, mapStream)
}
}
Http().bindAndHandle(route, "localhost", 8080)
}
}
With this code I have two compile error:
Error:(29, 28) value fromIterator is not a member of object scala.collection.immutable.Stream
val mapStream = Stream.fromIterator(() => map.toIterator)
Error:(38, 11) overloaded method value apply with alternatives:
(contentType: akka.http.scaladsl.model.ContentType,file: java.io.File,chunkSize: Int)akka.http.scaladsl.model.UniversalEntity <and>
(contentType: akka.http.scaladsl.model.ContentType,data: akka.stream.scaladsl.Source[akka.util.ByteString,Any])akka.http.scaladsl.model.HttpEntity.Chunked <and>
(contentType: akka.http.scaladsl.model.ContentType,data: akka.util.ByteString)akka.http.scaladsl.model.HttpEntity.Strict <and>
(contentType: akka.http.scaladsl.model.ContentType,bytes: Array[Byte])akka.http.scaladsl.model.HttpEntity.Strict <and>
(contentType: akka.http.scaladsl.model.ContentType.NonBinary,string: String)akka.http.scaladsl.model.HttpEntity.Strict
cannot be applied to (akka.http.scaladsl.model.ContentType.WithCharset, List[akka.util.ByteString])
HttpEntity(`text/csv`, mapStream)
I used a List of tuples to get arround the first issue (hower i do not know how to stream a map in Scala)
No idea for the second
Thanks for your help.
(I am using scala 2.11.8)
Use the apply function in HttpEntity that takes in a Source[ByteString,Any]. The apply creates a Chunked entity. You can read your file using code based on the documentation for streaming file IO using an akka stream Source:
import akka.stream.scaladsl._
val file = Paths.get("yourFile.csv")
val entity = HttpEntity(`txt/csv`, FileIO.fromPath(file))
The stream will break up your file into chunk sizes, default is currently set to 8192.
To stream the map that you've created you can use a similar trick:
val mapStream = Source.fromIterator(() => map.toIterator)
.map( (k : String, v : Int) => s"$k,$v" )
.map(ByteString.apply)
val mapEntity = HttpEntity(`test/csv`, mapStream)
I need to flatten a sequence of cats.data.ValidatedNel[E, T] values to a single ValidatedNel value:
val results: Seq[cats.data.ValidatedNel[E, T]] = ???
val flattenedResult: cats.data.ValidatedNel[E, T]
I can do it like this:
import cats.std.list._, cats.syntax.cartesian._
results.reduce(_ |#| _ map { case _ => validatedValue })
but wonder if a pre-defined library methods exists.
It depends on how you want to combine them (what is validatedValue in your question ?)
import cats.data.{Validated, ValidatedNel}
import cats.implicits._
val validations1 = List(1.validNel[String], 2.valid, 3.valid)
val validations2 = List(1.validNel[String], "kaboom".invalidNel, "boom".invalidNel)
If you want to combine the Ts, you can use Foldable.combineAll which uses a Monoid[T] :
val valSum1 = validations1.combineAll
// Valid(6)
val valSum2 = validations2.combineAll
// Invalid(OneAnd(kaboom,List(boom)))
If you want to get a ValidationNel[String, List[T]], you can use Traverse.sequence :
val valList1: ValidatedNel[String, List[Int]] = validations1.sequence
// Valid(List(1, 2, 3))
val valList2: ValidatedNel[String, List[Int]] = validations2.sequence
// Invalid(OneAnd(kaboom,List(boom)))
If you don't care about the result, which seems to be the case, you can use Foldable.sequence_.
val result1: ValidatedNel[String, Unit] = validations1.sequence_
// Valid(())
val result2: ValidatedNel[String, Unit] = validations2.sequence_
// Invalid(OneAnd(kaboom,List(boom)))
validations1.sequence_.as(validatedValue) // as(x) is equal to map(_ => x)
Is it possible to transform a List[F[G[A]]] into F[G[List[A]]] nicely?
I can do this in Scalaz the following way:
val x: List[Future[Option[Int]]] = ???
val transformed: Future[Option[List[Int]]] = x.sequenceU.map(_.sequenceU)
I'm just wondering if there's a nicer way to do this rather than .sequenceU.map(_.sequenceU) Perhaps using a monad transformer? I did attempt this, without much luck.
Monad transformers are the way to go if you want to avoid the nested sequencing. In this case you want an OptionT[Future, A] (which is equivalent to Future[Option[A]]):
import scalaz._, Scalaz._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future
val xs = List(OptionT(Future(some(1))), OptionT(Future(some(2))))
val ys = OptionT(Future(none[Int])) :: xs
val sequencedXs: Future[Option[List[Int]]] = xs.sequenceU.run
val sequencedYs: Future[Option[List[Int]]] = ys.sequenceU.run
And then:
scala> sequencedXs.foreach(println)
Some(List(1, 2))
scala> sequencedYs.foreach(println)
None
As expected.
I've realised that while a monad transformer is great here, I can take advantage that applicatives compose:
def transform[F[_], G[_], A](list: List[F[G[A]]])(implicit af: Applicative[F], ao: Applicative[G]): F[G[List[A]]] = {
type λ[α] = F[G[α]]
implicit val applicative: Applicative[λ] = af compose ao
list.sequence[λ, A]
}
val lfo: List[Future[Option[Int]]] = List(Future(1.some), Future(2.some))
val future: Future[Option[List[Int]]] = transform(lfo)
And then:
scala> future.foreach(println)
Some(List(1, 2))