I have the following function, that do recursion:
#tailrec
private def pool[F[_]: Monad, A]
: Consumer[String, String] => (Vector[KkConsumerRecord] => F[A]) => IO[Unit]
= consumer => cb => {
val records: ConsumerRecords[String, String] = consumer.poll(Long.MaxValue)
val converted = records.iterator().asScala.map(rec => {
KkConsumerRecord(rec.key(), rec.value(), rec.offset(), rec.partition(), rec.topic())
})
val vec = converted.foldLeft(Vector.empty[KkConsumerRecord]) { (b, a) =>
a +: b
}
cb(vec)
pool(consumer)(cb)
}
The compiler complains:
[error] /home/developer/Desktop/microservices/bary/kafka-api/src/main/scala/io/khinkali/Consumer/KkConsumer.scala:57:10: type mismatch;
[error] found : org.apache.kafka.clients.consumer.Consumer[String,String]
[error] required: cats.Monad[?]
[error] pool(consumer)(cb)
[error] ^
[error] two errors found
What am I doing wrong?
The following code compiles:
import cats.Monad
import cats.effect.IO
import org.apache.kafka.clients.consumer.{Consumer, ConsumerRecords}
import scala.collection.JavaConverters._
import scala.annotation.tailrec
object App {
case class KkConsumerRecord(key: String, value: String, offset: Long, partition: Int, topic: String)
// #tailrec
private def pool[F[_]: Monad, A]
: Consumer[String, String] => (Vector[KkConsumerRecord] => F[A]) => IO[Unit]
= consumer => cb => {
val records: ConsumerRecords[String, String] = consumer.poll(Long.MaxValue)
val converted = records.iterator().asScala.map(rec => {
KkConsumerRecord(rec.key(), rec.value(), rec.offset(), rec.partition(), rec.topic())
})
val vec = converted.foldLeft(Vector.empty[KkConsumerRecord]) { (b, a) =>
a +: b
}
cb(vec)
pool.apply(consumer)(cb)
}
}
def pool[F[_]: Monad, A] means def pool[F[_], A](implicit monad: Monad[F]) so compiler mistreated consumer as implicit parameter.
tailrec annotation is removed since pool is not tail recursive (the last operation is constructing lambda, I guess it's called tail recursion modulo cons).
If you want to make it tail-recursive you can rewrite it as
#tailrec
private def pool[F[_]: Monad, A](consumer: Consumer[String, String])(cb: Vector[KkConsumerRecord] => F[A]): IO[Unit] = {
val records: ConsumerRecords[String, String] = consumer.poll(Long.MaxValue)
val converted = records.iterator().asScala.map(rec => {
KkConsumerRecord(rec.key(), rec.value(), rec.offset(), rec.partition(), rec.topic())
})
val vec = converted.foldLeft(Vector.empty[KkConsumerRecord]) { (b, a) =>
a +: b
}
cb(vec)
pool(consumer)(cb)
}
Related
In my project I'm working with various handlers to perform logic on arrays of different primitive types, and I came across this runtime error:
[error] java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class [B ([Ljava.lang.Object; and [B are in module java.base of loader 'bootstrap')
In this case I was working with Bytes, hence the [B, but one of my functions returned [Ljava.lang.Object instead (at runtime!). How can I ensure my generic functions return primitive arrays instead of object arrays?
Here's a minimal example to reproduce it:
package asdf
import scala.reflect.ClassTag
import java.nio.charset.StandardCharsets
object Main {
trait Helper[A, B] {
def decode(bytes: Array[Byte]): Array[A]
def aggregate(values: Array[A]): B
}
object StringHelper extends Helper[Byte, String] {
def decode(bytes: Array[Byte]): Array[Byte] = bytes.filter(_ != 0)
def aggregate(values: Array[Byte]): String = new String(values, StandardCharsets.UTF_8)
}
object IntHelper extends Helper[Int, Int] {
def decode(bytes: Array[Byte]): Array[Int] = bytes.map(_.toInt)
def aggregate(values: Array[Int]): Int = values.sum
}
def decodeAgg[A, B](bytes: Array[Byte], helper: Helper[A, B])(implicit ev: ClassTag[A]): B = {
val decoded = helper.decode(bytes)
val notFirstDecoded = decoded
.zipWithIndex
.filter({case (_, i) => i != 0})
.map(_._1)
.toArray
helper.aggregate(notFirstDecoded)
}
def main(args: Array[String]) {
val helper: Helper[_, _] = args(1) match {
case "int" => IntHelper
case "string" => StringHelper
case _ => throw new Exception("unknown data type")
}
val bytes = Array(97, 98, 99).map(_.toByte)
val aggregated = decodeAgg(bytes, helper)
println(s"aggregated to $aggregated")
}
}
Run with sbt "run -- string".
Full stack trace on this example:
[error] java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class [B ([Ljava.lang.Object; and [B are in module java.base of loader 'bootstrap')
[error] at asdf.Main$StringHelper$.aggregate(Main.scala:12)
[error] at asdf.Main$.decodeAgg(Main.scala:29)
[error] at asdf.Main$.main(Main.scala:39)
I was using Scala 2.12, JDK 13.
I've tried using #specialized to no effect.
The problem lies in your toArray in call:
val notFirstDecoded = decoded
.zipWithIndex
.filter({case (_, i) => i != 0})
.map(_._1)
.toArray
toArray takes implicit ClassTag argument - it needs to know the runtime type of array elements to create the Array.
As you provide implicit ClassTag argument to decodeAgg, toArray happily takes what you deliver.
def decodeAgg[A, B](bytes: Array[Byte], helper: Helper[A, B])(implicit ev: ClassTag[A]): B
You can see that ClassTag corresponds to first generic argument of Helper.
You pass following helper:
val helper: Helper[_, _] = args(1) match {
case "int" => IntHelper
case "string" => StringHelper
case _ => throw new Exception("unknown data type")
}
The ClassTag is thus inferred to Object, which is why you get an Array of objects.
Note that if you use an IntHelper directly, the ClassTag is constrained to the correct type, and the function works as expected.
val aggregated = decodeAgg(bytes, IntHelper)
Solution ideas
There might be multiple ways to resolve it.
One idea might be to provide the classTag explicitly via Helper
import scala.reflect.ClassTag
import java.nio.charset.StandardCharsets
object Main {
trait Helper[A, B] {
def decode(bytes: Array[Byte]): Array[A]
def aggregate(values: Array[A]): B
def classTag: ClassTag[A]
}
object StringHelper extends Helper[Byte, String] {
def decode(bytes: Array[Byte]): Array[Byte] = bytes.filter(_ != 0)
def aggregate(values: Array[Byte]): String = new String(values, StandardCharsets.UTF_8)
def classTag: ClassTag[Byte] = ClassTag(classOf[Byte])
}
object IntHelper extends Helper[Int, Int] {
def decode(bytes: Array[Byte]): Array[Int] = bytes.map(_.toInt)
def aggregate(values: Array[Int]): Int = values.sum
def classTag: ClassTag[Int] = ClassTag(classOf[Int])
}
def decodeAgg[A, B](bytes: Array[Byte], helper: Helper[A, B]): B = {
val decoded = helper.decode(bytes)
val notFirstDecoded = decoded
.zipWithIndex
.filter({case (_, i) => i != 0})
.map(_._1)
.toArray(helper.classTag)
helper.aggregate(notFirstDecoded)
}
def main(args: Array[String]) {
val helper = args(1) match {
case "int" => IntHelper
case "string" => StringHelper
case _ => throw new Exception("unknown data type")
}
val bytes = Array(97, 98, 99).map(_.toByte)
val aggregated = decodeAgg(bytes, helper)
println(s"aggregated to $aggregated")
}
}
I've defined a simple generic macro:
object MyMacro {
def readWrite[T](readParse: String => T, label: String, format: T => String): Unit = macro readWriteImpl[T]
def readWriteImpl[T: c.WeakTypeTag](c: Context)(readParse: c.Expr[String => T], label: c.Expr[String], format: c.Expr[T => String]): c.Tree = {
import c.universe._
q"""
def read[WIRE](path: Path, reader: Transceiver[WIRE], isMapKey: Boolean = false): T =
reader.readString(path) match {
case null => null.asInstanceOf[T]
case s => Try( $readParse(s) ) match {
case Success(d) => d
case Failure(u) => throw new ReadMalformedError(path, "Failed to parse "+$label+" from input '"+s+"'", List.empty[String], u)
}
}
def write[WIRE](t: T, writer: Transceiver[WIRE], out: Builder[Any, WIRE]): Unit =
t match {
case null => writer.writeNull(out)
case _ => writer.writeString($format(t), out)
}
"""
}
}
In a separate compilation unit I use it like this:
object DurationTypeAdapterFactory extends TypeAdapter.=:=[Duration] {
MyMacro.readWrite[Duration]((s: String) => Duration.parse(s), "Duration", (t: Duration) => t.toString)
}
When built, the compiler complains it doesn't know about T:
[error] /Users/me/git/ScalaJack/core/src/main/scala/co.blocke.scalajack/typeadapter/TimePrimitives.scala:13:30: not found: type T
[error] MyMacro.readWrite[Duration]((s: String) => Duration.parse(s), "Duration", (t: Duration) => t.toString)
[error]
It doesn't like the 'T' references in my quasiquote, which I kinda understand. How can I represent the T paraeter passed into readWriteImpl inside the quasiquote so that it unpacks properly?
Use the tag to inspect the type or compare it to other types using =:=, or use it in the expansion.
For example,
scala 2.13.0-M5> def fImpl[A: c.WeakTypeTag](c: Context)(a: c.Expr[A]) = { import c._, universe._
| q"null.asInstanceOf[${implicitly[c.WeakTypeTag[A]].tpe}]" }
fImpl: [A](c: scala.reflect.macros.blackbox.Context)(a: c.Expr[A])(implicit evidence$1: c.WeakTypeTag[A])c.universe.Tree
scala 2.13.0-M5> import language.experimental.macros ; def f[A](a: A): A = macro fImpl[A]
import language.experimental.macros
defined term macro f: [A](a: A)A
scala 2.13.0-M5> f(42)
res2: Int = 0
scala 2.13.0-M5> f("")
res3: String = null
The following should show the problem:
class Container[A](val xs: List[A]) {
def foo(fun: A => A)(implicit ord: Ordering[A]): List[A] =
macro ContainerMacros.fooImpl // how to pass `xs`?
}
object ContainerMacros {
def fooImpl(c: blackbox.Context)
(fun: c.Expr[Nothing => Any])
(ord: c.Expr[Ordering[_]]): c.Expr[List[Nothing]] = {
import c.universe._
reify {
// val cont = c.prefix.splice.asInstanceOf[Container[_]]
// cont.xs.map(fun.splice).sorted(ord.splice)
???
}
}
}
That is, there seems no way to add A as a type parameter to fooImpl (if I do that, the compiler complains). So I have to remove it, but then the question is how to get the things in reify working the intended way, i.e. how to reintroduce the missing type parameter A.
Here is an attempt:
def fooImpl(c: blackbox.Context)
(fun: c.Expr[Nothing => Any])
(ord: c.Expr[Ordering[_]]): c.Expr[List[Nothing]] = {
import c.universe._
reify {
val ext = c.prefix.splice.asInstanceOf[ExtensionTest[_]]
import ext.{A1 => A}
val extC = ext .asInstanceOf[ExtensionTest[A]]
val funC = fun.splice.asInstanceOf[A => A]
val ordC = ord.splice.asInstanceOf[Ordering[A]]
val xs = extC.xs.asInstanceOf[List[A]] // computer says no
xs.map(funC).sorted(ordC)
}
}
The xs.map(funC) doesn't work with one of those legendary scalac error messages:
Error:(27, 16) type mismatch;
found : _$2 => Any where type _$2
required: Any => ?
xs.map(funC).sorted(ordC)
Answer actually thanks to #edmundnoble via Scala Gitter channel:
You can add type parameters in the impl call, thus it's possible to pass the A from Container to the macro expansion:
class Container[A](val xs: List[A]) {
def foo(fun: A => A)(implicit ord: Ordering[A]): List[A] =
macro ContainerMacros.fooImpl[A]
}
object ContainerMacros {
def fooImpl[A](c: blackbox.Context)
(fun: c.Expr[A => A])
(ord: c.Expr[Ordering[A]])
(implicit t: c.WeakTypeTag[A]): c.Expr[List[A]] = {
import c.universe._
reify {
val extC = c.prefix.splice.asInstanceOf[Container[A]]
val funC = fun.splice
val ordC = ord.splice
val xs = extC.xs
xs.map(funC).sorted(ordC)
}
}
}
I would like to do something like this:
implicit class MyString(s: String) {
def getAs[T]: T = {
T match {
case q if q == classOf[Int] => s.toInt
case q if q == classOf[Boolean] => s.toBoolean
}
}
}
This doesn't compile, of course. How do I write this so it does?
Consider this approach:
object Parse {
def parse[T](f:String => Option[T]) = f
implicit val parseInt = parse(s => Try(s.toInt).toOption)
implicit val parseLong = parse(s => Try(s.toLong).toOption)
implicit val parseDouble = parse(s => Try(s.toDouble).toOption)
implicit val parseBoolean = parse(s => Try(s.toBoolean).toOption)
}
implicit class MyString(s:String) {
def getAs[T]()(implicit run: String => Option[T]): Option[T] = run(s)
}
Usage:
def main(args: Array[String]) {
import Parse._
"true".getAs[Boolean].foreach(println)
"12345".getAs[Int].foreach(println)
}
When use classOf for type match, there maybe will have some issues, example:
scala> classOf[List[Int]] == classOf[List[String]]
res17: Boolean = true
scala> typeOf[List[Int]] =:= typeOf[List[String]]
res18: Boolean = false
classOf only store the class information not with generics type
typeOf will store full type information
TypeTags and Manifests
class MyString(s: String) {
def getAs[T](implicit tag: TypeTag[T]): T = {
tag.tpe match {
case t if t =:= typeOf[Int] => s.toInt.asInstanceOf[T]
}
}
}
scala> new MyString("123")
res2: MyString = MyString#7fca02
scala> res6.getAs[Int]
res3: Int = 123
use TypeType tag to get type info, and call getAs with type information.
This is what I have come up with so far:
import reflect.runtime.universe.TypeTag
import scala.reflection.runtime.universe._
implicit class MyString(s: String) {
def getAs[T : TypeTag]: T = {
typeOf[T] match {
case t if t =:= typeOf[Int] => s.toInt.asInstanceOf[T]
case t if t =:= typeOf[Boolean] => s.toBoolean.asInstanceOf[T]
}
}
}
Running with this in REPL:
scala> "32".getAs[Int]
res25: Int = 32
scala> "32".getAs[Boolean]
java.lang.IllegalArgumentException: For input string: "32"
at scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:290)
at scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:260)
at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:30)
at MyString.getAs(<console>:33)
... 43 elided
scala> "false".getAs[Int]
java.lang.NumberFormatException: For input string: "false"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:30)
at MyString.getAs(<console>:32)
... 43 elided
scala> "false".getAs[Boolean]
res28: Boolean = false
scala> "false".getAs[String]
scala.MatchError: String (of class scala.reflect.internal.Types$AliasNoArgsTypeRef)
at MyString.getAs(<console>:31)
... 43 elided
Better, I think, would be to return an option[T], and catch any issues such as a type T that isn't catered for, or attempting a cast that will fail. I started mucking around with scala's Try (and its .toOption method), but hit some odd errors so haven't gotten any further with that.
EDIT: just using a simple try-catch, we can get:
implicit class MyString(s: String) {
def getAs[T : TypeTag]: Option[T] = try {
typeOf[T] match {
case t if t =:= typeOf[Int] => Some(s.toInt.asInstanceOf[T])
case t if t =:= typeOf[Boolean] => Some(s.toBoolean.asInstanceOf[T])
}
} catch {
case ex: Exception => None
}
}
Resulting in:
scala> "false".getAs[String]
res30: Option[String] = None
scala> "32".getAs[Boolean]
res31: Option[Boolean] = None
scala> "32".getAs[Int]
res32: Option[Int] = Some(32)
scala> "true".getAs[Boolean]
res33: Option[Boolean] = Some(true)
scala> "true".getAs[Int]
res34: Option[Int] = None
Based on the answers given by #chengpohi and #Shadowlands, this is what I came up with.
object ImplicitsStartingWithS {
implicit class MyString(s: String) {
val str = s.trim
import reflect.runtime.universe.TypeTag
import scala.reflect.runtime.universe._
def getAs[T](implicit tag: TypeTag[T]): Option[T] = {
val value = tag.tpe match {
case t if t =:= typeOf[Int] => str.toIntStr.map(_.toInt)
case t if t =:= typeOf[Long] => str.toIntStr.map(_.toLong)
case t if t =:= typeOf[Float] => str.toNumericStr.map(_.toFloat)
case t if t =:= typeOf[Double] => str.toNumericStr.map(_.toDouble)
case _ => None
}
value.asInstanceOf[Option[T]]
}
def toDecimalStr = "^-*\\d+\\.\\d+$".r.findFirstIn(s)
def toIntStr = "^-*\\d+$".r.findFirstIn(s)
def toNumericStr = {
s.toDecimalStr match {
case Some(decimalStr) => Some(decimalStr)
case None => s.toIntStr
}
}
}
}
This avoids exception handling for faster response.
I was wondering if anyone could provide some insight on a problem I'm having. I've made a gist with some code and explanation of my problem: https://gist.github.com/tbrown1979/9993f07c8f4fa2786c83
Basically I'm trying to make something that will allow me to convert List[String] to a case class. I've made a Reader that will allow me to do so, but I've run into the issue where a Reader defined for a case class can't contain a reader for a separate case class.
Looking at the 'non-working example' below - I encounter an issue where, when reading, I don't know how many items to pull out of the list. With Bar, which holds a Test, I would need to pull 2 elements out (because Test has two parameters). Is there a way for me to know the amount of fields a case class has just from its type? Is there a better way to do this?
Here is an example of how to use the Reader. I've included a non-working example as well.
////Working Example////
case class Foo(a: Int, s: String)
object Foo {
implicit val FooReader : Reader[Foo] =
Reader[Int :: String :: HNil].map(Generic[Foo].from _)
}
val read: ValidationNel[String, Foo] = Reader.read[Foo](List("12","text"))
println(read)//Success(Foo(12, "text"))
///////////////////////////
////Non-working Example////
case class Test(a: Int, b: String)
object Test {
implicit val TestReader: Reader[Test] =
Reader[Int :: String :: HNil].map(Generic[Test].from _)
}
case class Bar(c: Test)
object Bar {
implicit val BarReader: Reader[Bar] =
Reader[Test :: HNil].map(Generic[Bar].from _)
}
val barRead = Reader.read[Bar](List("21", "someString"))
println(barRead) //Failure(NonEmptyList("Invalid String: List()", "Exepected empty, but contained value"))
//////////////////////////
Something like this works for me (modification of this)
object ShapelessStringToTypeConverters {
import cats._, implicits._, data.ValidatedNel
import mouse._, string._, option._
import shapeless._, labelled._
private type Result[A] = ValidatedNel[ParseFailure, A]
case class ParseFailure(error: String)
trait Convert[V] {
def parse(input: String): Result[V]
}
object Convert {
def to[V](input: String)(implicit C: Convert[V]): Result[V] =
C.parse(input)
def instance[V](body: String => Result[V]): Convert[V] = new Convert[V] {
def parse(input: String): Result[V] = body(input)
}
implicit def booleans: Convert[Boolean] =
Convert.instance(
s =>
s.parseBooleanValidated
.leftMap(e => ParseFailure(s"Not a Boolean ${e.getMessage}"))
.toValidatedNel)
implicit def ints: Convert[Int] =
Convert.instance(
s =>
s.parseIntValidated
.leftMap(e => ParseFailure(s"Not an Int ${e.getMessage}"))
.toValidatedNel)
implicit def longs: Convert[Long] =
Convert.instance(
s =>
s.parseLongValidated
.leftMap(e => ParseFailure(s"Not an Long ${e.getMessage}"))
.toValidatedNel)
implicit def doubles: Convert[Double] =
Convert.instance(
s =>
s.parseDoubleValidated
.leftMap(e => ParseFailure(s"Not an Double ${e.getMessage}"))
.toValidatedNel)
implicit def strings: Convert[String] = Convert.instance(s => s.validNel)
}
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
sealed trait SchemaMap[A] {
def readFrom(input: Map[String, String]): ValidatedNel[ParseFailure, A]
}
object SchemaMap {
def of[A](implicit s: SchemaMap[A]): SchemaMap[A] = s
private def instance[A](body: Map[String, String] => Result[A]): SchemaMap[A] = new SchemaMap[A] {
def readFrom(input: Map[String, String]): Result[A] =
body(input)
}
implicit val noOp: SchemaMap[HNil] =
SchemaMap.instance(_ => HNil.validNel)
implicit def parsing[K <: Symbol, V: Convert, T <: HList](implicit key: Witness.Aux[K], next: SchemaMap[T]): SchemaMap[FieldType[K, V] :: T] =
SchemaMap.instance { input =>
val fieldName = key.value.name
val parsedField = input
.get(fieldName)
.cata(entry => Convert.to[V](entry), ParseFailure(s"$fieldName is missing").invalidNel)
.map(f => field[K](f))
(parsedField, next.readFrom(input)).mapN(_ :: _)
}
implicit def classes[A, R <: HList](implicit repr: LabelledGeneric.Aux[A, R], schema: SchemaMap[R]): SchemaMap[A] =
SchemaMap.instance { input =>
schema.readFrom(input).map(x => repr.from(x))
}
}
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
sealed trait SchemaList[A] {
def readFrom(input: List[String]): ValidatedNel[ParseFailure, A]
}
object SchemaList {
def of[A](implicit s: SchemaList[A]): SchemaList[A] = s
private def instance[A](body: List[String] => Result[A]): SchemaList[A] = new SchemaList[A] {
def readFrom(input: List[String]): Result[A] = body(input)
}
implicit val noOp: SchemaList[HNil] =
SchemaList.instance(_ => HNil.validNel)
implicit def parsing[K <: Symbol, V: Convert, T <: HList](implicit key: Witness.Aux[K], next: SchemaList[T]): SchemaList[FieldType[K, V] :: T] =
SchemaList.instance { input =>
val fieldName = key.value.name
val parsedField = input
.headOption
.cata(entry => Convert.to[V](entry), ParseFailure(s"$fieldName is missing").invalidNel)
.map(f => field[K](f))
(parsedField, next.readFrom(input.tail)).mapN(_ :: _)
}
implicit def classes[A, R <: HList](implicit repr: LabelledGeneric.Aux[A, R], schema: SchemaList[R]): SchemaList[A] =
SchemaList.instance { input =>
schema.readFrom(input).map(x => repr.from(x))
}
}
}
/*
case class Foo(a: String, b: Int, c: Boolean)
def m: Map[String, String] = Map("a" -> "hello", "c" -> "true", "b" -> "100")
def e: Map[String, String] = Map("c" -> "true", "b" -> "a100")
val result = SchemaMap.of[Foo].readFrom(m)
val lst = List("145164983", "0.01862523", "16.11681596", "21:38:57", "bid")
case class Trade0(tid: Long, price: Double, amount: Double, time: String, tpe: String)
val result2 = SchemaList.of[Trade0].readFrom(lst)
*/