I have a function like this:
def getSomething: (String, Future[String]) => String = {
case (name, surname) if (name == "Joe", surname.map(s => s == "Doe")) => "Is ok"
}
But compiler said he needs executionContext here in map function. I tried to do some magic with:
def getSomething (implicit e: ExecutionContext): (String, Future[String]) => String{...}
or
def getSomething: (String, Future[String])(implicit e: ExecutionContext) => String{...}
But it not works. Is it possible to pass implicit parameter to function like this? Or could I do it in other way?
def getSomething (implicit e: ExecutionContext): (String, Future[String]) => String = ...
should work.
You can't write
(String, Future[String]) => (implicit e: ExecutionContext) => String
Implicit functions will appear in Scala 3.
http://dotty.epfl.ch/docs/reference/contextual/context-functions.html
http://dotty.epfl.ch/blog/2016/12/05/implicit-function-types.html
In Scala 3 we could return a function with implicit parameter, for example
def getSomething: ExecutionContext ?=> Future[String] => Future[Int] = {
(using ec: ExecutionContext) => (f: Future[String]) => f.map(_.toInt)
}
given fiveThreadsEc as ExecutionContext = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(5))
getSomething(Future("42"))
Note how ?=> in function type corresponds to using in function literal. Consider the example from Change to (using x: T) => E for context functions #8305
val f: Int ?=> Int = (using x: Int) => 2 * x
f(using 2)
I have this typeclass
import simulacrum._
#typeclass trait Functor[F[_]] {
def map[A, B](fa: F[A])(f: A => B) : F[B]
def lift[A, B](fa: F[A])(f: A => B) : F[A] => F[B] = fa => map(fa)(f)
def as[A, B](fa: F[A], b: => B) : F[B] = map(fa)(_ => b)
def void[A](fa: F[A]) : F[Unit] = as(fa, ())
}
and this is the implementation
object Functor {
implicit val listFunctor: Functor[List] = new Functor[List] {
def map[A, B](fa: List[A])(f: A => B) = fa.map(f)
}
implicit def functionFunctor[X]: Functor[X => ?] = new Functor[X => ?] {
def map[A, B](fa : X => A)(f : A => B) = fa andThen f
}
}
I can easily discover the List implicit implementation as
object Chapter1 extends App {
import Functor.ops._
List(1, 2, 3).as("foo").foreach(println)
}
The above works perfectly fine. I can also do
object Chapter1 extends App {
import Functor._
val func : Int => String = implicitly[Functor[Int => ?]].map(_ + 2)(_.toString)
println(func(5))
}
But when I try
object Chapter1 extends App {
import Functor.ops._
val x : Int => Int = _ + 2
val y : Int => String = x.map(_.toString)
}
It doesn't find my implicit implementation and says that the value map is not a member of Int => Int
Compiler can't see that Int => Int is Int => ? applied to Int.
Add
scalacOptions += "-Ypartial-unification"
to build.sbt.
It's necessary for normal work with higher-kinded types with Cats, Scalaz or manually.
By the way, there's no sense to import Functor._
I'm trying to consume a Kafka topic from Spark with KafkaUtils.createDirectStream. I don't know if it is a Scala or KafkaUtils/Spark issue.
Here is my call to createDirectStream:
val messages = KafkaUtils.createDirectStream[String, String, KafkaAvroDecoder, KafkaAvroDecoder, EvtAct](
ssc,
kafkaParams,
fromOffsets,
messageHandler
)
When I compile the code (from maven), I have this error:
[ERROR] C:\...\reader\Main.scala:60: error: type arguments [String,String,io.confluent.kafka.serializers.KafkaAvroDecoder,io.confluent.kafka.serializers.KafkaAvroDecoder,fr.sihm.reader.EvtAct] conform to the bounds of none of the overloaded alternatives of
[INFO] value createDirectStream: [K, V, KD <: kafka.serializer.Decoder[K], VD <: kafka.serializer.Decoder[V], R](jssc: org.apache.spark.streaming.api.java.JavaStreamingContext, keyClass: Class[K], valueClass: Class[V], keyDecoderClass: Class[KD], valueDecoderClass: Class[VD], recordClass: Class[R], kafkaParams: java.util.Map[String,String], fromOffsets: java.util.Map[kafka.common.TopicAndPartition,Long], messageHandler: org.apache.spark.api.java.function.Function[kafka.message.MessageAndMetadata[K,V],R])org.apache.spark.streaming.api.java.JavaInputDStream[R] <and> [K, V, KD <: kafka.serializer.Decoder[K], VD <: kafka.serializer.Decoder[V], R](ssc: org.apache.spark.streaming.StreamingContext, kafkaParams: Map[String,String], fromOffsets: Map[kafka.common.TopicAndPartition,Long], messageHandler: kafka.message.MessageAndMetadata[K,V] => R)(implicit evidence$14: scala.reflect.ClassTag[K], implicit evidence$15: scala.reflect.ClassTag[V], implicit evidence$16: scala.reflect.ClassTag[KD], implicit evidence$17: scala.reflect.ClassTag[VD], implicit evidence$18: scala.reflect.ClassTag[R])org.apache.spark.streaming.dstream.InputDStream[R]
[INFO] val messages = KafkaUtils.createDirectStream[String, String, KafkaAvroDecoder, KafkaAvroDecoder, EvtAct](
[INFO] ^
[ERROR] one error found
createDirectStream has 4 declarations:
def createDirectStream[
K: ClassTag,
V: ClassTag,
KD <: Decoder[K]: ClassTag,
VD <: Decoder[V]: ClassTag,
R: ClassTag] (
ssc: StreamingContext,
kafkaParams: Map[String, String],
fromOffsets: Map[TopicAndPartition, Long],
messageHandler: MessageAndMetadata[K, V] => R
): InputDStream[R] = {
val cleanedHandler = ssc.sc.clean(messageHandler)
new DirectKafkaInputDStream[K, V, KD, VD, R](
ssc, kafkaParams, fromOffsets, cleanedHandler)
}
def createDirectStream[
K: ClassTag,
V: ClassTag,
KD <: Decoder[K]: ClassTag,
VD <: Decoder[V]: ClassTag] (
ssc: StreamingContext,
kafkaParams: Map[String, String],
topics: Set[String]
): InputDStream[(K, V)] = {
val messageHandler = (mmd: MessageAndMetadata[K, V]) => (mmd.key, mmd.message)
val kc = new KafkaCluster(kafkaParams)
val fromOffsets = getFromOffsets(kc, kafkaParams, topics)
new DirectKafkaInputDStream[K, V, KD, VD, (K, V)](
ssc, kafkaParams, fromOffsets, messageHandler)
}
def createDirectStream[K, V, KD <: Decoder[K], VD <: Decoder[V], R](
jssc: JavaStreamingContext,
keyClass: Class[K],
valueClass: Class[V],
keyDecoderClass: Class[KD],
valueDecoderClass: Class[VD],
recordClass: Class[R],
kafkaParams: JMap[String, String],
fromOffsets: JMap[TopicAndPartition, JLong],
messageHandler: JFunction[MessageAndMetadata[K, V], R]
): JavaInputDStream[R] = {
implicit val keyCmt: ClassTag[K] = ClassTag(keyClass)
implicit val valueCmt: ClassTag[V] = ClassTag(valueClass)
implicit val keyDecoderCmt: ClassTag[KD] = ClassTag(keyDecoderClass)
implicit val valueDecoderCmt: ClassTag[VD] = ClassTag(valueDecoderClass)
implicit val recordCmt: ClassTag[R] = ClassTag(recordClass)
val cleanedHandler = jssc.sparkContext.clean(messageHandler.call _)
createDirectStream[K, V, KD, VD, R](
jssc.ssc,
Map(kafkaParams.asScala.toSeq: _*),
Map(fromOffsets.asScala.mapValues(_.longValue()).toSeq: _*),
cleanedHandler
)
}
def createDirectStream[K, V, KD <: Decoder[K], VD <: Decoder[V]](
jssc: JavaStreamingContext,
keyClass: Class[K],
valueClass: Class[V],
keyDecoderClass: Class[KD],
valueDecoderClass: Class[VD],
kafkaParams: JMap[String, String],
topics: JSet[String]
): JavaPairInputDStream[K, V] = {
implicit val keyCmt: ClassTag[K] = ClassTag(keyClass)
implicit val valueCmt: ClassTag[V] = ClassTag(valueClass)
implicit val keyDecoderCmt: ClassTag[KD] = ClassTag(keyDecoderClass)
implicit val valueDecoderCmt: ClassTag[VD] = ClassTag(valueDecoderClass)
createDirectStream[K, V, KD, VD](
jssc.ssc,
Map(kafkaParams.asScala.toSeq: _*),
Set(topics.asScala.toSeq: _*)
)
}
}
Note that when I replace the two firsts types [String, String, ...] by [Object, Object, ...] it works great.
Any Idea?
Thank you
From kafka message declaration:
package kafka.message
import kafka.serializer.Decoder
import kafka.utils.Utils
case class MessageAndMetadata[K, V](topic: String, partition: Int,
private val rawMessage: Message, offset: Long,
keyDecoder: Decoder[K], valueDecoder: Decoder[V]) {
/**
* Return the decoded message key and payload
*/
def key(): K = if(rawMessage.key == null) null.asInstanceOf[K] else keyDecoder.fromBytes(Utils.readBytes(rawMessage.key))
def message(): V = if(rawMessage.isNull) null.asInstanceOf[V] else valueDecoder.fromBytes(Utils.readBytes(rawMessage.payload))
}
and valudecoder as defined fromBytes so my suggestion for you try second argument as Array[Byte]
val messageHandler: MessageAndMetadata[String, Array[Byte]] => EvtAct = mmd => new EvtAct(mmd.message)
val messages = KafkaUtils.createDirectStream[String, Array[Byte], KafkaAvroDecoder, KafkaAvroDecoder, EvtAct](
ssc,
kafkaParams,
fromOffsets,
messageHandler
)
[error] test.scala:31: ambiguous implicit values:
[error] both method taggedQueryParamDecoder in trait ExternalInstances0 of type [A, T](implicit evidence$2: org.http4s.QueryParamDecoder[A])org.http4s.QueryParamDecoder[scalaz.##[A,T]]
[error] and method iiQueryParamDecoder in trait ExternalInstances1 of type [B](implicit ii: foo.InvariantInstances[B])org.http4s.QueryParamDecoder[B]
[error] match expected type org.http4s.QueryParamDecoder[scalaz.##[String,foo.tags.Social]]
[error] implicitly[QueryParamDecoder[String ## Social]]
[error] ^
I import instances._; instances extends ExternalInstances1, and ExternalInstances1 extends ExternalInstances0. Because of this inheritance I would expect ExternalInstances1's members to win out over ExternalInstances0's, rather than yielding an ambiguity.
Why is this happening, and how can I fix it? Thanks.
The source is at http://scastie.org/12233, reproduced below:
/***
scalaVersion := "2.11.7"
libraryDependencies += "org.http4s" %% "http4s-core" % "0.10.0"
resolvers ++= Seq(
"tpolecat" at "http://dl.bintray.com/tpolecat/maven",
"Scalaz Bintray Repo" at "http://dl.bintray.com/scalaz/releases",
Resolver.sonatypeRepo("releases")
)
libraryDependencies += "org.tpolecat" %% "doobie-core" % "0.2.2"
libraryDependencies += "com.github.alexarchambault" %% "argonaut-shapeless_6.1" % "0.3.1"
*/
import java.time.LocalDate
import argonaut._, Argonaut._
import doobie.imports._
import doobie.util.composite.Composite
import tags._
import instances._
import org.http4s._
import scala.reflect.runtime.universe.TypeTag
import scalaz._
object Main extends App {
implicitly[QueryParamDecoder[String ## Social]]
println("ok")
}
object tags {
trait Social; val Social = Tag.of[Social]
}
object instances extends ExternalInstances1 {
implicit val ssn: InvariantInstances[String ## Social] =
InvariantInstances { (raw: String) ⇒
val digitsOnly = raw.filter(_.isDigit)
require(digitsOnly.length == 9)
Social(digitsOnly)
}(Social.unwrap)
}
trait ExternalInstances1 extends ExternalInstances0 {
implicit def iiCodecJson[B](implicit ii: InvariantInstances[B]): CodecJson[B] = ii.codecJson
implicit def iiEncodeJson[B](implicit ii: InvariantInstances[B]): EncodeJson[B] = ii.encodeJson
implicit def iiDecodeJson[B](implicit ii: InvariantInstances[B]): DecodeJson[B] = ii.decodeJson
implicit def iiMeta[B](implicit ii: InvariantInstances[B]): Meta[B] = ii.meta
implicit def iiQueryParamEncoder[B](implicit ii: InvariantInstances[B]): QueryParamEncoder[B] = ii.queryParamEncoder
implicit def iiQueryParamDecoder[B](implicit ii: InvariantInstances[B]): QueryParamDecoder[B] = ii.queryParamDecoder
}
trait ExternalInstances0 {
implicit def taggedEncodeJson[A, T](implicit A: EncodeJson[A]): EncodeJson[A ## T] =
A.contramap(Tag.of[T].unwrap)
implicit def taggedDecodeJson[A, T](implicit A: DecodeJson[A]): DecodeJson[A ## T] =
A.map(Tag.of[T].apply)
implicit def taggedComposite[A: Composite, T]: Composite[A ## T] =
Composite[A].xmap(a => Tag[A, T](a), Tag.unwrap(_))
implicit def taggedQueryParamDecoder[A: QueryParamDecoder, T]: QueryParamDecoder[A ## T] =
QueryParamDecoder.decodeBy(Tag.of[T](_: A))
}
trait InvariantInstances[B] {
def codecJson: CodecJson[B]
def decodeJson: DecodeJson[B]
def encodeJson: EncodeJson[B]
def meta: Meta[B]
def queryParamEncoder: QueryParamEncoder[B]
def queryParamDecoder: QueryParamDecoder[B]
}
object InvariantInstances {
def apply[A: EncodeJson: DecodeJson: Meta: QueryParamDecoder: QueryParamEncoder, B: TypeTag](f: A ⇒ B)(g: B ⇒ A) =
new InvariantInstances[B] {
def codecJson: CodecJson[B] =
CodecJson.derived[B](encodeJson, decodeJson)
def decodeJson: DecodeJson[B] =
implicitly[DecodeJson[A]].map(f)
def encodeJson: EncodeJson[B] =
implicitly[EncodeJson[A]].contramap(g)
def meta: Meta[B] =
implicitly[Meta[A]].xmap(f, g)
def queryParamDecoder: QueryParamDecoder[B] =
CovariantInstances.queryParamDecoder(f)
def queryParamEncoder: QueryParamEncoder[B] =
ContravariantInstances.queryParamEncoder(g)
}
}
object CovariantInstances {
def queryParamDecoder[A, B](f: A ⇒ B)(implicit A: QueryParamDecoder[A],
B: reflect.runtime.universe.TypeTag[B]): QueryParamDecoder[B] =
new QueryParamDecoder[B] {
import scalaz.Validation.FlatMap._ // suppress deprecation warning
def decode(value: QueryParameterValue): ValidationNel[ParseFailure, B] =
A.decode(value).flatMap(
a ⇒ Validation.fromTryCatchNonFatal(f(a)).leftMap(t =>
ParseFailure(s"Query decoding ${B.tpe.typeSymbol} failed", t.getMessage)
).toValidationNel
)
}
}
object ContravariantInstances {
def queryParamEncoder[A, B](g: B ⇒ A)(implicit A: QueryParamEncoder[A]): QueryParamEncoder[B] =
new QueryParamEncoder[B] {
def encode(value: B): QueryParameterValue = A.encode(g(value))
}
}
Based on Sasha Kolberg's comment and http://eed3si9n.com/revisiting-implicits-without-import-tax, I suppose I have a "more specific" definition in ExternalInstances0 (+1 point) and a "higher priority" definition in ExternalInstances1 (+1 point) leading to a tie and ambiguity.
My workaround was to add a "more specific, higher priority" definition in ExternalInstances1 (+2 points?) to break the tie, even though it is duplicating code which was basically boilerplate to begin with.
I'd love to know a better solution. Thanks!
trait ExternalInstances1 extends ExternalInstances0 {
implicit def iiCodecJson[B](implicit ii: InvariantInstances[B]): CodecJson[B] = ii.codecJson
implicit def iiEncodeJson[B](implicit ii: InvariantInstances[B]): EncodeJson[B] = ii.encodeJson
implicit def iiDecodeJson[B](implicit ii: InvariantInstances[B]): DecodeJson[B] = ii.decodeJson
implicit def iiMeta[B](implicit ii: InvariantInstances[B]): Meta[B] = ii.meta
implicit def iiQueryParamEncoder[B](implicit ii: InvariantInstances[B]): QueryParamEncoder[B] = ii.queryParamEncoder
implicit def iiQueryParamDecoder[B](implicit ii: InvariantInstances[B]): QueryParamDecoder[B] = ii.queryParamDecoder
implicit def iiCodecJsonT[B,T](implicit ii: InvariantInstances[B ## T]): CodecJson[B ## T] = ii.codecJson
implicit def iiEncodeJsonT[B,T](implicit ii: InvariantInstances[B ## T]): EncodeJson[B ## T] = ii.encodeJson
implicit def iiDecodeJsonT[B,T](implicit ii: InvariantInstances[B ## T]): DecodeJson[B ## T] = ii.decodeJson
implicit def iiMetaT[B,T](implicit ii: InvariantInstances[B ## T]): Meta[B ## T] = ii.meta
implicit def iiQueryParamEncoderT[B,T](implicit ii: InvariantInstances[B ## T]): QueryParamEncoder[B ## T] = ii.queryParamEncoder
implicit def iiQueryParamDecoderT[B,T](implicit ii: InvariantInstances[B ## T]): QueryParamDecoder[B ## T] = ii.queryParamDecoder
}
Given simplified code example:
sealed trait A {
val c1: String
val c2: Int
def copy[Z <: A](src: File) : Z
}
case class B(c1: String, c2: Int, src: File) extends A
case class C(c1: String, c2: Int, c3: MyClass, src: File) extends A
how do I define copy method in trait A so it will match generated one for case class and 'target' file? Given definition does typecheck and complains about missing method copy in classes B and C.
scala compiler will not generate copy methods for case class that defines method with name copy.
scala -Xprint:typer -e "sealed trait A { def copy[T <: A](s: String):T }; case class B(x: Int, y:Int) extends A"
outputs:
/var/folders/fm/fm4b21vj6jl995ywlrd99t49tjthjc/T/scalacmd3413244935208502669.scala:1: error: class B needs to be abstract, since method copy in trait A of type [T <: this.A](s: String)T is not defined
sealed trait A { def copy[T <: A](s: String):T }; case class B(x: Int, y: Int) extends A
^
one error found
[[syntax trees at end of typer]] // scalacmd3413244935208502669.scala
package <empty> {
object Main extends scala.AnyRef {
def <init>(): Main.type = {
Main.super.<init>();
()
};
def main(argv: Array[String]): Unit = {
val args: Array[String] = argv;
{
final class $anon extends scala.AnyRef {
def <init>(): anonymous class $anon = {
$anon.super.<init>();
()
};
sealed abstract trait A extends scala.AnyRef {
def copy[T >: Nothing <: this.A](s: String): T
};
case class B extends AnyRef with this.A with Product with Serializable {
<caseaccessor> <paramaccessor> private[this] val x: Int = _;
<stable> <caseaccessor> <accessor> <paramaccessor> def x: Int = B.this.x;
<caseaccessor> <paramaccessor> private[this] val y: Int = _;
<stable> <caseaccessor> <accessor> <paramaccessor> def y: Int = B.this.y;
def <init>(x: Int, y: Int): this.B = {
B.super.<init>();
()
};
override <synthetic> def productPrefix: String = "B";
<synthetic> def productArity: Int = 2;
<synthetic> def productElement(x$1: Int): Any = x$1 match {
case 0 => B.this.x
case 1 => B.this.y
case _ => throw new IndexOutOfBoundsException(x$1.toString())
};
override <synthetic> def productIterator: Iterator[Any] = runtime.this.ScalaRunTime.typedProductIterator[Any](B.this);
<synthetic> def canEqual(x$1: Any): Boolean = x$1.$isInstanceOf[this.B]();
override <synthetic> def hashCode(): Int = {
<synthetic> var acc: Int = -889275714;
acc = Statics.this.mix(acc, x);
acc = Statics.this.mix(acc, y);
Statics.this.finalizeHash(acc, 2)
};
override <synthetic> def toString(): String = ScalaRunTime.this._toString(B.this);
override <synthetic> def equals(x$1: Any): Boolean = B.this.eq(x$1.asInstanceOf[Object]).||(x$1 match {
case (_: this.B) => true
case _ => false
}.&&({
<synthetic> val B$1: this.B = x$1.asInstanceOf[this.B];
B.this.x.==(B$1.x).&&(B.this.y.==(B$1.y)).&&(B$1.canEqual(B.this))
}))
};
<synthetic> private object B extends scala.runtime.AbstractFunction2[Int,Int,this.B] with Serializable {
def <init>(): this.B.type = {
B.super.<init>();
()
};
final override <synthetic> def toString(): String = "B";
case <synthetic> def apply(x: Int, y: Int): this.B = new B(x, y);
case <synthetic> def unapply(x$0: this.B): Option[(Int, Int)] = if (x$0.==(null))
scala.this.None
else
Some.apply[(Int, Int)](Tuple2.apply[Int, Int](x$0.x, x$0.y));
<synthetic> private def readResolve(): Object = $anon.this.B
}
};
{
new $anon();
()
}
}
}
}
}