Configuring implicits in Scala - scala

I have typeclass:
trait ProcessorTo[T]{
def process(s: String): T
}
and its implementation
class DefaultProcessor extends ProcessorTo[String]{
def process(s: String): String = s
}
trait DefaultProcessorSupport{
implicit val p: Processor[String] = new DefaultProcessor
}
To make it available for using I created
object ApplicationContext
extends DefaultProcessorSupport
with //Some other typeclasses
But now I have to add a processor which performs some DataBase - read. The DB URL etc are placed in condifguration file that is available only a runtime. For now I did the following.
class DbProcessor extends ProcessorTo[Int]{
private var config: Config = _
def start(config: Config) = //set the configuration, open connections etc
//Other implementation
}
object ApplicationContext{
implicit val p: ProcessorTo[Int] = new DbProcessor
def configure(config: Config) = p.asInstanceOf[DbProcessor].start(config)
}
It works for me, but I'm not sure about this technique. Looks strange for me a little bit. Is it a bad practice? If so, what would be a good solution?

I am a bit confused by the requirements as DbProcessor is missing the process implementation(???) and trait ProcessorTo[T] is missing start method which is defined in DbProcessor. So, I will assume the following while answering: the type class has both process and start methods
Define a type class:
trait ProcessorTo[T]{
def start(config: Config): Unit
def process(s: String): T
}
Provide implementations for the type class in the companion objects:
object ProcessorTo {
implicit object DbProcessor extends ProcessorTo[Int] {
override def start(config: Config): Unit = ???
override def process(s: String): Int = ???
}
implicit object DefaultProcessor extends ProcessorTo[String] {
override def start(config: Config): Unit = ???
override def process(s: String): String = s
}
}
and use it in your ApplicationContext as follows:
object ApplicationContext {
def configure[T](config: Config)(implicit ev: ProcessorTo[T]) = ev.start(config)
}
This is a nice blog post about Type Classes: http://danielwestheide.com/blog/2013/02/06/the-neophytes-guide-to-scala-part-12-type-classes.html

I don't really see why you need start. If your implicit DbProcessor has a dependency, why not make it an explicit dependency via constructor? I mean something like this:
class DbConfig(val settings: Map[String, Object]) {}
class DbProcessor(config: DbConfig) extends ProcessorTo[Int] {
// here goes actual configuration of the processor using config
private val mappings: Map[String, Int] = config.settings("DbProcessor").asInstanceOf[Map[String, Int]]
override def process(s: String): Int = mappings.getOrElse(s, -1)
}
object ApplicationContext {
// first create config then pass it explicitly
val config = new DbConfig(Map[String, Object]("DbProcessor" -> Map("1" -> 123)))
implicit val p: ProcessorTo[Int] = new DbProcessor(config)
}
Or if you like Cake pattern, you can do something like this:
trait DbConfig {
def getMappings(): Map[String, Int]
}
class DbProcessor(config: DbConfig) extends ProcessorTo[Int] {
// here goes actual configuration of the processor using config
private val mappings: Map[String, Int] = config.getMappings()
override def process(s: String): Int = mappings.getOrElse(s, -1)
}
trait DbProcessorSupport {
self: DbConfig =>
implicit val dbProcessor: ProcessorTo[Int] = new DbProcessor(self)
}
object ApplicationContext extends DbConfig with DbProcessorSupport {
override def getMappings(): Map[String, Int] = Map("1" -> 123)
}
So the only thing you do in your ApplicationContext is providing actual implementation of the DbConfig trait.

Related

Flink KafkaSource & KafkaSink with GenericRecords and Confluent Schema Registry

I was faced with the problem of reading / writing from and to Kafka using KafkaSource and KafkaSink with Flink v1.16 (Scala 2.12) and the Confluent Schema Registry. The events should be read and written as GenericRecords. Below is an overview of the approach that I came up with. It is far from perfect, but I hope it helps someone to get a general idea of how things can be tied together.
KafkaSource
First I created a POJO class to deserialize the event into (Why POJO). Be careful to adhere to the POJO conventions.
class InputEvent
(
var timestamp: Long
, var someValue: String
) extends Serializable {
def this() = this(Long, String)
def canEqual(other: Any): Boolean = ...
override def equals(other: Any): Boolean = ...
override def hashCode(): Int = ...
override def toString = ...
}
I then created a companion object that extends the SchemaProjectable[T] trait. It is important that the Schema is a string and only later parsed with the getSchema method (serialization issues).
object InputEvent extends SchemaProjectable[InputEvent] with Serializable {
val SCHEMA = "{\"type\":\"record\",\"name\":\"InputEvent\",\"namespace\":\"com.blog.post\",\"doc\":\"Input Events\",\"fields\":[{\"name\":\"timestamp\",\"type\":\"long\",\"doc\":\"Event timestamp in seconds\"},{\"name\":\"someValue\",\"type\":\"string\",\"doc\":\"Some Value Field\"}]}"
override def getSchema: Schema = new Schema.Parser().parse(SCHEMA)
override def projectFromGeneric(in: GenericRecord): InputEvent = {
new InputEvent(
in.getLong("timestamp"),
in.getString("someValue")
)
}
override def projectToGeneric(in: InputEvent): GenericRecord = new GenericData.Record(getSchema)
}
The SchemaProjectable[T] trait is used later for the de/serializer to have a common interface and to allow an event to be converted from and to a GenericRecord easily.
abstract class SchemaProjectable[T] extends Serializable {
def getSchema : Schema
def projectFromGeneric(in: GenericRecord) : T
def projectToGeneric(in: T) : GenericRecord
}
Subsequently, the builder of the KafkaSource is put into a separate object. The getKafkaSource[T] method requires the configuration properties as well as an implementation of trait SchemaProjectable[T].
object GenericKafkaSource {
def getKafkaSource[T]
(properties: Properties, schemaProjectable: SchemaProjectable[T])
(implicit tInfo: TypeInformation[T]): KafkaSource[T] = {
KafkaSource.builder[T]
.setProperties(properties)
.setTopics(properties.getProperty("topicName"))
.setStartingOffsets(configToOffset(properties.getProperty("offset")))
.setDeserializer(
new GenericDeserializationSchema(schemaProjectable, properties.getProperty("schema.registry.url"))
)
.build
}
}
As a last step, the generic deserializer is implemented using the ConfluentRegistryAvroDeserializationSchema
class GenericDeserializationSchema[T]
(schemaProjectable: SchemaProjectable[T], url: String)
(implicit tInfo: TypeInformation[T]) extends KafkaRecordDeserializationSchema[T]{
private val deserializationSchema = ConfluentRegistryAvroDeserializationSchema.forGeneric(schemaProjectable.getSchema,url)
override def deserialize(record: ConsumerRecord[Array[Byte], Array[Byte]], out: Collector[T]): Unit = {
out.collect(
schemaProjectable.projectFromGeneric(deserializationSchema.deserialize(record.value()))
)
}
override def getProducedType: TypeInformation[T] = Types.of[T]
}
Finally, the KafkaSource can be used as follows:
val source = GenericKafkaSource.getKafkaSource(consumerProperties, InputEvent)
KafkaSink
For the KafkaSink, we take a similar approach than with the KafkaSource. Again, we create a class and companion object for the output event.
class OutputEvent
(
var timestamp: Long
, var resultValue: String
) extends Serializable {
def this() = this(Long, String)
def canEqual(other: Any): Boolean = ...
override def equals(other: Any): Boolean = ...
override def hashCode(): Int = ...
override def toString = ...
}
object OutputEvent extends SchemaProjectable[OutputEvent] with Serializable{
val SCHEMA = "{\"type\":\"record\",\"name\":\"OutputEvent\",\"namespace\":\"com.blog.post\",\"fields\":[{\"name\":\"timestamp\",\"type\":[\"null\",\"long\"],\"doc\":\"Timestamp of the event\"},{\"name\":\"resultValue\",\"type\":[\"null\",\"string\"],\"doc\":\"Result value\"}]}"
override def getSchema: Schema = new Schema.Parser().parse(SCHEMA)
override def projectFromGeneric(in: GenericRecord): OutputEvent = new OutputEvent()
override def projectToGeneric(in: OutputEvent): GenericRecord = {
val record = new GenericData.Record(getSchema)
record.put("timestamp", in.timestamp)
record.put("resultValue", in.resultValue)
record
}
}
The KafkaSink builder is factored out to a separate object.
object GenericKafkaSink {
def getGenericKafkaSink[T]
(properties: Properties, schemaProjectable: SchemaProjectable[T])
(implicit tInfo: TypeInformation[T]): KafkaSink[T] = {
val topicName = properties.getProperty("topicName")
val url = properties.getProperty("schema.registry.url")
KafkaSink.builder[T]
.setKafkaProducerConfig(properties)
.setBootstrapServers(properties.getProperty("bootstrap.servers"))
.setRecordSerializer(
new GenericSerializationSchema[T](topicName, schemaProjectable, url)
)
.setTransactionalIdPrefix(properties.getProperty("transactionId"))
.build
}
}
Subsequently, the SerializationSchema is implemented as follows using the ConfluentRegistryAvroSerializationSchema.
class GenericSerializationSchema[T]
(topicName: String, schemaProjectable: SchemaProjectable[T], url: String)
(implicit tInfo: TypeInformation[T]) extends KafkaRecordSerializationSchema[T] with Serializable {
lazy private val serializationSchema: ConfluentRegistryAvroSerializationSchema[GenericRecord] = ConfluentRegistryAvroSerializationSchema.forGeneric(topicName+"-value",schemaProjectable.getSchema,url)
override def serialize(element: T, context: KafkaRecordSerializationSchema.KafkaSinkContext, timestamp: lang.Long): ProducerRecord[Array[Byte], Array[Byte]] = {
new ProducerRecord[Array[Byte], Array[Byte]](
topicName
, serializationSchema.serialize(schemaProjectable.projectToGeneric(element))
)
}
}
Finally, the KafkaSink can be used as follows:
val sink = GenericKafkaSink.getGenericKafkaSink(producerProperties, OutputEvent)
Any opinions on the approach? I'm glad for any feedback!
Kind Regards
Dominik

How to override an implicit value, that is imported?

I have tried solutions, described in How to override an implicit value?, but it does not help. Here is a code example.
A definition of TestImplicit abstraction with 2 different implementations (analogue of ExecutionContextExecutor):
trait TestImplicit {
def f(s:String):Unit
}
object TestImplicitImpl1 extends TestImplicit {
override def f(s: String): Unit = println(s"1: $s")
}
object TestImplicitImpl2 extends TestImplicit {
override def f(s: String): Unit = println(s"2: $s")
}
And in the ImplDefinition object a q variable is defined to be used implicitly via import (analogue of ExecutionContext.Implicits.global):
object ImplDefinition {
implicit val q:TestImplicit = TestImplicitImpl1
}
Client that defines a method, accepting TestImplicit implicitly (analogue of scala.concurrent.Future):
trait TestImplicitClient {
def fu(implicit ti:TestImplicit):Unit
}
object TestImplicitClient extends TestImplicitClient {
override def fu(implicit ti: TestImplicit): Unit = {
println("client")
ti.f("param")
}
}
The next step, a client of client, that chooses which implementation of TestImplicit should be used, the decision is done via import (analogue of API that uses Future):
object ClientOfClient {
import somepackage.ImplDefinition.q
def t():Unit =
TestImplicitClient.fu
}
Now in test, I want to use this ClientOfClient.t(), but I need to override implicit, and use TestImplicitImpl2 instead. The main idea behind - implicits should be defined/overridable by the client of API, but not by API itself:
import somepackage.{ClientOfClient, TestImplicit, TestImplicitImpl2}
import org.junit.Test
class ImplTest {
// trying to hide it via import, does not help
import somepackage.ImplDefinition.{q => _,_}
#Test def test(): Unit ={
//trying to hide it via downgrading to non-implicit, does not work either
val q = somepackage.ImplDefinition.q
implicit val ti = TestImplicitImpl2
ClientOfClient.t()
}
}
Each time I run test, I get in the output:
client
1: param
But not expected:
client
2: param
How can I fix it? I need a way to allow clients to override implicits and stay with as simple API as possible. Which means, I do not want to add additional implicit parameter into ClientOfClient.t() method.
As soon as you have a singleton object ClientOfClient with a hard-coded constant TestImplicitImpl1 everywhere, there is essentially nothing you can do. But there are several work-arounds.
1. Use default implicit parameters
object ClientOfClient {
def t()(implicit ti: TestImplicit = ImplDefinition.q): Unit =
TestImplicitClient.fu
}
object ImplTest {
def test(): Unit = {
implicit val ti2 = TestImplicitImpl2
ClientOfClient.t()
}
}
ImplTest.test() // 2: param
2. Supply the implicit through a separate method that can be overridden
If you want to make the implicit overridable, then make ClientOfClient extendable, and create a method (here "cocti") that returns the implicit, instead of importing the implicit directly. You can then override the method (whereas you cannot override an import).
This here produces 2: param in the end of the day:
trait TestImplicit {
def f(s: String): Unit
}
object TestImplicitImpl1 extends TestImplicit {
override def f(s: String): Unit = println(s"1: $s")
}
object TestImplicitImpl2 extends TestImplicit {
override def f(s: String): Unit = println(s"2: $s")
}
object ImplDefinition {
implicit val q: TestImplicit = TestImplicitImpl1
}
trait TestImplicitClient {
def fu(implicit ti: TestImplicit): Unit
}
object TestImplicitClient extends TestImplicitClient {
override def fu(implicit ti: TestImplicit): Unit = {
println("client")
ti.f("param")
}
}
class ClientOfClient {
implicit def cocti: TestImplicit = {
ImplDefinition.q
}
def t():Unit =
TestImplicitClient.fu
}
object ImplTest {
def test(): Unit = {
implicit val ti2 = TestImplicitImpl2
new ClientOfClient {
override def cocti = ti2
}.t()
}
}
ImplTest.test() // 2: param

Immutable composable builder in scala

I'm interested how to improve the piece of code bellow. The idea is to build an immutable composable builder. In the end the builder just builds a Map[String, Object]. I want to be able to define core builder components that can be reused and to also let people define their own additional builders to extend the main builder. I'm able to do so but not without the ugly use of reflection. The code looks as follows:
object TestPizzaBuilder {
def main(args: Array[String]): Unit = {
val build = new PizzaBuilder()
.withCheese("blue")
.withSauce("tomato")
.build()
println(build)
}
}
object PizzaBuilder {
type BuilderParams = Map[String, Object]
}
class PizzaBuilder(params: BuilderParams = Map.empty) extends BaseBuilder[PizzaBuilder](params: BuilderParams)
with CheeseBuilder[PizzaBuilder]
with SauceBuilder[PizzaBuilder] {
}
abstract class BaseBuilder[A <: BaseBuilder[A]](params: BuilderParams)(implicit tag: ClassTag[A]) {
protected def _copy(tuples: (String, Object)*): A = {
val constr = tag.runtimeClass.getConstructors()(0)
constr.newInstance(params ++ tuples).asInstanceOf[A]
}
def build(): Map[String, Object] = {
params
}
}
trait CheeseBuilder[A <: BaseBuilder[A]] {
this: BaseBuilder[A] =>
def withCheese(cheese: String): A = _copy("cheese" -> cheese)
}
trait SauceBuilder[A <: BaseBuilder[A]] {
this: BaseBuilder[A] =>
def withSauce(sauce: String): A = _copy("sauce" -> sauce)
}
Do you have suggestions how reflection can be avoided in this scenario yet keeping the builder immutable and also allowing to compose the builder of tiny other builders.
Smallest change would be to pass the constructor (as a function) instead of a ClassTag:
abstract class BaseBuilder[A <: BaseBuilder[A]](params: BuilderParams)(constructor: BuilderParams => A) {
protected def _copy(tuples: (String, Object)*): A = constructor(params ++ tuples)
def build(): Map[String, Object] = {
params
}
}
// or class PizzaBuilder(params: BuilderParams = Map.empty) extends BaseBuilder[PizzaBuilder](params)(new PizzaBuilder(_))
case class PizzaBuilder(params: BuilderParams = Map.empty) extends BaseBuilder[PizzaBuilder](params)(PizzaBuilder)
with CheeseBuilder[PizzaBuilder]
with SauceBuilder[PizzaBuilder] {
}

Scala Implicit class and overloading an existing function

Any idea how to get the following to work:
trait HttpTransport {
def doGet(str: String): String
}
trait CoreGet {
def GET(str: String)(implicit imp:String): List[String]
}
trait VerbGet extends CoreGet with HttpTransport {
override def GET(str: String)(implicit imp:String): List[String]= {
println("->VerbGet.GET")
val str1 = doGet(str)
// and more biz logic calls here
List(s"\nverb (biz logic) called url $str and got '${str1}'>")
}
}
// PlayGet {
implicit class ExtendCoreGet(coreGet: CoreGet) {
def GET[A](url: String)(implicit imp:String, imp2: List[A]): List[A]= {
println(s"->ExtendCoreGet.GET($url)")
val coreGetResult = coreGet.GET(url)
coreGetResult.flatMap(_ => imp2)
}
}
trait Play extends HttpTransport {
override def doGet(str: String): String = {
println("->Play.doGet")
s"\nPlay.doGet($str)>"
}
}
val client = new VerbGet with Play
client.GET("www.go.com")("hello", List("1")) //<-- does not compile
Compiler error:
too many arguments (2) for method GET: (implicit imp:
String)List[String]
You can play with the code here:
https://scastie.scala-lang.org/arminio/tN9NfdxGQUmusrNL0lJ78w
It looks like you are trying extend functionality of VerbGet. You need to fix two things:
1. ExtendCoreGet must extend AnyVal to add more methods.
2. You can add new functionality by adding new method say GET2 but you can't overload existing method. Renmaed your GET to GET2 or something meaningful.
ExtendCoreGet definition should be
implicit class ExtendCoreGet(val coreGet: CoreGet) extends AnyVal {
def GET2[A](url: String)(implicit imp:String, imp2: List[A]): List[A]= {
coreGet.GET(url).flatMap(_ => imp2)
}
}

Implicit resolution and companion objects for case classes

I'm trying to add an implicit value to (what I believe is) the companion object of a case class, but this implicit value is not found.
I'm trying to achieve something like the following:
package mypackage
object Main {
def main(args: Array[String]): Unit = {
val caseClassInstance = MyCaseClass("string")
val out: DataOutput = ...
serialize(out, caseClassInstance)
// the above line makes the compiler complain that there is no
// Serializer[MyCaseClass] in scope
}
def serialize[T : Serializer](out: DataOutput, t: T): Unit = {
...
}
}
object MyCaseClass {
// implicits aren't found here
implicit val serializer: Serializer[MyCaseClase] = ...
}
case class MyCaseClass(s: String) {
// some other methods
}
I've explicitly added the package here to show that both the MyCaseClass case class and object should be in scope. I know that the object is actually being constructed because I can get this to compile if I add
implicit val serializer = MyCaseClass.serializer
to main (though notably not if I add import MyCaseClass.serializer).
I'm concerned that the MyCaseClass object is not actually a companion of the case class, because if I explicitly define apply and unapply on the object and then attempt to call MyCaseClass.apply("string") in main, the compiler gives the following error:
ambiguous reference to overloaded definition,
both method apply in object MyCaseClass of type (s: String)mypackage.MyCaseClass
and method apply in object MyCaseClass of type (s: String)mypackage.MyCaseClass
match argument types (String)
val a = InputRecord.apply("string")
^
If it's not possible to take this approach, is there a way to use type classes with case classes without creating an implicit value every time it must be brought into scope?
EDIT: I'm using scala 2.10.3.
EDIT 2: Here's the example fleshed out:
package mypackage
import java.io.{DataOutput, DataOutputStream}
object Main {
def main(args: Array[String]): Unit = {
val caseClassInstance = MyCaseClass("string")
val out: DataOutput = new DataOutputStream(System.out)
serialize(out, caseClassInstance)
// the above line makes the compiler complain that there is no
// Serializer[MyCaseClass] in scope
}
def serialize[T : Serializer](out: DataOutput, t: T): Unit = {
implicitly[Serializer[T]].write(out, t)
}
}
object MyCaseClass {
// implicits aren't found here
implicit val serializer: Serializer[MyCaseClass] = new Serializer[MyCaseClass] {
override def write(out: DataOutput, t: MyCaseClass): Unit = {
out.writeUTF(t.s)
}
}
}
case class MyCaseClass(s: String) {
// some other methods
}
trait Serializer[T] {
def write(out: DataOutput, t: T): Unit
}
This actually compiles, though. I am getting this issue when using Scoobi's WireFormat[T] instead of Serializer, but can't provide a concise, runnable example due to complexity and the Scoobi dependency. I will try to create a more relevant example, but it seems as though the issue is not as general as I thought.
It turns out that the type class instances actually need to be implicit values, rather than objects. The MyCaseClass object above works because its serializer is assigned to an implicit value. However, this implementation
object MyCaseClass {
implicit object MyCaseClassSerializer extends Serializer[MyCaseClass] {
override def write(out: DataOutput, t: MyCaseClass): Unit = {
out.writeUTF(t.s)
}
}
}
fails with the error
Main.scala:9: error: could not find implicit value for evidence parameter of type mypackage.Serializer[mypackage.MyCaseClass]
serialize(out, caseClassInstance)
^
In my real code, I was using an auxiliary function to generate the Serializer[T] (see https://github.com/NICTA/scoobi/blob/24f48008b193f4e87b9ec04d5c8736ce0725d006/src/main/scala/com/nicta/scoobi/core/WireFormat.scala#L137). Despite the function having its own explicit return type, the type of the assigned value was not being inferred correctly by the compiler.
Below is the full example from the question with such a Serializer-generator.
package mypackage
import java.io.{DataOutput, DataOutputStream}
object Main {
import Serializer._
def main(args: Array[String]): Unit = {
val caseClassInstance = MyCaseClass("string")
val out: DataOutput = new DataOutputStream(System.out)
serialize(out, caseClassInstance)
}
def serialize[T : Serializer](out: DataOutput, t: T): Unit = {
implicitly[Serializer[T]].write(out, t)
}
}
object MyCaseClass {
import Serializer._
// does not compile without Serializer[MyCaseClass] type annotation
implicit val serializer: Serializer[MyCaseClass] =
mkCaseSerializer(MyCaseClass.apply _, MyCaseClass.unapply _)
}
case class MyCaseClass(s: String)
trait Serializer[T] {
def write(out: DataOutput, t: T): Unit
}
object Serializer {
// does not compile without Serializer[String] type annotation
implicit val stringSerializer: Serializer[String] = new Serializer[String] {
override def write(out: DataOutput, s: String): Unit = {
out.writeUTF(s)
}
}
class CaseClassSerializer[T, A : Serializer](
apply: A => T, unapply: T => Option[A]) extends Serializer[T] {
override def write(out: DataOutput, t: T): Unit = {
implicitly[Serializer[A]].write(out, unapply(t).get)
}
}
def mkCaseSerializer[T, A : Serializer]
(apply: A => T, unapply: T => Option[A]): Serializer[T] =
new CaseClassSerializer(apply, unapply)
}
This related, simple code below prints 1.
object A{
implicit def A2Int(a:A)=a.i1
}
case class A(i1:Int,i2:Int)
object Run extends App{
val a=A(1,2)
val i:Int=a
println(i)
}