I am building an application using Kafka Streams with Scala. In it, I have a use case where I have to apply map() on a KStream.
Since Kafka Streams does not provide a Scala API, so, I have to write map function like this:
val builder = new KStreamBuilder()
val originalStream = builder.stream("SourceTopic")
val mappedStream =
originalStream.map[String, Integer] {
new KeyValueMapper[String, String, KeyValue[String, Integer]] {
override def apply(key: String, value: String): KeyValue[String, Integer] = {
new KeyValue(key, new Integer(value.length))
}
}
}
Above code compiles/runs fine. But, gives a warning - Convert Expression to Single Abstract Method.
So, my question is to how to convert above map expression into a SAM?
Any help is appreciated!
Related
I’m just heading an issue when I’m trying to create an endpoint with multiple bodies shape.
My model looks like this:
sealed trait FileExampleTrait {
def kind: String
}
case class FileExampleOne(name: String, length: Int) extends FileExampleTrait {
override def kind: String = “one”
}
case class FileExampleTwo(name: String) extends FileExampleTrait {
override def kind: String = “two”
}
case class FileExampleResponse(message: String)
And I’m trying to create this endpoint:
val fileExample = baseEndpoint.post
.in(“example”)
.in(jsonBody[FileExampleTrait])
.out(jsonBody[FileExampleResponse])
.summary(“something”)
.description(“something”)
The implementation of the endpoint looks like this:
private val fileExample = toAkkaRoute(jwtConsumer, errorHandler)(
FileApi.fileExample, { (scope: RequestScope, input: (FileExampleTrait)) =>
print(scope)
input match {
case FileExampleOne(name, _) => Future.successful(FileExampleResponse(name).asRight)
case FileExampleTwo(name) => Future.successful(FileExampleResponse(name).asRight)
}
}
)
This is just an example on what I’m trying to create. I added the schema derivation based on this:
val sOne = Schema.derived[FileExampleOne]
val sTwo = Schema.derived[FileExampleTwo]
implicit val sExampleTrait: Schema[FileExampleTrait] =
Schema.oneOfUsingField[FileExampleTrait, String](_.kind, _.toString)(“one” -> sOne, “two” -> sTwo)
I created a test for trying the endpoint based on Akka HTTP:
test(“Example test”) {
new Fixture() {
val request = FileExampleOne(“name”, 1)
Post(s”/api/v1/files/example”, jsonEntity(request)).withHeaders(requestHeaders) ~> wrappedRoute ~> check {
response should be(successful)
contentType shouldEqual ContentTypes.`application/json`
}
}
}
The error I’m getting is the following:
Response error: {“code”:400,“message”:“Invalid value for: body (No constructor for type FileExampleTrait, JObject(List((name,JString(name)), (length,JInt(1)))))“}
I was following this documentation.
Well that's because a trait doesn't have a constructor as indicated in the error itself. I think I see where you're going, you want to try parsing the body as one of the traits subclasses. So imagine you have this type/class hierarchy:
T // parent trait
/ \
C1 C2 // class 1, etc...
/
C3
Now you want to deserialize some JSON into trait T, you need to define your custom behavior, like "First try converting into C3, if failed, try converting to C2, if failed again, try converting to C1", and you'll get your T value. Now depending on the JSON library you use, the implementation might differ, see the documentation by softwaremill to get more information about how to deal with JSONs in tapir, and if you use Play Json, I can recommend:
object FileExampleOne {
implicit val reader: Reads[FileExampleOne] = Json.reads
}
object FileExampleTwo {
implicit val reader: Reads[FileExampleTwo] = Json.reads
}
object FileExampleTrait {
implicit val reads: Reads[FileExampleTrait] = json => {
json.validate[FileExampleOne] orElse json.validate[FileExampleTwo]
}
}
You can see it running on scastie. And based on the tapir documentations, you need a Codec for your types, and one of the approaches to create your coded for JSON, is using one of tapir's supported libraries (circe, Play Json, ...).
We've got a Flink job written in Scala using case classes (generated from avsc files by avrohugger) to represent our state. We would like to use Avro for serialising our state so state migration will work when we update our models. We understood since Flink 1.7 Avro serialization is supported OOTB. We added the flink-avro module to the classpath, but when restoring from a saved snapshot we notice that it's still trying to use Kryo serialization. Relevant Code snippet
case class Foo(id: String, timestamp: java.time.Instant)
val env = StreamExecutionEnvironment.getExecutionEnvironment
val conf = env.getConfig
conf.disableForceKryo()
conf.enableForceAvro()
val rawDataStream: DataStream[String] = env.addSource(MyFlinkKafkaConsumer)
val parsedDataSteam: DataStream[Foo] = rawDataStream.flatMap(new JsonParser[Foo])
// do something useful with it
env.execute("my-job")
When performing a state migration on Foo (e.g. by adding a field and deploying the job) I see that it tries to deserialize using Kryo, which obviously fails. How can I make sure Avro serialization is being used?
UPDATE
Found out about https://issues.apache.org/jira/browse/FLINK-10897, so POJO state serialization with Avro is only supported from 1.8 afaik. I tried it using the latest RC of 1.8 with a simple WordCount POJO that extends from SpecificRecord:
/** MACHINE-GENERATED FROM AVRO SCHEMA. DO NOT EDIT DIRECTLY */
import scala.annotation.switch
case class WordWithCount(var word: String, var count: Long) extends
org.apache.avro.specific.SpecificRecordBase {
def this() = this("", 0L)
def get(field$: Int): AnyRef = {
(field$: #switch) match {
case 0 => {
word
}.asInstanceOf[AnyRef]
case 1 => {
count
}.asInstanceOf[AnyRef]
case _ => new org.apache.avro.AvroRuntimeException("Bad index")
}
}
def put(field$: Int, value: Any): Unit = {
(field$: #switch) match {
case 0 => this.word = {
value.toString
}.asInstanceOf[String]
case 1 => this.count = {
value
}.asInstanceOf[Long]
case _ => new org.apache.avro.AvroRuntimeException("Bad index")
}
()
}
def getSchema: org.apache.avro.Schema = WordWithCount.SCHEMA$
}
object WordWithCount {
val SCHEMA$ = new org.apache.avro.Schema.Parser().parse(" .
{\"type\":\"record\",\"name\":\"WordWithCount\",\"fields\":
[{\"name\":\"word\",\"type\":\"string\"},
{\"name\":\"count\",\"type\":\"long\"}]}")
}
This, however, also didn’t work out of the box. We then tried to define our own type information using flink-avro’s AvroTypeInfo but this fails because Avro looks for a SCHEMA$ property (SpecificData:285) in the class and is unable to use Java reflection to identify the SCHEMA$ in the Scala companion object.
I could never get reflection to work due to Scala's fields being private under the hood. AFAIK the only solution is to update Flink to use avro's non-reflection-based constructors in AvroInputFormat (compare).
In a pinch, other than Java, one could fall back to avro's GenericRecord, maybe use avro4s to generate them from avrohugger's Standard format (note that Avro4s will generate it's own schema from the generated Scala types)
I want to share a HashMap across every node in Flink and allow the nodes to update that HashMap. I have this code so far:
object ParallelStreams {
val env = StreamExecutionEnvironment.getExecutionEnvironment
//Is there a way to attach a HashMap to this config variable?
val config = new Configuration()
config.setClass("HashMap", Class[CustomGlobal])
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
class CustomGlobal extends ExecutionConfig.GlobalJobParameters {
override def toMap: util.Map[String, String] = {
new HashMap[String, String]()
}
}
class MyCoMap extends RichCoMapFunction[String, String, String] {
var users: HashMap[String, String] = null
//How do I get access the HashMap I attach to the global config here?
override def open(parameters: Configuration): Unit = {
super.open(parameters)
val globalParams = getRuntimeContext.getExecutionConfig.getGlobalJobParameters
val globalConf = globalParams[Configuration]
val hashMap = globalConf.getClass
}
//Other functions to override here
}
}
I was wondering if you can attach a custom object to the config variable created here val config = new Configuration()? (Please see comments in the code above).
I noticed you can only attach primitive values. I created a custom class that extends ExecutionConfig.GlobalJobParameters and attached that class by doing config.setClass("HashMap", Class[CustomGlobal]) but I am not sure if that is how you are supposed to do it?
The common way to distribute parameters to operators is to have them as regular member variables in the function class. The function object that is created and assigned during plan construction is serialized and shipped to all workers. So you don't have to pass parameters via a configuration.
This would look as follows
class MyMapper(map: HashMap) extends MapFunction[String, String] {
// class definition
}
val inStream: DataStream[String] = ???
val myHashMap: HashMap = ???
val myMapper: MyMapper = new MyMapper(myHashMap)
val mappedStream: DataStream[String] = inStream.map(myMapper)
The myMapper object is serialized (using Java serialization) and shipped for execution. So the type of map must implement the Java Serializable interface.
EDIT: I missed the part that you want the map to be updatable from all parallel tasks. That is not possible with Flink. You would have to either fully replicate the map and all updated (by broadcasting) or use an external system (key-value store) for that.
Hello all :) I'm about 16 hours new to Scala and Play!Framework (version 2.1). I'm following this Play!2.0 tutorial with Anorm, which uses Jerkson. From what I understand, in 2.1 you can do that out of the box, as long as you have the right JSON Formatters.
So here is the JSON service:
def listBars() = Action {
val bars = Bar.findAll()
Ok(Json.toJson(bars)).as("application/json")
}
And here is Bar.scala:
case class Bar(id: Pk[Long], name: String)
object Bar {
implicit var anormLongPkFormat = new Format[Pk[Long]] {
def writes(key: Pk[Long]): JsValue = Json.toJson(key.toString)
def reads(jv: JsValue): JsResult[Pk[Long]] = JsSuccess( -?- )
}
implicit val barFormat = Json.format[Bar]
def findAll(): Seq[Bar] = {...}
}
I'm using Json.format[Bar], but it tells me he needs another formatter for anorm.Pk[Long]. I don't need the reads method, for the moment, I only want to serve the values; But the compiler needs a reads method. I'm totally at a loss at how to make it compile, let alone at how to write a good reads.
Best regards
If you don't need reads now, then the easiest way is not to implement it's logic and return an error:
def reads(jv: JsValue): JsResult[Pk[Long]] = JsError()
(Essentially I need some kind of a synthesis of these two questions (1, 2), but I'm not smart enough to combine them myself.)
I have a set of JAXB representations in Scala like this:
abstract class Representation {
def marshalToXml(): String = {
val context = JAXBContext.newInstance(this.getClass())
val writer = new StringWriter
context.createMarshaller.marshal(this, writer)
writer.toString()
}
}
class Order extends Representation {
#BeanProperty
var name: String = _
...
}
class Invoice extends Representation { ... }
The problem I have is with my unmarshalling "constructor" methods:
def unmarshalFromJson(marshalledData: String): {{My Representation Subclass}} = {
val mapper = new ObjectMapper()
mapper.getDeserializationConfig().withAnnotationIntrospector(new JaxbAnnotationIntrospector())
mapper.readValue(marshalledData, this.getClass())
}
def unmarshalFromXml(marshalledData: String): {{My Representation Subclass}} = {
val context = JAXBContext.newInstance(this.getClass())
val representation = context.createUnmarshaller().unmarshal(
new StringReader(marshalledData)
).asInstanceOf[{{Type of My Representation Subclass}}]
representation // Return the representation
}
Specifically, I can't figure out how to attach these unmarshalling methods in a typesafe and DRY way to each of my classes, and then to call them from Scala (and hopefully sometimes by using only abstract type information). In other words, I would like to do this:
val newOrder = Order.unmarshalFromJson(someJson)
And more ambitiously:
class Resource[R <: Representation] {
getRepresentation(marshalledData: String): R =
{{R's Singleton}}.unmarshalFromXml(marshalledData)
}
In terms of my particular stumbling blocks:
I can't figure out whether I should define my unmarshalFrom*() constructors once in the Representation class, or in a singleton Representation object - if the latter, I don't see how I can automatically inherit that down through the class hierarchy of Order, Invoice etc.
I can't get this.type (as per this answer) to work as a way of self-typing unmarshalFromJson() - I get a compile error type mismatch; found: ?0 where type ?0 required: Representation.this.type on the readValue() call
I can't figure out how to use the implicit Default[A] pattern (as per this answer) to work down my Representation class hierarchy to call the singleton unmarshalling constructors using type information only
I know this is a bit of a mammoth question touching on various different (but related) issues - any help gratefully received!
Alex
The key is to not try and attach the method to the class but rather pass it in as a parameter. To indicate the type you are expecting and let the type system handle passing it in. I tried to make the unmarshal invocation something that reads a little DSL like.
val order = UnMarshalXml( xml ).toRepresentation[Order]
The following is a fully testable code snippet
abstract class Representation {
def marshalToXml(): String = {
val context = JAXBContext.newInstance(this.getClass)
val writer = new StringWriter
context.createMarshaller.marshal(this, writer)
writer.toString
}
}
#XmlRootElement
class Order extends Representation {
#BeanProperty
var name: String = _
}
case class UnMarshalXml( xml: String ) {
def toRepresentation[T <: Representation](implicit m:Manifest[T]): T = {
JAXBContext.newInstance(m.erasure).createUnmarshaller().unmarshal(
new StringReader(xml)
).asInstanceOf[T]
}
}
object test {
def main( args: Array[String] ) {
val order = new Order
order.name = "my order"
val xml = order.marshalToXml()
println("marshalled: " + xml )
val received = UnMarshalXml( xml ).toRepresentation[Order]
println("received order named: " + received.getName )
}
}
You should see the following output if you run test.main
marshalled: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><order><name>my order</name></order>
received name: my order
Here's the updated version of Neil's code which I used to support the second use case as well as the first:
case class UnmarshalXml(xml: String) {
def toRepresentation[T <: Representation](implicit m: Manifest[T]): T =
toRepresentation[T](m.erasure.asInstanceOf[Class[T]])
def toRepresentation[T <: Representation](typeT: Class[T]): T =
JAXBContext.newInstance(typeT).createUnmarshaller().unmarshal(
new StringReader(xml)
).asInstanceOf[T]
}
This supports simple examples like so:
val order = UnmarshalXml(xml).toRepresentation[Order]
But also for abstract type based usage, you can use like this:
val order = UnmarshalXml(xml).toRepresentation[T](typeOfT)
(Where you have grabbed and stored typeOfT using another implicit Manifest at the point of declaring T.)