I am trying to use a ProcessWindowFunction in my Apache Flink project using Scala. Unfortunately, I already fail at implementing a basic ProcessWindowFunction like it is used in the Apache Flink Documentation.
This is my code:
import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.api.scala.{StreamExecutionEnvironment, _}
import org.apache.flink.streaming.api.windowing.time.Time
import org.fiware.cosmos.orion.flink.connector.{NgsiEvent, OrionSource}
import org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction
import org.apache.flink.streaming.api.windowing.windows.TimeWindow
import org.apache.flink.streaming.api.windowing.assigners.SlidingProcessingTimeWindows
import org.apache.flink.util.Collector
import scala.collection.TraversableOnce
object StreamingJob {
def main(args: Array[String]) {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val eventStream = env.addSource(new OrionSource(9001))
val processedDataStream = eventStream.flatMap(event => event.entities)
.map(entity => (entity.id, entity.attrs("temperature").value.asInstanceOf[String]))
.keyBy(_._1)
.window(SlidingProcessingTimeWindows.of(Time.seconds(10), Time.seconds(5)))
.process(new MyProcessWindowFunction())
env.execute("Socket Window NgsiEvent")
}
}
private class MyProcessWindowFunction extends ProcessWindowFunction[(String, String), String, String, TimeWindow] {
def process(key: String, context: Context, input: Iterable[(String, String)], out: Collector[String]): Unit = {
var count: Int = 0
for (in <- input) {
count = count + 1
}
out.collect(s"Window ${context.window} count: $count")
}
}
From IntelliJ I get the following hints:
1) This is shown where the new class object is created:
Type mismatch, expected: ProcessWindowFunction[(String, String), NotInferedR, String, TimeWindow], actual: MyProcessWindowFunction
2) This is shown directly at the class:
Class 'MyProcessWindowFunction' must either be declared abstract or implement abstract member 'process(key:KEY, context:ProcessWindowFunction.Context, iterable:Iterable<IN>, collector:Collector<OUT>):void' in 'org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction'
Building the code shows me the following error:
Error:(51, 16) type mismatch;
found : org.apache.flink.MyProcessWindowFunction
required:
org.apache.flink.streaming.api.scala.function.ProcessWindowFunction[(String, String),?,String,org.apache.flink.streaming.api.windowing.windows.TimeWindow]
.process(new MyProcessWindowFunction())
I am grateful for every help.
After spending some time debugging with 2 more people we finally managed to find the problem.
In my code I used the following import:
import org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction
But the correct import when using Scala seems to be:
import org.apache.flink.streaming.api.scala.function.ProcessWindowFunction
//package of ProcessWindowFunction is
import org.apache.flink.streaming.api.scala.function.ProcessWindowFunction
//The correct way to call this method
new MyProcessWindowFunction()[(String, String), String, String, TimeWindow]
//I know the official documents don't.This may be a bug
Related
I have the following:
Account.scala
package modules.accounts
import java.time.Instant
import reactivemongo.api.bson._
case class Account(id: String, name: String)
object Account {
type ID = String
implicit val accountHandler: BSONDocumentHandler[Account] = Macros.handler[Account]
// implicit def accountWriter: BSONDocumentWriter[Account] = Macros.writer[Account]
// implicit def accountReader: BSONDocumentReader[Account] = Macros.reader[Account]
}
AccountRepo.scala
package modules.accounts
import java.time.Instant
import reactivemongo.api.collections.bson.BSONCollection
import scala.concurrent.ExecutionContext
final class AccountRepo(
val coll: BSONCollection
)(implicit ec: ExecutionContext) {
import Account.{ accountHandler, ID }
def insertTest() = {
val doc = Account(s"account123", "accountName") //, Instant.now)
coll.insert.one(doc)
}
}
The error I am getting is:
could not find implicit value for parameter writer: AccountRepo.this.coll.pack.Writer[modules.accounts.Account]
[error] coll.insert.one(doc)
From what I understand the implicit handler that is generated by the macro should be enough and create the Writer. What am I doing wrong?
Reference: http://reactivemongo.org/releases/1.0/documentation/bson/typeclasses.html
The code is mismixing different versions.
The macro generated handler is using the new BSON API, as it can be seen with the import reactivemongo.api.bson, whereas the collection is using an old driver, as it can be seen as it uses reactivemongo.api.collections.bson instead of reactivemongo.api.bson.collection.
It's recommended to have a look at the documentation, and not mixing incompatible versions of related libraries.
I'm not really sure where this error is coming from. I'm trying to migrate from an old .find() method where I could pass Pattern: query: BSONDocument to find and call .one[T] on it. This worked perfectly fine, but I'd like to follow the recommendations.
However, I have a feeling that my desire to create a generic database access object is causing some of this headache... take a look below.
The object that i'm trying to read from:
package Models
import reactivemongo.api.bson.{BSONDocumentHandler, Macros}
case class Person(name: String)
object Person {
implicit val handler: BSONDocumentHandler[Person] = Macros.handler[Person]
}
The class containing the method I'm trying to execute (ignore the Await, etc.)
package Data
import reactivemongo.api.AsyncDriver
import reactivemongo.api.bson._
import reactivemongo.api.bson.collection.BSONCollection
import reactivemongo.api.commands.WriteResult
import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
import scala.reflect.ClassTag
object DataService extends DataService
trait DataService {
implicit val ec: scala.concurrent.ExecutionContext = scala.concurrent.ExecutionContext.global
private val driver = new AsyncDriver
private val connection = Await.result(driver.connect(List("localhost:32768")), 10.seconds)
private val database = Await.result(connection.database("database"), 10.seconds)
private def getCollection[T]()(implicit tag: ClassTag[T]): BSONCollection = database.collection(tag.runtimeClass.getSimpleName)
def Get[T]()(implicit handler: BSONDocumentHandler[T], tag: ClassTag[T]): Future[Option[T]] = {
val collection = getCollection[T]
collection.find(BSONDocument("name" -> "hello"), Option.empty).one[T]
}
}
How I'm trying to get it (ignore the Await, this is just for test purposes):
val personName = Await.result(dataService.Get[Person](), 10.seconds).map(_.name).getOrElse("Not found")
The error that I'm getting:
Error:(32, 19) ambiguous implicit values:
both object BSONDocumentIdentity in trait BSONIdentityHandlers of type reactivemongo.api.bson.package.BSONDocumentIdentity.type
and value handler of type reactivemongo.api.bson.BSONDocumentHandler[T]
match expected type collection.pack.Writer[J]
collection.find(BSONDocument("name" -> "hello"), Option.empty).cursor[T]
Thank you all for your help.
Try to add type to your option in your second argument find method, for example:
def Get[T]()(implicit handler: BSONDocumentHandler[T], tag: ClassTag[T]): Future[Option[T]] = {
val collection = getCollection[T]
collection.find(BSONDocument("name" -> "hello"), Option.empty[BSONDocument]).one[T]
}
I'd like to build a generic method for transforming Scala Case Classes to Mongo Documents.
A promising Document constructor is
fromSeq(ts: Seq[(String, BsonValue)]): Document
I can turn a case class into a Map[String -> Any], but then I've lost the type information I need to use the implicit conversions to BsonValues. Maybe TypeTags can help with this?
Here's what I've tried:
import org.mongodb.scala.bson.BsonTransformer
import org.mongodb.scala.bson.collection.immutable.Document
import org.mongodb.scala.bson.BsonValue
case class Person(age: Int, name: String)
//transform scala values into BsonValues
def transform[T](v: T)(implicit transformer: BsonTransformer[T]): BsonValue = transformer(v)
// turn any case class into a Map[String, Any]
def caseClassToMap(cc: Product) = {
val values = cc.productIterator
cc.getClass.getDeclaredFields.map( _.getName -> values.next).toMap
}
// transform a Person into a Document
def personToDocument(person: Person): Document = {
val map = caseClassToMap(person)
val bsonValues = map.toSeq.map { case (key, value) =>
(key, transform(value))
}
Document.fromSeq(bsonValues)
}
<console>:24: error: No bson implicit transformer found for type Any. Implement or import an implicit BsonTransformer for this type.
(key, transform(value))
def personToDocument(person: Person): Document = {
Document("age" -> person.age, "name" -> person.name)
}
Below code works without manual conversion of an object.
import reactivemongo.api.bson.{BSON, BSONDocument, Macros}
case class Person(name:String = "SomeName", age:Int = 20)
implicit val personHandler = Macros.handler[Person]
val bsonPerson = BSON.writeDocument[Person](Person())
println(s"${BSONDocument.pretty(bsonPerson.getOrElse(BSONDocument.empty))}")
You can use Salat https://github.com/salat/salat. A nice example can be found here - https://gist.github.com/bhameyie/8276017. This is the piece of code that will help you -
import salat._
val dBObject = grater[Artist].asDBObject(artist)
artistsCollection.save(dBObject, WriteConcern.Safe)
I was able to serialize a case class to a BsonDocument using the org.bson.BsonDocumentWriter. The below code runs using scala 2.12 and mongo-scala-driver_2.12 version 2.6.0
My quest for this solution was aided by this answer (where they are trying to serialize in the opposite direction): Serialize to object using scala mongo driver?
import org.mongodb.scala.bson.codecs.Macros
import org.mongodb.scala.bson.codecs.DEFAULT_CODEC_REGISTRY
import org.bson.codecs.configuration.CodecRegistries.{fromRegistries, fromProviders}
import org.bson.codecs.EncoderContext
import org.bson.BsonDocumentWriter
import org.mongodb.scala.bson.BsonDocument
import org.bson.codecs.configuration.CodecRegistry
import org.bson.codecs.Codec
case class Animal(name : String, species: String, genus: String, weight: Int)
object TempApp {
def main(args: Array[String]) {
val jaguar = Animal("Jenny", "Jaguar", "Panthera", 190)
val codecProvider = Macros.createCodecProvider[Animal]()
val codecRegistry: CodecRegistry = fromRegistries(fromProviders(codecProvider), DEFAULT_CODEC_REGISTRY)
val codec = Macros.createCodec[Animal](codecRegistry)
val encoderContext = EncoderContext.builder.isEncodingCollectibleDocument(true).build()
var doc = BsonDocument()
val writr = new BsonDocumentWriter(doc) // need to call new since Java lib w/o companion object
codec.encode(writr, jaguar, encoderContext)
print(doc)
}
};
I'm doing tests with reactivemongo
In my controller I have this:
package controllers
import models._
import models.JsonFormats._
import play.modules.reactivemongo.MongoController
import scala.concurrent.Future
import reactivemongo.api.Cursor
import org.slf4j.{LoggerFactory, Logger}
import javax.inject.Singleton
import play.api.mvc._
import reactivemongo.api.collections.default.BSONCollection
import reactivemongo.bson._
#Singleton
class Users extends Controller with MongoController {
private final val logger: Logger = LoggerFactory.getLogger(classOf[Users])
val collection = db[BSONCollection]("users")
// list all articles and sort them
def list = Action.async { implicit request =>
// get a sort document (see getSort method for more information)
val sort = getSort(request)
// build a selection document with an empty query and a sort subdocument ('$orderby')
val query = BSONDocument(
"$orderby" -> sort,
"$query" -> BSONDocument())
val activeSort = request.queryString.get("sort").flatMap(_.headOption).getOrElse("none")
// the cursor of documents
val found = collection.find(query).cursor[User]
// build (asynchronously) a list containing all the articles
found.collect[List]().map { users =>
Ok(views.html.admin.list(users, activeSort))
}.recover {
case e =>
e.printStackTrace()
BadRequest(e.getMessage())
}
}
...........
}
and in my model i have this:
package models
import reactivemongo.bson._
case class User(
nickName: String,
email: String,
password: String,
active: Boolean
)
object JsonFormats {
import play.api.libs.json.Json
// Generates Writes and Reads for Feed and User thanks to Json Macros
implicit val userFormat = Json.format[User]
}
When I compile the project returns the following error:
could not find implicit value for parameter reader: reactivemongo.bson.BSONDocumentReader[models.User]
in this line is the problem:
val found = collection.find(query).cursor[User]
Can anyone tell me where I'm wrong or what I'm missing please?
You have no implicit handler defined to map your model class to a BSONDocument. You can implement it yourself, or, just like you did for the JsonFormats, you could use the macros provided by ReactiveMongo.
object BsonFormats {
import reactivemongo.bson.Macros
implicit val userFormat = Macros.handler[User]
}
Alternatively, instead of the BSONCollection, you could use the JSONCollection provided by Play-ReactiveMongo to perform your mapping using the JSON format that you have already defined.
For me, I still get the error even after I have declared the implicits for both bson and json format. What I need to do is just import this:
import reactivemongo.api.commands.bson.BSONCountCommandImplicits._
I have the following route setup, but when my map is returned in the first complete block I get an error:
could not find implicit value for evidence parameter of type spray.httpx.marshalling.Marshaller[scala.collection.immutable.Map[String,String]]
import spray.routing.HttpService
import akka.actor.Actor
import spray.http.HttpRequest
import spray.routing.RequestContext
import spray.json.DefaultJsonProtocol._
class UserServiceActor extends Actor with RestUserService {
def actorRefFactory = context
def receive = runRoute(linkRoute)
}
trait RestUserService extends HttpService {
val userService = new LinkUserService
def linkRoute =
pathPrefix("user" / Segment) {
userId =>
path("link") {
parameters('service ! "YT") {
complete {
Map("status"-> "OK", "auth_url" -> "http://mydomain.com/auth")
}
}
}
}
}
According to this test I should be able to convert a Map to json when DefaultJsonProtocol._ is imported but even that's failing:
val map:Map[String, String] = Map("hi"->"bye")
map.toJson
Cannot find JsonWriter or JsonFormat type class for scala.collection.mutable.Map[String,String]
Not sure what's wrong :(
Someone on the spray mailing list pointed out that the Map being created was a mutable one, spray-json won't marshal that. I changed it to be scala.collection.immutable.Map and also added the following imports:
import spray.httpx.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
And now everything works great.