I'm trying to dynamically choose a function, as follows:
val yld: (Nothing => Object) = {
column.getType match {
case PrimitiveType.PrimitiveTypeName.BINARY => new String(creader.getBinary.getBytes)
case PrimitiveType.PrimitiveTypeName.BOOLEAN => creader.getBoolean
case PrimitiveType.PrimitiveTypeName.DOUBLE => creader.getDouble
case PrimitiveType.PrimitiveTypeName.FLOAT => creader.getFloat
case PrimitiveType.PrimitiveTypeName.INT32 => creader.getInteger
case PrimitiveType.PrimitiveTypeName.INT64 => creader.getLong
case PrimitiveType.PrimitiveTypeName.INT96 => new String(creader.getBinary.getBytes)
case PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY => new String(creader.getBinary.getBytes)
}
}
Question: However, the inner functions (e.g. creader.getBoolean, etc.) are (obviously) executed at the time of the match. How do I capture those functions as objects to execute at a later time rather than having them execute when yld is assigned?
Context:
This is so that in a later for loop, I don't have to match again:
for (i <- 0L to 900000) {
println(yld)
}
Because my object creader is an iterator on column-stored data, being able to make the type decision only one time should be more efficient than having to make it every time, i.e.
for (i <- 0L to 900000) {
column.getType match {
case PrimitiveType.PrimitiveTypeName.BINARY => println(new String(creader.getBinary.getBytes))
case PrimitiveType.PrimitiveTypeName.BOOLEAN => println(creader.getBoolean)
case PrimitiveType.PrimitiveTypeName.DOUBLE => println(creader.getDouble)
case PrimitiveType.PrimitiveTypeName.FLOAT => println(creader.getFloat)
case PrimitiveType.PrimitiveTypeName.INT32 => println(creader.getInteger)
case PrimitiveType.PrimitiveTypeName.INT64 => println(creader.getLong)
case PrimitiveType.PrimitiveTypeName.INT96 => println(new String(creader.getBinary.getBytes))
case PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY => println(new String(creader.getBinary.getBytes))
}
}
If you have a method def foo = 42 you can convert it to a function and assign it to a val by val func = foo _.
Related
I want to know how to replace x._1._2, x._1._3 by the name of the field using case class
def keyuid(l:Array[String]) : (String,Long,String) ={
//val l=s.split(",")
val ip=l(3).split(":")(1)
val values=Array("",0,0,0)
val uid=l(1).split(":")(1)
val timestamp=l(2).split(":")(1).toLong*1000
val impression=l(4).split(":")(1)
return (uid,timestamp,ip)
}
val cli_ip = click.map(_.split(","))
.map(x => (keyuid(x), 1.0)).assignAscendingTimestamps(x=>x._1._2)
.keyBy(x => x._1._3)
.timeWindow(Time.seconds(10))
.sum(1)
Use Scala pattern matching when writing lambda functions using curly braces and case keyword.
val cli_ip = click.map(_.split(","))
.map(x => (keyuid(x), 1.0)).assignAscendingTimestamps {
case ((_, timestamp, _), _) => timestamp
}
.keyBy { elem => elem match {
case ((_, _, ip), _) => ip
}
}
.timeWindow(Time.seconds(10))
.sum(1)
More information on Tuples and their pattern matching syntax here: https://docs.scala-lang.org/tour/tuples.html
Pattern Matching is indeed a good idea and makes the code more readable.
To answer your question, to make keyuid function returns a case class, you first need to define it, for instance :
case class Click(string uid, long timestamp, string ip)
Then instead of return (uid,timestamp,ip) you need to do return Click(uid,timestamp,ip)
Case class are not related to flink, but to scala : https://docs.scala-lang.org/tour/case-classes.html
I've got a problem with returning a list after handling futures in scala. My code looks like this:
def getElements(arrayOfIds: Future[Seq[Int]]): Future[Seq[Element]] = {
var elementArray: Seq[Element] = Seq()
arrayOfIds.map {
ids => ids.map(id => dto.getElementById(id).map {
case Some(element) => elementArray = elementArray :+ element
case None => println("Element not found")
})
}
arrayOfIds.onComplete(_ => elementArray)
}
I'd like to do something like .onComplete, however the return type is
Unit and I'd like to return a Future[Seq[Whatever]]. Is there clean way to handle futures like this? Thanks!
Please provide the type of function dto.getElementById. If it is Int => Future[Option[Element]], then:
def getElements(arrayOfIds: Future[Seq[Int]]): Future[Seq[Element]] = {
val allElements: Future[Seq[Option[Element]]] = arrayOfIds.flatMap( ids =>
Future.sequence(ids.map(dto.getElementById))
)
allElements.map(_.flatMap{
case None => println();None
case some => some
})
}
Without logging, it would be:
arrayOfIds.flatMap( ids => Future.traverse(ids.map(dto.getElementById))(_.flatten))
Instead of assigning the result to a mutable variable, return it from the continuation of the Future. You can use flatMap to extract only the Element results which actually contain a value:
def getElements(arrayOfIds: Future[Seq[Int]]): Future[Seq[Element]] = {
arrayOfIds.flatMap(id => Future.fold(id.map(getElementById))(Seq.empty[Element])(_ ++ _))
}
I have an piece of code where i have used the pattern matching inside it i have used map in all cases, i want to get the output of what map is giving to a variable. Below is my code:
override def run():List[Option[Student]] =
StudentDataCache.get(surname) match {
case Some(i) => i.otherSiblings.map(siblings =>
StudentDataCache.get(siblings) match {
case Some(i) => Some(i)
case None=> getStudentFromDatabase(siblings)
}
)
case None =>
getStudentFromDatabase(surname).get.otherSiblings.map(siblings => StudentDataCache.get(siblings) match {
case Some(i) => Some(i)
case None=> getStudentFromDatabase(siblings)
}
)
}
output of both the map statement inside case is List[Option[Student]], is there a way to get this into variable because i want to convert this list into a single object since HystrixCommand execute output does not supports List as output. I want to convert it to StudentListing(val listing: List[Option[Student]])
just... assign it to a value / variable:
override def run(): StudentListing = {
val result = StudentDataCache.get(surname) match { /* same code*/ }
StudentListing(result) // or however you wrap it into a StudentListing...
}
A match expression, like any other expression in Scala, is evaluated into a value - you can do whatever you want with this value.
I have some code to convert json string to objects like following:
def apply(line: String): PairEvent = {
val (name, json) = line.span(_ != ' ')
name match {
case "OpenTabEvent" => Serialization.read[OpenTabEvent](json)
case "CloseTabEvent" => Serialization.read[CloseTabEvent](json)
case "ChangeContentEvent" => Serialization.read[ChangeContentEvent](json)
case "ChangeMasterRequest" => Serialization.read[ChangeMasterRequest](json)
case "CreateFileEvent" => Serialization.read[CreateFileEvent](json)
case "DeleteFileEvent" => Serialization.read[DeleteFileEvent](json)
case "CreateDirEvent" => Serialization.read[CreateDirEvent](json)
case "DeleteDirEvent" => Serialization.read[DeleteDirEvent](json)
case "WatchingFiles" => Serialization.read[WatchingFiles](json)
case _ =>
ServerLogger.info("!!!!!!!!!!!!!!!!!!!!! unknown line from server: " + line)
???
}
}
You can see I hardcoded some strings and use them as types to convert to objects.
Is there any way to do the same without hardcode them? I'm not sure if it's possible, even with macro.
You could use the class name:
case OpenTabEvent.getClass.getSimpleName => Serialization.read[OpenTabEvent](json)
Is there really a way to transform an object of type Future[Either[Future[T1], Future[T2]]] to and object of type Either[Future[T1], Future[T2]] ??
Maybe something like flatMap that works on Either....
I'm trying to make this code work (I have similar code that implements wrapped chain-of actions, but it doesn't involve future. It works, much simpler). The code below is based on that, with necessary modification to make it work for situation that involves futures.
case class WebServResp(msg: String)
case class WebStatus(code: Int)
type InnerActionOutType = Either[Future[Option[WebServResp]], Future[WebStatus]]
type InnerActionSig = Future[Option[WebServResp]] => Either[Future[Option[WebServResp]], Future[WebStatus]]
val chainOfActions: InnerActionSig = Seq(
{prevRespOptFut =>
println("in action 1: " + prevRespOptFut)
//dont care about prev result
Left(Future.successful(Some(WebServResp("result from 1"))))
},
{prevRespOptFut =>
println("in action 2: " + prevFutopt)
prevRespOptFut.map {prevRespOpt =>
//i know prevResp contains instance of WebServResp. so i skip the opt-matching
val prevWebServResp = prevRespOpt.get
Left(Some(prevWebServResp.msg + " & " + " additional result from 2"))
}
//But the outcome of the map above is: Future[Left(...)]
//What I want is Left(Future[...])
}
)
type WrappedActionSig = InnerActionOutType => InnerActionOutType
val wrappedChainOfActions = chainOfActions.map {innerAction =>
val wrappedAction: WrappedActionSig = {respFromPrevWrappedAction =>
respFromPrevWrappedAction match {
case Left(wsRespOptFut) => {
innerAction(wsRespOptFut)
}
case Right(wsStatusFut) => {
respFromPrevWrappedAction
}
}
}
wrappedAction
}
wrappedChainOfActions.fold(identity[WrappedActionIOType] _) ((l, r) => l andThen r).apply(Left(None))
UPDATE UPDATE UPDATE
Based on comments from Didier below ( Scala Future, flatMap that works on Either )... here's a code that works:
//API
case class WebRespString(str: String)
case class WebStatus(code: Int, str: String)
type InnerActionOutType = Either[Future[Option[WebRespString]], Future[WebStatus]]
type InnerActionSig = Future[Option[WebRespString]] => InnerActionOutType
type WrappedActionSig = InnerActionOutType => InnerActionOutType
def executeChainOfActions(chainOfActions: Seq[InnerActionSig]): Future[WebStatus] = {
val wrappedChainOfActions : Seq[WrappedActionSig] = chainOfActions.map {innerAction =>
val wrappedAction: WrappedActionSig = {respFromPrevWrappedAction =>
respFromPrevWrappedAction match {
case Left(wsRespOptFut) => {
innerAction(wsRespOptFut) }
case Right(wsStatusFut) => {
respFromPrevWrappedAction
}
}
}
wrappedAction
}
val finalResultPossibilities = wrappedChainOfActions.fold(identity[InnerActionOutType] _) ((l, r) => l andThen r).apply(Left(Future.successful(None)))
finalResultPossibilities match {
case Left(webRespStringOptFut) => webRespStringOptFut.map {webRespStringOpt => WebStatus(200, webRespStringOpt.get.str)}
case Right(webStatusFut) => webStatusFut
}
}
//API-USER
executeChainOfActions(Seq(
{prevRespOptFut =>
println("in action 1: " + prevRespOptFut)
//dont care about prev result
Left(Future.successful(Some(WebRespString("result from 1"))))
},
{prevRespOptFut =>
println("in action 2: " + prevRespOptFut)
Left(prevRespOptFut.map {prevRespOpt =>
val prevWebRespString = prevRespOpt.get
Some(WebRespString(prevWebRespString.str + " & " + " additional result from 2"))
})
}
)).map {webStatus =>
println(webStatus.code + ":" + webStatus.str)
}
executeChainOfActions(Seq(
{prevRespOptFut =>
println("in action 1: " + prevRespOptFut)
//Let's short-circuit here
Right(Future.successful(WebStatus(404, "resource non-existent")))
},
{prevRespOptFut =>
println("in action 2: " + prevRespOptFut)
Left(prevRespOptFut.map {prevRespOpt =>
val prevWebRespString = prevRespOpt.get
Some(WebRespString(prevWebRespString.str + " & " + " additional result from 2"))
})
}
)).map {webStatus =>
println(webStatus.code + ":" + webStatus.str)
}
Thanks,
Raka
The type Future[Either[Future[T1], Future[T2]]] means that sometimes later (that's future) one gets an Either, so at that time, one will know which way the calculation will go, and whether one will, still later, gets a T1 or a T2.
So the knowledge of which branch will be chosen (Left or Right) will come later. The type Either[Future[T1], Future[T2] means that one has that knowledge now (don't know what the result will be, but knows already what type it will be). The only way to get out of the Future is to wait.
No magic here, the only way for later to become now is to wait, which is done with result on the Future, and not recommended. `
What you can do instead is say that you are not too interested in knowing which branch is taken, as long has it has not completed, so Future[Either[T1, T2]] is good enough. That is easy. Say you have the Either and you would rather not look not but wait for the actual result :
def asFuture[T1, T2](
either: Either[Future[T1], Future[T2]])(
implicit ec: ExecutionContext)
: Future[Either[T1, T2] = either match {
case Left(ft1) => ft1 map {t1 => Left(t1)}
case Right(ft2) => ft2 map {t2 => Right(t2)}
}
You don't have the Either yet, but a future on that, so just flatMap
f.flatMap(asFuture) : Future[Either[T1, T2]]
(will need an ExecutionContext implicitly available)
It seems like you don't actually need the "failure" case of the Either to be a Future? In which case we can use scalaz (note that the "success" case of an either should be on the right):
import scalaz._
import scalaz.Scalaz._
def futureEitherFutureToFuture[A, B](f: Future[Either[A, Future[B]]])(
implicit ec: ExecutionContext): Future[Either[A, B]] =
f.flatMap(_.sequence)
But it's probably best to always keep the Future on the outside in your API, and to flatMap in your code rather than in the client. (Here it's part of the foldLeftM):
case class WebServResp(msg: String)
case class WebStatus(code: Int)
type OWSR = Option[WebServResp]
type InnerActionOutType = Future[Either[WebStatus, OWSR]]
type InnerActionSig = OWSR => InnerActionOutType
def executeChain(chain: List[InnerActionSig]): InnerActionOutType =
chain.foldLeftM(None: OWSR) {
(prevResp, action) => action(prevResp)
}
//if you want that same API
def executeChainOfActions(chainOfActions: Seq[InnerActionSig]) =
executeChain(chainOfActions.toList).map {
case Left(webStatus) => webStatus
case Right(webRespStringOpt) => WebStatus(200, webRespStringOpt.get.str)
}
(If you need "recovery" type actions, so you really need OWSR to be an Either, then you should still make InnerActionOutType a Future[Either[...]], and you can use .traverse or .sequence in your actions as necessary. If you have an example of an "error-recovery" type action, I can put an example of that here)