I have a Finatra application that accesses Cassandra using Datastax driver which produces Guava Futures. The conversion is done by following code
import com.google.common.util.concurrent.{FutureCallback, Futures, ListenableFuture}
import com.twitter.util.{Future, Promise}
implicit def toTwitterFuture[A](f: ListenableFuture[A]): Future[A] = {
val p = Promise[A]()
val callback = new FutureCallback[A] {
override def onSuccess(result: A): Unit = p.setValue(result)
override def onFailure(t: Throwable): Unit = p.setException(t)
}
Futures.addCallback(f, callback)
p
}
Problem is, after this conversion the rest of the request processing happens in a thread pool created by Cassandra driver potentially blocking other I/O tasks. How can I access Finagle's executor where this kind of CPU-intensive work should be done?
Related
I have multiple files those are independent and need processing by spark. How could I load them into separate rdds in parallel? Thanks!
The coding language is scala.
If you want concurrent reading/processing of RDDs, you could leverage scala.concurrent.Future (or effects in ZIO, Cats etc).
Sample code for loading function is below:
def load(paths: Seq[String], spark: SparkSession)
(implicit ec: ExecutionContext): Seq[Future[RDD[String]]] = {
def loadSinglePath(path: String): Future[RDD[String]] = Future {
spark.sparkContext.textFile(path)
}
paths map loadSinglePath
}
Sample code for using this function:
import scala.concurrent.duration.{Duration, DurationInt}
val sc = SparkSession.builder.master("local[*]").getOrCreate()
implicit val ec = ExecutionContext.global
val result = load(Seq("t1.txt", "t2.txt", "t3.txt"), sc).zipWithIndex
.map { case (rddFuture, idx) =>
rddFuture.map( rdd =>
println(s"Rdd with index $idx has ${rdd.count()}")
)
}
Await.result(Future.sequence(result), 1 hour)
For example purposes, default global ExecutionContext is provided, but it could be configurable to run your code inside the custom one (just replace this implicit val ec with your own ExecutionContext)
I am trying to work with typed actors version 2.6.3 and akka http version 10.1.11
while all worked fine in non typed actors , now I am getting compilation error
object Main extends App {
def createServer(implicit system: ActorSystem[IdentityCalculated]) = {
implicit val materializer = ActorMaterializer()
}
def apply(): Behavior[IdentityCalculated] = {
Behaviors.setup { context =>
val identityManager = context.spawn(IdentityManager(), "identity-manager")
implicit val timeout = Timeout(10, TimeUnit.SECONDS)
implicit val scheduler = context.system.scheduler
identityManager.tell(CalculateIdentity(context.self))
Behaviors.receiveMessage{
case IdentityCalculated(_,_,_) =>
println("got response")
Behaviors.same
}
}
}
ActorSystem(Main(), "credentials-manager")
}
what am I missing?
I want to create http server when I get the message that the identity was calculated how ever i am getting compilation error in the materializer
creation
Main.scala:19:50: implicit ActorRefFactory required: if outside of an Actor you need an implicit ActorSystem, inside of an actor this should be the implicit ActorContext
[error] implicit val materializer = ActorMaterializer()
[error] ^
[error] one error found
thanks
You can use classic streams (which are typed, despite running on an untyped ActorSystem) by replacing:
implicit val materializer = ActorMaterializer()
with
implicit val materializer = system.classicSystem
In Akka 2.6 the ActorMaterializer API has been deprecated and a new system wide materializer that is always available have been introduced. To run a stream you should only need the implicit ActorSystem[T] in scope.
akka-stream-typed is only needed if you want to use the specific stream operators for interacting with Akka typed actors (ActorSource, ActorFlow and ActorSink)
However the Akka HTTP APIs is still depending on the classic APIs so you may need to adapt them to be able to bind a HTTP endpoint for example.
If that is your issue, you can see how to do that in the Akka HTTP quickstart guide here: https://developer.lightbend.com/guides/akka-http-quickstart-scala/http-server.html
I had to use akka-stream-typed and not akka-stream
I'd like to convert fs2.Stream to java.io.InputStream so I can pass that input stream to an http framework (Finch and Akka Http).
I found a fs2.io.toInputStream, but this doesn't work (it prints nothing):
import java.io.{ByteArrayInputStream, InputStream}
import cats.effect.IO
import scala.concurrent.ExecutionContext.Implicits.global
object IOTest {
def main(args: Array[String]): Unit = {
val is: InputStream = new ByteArrayInputStream("test".getBytes)
val stream: fs2.Stream[IO, Byte] = fs2.io.readInputStream(IO(is), 128)
val test: Seq[InputStream] = stream.through(fs2.io.toInputStream).compile.toList.unsafeRunSync()
println(scala.io.Source.fromInputStream(test.head).mkString)
}
}
As far as I understand when I run .unsafeRunSync() it's consuming the whole stream, so even though it returns a Seq[InputStream] the under-laying input stream is already consumed.
Is there any way I can convert fs2.Stream[IO, Byte] to java.io.InputStream without it being consumed?
Thnaks!
The problem is that compile is being invoked prematurely. I'm sure that under the hood fs2.io.toInputStream does the correct thing and brackets the created InputStream. Which means that the InputStream must be accessed inside the Stream itself (e.g., in a map/flatMap call):
val wire: fs2.Stream[IO, Byte] = ???
val result: fs2.Stream[IO, String] = for {
is <- wire.through(fs2.io.toInputStream)
str = scala.io.Source.fromInputStream(is).mkString //<--- use the InputStream here
} yield str
println( result.compile.lastOrError.unsafeRunSync() ) //<--- compile at the _very_ end
Outputs:
test
It looks that Finch has fs2 support https://github.com/finagle/finch/tree/master/fs2 and Akka also has it's stream implementation and there are fs2 - Akka Stream interop libraries like https://github.com/krasserm/streamz/tree/master/streamz-converter
So i recommend you to take a look to the implementations because they take care of the resources life cycle. Probably you don't need the whole library but it serves as guideline.
And if you are starting at the "safe zone" with fs2, why moving out of there :)
As "MongoDb Scala Driver" is the only official Scala driver now, I plan to switch from Casbah. However, MongoDb Scala Driver seems to support only Asynchronous API (at least in its documentation).
Is there a way to make synchronous queries?
I had the same problems some days ago when I was moving from Casbah. Apparently the official Mongodb driver uses the observer pattern. I wanted to retrieve a sequence number from a collection and I had to wait for the value to be retrieved to continue the operation. I am not sure if that's the correct way, but at least this is one way of doing it:
def getSequenceId(seqName: String): Int = {
val query = new BsonDocument("seq_id", new BsonString(seqName))
val resultado = NewMongo.SequenceCollection.findOneAndUpdate(query,inc("nextId",1))
resultado.subscribe(new Observer[Document] {
override def onNext(result: Document): Unit ={}
override def onError(e: Throwable): Unit ={}
override def onComplete(): Unit = {}
})
val awaitedR = Await.result(resultado.toFuture, Duration.Inf).asInstanceOf[List[Document]](0)
val ret = awaitedR.get("nextId").getOrElse(0).asInstanceOf[BsonDouble].intValue();
return ret;
}
Apparently you can convert the result observer into a future and wait for its return using the Await function as I did. Then you can manipulate the result as you wish.
The database configuration is the following:
private val mongoClient: MongoClient = MongoClient("mongodb://localhost:27017/?maxPoolSize=30")
private val database: MongoDatabase = mongoClient.getDatabase("mydb");
val Sequence: MongoCollection[Document] = database.getCollection(SEQUENCE);
Hope my answer was helpful
I am working on Play framework 2.5.3 and Akka 2.4.7 using Scala. I have an application made in play based webSockets using Akka. I am using the project as a jar in another project.
My code in new project looks like this:
For accepting connections:
val postActor = actorSystem.actorOf(Props[PostActorClass])
def wsSocket = WebSocket.accept[JsValue,JsValue]{ request =>
ActorFlow.actorRef(out => wsConnObj.HandlerClass.props(out,postActor))
}
For getting references of all connections stored in my jar project:
def getConnReferences:Set[ActorRef] = {
implicit val timeout = Timeout(5 seconds)
val future = postActor ? GetReferences
val result = Await.result(future, timeout.duration).asInstanceOf[Set[ActorRef]]
(result)
}
For sending messages to all the references:
def sendMsg = {
val msg = Json.parse("""{"username":"Server","message":"Reached finally.","pub_key":"empty","target":"user"}""")
val refs = getConnReferences
refs.foreach( _ ! msg)
}
I am trying to use these methods in my new project's HomeController. But unable to work them together, like wsSocket should be executed , then sendMsg (which itself calls getConnReferences also).
How will I achieve that? If am calling all these blocks in single action method, I can't establish connection even.
Code for WebSocketConnect class:
class WebSocketConnect #Inject() (cache:CacheApi) (implicit actorSystem: ActorSystem,materializer: Materializer){
object HandlerClass {
homelogger.info(logMessages.passingActorRef)
def props(out: ActorRef, postActor: ActorRef) = {
Props(new SocketHandlerClass(out, postActor, cache, postActorToUsernameMap))
}
}
}