Proper usage of Monix 3.2.2 Observable with Doobie 0.9.0 - scala

I would like to use Monix Observable with Doobie (fs2) stream, but can't seem to get it working properly. Without streaming, my test app exits just fine but after using streaming, my TaskApp seems to hang on shutdown and can't figure out why.
Here is a minimal example to re-produce the problem:
package example
import java.util.concurrent.Executors
import doobie.implicits._
import cats.effect.{Blocker, ContextShift, ExitCode, Resource}
import doobie.hikari.HikariTransactor
import monix.eval.{Task, TaskApp}
import com.typesafe.scalalogging.StrictLogging
import fs2.interop.reactivestreams._
import monix.reactive.Observable
import scala.concurrent.ExecutionContext
object Hello extends TaskApp with StrictLogging {
private def resources()(implicit contextShift: ContextShift[Task]): Resource[Task, Resources] = {
for {
transactor <- Database.transactor("org.postgresql.Driver", "jdbc:postgresql://localhost/fubar", "fubar", "fubar")
} yield Resources(transactor)
}
def run(args: List[String]): Task[ExitCode] = resources().use(task)
.flatMap(_ => Task { println("All Done!") })
.flatMap(_ => Task(ExitCode.Success))
def task(resources: Resources): Task[Unit] = {
val publisher =
sql"""select id from message;"""
.query[(Long)]
.stream
.transact(resources.transactor)
.toUnicastPublisher()
Observable.fromReactivePublisher(publisher)
.foreachL(id => logger.info(id.toString))
}
}
case class Resources(transactor: HikariTransactor[Task])
object Database {
val ecBlocking = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(8))
def transactor(dbDriver: String, dbUrl: String, dbUser: String, dbPassword: String)(implicit contextShift: ContextShift[Task]): Resource[Task, HikariTransactor[Task]] = {
HikariTransactor.newHikariTransactor[Task](dbDriver, dbUrl, dbUser, dbPassword, ecBlocking, Blocker.liftExecutionContext(ecBlocking))
}
}
I have converted fs2 stream to Monix observable according to Monix documentation: https://monix.io/docs/current/reactive/observable.html#fs2
Do I need to somehow close the fs2 stream or the Observable to get the application exit cleanly?
Appreciate any tips to get this working or tips how to properly debug this.

The problem was that ExecutionContext needs to be shutdown. See the authors' answer here.
Correct usage can been seen in the documentation.

Related

How should I test akka-streams RestartingSource usage

I'm working on an application that has a couple of long-running streams going, where it subscribes to data about a certain entity and processes that data. These streams should be up 24/7, so we needed to handle failures (network issues etc).
For that purpose, we've wrapped our sources in RestartingSource.
I'm now trying to verify this behaviour, and while it looks like it functions, I'm struggling to create a test where I push in some data, verify that it processes correctly, then send an error, and verify that it reconnects after that and continues processing.
I've boiled that down to this minimal case:
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{RestartSource, Sink, Source}
import akka.stream.testkit.TestPublisher
import org.scalatest.concurrent.Eventually
import org.scalatest.{FlatSpec, Matchers}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext
class MinimalSpec extends FlatSpec with Matchers with Eventually {
"restarting a failed source" should "be testable" in {
implicit val sys: ActorSystem = ActorSystem("akka-grpc-measurements-for-test")
implicit val mat: ActorMaterializer = ActorMaterializer()
implicit val ec: ExecutionContext = sys.dispatcher
val probe = TestPublisher.probe[Int]()
val restartingSource = RestartSource
.onFailuresWithBackoff(1 second, 1 minute, 0d) { () => Source.fromPublisher(probe) }
var last: Int = 0
val sink = Sink.foreach { l: Int => last = l }
restartingSource.runWith(sink)
probe.sendNext(1)
eventually {
last shouldBe 1
}
probe.sendNext(2)
eventually {
last shouldBe 2
}
probe.sendError(new RuntimeException("boom"))
probe.expectSubscription()
probe.sendNext(3)
eventually {
last shouldBe 3
}
}
}
This test consistently fails on the last eventually block with Last failure message: 2 was not equal to 3. What am I missing here?
Edit: akka version is 2.5.31
I figured it out after having had a look at the TestPublisher code. Its subscription is a lazy val. So when RestartSource detects the error, and executes the factory method () => Source.fromPublisher(probe) again, it gets a new Source, but the subscription of the probe is still pointing to the old Source. Changing the code to initialize both a new Source and TestPublisher works.

how to mock external WS API calls in Scala Play framework

I have an existing Scala play application which has a REST API that calls another external REST API. I want to mock the external Web service returning fake JSON data for internal tests. Based on example from: https://www.playframework.com/documentation/2.6.x/ScalaTestingWebServiceClients
I followed example exactly as in Documentation and I'm getting compiler errors due to deprecated class Action.
import play.core.server.Server
import play.api.routing.sird._
import play.api.mvc._
import play.api.libs.json._
import play.api.test._
import scala.concurrent.Await
import scala.concurrent.duration._
import org.specs2.mutable.Specification
import product.services.market.common.GitHubClient
class GitHubClientSpec extends Specification {
import scala.concurrent.ExecutionContext.Implicits.global
"GitHubClient" should {
"get all repositories" in {
Server.withRouter() {
case GET(p"/repositories") => Action {
Results.Ok(Json.arr(Json.obj("full_name" -> "octocat/Hello-World")))
}
} { implicit port =>
WsTestClient.withClient { client =>
val result = Await.result(
new GitHubClient(client, "").repositories(), 10.seconds)
result must_== Seq("octocat/Hello-World")
}
}
}
}
}
object Action in package mvc is deprecated: Inject an ActionBuilder
(e.g. DefaultActionBuilder) or extend
BaseController/AbstractController/InjectedController
And this is the primary example from latest official docs which in fact contains a compile time error, given this example doesn't work how should be the proper way to easily mock an external API using Scala Play?
You may change your example to:
Server.withRouterFromComponents() { cs => {
case GET(p"/repositories") => cs.defaultActionBuilder {
Results.Ok(Json.arr(Json.obj("full_name" -> "octocat/Hello-World")))
}
}
} { implicit port =>
WsTestClient.withClient { client =>
val result = Await.result(
new GitHubClient(client, "").repositories(), 10.seconds)
result should be(Seq("octocat/Hello-World"))
}
}
To be honest, I'm not 100% sure if this is the nicest way. However I have submitted a PR to the play framework so you might watch that space for comments from the makers.
If you're using standalone version of play-ws you can use this library https://github.com/f100ded/play-fake-ws-standalone like this
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import org.f100ded.play.fakews._
import org.scalatest._
import play.api.libs.ws.JsonBodyWritables._
import scala.concurrent.duration.Duration
import scala.concurrent._
import scala.language.reflectiveCalls
/**
* Tests MyApi HTTP client implementation
*/
class MyApiClientSpec extends AsyncFlatSpec with BeforeAndAfterAll with Matchers {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
import system.dispatcher
behavior of "MyApiClient"
it should "put access token to Authorization header" in {
val accessToken = "fake_access_token"
val ws = StandaloneFakeWSClient {
case request # GET(url"http://host/v1/foo/$id") =>
// this is here just to demonstrate how you can use URL extractor
id shouldBe "1"
// verify access token
request.headers should contain ("Authorization" -> Seq(s"Bearer $accessToken"))
Ok(FakeAnswers.foo)
}
val api = new MyApiClient(ws, baseUrl = "http://host/", accessToken = accessToken)
api.getFoo(1).map(_ => succeed)
}
// ... more tests
override def afterAll(): Unit = {
Await.result(system.terminate(), Duration.Inf)
}
}

Composing and outputting two future asynchronous actions on screen

I hate to ask - I really do but this one has got me for the moment..
I'm trying to compose some actions (in Play Framework & scala) with my main guide being this vid. However it was made a few years back so some of the functionality has since been deprecated and therefore I have had to find work-arounds as I go. Currently I am trying to output two asynchronous actions within some HTML markup.
I successfully outputted one action with this controller:
package controllers
import akka.actor.ActorSystem
import javax.inject._
import play.api.mvc._
import services.ServiceClient
import scala.concurrent.ExecutionContext
#Singleton
class AsyncController #Inject() (sc: ServiceClient)(actorSystem: ActorSystem)(implicit exec: ExecutionContext) extends Controller {
def index = Action.async { request =>
val asy1 = sc.makeServiceCall("async1")
for {
async1Message <- asy1
} yield {
Ok(views.html.async1.async1(async1Message))
}
}
}
In case you are wondering the sc.makeServiceCall refers to this file:
class ServiceClient #Inject() (ws: WSClient) {
def makeServiceCall(serviceName: String): Future[String] = {
ws.url(s"http://localhost:9000/mock/$serviceName").get().map(_.body)
}
}
So I followed the video in its' guidance to compose two asynchronous actions with some HTML. And this is where it gets difficult/interesting/upsetting:
package controllers
import javax.inject.Inject
import akka.actor.ActorSystem
import play.api.mvc._
import scala.concurrent.{ExecutionContext}
import Ui.Pagelet
class AsyncHomeController #Inject() (as1: AsyncController)(as2: Async2Controller)(actorSystem: ActorSystem)(implicit exec: ExecutionContext) extends Controller {
def index = Action.async { request =>
val asy1 = as1.index(request)
val asy2 = as2.index(request)
for {
async1Result <- asy1
async2Result <- asy2
async1Body <- Pagelet.readBody(async1Result)
async2Body <- Pagelet.readBody(async2Result)
} yield {
Ok(views.html.home2(async1Body, async2Body))
}
}
}
So Async2Controller is very similar to AsyncController and Pagelet.readBody refers to this:
package Ui
import play.api.libs.iteratee.Iteratee
import play.api.mvc.{Codec, Result}
import play.twirl.api.Html
import scala.concurrent._
object Pagelet {
def readBody(result: Result)(implicit codec: Codec): Future[Html] = {
result.body.run(Iteratee.consume()).map(bytes => Html(new String(bytes, codec.charset)))
}
}
And this is wherein the error lies - which is:
value run is not a member of play.api.http.HttpEntity
I cannot find documentation on whether it needs to be injected or any indication that it has since been deprecated. If someone has got an answer to this or a work-around please divulge. Many thanks
The Iteratee lib is deprecated and was replaced by akka-stream. You need to change the implementation of readBody:
def readBody(result: Result)(implicit mat: Materializer, ec: ExecutionContext, codec: Codec): Future[Html] = {
result.body.consumeData.map(byteString => Html(codec.decode(byteString))
}
You also need to change the dependencies of the controller to get a Materializer:
class AsyncHomeController #Inject() (as1: AsyncController, as2: Async2Controller)(actorSystem: ActorSystem)(implicit exec: ExecutionContext, mat: Materializer)
Edit: code updated

How to unit test Akka Agents?

I'm trying to write a scala class which periodically retrieves data from an external location. I don't care when the data is retrieved, just that it happens every hour. Akka agents are a perfect solution for this but I have no idea how to test them. Here's an example to illustrate my problem:
import akka.actor.ActorSystem
import akka.agent.Agent
import scala.concurrent.duration._
abstract class Fetcher(implicit actorSystem: ActorSystem) {
import actorSystem.dispatcher
private val data: Agent[String] = Agent("")
def current: String = data.get()
protected def retrieveData(): String
actorSystem.scheduler.schedule(0.seconds, 1.hour) {
data send retrieveData()
}
}
import akka.actor.ActorSystem
import bbc.scala.whitelist.dynamic.Fetcher
import org.specs2._
class FetcherSpec extends mutable.Specification {
implicit val actorSystem = ActorSystem("TestActorSystem")
class TestFetcher extends Fetcher {
override protected def retrieveData(): String = "data!"
}
"The Fetcher" should {
"contain the empty string initially" in {
val fetcher = new TestFetcher
fetcher.current must be equalTo ""
}
"contain data after the first scheduled update" in {
val fetcher = new TestFetcher
Thread.sleep(100)
fetcher.current must be equalTo "data!"
}
}
}
That second unit test currently passes but I do not like the Thread.sleep at all! How can I force the scheduled task to happen before making my assertion? Any help would be much appreciated. Thank you very much.

Implicits getting out of scope in Spray example code: what's happening here?

I have copied Spray Client's example code into my own project, to have it easily available. I use IntelliJ 13.
Here is the code I have:
package mypackage
import scala.util.Success
import scala.concurrent.duration._
import akka.actor.ActorSystem
import akka.pattern.ask
import akka.event.Logging
import akka.io.IO
import spray.json.{JsonFormat, DefaultJsonProtocol}
import spray.can.Http
import spray.util._
import spray.client.pipelining._
import scala.util.Success
import scala.util.Failure
case class Elevation(location: Location, elevation: Double)
case class Location(lat: Double, lng: Double)
case class GoogleApiResult[T](status: String, results: List[T])
object ElevationJsonProtocol extends DefaultJsonProtocol {
implicit val locationFormat = jsonFormat2(Location)
implicit val elevationFormat = jsonFormat2(Elevation)
implicit def googleApiResultFormat[T :JsonFormat] = jsonFormat2(GoogleApiResult.apply[T])
}
object SprayExample extends App {
// we need an ActorSystem to host our application in
implicit val system = ActorSystem("simple-spray-client")
import system.dispatcher // execution context for futures below
val log = Logging(system, getClass)
log.info("Requesting the elevation of Mt. Everest from Googles Elevation API...")
val pipeline = sendReceive ~> unmarshal[GoogleApiResult[Elevation]]
val responseFuture = pipeline {
Get("http://maps.googleapis.com/maps/api/elevation/json?locations=27.988056,86.925278&sensor=false")
}
responseFuture onComplete {
case Success(GoogleApiResult(_, Elevation(_, elevation) :: _)) =>
log.info("The elevation of Mt. Everest is: {} m", elevation)
shutdown()
case Success(somethingUnexpected) =>
log.warning("The Google API call was successful but returned something unexpected: '{}'.", somethingUnexpected)
shutdown()
case Failure(error) =>
log.error(error, "Couldn't get elevation")
shutdown()
}
def shutdown(): Unit = {
IO(Http).ask(Http.CloseAll)(1.second).await
system.shutdown()
}
}
As it stands, this works perfectly and it prints the height of Mt.Everest as expected.
The strange thing happens if I move the file down one level in the package structure, that is I create a mypackage.myinnerpackage and move the file inside it.
IDEA changes my first line of code into package mypackage.myinnerpackage and that's it.
Then I try to run the app and the compilation will fail with the following message:
could not find implicit value for evidence parameter of type spray.httpx.unmarshalling.FromResponseUnmarshaller[courserahelper.sprayexamples.GoogleApiResult[courserahelper.sprayexamples.Elevation]]
val pipeline = sendReceive ~> unmarshal[GoogleApiResult[Elevation]]
^
I did not change anything in the code, I effectively just changed the package! Additionally, this code is self contained, it does not rely on any other implicit I declared in any other part of my code....
What am I missing?
Thanks!
(Replaced the comment by this answer which supports proper formatting.)
The code you posted is missing these two imports before the usage of unmarshal:
import ElevationJsonProtocol._
import SprayJsonSupport._
val pipeline = sendReceive ~> unmarshal[GoogleApiResult[Elevation]]
which exist in the original code. IntelliJ is sometimes messing with imports so that may be the reason they got lost in the transition.
You need to provide a Json Formatter for your case class.
case class Foo(whatever: Option[String])
object FooProtocol extends DefaultJsonProtocol {
implicit val fooJsonFormat = jsonFormat1(Foo)
}
Then include the following near the implementation...
import SprayJsonSupport._
import co.xxx.FooProtocol._