I'm trying to write a scala class which periodically retrieves data from an external location. I don't care when the data is retrieved, just that it happens every hour. Akka agents are a perfect solution for this but I have no idea how to test them. Here's an example to illustrate my problem:
import akka.actor.ActorSystem
import akka.agent.Agent
import scala.concurrent.duration._
abstract class Fetcher(implicit actorSystem: ActorSystem) {
import actorSystem.dispatcher
private val data: Agent[String] = Agent("")
def current: String = data.get()
protected def retrieveData(): String
actorSystem.scheduler.schedule(0.seconds, 1.hour) {
data send retrieveData()
}
}
import akka.actor.ActorSystem
import bbc.scala.whitelist.dynamic.Fetcher
import org.specs2._
class FetcherSpec extends mutable.Specification {
implicit val actorSystem = ActorSystem("TestActorSystem")
class TestFetcher extends Fetcher {
override protected def retrieveData(): String = "data!"
}
"The Fetcher" should {
"contain the empty string initially" in {
val fetcher = new TestFetcher
fetcher.current must be equalTo ""
}
"contain data after the first scheduled update" in {
val fetcher = new TestFetcher
Thread.sleep(100)
fetcher.current must be equalTo "data!"
}
}
}
That second unit test currently passes but I do not like the Thread.sleep at all! How can I force the scheduled task to happen before making my assertion? Any help would be much appreciated. Thank you very much.
Related
I am trying to have a source which poll http endpoint every 1 hour and keep that as flink source to broadcast to operators.
I tried to make it as simple function but seems not working as expected.
Code is :
import org.apache.flink.streaming.api.functions.source.SourceFunction
import org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext
import org.apache.http.{HttpRequest, HttpResponse}
import org.apache.http.entity.StringEntity
import org.apache.http.impl.bootstrap.{HttpServer, ServerBootstrap}
import org.apache.http.protocol.{HttpContext, HttpRequestHandler}
import java.util.concurrent.TimeUnit
class HttpStreamFun(url: String) extends SourceFunction[String] {
#transient private var server: HttpServer = _
override def run(ctx: SourceContext[String]): Unit = {
server = ServerBootstrap
.bootstrap()
.registerHandler(
url,
new HttpRequestHandler() {
override def handle(req: HttpRequest,
rep: HttpResponse,
context: HttpContext): Unit = {
ctx.collect(req.getRequestLine.getUri)
rep.setStatusCode(200)
rep.setEntity(new StringEntity("OK"))
}
}
)
.create()
server.start()
server.awaitTermination(1, TimeUnit.HOURS)
}
override def cancel(): Unit = {
server.stop()
}
}
Main job has these to add the source as datastream:
val text: DataStream[String] = env.addSource(new HttpStreamFun(config.baseUri))
text.print()
Maybe you can try to use BroadcastStream
For more information, refer to
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/
I am trying to create an (Akka HTTP) stream procsesing flow using the classes akka.stream.scaladsl.Source and Sink queues.
I am using a queue because I have a processing step in my flow that issues http requests and I want this step to take as many
items off the queue as there are max-open-requests, and stop taking off the queue once max-open-requests are in flight.
The result is that backpressure is applied when my connection pool is overloaded.
Below, I have a very simplified test that reflects the main logic of my app. In the test 'Stress Spec' (below)
I am simulating a number of simultaneous connections via which I will send a 'Source' of 'Requesto' objects
to the getResponses method of the class ServiceImpl.
In the processing step 'pullOffSinkQueue' you will note that I am incrementing a counter to see how many items
I have pulled off the queue.
The test will send Serviceimpl a set of requests whose cardinality is set to equal
streamedRequestsPerConnection * numSimultaneousConnections.
When I send 20 requests my test passes fine. In particular the count of requests pulled off the
Sink.queue will be equal to the number of requests I send out. However, if
I increase the number of requests I send to above 50 or so, I see consistent failures in the test.
I get a message such as the one below
180 was not equal to 200
ScalaTestFailureLocation: com.foo.StressSpec at (StressSpec.scala:116)
Expected :200
Actual :180
<Click to see difference>
This indicates that the number of items pulled off the queue does not equal the number of items put on the queue.
I have a feeling this might be due to the fact that my test is not properly waiting for all items put into the stream
to be processed. If anyone has any suggestions, I'd be all ears ! Code is below.
package com.foo
import java.util.concurrent.atomic.AtomicInteger
import akka.stream.ActorAttributes.supervisionStrategy
import akka.stream.{Attributes, Materializer, QueueOfferResult}
import akka.stream.Supervision.resumingDecider
import akka.stream.scaladsl.{Flow, Keep, Sink, Source}
import scala.concurrent.{ExecutionContext, Future}
import akka.NotUsed
import akka.actor.ActorSystem
import akka.event.{Logging, LoggingAdapter}
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{Sink, Source}
import org.scalatest.mockito.MockitoSugar
import org.scalatest.{FunSuite, Matchers}
import scala.collection.immutable
import scala.concurrent.duration._
import scala.concurrent.{Await, Future, _}
final case class Responso()
final case class Requesto()
object Handler {
val dbRequestCounter = new AtomicInteger(0)
}
class Handler(implicit ec: ExecutionContext, mat: Materializer) {
import Handler._
private val source =
Source.queue[(Requesto, String)](8, akka.stream.OverflowStrategy.backpressure)
private val sink =
Sink.queue[(Requesto, String)]().withAttributes(Attributes.inputBuffer(8, 8))
private val (sourceQueue, sinkQueue) = source.toMat(sink)(Keep.both).run()
def placeOnSourceQueue(ar: Requesto): Future[QueueOfferResult] = {
sourceQueue.offer((ar, "foo"))
}
def pullOffSinkQueue(qofr: QueueOfferResult): Future[Responso] = {
dbRequestCounter.incrementAndGet()
qofr match {
case QueueOfferResult.Enqueued =>
sinkQueue.pull().flatMap { maybeRequestPair: Option[(Requesto, String)] =>
Future.successful(Responso())
}
case error =>
println("enqueuing error: " + error)
Future.failed(new RuntimeException("enqueuing error: " + error))
}
}
}
class ServiceImpl(readHandler: Handler, writeHandler: Handler)
(implicit log: LoggingAdapter, mat: Materializer) {
private val readAttributeFlow: Flow[Requesto, Responso, NotUsed] = {
Flow[Requesto]
.mapAsyncUnordered(1)(readHandler.placeOnSourceQueue)
.mapAsyncUnordered(1)(readHandler.pullOffSinkQueue)
}
def getResponses(request: Source[Requesto, NotUsed]): Source[Responso, NotUsed] =
request
.via(readAttributeFlow)
.withAttributes(supervisionStrategy(resumingDecider))
}
class StressSpec
extends FunSuite
with MockitoSugar
with Matchers {
val streamedRequestsPerConnection = 10
val numSimultaneousConnections = 20
implicit val actorSystem: ActorSystem = ActorSystem()
implicit val materializer: ActorMaterializer = ActorMaterializer()
implicit val log: LoggingAdapter = Logging(actorSystem.eventStream, "test")
implicit val ec: ExecutionContext = actorSystem.dispatcher
import Handler._
lazy val requestHandler = new Handler()
lazy val svc: ServiceImpl =
new ServiceImpl(requestHandler, requestHandler)
test("can handle lots of simultaneous read requests") {
val totalExpected = streamedRequestsPerConnection * numSimultaneousConnections
def sendRequestAndAwaitResponse(): Unit = {
def getResponses(i: Integer) = {
val requestStream: Source[Requesto, NotUsed] =
Source(1 to streamedRequestsPerConnection)
.map { i =>
Requesto()
}
svc.getResponses(requestStream).runWith(Sink.seq)
}
val responses: immutable.Seq[Future[immutable.Seq[Responso]]] =
(1 to numSimultaneousConnections).map { getResponses(_) }
val flattenedResponses: Future[immutable.Seq[Responso]] =
Future.sequence(responses).map(_.flatten)
Await.ready(flattenedResponses, 1000.seconds).value.get
}
sendRequestAndAwaitResponse()
dbRequestCounter.get shouldBe(totalExpected)
}
}
Using Scala play version 2.5 and trying to follow the guidelines for unit testing controllers as per documentation at: https://www.playframework.com/documentation/2.5.x/ScalaTestingWithScalaTest
there's no example for unit testing an async controller.
I'm trying to create a unit test for my controller which has an async action method, I ended up mocking some objects
class ProductController #Inject()(
action: ProductAction,
handler: ProductResourceHandler)(implicit ec: ExecutionContext)
extends Controller {
/**
* Fetch a list of products
*/
def index: Action[AnyContent] = {
action.async { implicit request =>
handler.find.map { list =>
Ok(Json.toJson(list))
}
}
}
// ...
}
My unit test:
import scala.concurrent.Future
import org.scalatestplus.play._
import play.api.mvc._
import play.api.test._
import play.api.test.Helpers._
import org.scalatest.mockito.MockitoSugar
import product.ProductAction
import product.ProductController
import product.services.maps.GeolocationService
import product.ProductResourceHandler
import play.api.libs.concurrent.Execution.Implicits._
import scala.io.Source
import play.api.libs.json.Json
import product.model.OfferList
import product.model.OfferDetail
import org.mockito.Mockito._
class ProductControllerSpec extends PlaySpec with Results with MockitoSugar {
private val productList = Json.parse(Source.fromFile("conf/app/sample_products.json").getLines.mkString).as[ProductList]
"Example Page#index" should {
"should be valid" in {
val action = new ProductAction()
val handler = mock[ProductResourceHandler]
when(handler.find) thenReturn Future.successful(productList)
val controller = new ProductController(action, handler)
val result: Future[Result] = controller.index().apply(FakeRequest())
val bodyText: String = contentAsString(result)
bodyText != null mustBe true
}
}
}
up till now it's working but I'm wondering if this follows the best practices or guidelines for this type of test. Is this the right way to unit test an async controller in Scala play framework?
Some of my suggestions are
use contentAsJson instead of contentAsString and inspect the returned json.
use route to directly invoke the controller and test response (for eg route(app, FakeRequest..)
use status method to check if the returned status is HTTP OK (status code 200)
val Some(result) = route(app, FakeRequest(GET,
controllers.routes. ProductController.index.path()))
status(result) must be (OK)
val json = contentAsJson(result)
// inspect json fields like if you have to check if the json
// has string field called id you can do (json \ "id").as[String] must be ("<id value>")
According to Play documentation:
Play actions are asynchronous by default.
This means even if you are not using Action.async { Future { myAnonymousFunction } } but just Action { myAnonymousFunction }, internally the result of myAnonymousFunction will be enclosed in a Future.
For instance, say you have
class HelloWorld extends Controller {
def index = Action { request => Ok("") }
}
then
(new HelloWorld).index().apply(FakeRequest())
still has type
Future[Result]
This leads me to believe that your unit test is indeed appropriate way of testing controllers, that is, Play's documentation is implicitly covering also the case of Action.async.
In "Dependency Injecting Actors" it's shown how to inject a parameter into the constructor of a child actor. The parent actor uses injectedChild to be allowed to pass to the child (at child creation time) only the non-injected parameter and then let Guice inject the rest. To do this, it extends InjectedActorSupport and gets the child's factory injected in the constructor:
class MyParent #Inject() (childFactory: MyChild.Factory,
#Assisted something: Something,
#Assisted somethingElse: SomethingElse) extends Actor with InjectedActorSupport
[..]
val child: ActorRef = injectedChild(childFactory(something, somethingElse), childName)
But what about the class that starts the parent and is not an actor but a custom ApplicationLoader?
How can I start the parent actor from there? No mention of this is in the documentation.
I tried doing the same for the loader as I did for parent:
class MyLoader #Inject() (parentFactory: MyParent.Factory) extends ApplicationLoader with Actor with InjectedActorSupport {
[..]
val parent = injectedChild(parentFactory(something, somethingElse), parentName)
would this be correct? How can I test it?
class MyModule extends AbstractModule with AkkaGuiceSupport {
def configure = {
bindActor[MyParent](parentName)
bindActor[MyLoader](loaderName)
bindActorFactory[MyChild, MyChild.Factory]
bindActorFactory[MyParent, MyParent.Factory]
}
}
So:
How do I start the parent from MyLoader while letting Guice dependency-inject what's required?
How can I test MyLoader?
This has been my test so far but now I need to pass the injected thingy to MyLoader and I don't know how (note the ***???**** in place of the argument which I do not know where to find):
class MyLoaderSpec(_system: ActorSystem, implicit val ec: ExecutionContext) extends TestKit(_system) with WordSpecLike with BeforeAndAfterAll with Matchers {
val loader = new SimstimLoader(???)
override def beforeAll(): Unit = {
loader.load(ApplicationLoader.createContext(new Environment(new File("."), ApplicationLoader.getClass.getClassLoader, Mode.Test)))
}
Thanks a million in advance!
Here is how I solved this issue.
--> How to start a parent actor who needs dependency-injection.
First of all, manually starting such an actor is impossible if you, like me, need to dependency-inject an instance which you do not know how to pass and where from. The solution is to let Guice start the actor automagically. Here is how.
First, create your binder module for Guice:
class MyModule extends AbstractModule with AkkaGuiceSupport{
override def configure(): Unit = {
bindActor[Root](Root.NAME)
bind(classOf[StartupActors]).asEagerSingleton()
}
}
Then, tell Play where your binder module is located by adding the following in your conf/application.conf:
play.modules={
enabled += "my.path.to.MyModule"
}
The StartupActors is simply a class I use to log whenever the automagic start of dependency-injected actors actually takes place. I log the event so that I can be sure of when and whether it occurs:
class StartupActors #Inject() (#Named(Root.NAME) root: ActorRef) {
play.api.Logger.info(s"Initialised $root")
}
The Root actor in my case takes care of parsing a custom configuration. Since the resulting vars from the parsing is required by my parent actor and during the tests I need to mock such resulting vars, I delegate the parsing to an actor other than the parent actor, i.e., the Root actor:
object Root {
final val NAME = "THERoot"
case class ParseConfiguration()
}
class Root #Inject()(configuration: Configuration, projectDAO: ProjectDAO) extends Actor {
val resultingVar: Something = myConfigParsing()
override def preStart(): Unit = {
context.actorOf(Props(new MyParent(resultingVar: Something, somethingElse: SomethingElse, projectDAO: ProjectDAO)))
}
override def receive: Receive = {
case ParseConfiguration => sender ! myConfigParsing()
case _ => logger.error("Root actor received an unsupported message")
}
}
The ParseConfiguration message is used uniquely for testing purposes. Normally the configuration parsing occurs instead because of the initialisation of the resultingVar attribute.
This way, MyParent wont need to get anything injected. Only StartupActors and Root will get injected. MyParent will simply get projectDAO from Root and pass it on to all its children.
class MyParent(something: Something, somethingElse: SomethingElse, projectDAO: ProjectDAO) extends Actor { ... }
Finally, for completion, I'm reporting here how I wrote the tests since I had troubles finding enough information online around this as well.
import akka.actor.{ActorRef, ActorSystem, Props}
import akka.testkit.{TestKit, TestProbe}
import com.typesafe.config.ConfigFactory
import org.mockito.Mockito.mock
import org.scalatest.{BeforeAndAfterAll, WordSpecLike}
import org.specs2.matcher.MustMatchers
import play.api.Configuration
import scala.concurrent.ExecutionContext
class RootSpec(_system: ActorSystem) extends TestKit(_system)
with WordSpecLike with BeforeAndAfterAll with MustMatchers {
implicit val ec: ExecutionContext = scala.concurrent.ExecutionContext.global
val conf: com.typesafe.config.Config = ConfigFactory.load()
val configuration: Configuration = Configuration(conf)
val projectDAOMock: ProjectDAO = mock(classOf[ProjectDAO])
private var mainActor: ActorRef = _
private var something: Something = Something.empty
def this() = this(ActorSystem("MySpec"))
override def afterAll: Unit = {
system.shutdown()
}
override def beforeAll(): Unit = {
mainActor = system.actorOf(Props(new Root(configuration, projectDAOMock)), Root.NAME)
}
"RootSpec: Root Actor" should {
val probe = TestProbe()
"successfully parse the configuration file" in {
probe.send(mainActor, ParseConfiguration)
something = probe.expectMsgPF() {
case msg => msg.asInstanceOf[Something]
}
}
}
}
and then I test MyParent by conveniently providing mock objects in place of vars resulting from the configuration parsing:
import akka.actor.{ActorRef, ActorSystem, Props}
import akka.testkit.{TestKit, TestProbe}
import org.mockito.Mockito
import org.mockito.Mockito._
import org.scalatest.{BeforeAndAfterAll, WordSpecLike}
import org.specs2.matcher.MustMatchers
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.{ExecutionContext, Future}
case class AnyProjectAPI(val projectAPI: ProjectAPI) extends AnyVal
class MyParentSpec(_system: ActorSystem, implicit val ec: ExecutionContext) extends TestKit(_system)
with WordSpecLike with BeforeAndAfterAll with MustMatchers {
val something = mock(classOf[Something])
val somethingElse = mock(classOf[somethingElse])
val projectDAOMock: ProjectDAO = mock(classOf[ProjectDAO])
val projectTest: ProjectAPI = new ProjectAPI(allMyRandomConstructorArguments),
val projectsList: List[ProjectAPI] = List(projectTest)
val expectedCreationId = 1
private var parent: ActorRef = _
def this() = this(ActorSystem("MySpec"), scala.concurrent.ExecutionContext.global)
override def afterAll: Unit = {
system.shutdown()
}
override def beforeAll(): Unit = {
parent = system.actorOf(Props(new MyParent(something, somethingElse, projectDAOMock)), MyParent.NAME)
}
"MyParentTesting: parent's pull request" should {
when(myProjApi.getAllProjects).thenReturn(Future {projectsList})
val anyProject: AnyProjectAPI = AnyProjectAPI(org.mockito.Matchers.any[ProjectAPI])
Mockito.when(projectDAOMock.create(org.mockito.Matchers.any[ProjectAPI]))
.thenReturn(Future {expectedCreationId}: Future[Int])
val probe = TestProbe()
val probe1 = TestProbe()
"be successfully satisfied by all children when multiple senders are waiting for an answer" in {
probe.send(parent, UpdateProjects)
probe1.send(parent, UpdateProjects)
allChildren.foreach(child =>
probe.expectMsg(expectedCreationId))
allChildren.foreach(child =>
probe1.expectMsg(expectedCreationId))
}
}
}
Here is the code:
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.Sink
import com.google.inject.Inject
import org.bytedeco.javacv.{CanvasFrame, Frame}
class WebCamWindow #Inject()(webCam: WebCam) {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
val canvas = new CanvasFrame("Webcam")
// Set Canvas frame to close on exit
canvas.setDefaultCloseOperation(javax.swing.JFrame.EXIT_ON_CLOSE)
val imageDimensions = Dimensions(width = 640, height = 480)
val webcamSource = webCam.source
val graph = webcamSource
.map(MediaConversion.toMat) // most OpenCV manipulations require a Matrix
.map(Flip.horizontal)
.map(MediaConversion.toFrame) // convert back to a frame
.map(canvas.showImage)
.to(Sink.ignore)
def run(): Unit = {
graph.run()
}
}
I would like to make this an App object and run it, but I don't know how to deal with the dependency injection here. Could anyone help?
Full project can be found here: https://bitbucket.org/kindlychung/testscalacv
First step is to make it into a trait (just for convenience):
trait WebCamWindowLike {
def webCam: WebCam
implicit val system ...//all the code is here
}
Then you can have old injectable WebCamWindow:
class WebCamWindow #Inject()(val webCam: WebCam) extends WebCamWindowLike
As well as independent runnable object:
object WebCamWindowApp extends App with WebCamWindowLike {
private val injector = Guice.createInjector(new AppInjectory())
override val webCam = injector.getInstance(classOf[WebCam])
run()
}
Where AppInjectory extends AbstractModule is your actual injectory for Guice that takes care of all of your dependencies.
Another option is (if you want to get rid of Guice) manual "injection":
object WebCamWindowApp extends App with WebCamWindowLike {
override val webCam = new WebCam(...)
run()
}