I have been trying to Log things within my scalaTest as such:
class ChangeSetActorTest extends PersistenceSpec(ActorSystem("Persistent-test-System")) with PersistenceCleanup {
val log = Logging(system, this)
Basically let's just say that ChangesetActorTest inherit from TestKit(system)
Unfortunately Logging(system, this) does not work with the this.
I get the following error:
[error]
/Users/maatary/Dev/IdeaProjects/PoolpartyConnector/src/test/scala/org/iadb/poolpartyconnector/changepropagation/ChangeSetActorTest.scala:22:
Cannot find LogSource for
org.iadb.poolpartyconnector.changepropagation.ChangeSetActorTest
please see ScalaDoc for LogSource for how to obtain or construct one.
[error] val log = Logging(system, this)
I believe in the Akka Logging Doc this is the following point:
and in all other cases a compile error occurs unless and implicit LogSource[T] is in scope for the type in question.
In other words there is no LogSource[TestKit]
I would like the simplest solution to deal with that issue, with minimal additional configuration. So far what i did is the following and everything works as expected:
class ChangeSetActorTest extends PersistenceSpec(ActorSystem("Persistent-test-System")) with PersistenceCleanup {
val log = system.log
From there I just go and do things like
val received = chgtFetcher.receiveWhile((requestInterval + ProcessingLag).*(3)) {
case msg:FetchNewChangeSet => log.info(s"received: ${msg}" ) ; chgtFetcher.reply(NoAvailableChangeSet); msg
}
My question, is this recommended approach. So far the order of the message coming from my actor and the one from the test are well ordered.
What is the recommended approach to log in a unified way:
From the Test class (e.g. above) and the Actor at the same time ?
If one uses a system where external class needs to log as well and we need one unified logging (asynchronous) going on.
Have a look at this comment:
https://github.com/akka/akka/blob/master/akka-actor/src/main/scala/akka/event/Logging.scala#L196-L237
I believe a more straight forward approach would be to define your implicit LogSource[ChangeSetActorTest] locally.
I.E:
val log = {
implicit val logSource = new LogSource[ChangeSetActorTest] {
override def genString(t: ChangeSetActorTest) = "ChangeSetActorTest"
}
Logging(system, this)
}
Simplest way to log in a TestKit is either:
Get the logger from underlyingActor:
val mockActor = TestActorRef(new XXXActor)
val log = mockActor.underlyingActor.log
Use FeatureSpecLike
http://doc.scalatest.org/3.0.1-2.12/org/scalatest/FeatureSpecLike.html
class ChangeSetActorTest extends PersistenceSpec(ActorSystem("Persistent-test-System")) with PersistenceCleanup with FeatureSpecLike {
//...
alert("Something like warning")
info("Infos")
note("Green infos")
markup("documents")
}
Related
I have a Route defined using akka-http that uses an actor inside to send messages.
My route looks like this:
path("entity") {
post {
entity(as[Enrtity]) {
entity =>
val result: Future[Message] = mainActor.ask {
ref: akka.actor.typed.ActorRef[Message] =>
Message(
entity = entity,
replyRef = ref
)
}
complete("OK")
}
}
}
My test spec:
class APITest
extends ScalaTestWithActorTestKit(ManualTime.config)
with ScalatestRouteTest
with AnyWordSpecLike {
val manualTime: ManualTime = ManualTime()
// my tests here ...
}
Compiling the test fails since there are conflicting actor systems:
class APITest inherits conflicting members:
[error] implicit def system: akka.actor.typed.ActorSystem[Nothing] (defined in class ActorTestKitBase) and
[error] implicit val system: akka.actor.ActorSystem (defined in trait RouteTest)
Overriding the actor system doesn't help either since the inherited actor systems are of both typed and untyped ones.
How can I resolve this easily?
Update:
This is related to conflicting inherited members with different types, but we might be able to solve what I want to achieve in this context differently.
I spent a little time here while moving over to typed. For anyone still looking, there's a nice hint at https://developer.lightbend.com/guides/akka-http-quickstart-scala/testing-routes.html
// the Akka HTTP route testkit does not yet support a typed actor system (https://github.com/akka/akka-http/issues/2036)
// so we have to adapt for now
lazy val testKit = ActorTestKit()
implicit def typedSystem = testKit.system
override def createActorSystem(): akka.actor.ActorSystem =
testKit.system.classicSystem
Looking at the first comment at https://github.com/akka/akka-http/issues/2036 it notes
Perhaps just a docs addition to show that you don't need to use the ActorTestKit from Akka Typed and can just use TestProbes e.g. https://gist.github.com/chbatey/964b80adc2cd124fa4bf4624927b5be0
or val probe = TestProbe[Ping]() > val probe = testKit.createTestProbe[Ping]()
I have a Scala App that has a trait that implements some function(s) and a class which extends that trait.
The class mentioned above also has a function which calls the function that is defined in the parent trait using it's parameter.
I observed this in Spark + Kafka implementation using Scala. I'm guessing this is some kind of design pattern but I don't know which one. Is it Cake Pattern? Dependency Injection Pattern? Or something else?
Below is the code I'm referring to:
trait SparkApplication {
def sparkConfig: Map[String, String]
def withSparkContext(f: SparkContext => Unit): Unit = {
val conf = new SparkConf()
sparkConfig.foreach { case (k, v) => conf.setIfMissing(k, v) }
val sc = new SparkContext(conf)
f(sc)
}
}
trait SparkStreamingApplication extends SparkApplication {
def withSparkStreamingContext(f: (SparkContext, StreamingContext) => Unit): Unit = {
withSparkContext { sc =>
val ssc = new StreamingContext(sc, Seconds(streamingBatchDuration.toSeconds))
ssc.checkpoint(streamingCheckpointDir)
f(sc, ssc)
ssc.start()
ssc.awaitTermination()
}
}
}
What is being used here (albeit, with a possible error) is the so-called Loan Pattern, called in such way because it's useful when you want to manage the lifecycle of a resource (in your case a SparkContext), while allowing the user to define how the resource is going to be used.
A classical example of this is files: you want to open a file, read it's content and then close it as soon as you are done, without allowing the user to make some mistake and forget to close the resource. You may implement this as follows:
import scala.io.Source
// Read a file at `path` and allow to pass a function that iterates over lines
def consume[A](path: String)(f: Iterator[String] => A): A = {
val source = Source.fromFile(path)
try {
f(source.getLines)
} finally {
source.close()
}
}
Then you'd use this as follows (in the example, to just print all the lines paired with their numbers):
consume("/path/to/some/file")(_.zipWithIndex.foreach(println))
As you may notice, there is something very close to this going on in your code, with the only difference that the resource whose lifecycle you are managing is a SparkContext.
Regarding the possible error I mentioned initially, it regards the fact that you are loaning a SparkContext that you never close. That is probably ok, but the main aspect of the Loan Pattern is precisely that of minimizing the error surface when it comes to managing resources. You may be interested in doing something like the following (you want to check the last line in the method):
def withSparkContext(f: SparkContext => Unit): Unit = {
val conf = new SparkConf()
sparkConfig.foreach { case (k, v) => conf.setIfMissing(k, v) }
val sc = new SparkContext(conf)
f(sc)
sc.stop() // shutdown the context after the user is done
}
You may read more regarding this pattern here.
As a side note, you may be interested in this project that creates a very nice and idiomatic interface around managed resources.
for creating datasource I have
object MyDataSource {
priavte lazy val dataSource: javax.sql.DataSource = {
val ds = new BasicDataSource
val conf = ConfigFactory.load()
val url = conf.getString("jdbc-url")
val driver = conf.getString("jdbc-driver")
val username = conf.getString("db-username")
val password = conf.getString("db-password")
val port = conf.getString("db-port")
val maxActive = conf.getInt("max-active")
val maxIdle = conf.getInt("max-idle")
val initSize = conf.getInt("init-size")
ds.setDriverClassName(driver)
ds.setUsername(username)
ds.setPassword(password)
ds.setMaxActive(maxActive)
ds.setMaxIdle(maxIdle)
ds.setInitialSize(initSize)
ds.setUrl(url)
ds
}
lazy val database = Database.forDataSource(dataSource)
}
MyDataSource is used as below
def insertCompany = {
MyDataSource.database.withSession{ implicit session =>
company.insert(companyRow)
}
}
Now for testing I have trait DatabaseSpec which loads test database(pointing to test db) config and has following fixture
def withSession(testCode: Session => Any) {
val session = postgres.createSession()
session.conn.setAutoCommit(false)
try {
testCode(session)
} finally {
session.rollback()
session.close()
}
}
And test code can then mix in DatabaseSpec and use withSession to test transactional code.
Now question is what's the best practice in keeping MyDataSource.database.withSession abstracted away from DataSource in insertCompany so that method can be tested with DatabaseSpec and pointing to test db?
The best way to be able to exchange a value, .e.g for prod and testing is by parameterizing your code in that value. E.g.
def insertCompany(db: Database) = db.withSession(company.insert(companyRow)(_))
or
class DAO(db:Database){
def insertCompany = db.withSession(company.insert(companyRow)(_))
}
Keep it simple. Avoid unnecessary complexity like the Cake pattern, DI frameworks or mixin composition for this.
If you need to pass multiple values around... aggregate them into a "config"-class. Compose multiple config classes with different purposes to target different things, if you want to avoid writing one huge config class as stuff accumulates.
If you find yourself passing config objects to all your functions, you can mark them as implicit, that saves you at least the call-site code overhead. Or you can use something like scalaz's monadic function composition to avoid call site and definition site code overhead for passing config around. It is sometimes called the Reader monad, but it is simply for-comprehension enabled composition of 1-argument functions.
Slick 2.2 will ship with something like that out-of-the-box and make what you want very easy.
Also, here is an approach I am currently playing around with, a composable configuration object "TMap". This code example shows step by step how you get from global imports over parameterized functions and making them implicit to using TMap and removing most boilerplate: https://github.com/cvogt/slick-action/blob/0.1/src/test/scala/org/cvogt/di/TMapTest.scala#L49
I've had a situation come up and bite me a few times where I'm testing an Actor and the Actor throws an exception unexpectedly (due to a bug), but the test still passes. Now most of the time the exception in the Actor means that whatever the test is verifying won't come out properly so it the test fails, but in rare cases that's not true. The exception occurs in a different thread than the test runner so the test runner knows nothing about it.
One example is when I'm using a mock to verify some dependency gets called, and due to a mistake in the Actor code I call an unexpected method in the mock. That causes the mock to throw an exception which blows up the actor but not the test. Sometimes this can even cause downstream tests to fail mysteriously because of how the Actor blew up. For example:
// using scala 2.10, akka 2.1.1, scalatest 1.9.1, easymock 3.1
// (FunSpec and TestKit)
class SomeAPI {
def foo(x: String) = println(x)
def bar(y: String) = println(y)
}
class SomeActor(someApi: SomeAPI) extends Actor {
def receive = {
case x:String =>
someApi.foo(x)
someApi.bar(x)
}
}
describe("problem example") {
it("calls foo only when it receives a message") {
val mockAPI = mock[SomeAPI]
val ref = TestActorRef(new SomeActor(mockAPI))
expecting {
mockAPI.foo("Hi").once()
}
whenExecuting(mockAPI) {
ref.tell("Hi", testActor)
}
}
it("ok actor") {
val ref = TestActorRef(new Actor {
def receive = {
case "Hi" => sender ! "Hello"
}
})
ref.tell("Hi", testActor)
expectMsg("Hello")
}
}
"problemExample" passes, but then downstream "ok actor" fails for some reason I don't really understand... with this exception:
cannot reserve actor name '$$b': already terminated
java.lang.IllegalStateException: cannot reserve actor name '$$b': already terminated
at akka.actor.dungeon.ChildrenContainer$TerminatedChildrenContainer$.reserve(ChildrenContainer.scala:86)
at akka.actor.dungeon.Children$class.reserveChild(Children.scala:78)
at akka.actor.ActorCell.reserveChild(ActorCell.scala:306)
at akka.testkit.TestActorRef.<init>(TestActorRef.scala:29)
So, I can see ways of catching this sort of thing by examining the logger output in afterEach handlers. Definitely doable, although a little complicated in cases where I actually expect an exception and that's what I'm trying to test. But is there any more direct way of handling this and making the test fail?
Addendum: I have looked at the TestEventListener and suspect there's maybe something there that would help, but I can't see it. The only documentation I could find was about using it to check for expected exceptions, not unexpected ones.
Thinking in Actors there is also another solution: failures travel to the supervisor, so that is the perfect place to catch them and feed them into the test procedure:
val failures = TestProbe()
val props = ... // description for the actor under test
val failureParent = system.actorOf(Props(new Actor {
val child = context.actorOf(props, "child")
override val supervisorStrategy = OneForOneStrategy() {
case f => failures.ref ! f; Stop // or whichever directive is appropriate
}
def receive = {
case msg => child forward msg
}
}))
You can send to the actor under test by sending to failureParent and all failures—expected or not—go to the failures probe for inspection.
Other than examining the logs, I can think of two ways to fail tests when an actor crashes:
Ensure that no Terminated message is received
Check the TestActorRef.isTerminated property
The latter option is deprecated, so I'll ignore it.
Watching Other Actors from Probes describes how to setup a TestProbe. In this case it might look something like:
val probe = TestProbe()
probe watch ref
// Actual test goes here ...
probe.expectNoMessage()
If the actor dies due to an exception it will generate the Terminated message. If that happens during the test and you expect something else, the test will fail. If it happens after your last message expectation, then the expectNoMessage() should fail when Terminated is received.
Okay, I've had a little time to play with this. I've got a nice solution that uses an event listener and filter to catch errors. (Checking isTerminated or using TestProbes is probably good in more focused cases but seems awkward when trying to make something to mix into any old test.)
import akka.actor.{Props, Actor, ActorSystem}
import akka.event.Logging.Error
import akka.testkit._
import com.typesafe.config.Config
import org.scalatest._
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.mock.EasyMockSugar
import scala.collection.mutable
trait AkkaErrorChecking extends ShouldMatchers {
val system:ActorSystem
val errors:mutable.MutableList[Error] = new mutable.MutableList[Error]
val errorCaptureFilter = EventFilter.custom {
case e: Error =>
errors += e
false // don't actually filter out this event - it's nice to see the full output in console.
}
lazy val testListener = system.actorOf(Props(new akka.testkit.TestEventListener {
addFilter(errorCaptureFilter)
}))
def withErrorChecking[T](block: => T) = {
try {
system.eventStream.subscribe(testListener, classOf[Error])
filterEvents(errorCaptureFilter)(block)(system)
withClue(errors.mkString("Akka error(s):\n", "\n", ""))(errors should be('empty))
} finally {
system.eventStream.unsubscribe(testListener)
errors.clear()
}
}
}
You can just use withErrorChecking inline at specific spots, or mix it into a Suite and use withFixture to do it globally across all tests, like this:
trait AkkaErrorCheckingSuite extends AkkaErrorChecking with FunSpec {
override protected def withFixture(test: NoArgTest) {
withErrorChecking(test())
}
}
If you use this in my original example, then you will get the first test "calls foo only when it receives a message" to fail, which is nice because that's where the real failure is. But the downstream test will still fail as well due to the system blowing up. To fix that, I went a step further and used a fixture.Suite to instance a separate TestKit for each test. That solves lots of other potential test isolation issues when you have noisy actors. It requires a little more ceremony declaring each test but I think it's well worth it. Using this trait with my original example I get the first test failing and the second one passing which is just what I want!
trait IsolatedTestKit extends ShouldMatchers { this: fixture.Suite =>
type FixtureParam = TestKit
// override this if you want to pass a Config to the actor system instead of using default reference configuration
val actorSystemConfig: Option[Config] = None
private val systemNameRegex = "[^a-zA-Z0-9]".r
override protected def withFixture(test: OneArgTest) {
val fixtureSystem = actorSystemConfig.map(config => ActorSystem(systemNameRegex.replaceAllIn(test.name, "-"), config))
.getOrElse (ActorSystem (systemNameRegex.replaceAllIn(test.name, "-")))
try {
val errorCheck = new AkkaErrorChecking {
val system = fixtureSystem
}
errorCheck.withErrorChecking {
test(new TestKit(fixtureSystem))
}
}
finally {
fixtureSystem.shutdown()
}
}
}
Given the trait (simplified)
trait A {
val eventStream: EventStream
val credentialsStorage = // something here
val userStorage = // something here
val crypto = // something here
...
lazy val authSvc = new CoreAuthentication(credentialsStorage, new AuthenticationProviderResolver, userStorage, eventStream, crypto)
}
class T extends A with TraitProvidingEventStream with FlatSpec with [lot of another traits here] {
val eventStream = systemFromTraitProvidingEventStream.eventStream
"This" should "work" in {
println(authSvc) // this is "magic"
val user = authSvc.doSomethingWithUser(...);
}
}
if I remove line marked as //this is "magic", then I will get NullPointerException on the next line, so authSvc is null.
What may be wrong there?
I wasn't be able to create clean small test case for that, usually this works well
This came up once on the ML: If an exception is thrown when initializing a lazy val, the val is null; but you can attempt to init again and it can work magically. (That is, the "initialized" bit flag for the lazy val is not set on the first failed attempt to initialize.)
I think the case on the ML had to do with init order of vals in traits, so maybe that's your problem. It's infamously dangerous to rely on it, hence the advice to use defs in traits. See Luigi's comment on DelayedInit.