I'm new to scala and same for scalatest.
My aim is to write a Unit test for the following small piece of code:
import java.sql.SQLException
import com.typesafe.scalalogging.LazyLogging
import slick.jdbc.MySQLProfile.api._
import scala.concurrent.{ExecutionContext, ExecutionContextExecutor, Future}
object DbCreator extends LazyLogging {
implicit val ex: ExecutionContextExecutor = ExecutionContext.global
def createDatabaseIfNotExist(): Future[String] = {
Database
.forURL(url = "some host", user = "user", password = "pass", driver = "driver")
.run(sqlu"CREATE DATABASE ...").map(_ => "created")
.recover {
case e: Throwable => {
logger.error("Error!", e)
throw new SQLException(e.getMessage)
}
}
}
}
I've previously used python, which has the patch concept.
So my idea was to patch the Database class being imported and used in createDatabaseIfNotExist, so I can verify the different scenarios.
Unfortunately, I could not find equivalent/similar concept in scalatest.
Did I miss it?
Is my approach wrong? If so, how would u suggest me write a UT for the createDatabaseIfNotExist method?
Is the current DbCreator implementation not testable enough?
Many thanks for all help!
I see several options:
scalamock or some java mocking library
acolyte JDBC driver (has scala DSL)
embedded mysql
Related
I try to upgrade a Scala/Play project to Play 2.7, Scala 2.12.11, Specs2 4.5.1.
In the project there is the following Specs2 test that I cannot understand in the sense of its Specs2 specification structure (could be that the Specs2 API changed a lot since the test was written).
When I looked at the structure of specifications in the current API, I could not see any example of is method combined together with should.
What was it supposed to mean?
How can I rewrite such a specification in the current Specs2 API?
I also noticed that the test code used import org.specs2.mutable.Specification instead of import org.specs2.Specification which is supposed to be used when using the is method.
And it uses def is(implicit ee: ExecutionEnv), instead of def is.
Here is the old test:
package services
import org.specs2.concurrent.ExecutionEnv
import org.specs2.mutable.Specification
import org.specs2.specification.mutable.ExecutionEnvironment
import play.api.inject.guice.GuiceApplicationBuilder
import play.api.test.WithApplication
import play.modules.reactivemongo._
import scala.concurrent.duration._
class StatisticsServiceSpec() extends Specification with ExecutionEnvironment {
def is(implicit ee: ExecutionEnv) = {
"The StatisticsService" should {
"compute and publish statistics" in new WithApplication() {
val repository = new MongoStatisticsRepository(configuredAppBuilder.injector.instanceOf[ReactiveMongoApi])
val wsTwitterService = new WSTwitterService
val service = new DefaultStatisticsService(repository, wsTwitterService)
val f = service.createUserStatistics("elmanu")
f must beEqualTo(()).await(retries = 0, timeout = 5.seconds)
}
}
def configuredAppBuilder = {
import scala.collection.JavaConversions.iterableAsScalaIterable
val env = play.api.Environment.simple(mode = play.api.Mode.Test)
val config = play.api.Configuration.load(env)
val modules = config.getStringList("play.modules.enabled").fold(
List.empty[String])(l => iterableAsScalaIterable(l).toList)
new GuiceApplicationBuilder().
configure("play.modules.enabled" -> (modules :+
"play.modules.reactivemongo.ReactiveMongoModule")).build
}
}
}
To simplify the code down to the actual Specs2 API, I think it could be reduced to something like this:
package services
import org.specs2.concurrent.ExecutionEnv
import org.specs2.mutable.Specification
import scala.concurrent.Future
import play.api.test.WithApplication
import scala.concurrent.duration._
class StatisticsServiceSpec(implicit ee: ExecutionEnv) extends Specification /* with ExecutionEnvironment */ {
def is(implicit ee: ExecutionEnv) = {
"The StatisticsService" should {
"compute and publish statistics" in new WithApplication() {
val f = Future(1)
f must beEqualTo(1).await(retries = 0, timeout = 5.seconds)
}
}
}
}
Pay attention that I removed the ExecutionEnvironment trait, since it seems to have been removed from the library.
Now, the code finally compiles, but when I try to run the test, there are no errors, but no test is actually run: the output is Empty test suite.
The new specification should be
package services
import org.specs2.concurrent.ExecutionEnv
import org.specs2.mutable.Specification
import scala.concurrent.Future
import play.api.test.WithApplication
import scala.concurrent.duration._
class StatisticsServiceSpec(implicit ee: ExecutionEnv) extends Specification {
"The StatisticsService" should {
"compute and publish statistics" in new WithApplication() {
val f = Future(1)
f must beEqualTo(1).await(retries = 0, timeout = 5.seconds)
}
}
}
The ExecutionEnv is now really supposed to be retrieved as a specification member directly (with an implicit to be make it available to the await method).
is is not necessary in a "mutable" specification. is is the function in a Specification where you declare all the "Fragments" of your specification (a Fragment is a Description + an Execution). In a mutable specification this function is automatically populated from the fact that you trigger method calls directly in the body of the class when the specification is instantiated. The fragments created by should and in are collected in a mutable variable, hence the name "mutable specification".
If you define def is(implicit ee: ExecutionEnv), this is like defining another, valid, is definition that specs2 doesn't know about, while not creating anything for the def is: Fragments method. That's why you end up with an empty specification.
Hello I'm writing scala code to pull the data from API.
Data is paginated, so I'm pulling a data sequentially.
Now, I'm looking a solution to pulling multiple page parallel and stuck to create WSClient programatically instead of Inject.
Anyone have a solution to create WSClient ?
I found a AhcWSClient(), but it required to implicitly import actor system.
When you cannot Inject one as suggested in the other answer, you can create a Standalone WS client using:
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import play.api.libs.ws._
import play.api.libs.ws.ahc.StandaloneAhcWSClient
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
val ws = StandaloneAhcWSClient()
No need to reinvent the wheel here. And I'm not sure why you say you can't inject a WSClient. If you can inject a WSClient, then you could do something like this to run the requests in parallel:
class MyClient #Inject() (wsClient: WSClient)(implicit ec: ExecutionContext) {
def getSomething(urls: Vector[String]): Future[Something] = {
val futures = urls.par.map { url =>
wsClient.url(url).get()
}
Future.sequence(futures).map { responses =>
//process responses here. You might want to fold them together
}
}
}
Slick 3.0.0
play 2.6.2
I am experimenting with Slick and run into an interesting issue. I hope the solution is just trivial and I am thinking too much about this
I have implemented the following simple code.
case class Content(content:String)
class ContentTable(tag: Tag) extends Table[Content](tag, "content"){
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def content = column[String]("content")
override def * : ProvenShape[Content] = (content).mapTo[Content]
}
object ContentDb {
val db = Database.forConfig("databaseConfiguration")
lazy val contents = TableQuery[ContentTable]
def all: Seq[Content] = Await.result(db.run(contents.result), 2 seconds)
}
So for this code to work, it requires the following import among others of course.
import slick.jdbc.H2Profile.api._
or alternatively
import slick.jdbc.PostgresProfile.api._
Now, I think, and please correct me if I am wrong, that the database driver should be a configuration detail. That is, I'd choose to run H2 in memory database while developing. Run against test PostgreSQL instance when testing and then run against another instance when in production. I mean the whole idea of abstracting the driver is to have this flexibility... I think.
Now, I have done some research and found that I could do something like this:
trait DbComponent {
val driver: JdbcProfile
import driver.api._
val db: Database
}
trait H2DbComponent extends DbComponent {
val driver: JdbcProfile = slick.jdbc.H2Profile
import driver.api._
val db = Database.forConfig("databaseConfiguration")
}
trait Contents {
def all: Seq[Content]
}
object Contents {
def apply: Contents = new ContentsDb with H2DbComponent
}
trait ContentsDb extends Contents {
this: DbComponent =>
import driver.api._
class ContentTable(tag: Tag) extends Table[Content](tag, "content") {
def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
def content = column[String]("content")
override def * : ProvenShape[Content] = content.mapTo[Content]
}
lazy val contents = TableQuery[ContentTable]
def all: Seq[Content] = Await.result(db.run(contents.result), 2 seconds)
}
Then I can use dependency injection to inject the right instance for each entity that I have. Not ideal, but possible. So, I start digging on how to have conditional dependency injection based on which environment is running in Play Framework.
I was expecting something similar to the following:
#Component(env=("prod","test")
class ProductionContentsDb extends ContentsDb with PostgresDbComponent
#Component(env="dev")
class ProductionContentsDb extends ContentsDb with H2DbComponent
But, no luck...
EDIT
Just after I have finished writing this and started reading it again, I am curious if we can just have the something similar to:
class DbComponent #Inject (driver: JdbcProfile) {
import driver.api._
val db = Database.forConfig("databaseConfiguration")
}
You can create separate configuration files for each environment. like
application.conf --local
application.prod.conf -- prod with contents below
include "aplication.conf"
###update slick configurations
Then while running application in different stages feed in with -Dconfig.resource=
Following this answer: https://stackoverflow.com/a/30806548/4496364
I use Play's ExecutionContext in my project.
Recently I needed to use Mockito to test some services in Play.
So, this is simplified version of it:
import scala.concurrent.{ Future, ExecutionContext }
import play.api.libs.concurrent.Execution.Implicits.defaultContext
case class Model(id: Int, name: String)
trait DAO {
def findAll(implicit ec: ExecutionContext): Future[List[Model]]
}
class Service(dao: DAO) {
def findAll: Future[List[Model]] = dao.findAll
}
Test:
import play.api.libs.concurrent.Execution.Implicits.defaultContext
// doesn't work when different ExecutionContext
// import scala.concurrent.ExecutionContext.Implicits.global
class FuturesTest extends PlaySpec with MockitoSugar with ScalaFutures {
"Service" should {
"return all future data" in {
val mockModel = Model(1, "name")
val mockDAO = mock[DAO]
when(mockDAO.findAll) thenReturn Future.successful(List(mockModel))
val service = new Service(mockDAO)
val futureData = service.findAll
whenReady(futureData) { data =>
data.map(_.name) must contain(mockModel.name)
}
}
}
}
Note the comment in test, i get a NullPointException when calling dao.findAll in the Service. At first I thought that Mockito can't handle Scala's Futures but I figured out that the ExecutionContext is the problem. Since I'm not a concurrency expert, can someone please explain why does this happen?
In case someone is looking, the answer was obvious...
import org.mockito.Matchers.any
..
mockDAO.findAll(any[ExecutionContext])
I wasn't familiar with how Mockito works, or with Scala implicits.
When you don't pass any[ExecutionContext] Scala will fill it with the implicit one from the test.
I'm working on a web scraper using this project (based on Scala, Spray, Akka and PhantomJS)
The problem is that I can't find a more specific example of how to use it, and the documentation is missing a lot of details
1- I would like to know how to give an specific URL so I can get data from it
2- How can I excecute, or pass a javascript file or function so that phantom can run and do some stuff(return specific data or whatever, from the site in point 1- )
Here is my Main.scala file: (Is almost the same as the one in the project)
package com.typesafe.webdriver.tester
import akka.actor.{ActorRef, ActorSystem}
import akka.pattern.ask
import com.typesafe.webdriver.{Session, PhantomJs, LocalBrowser}
import akka.util.Timeout
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import spray.json._
import spray.http._
object Main {
def main(args: Array[String]) {
implicit val system = ActorSystem("webdriver-system")
implicit val timeout = Timeout(5.seconds)
system.scheduler.scheduleOnce(7.seconds) {
system.shutdown()
System.exit(1)
}
val browser = system.actorOf(PhantomJs.props(system), "localBrowser")
browser ! LocalBrowser.Startup
for (
session <- (browser ? LocalBrowser.CreateSession).mapTo[ActorRef];
result <- (session ? Session.ExecuteNativeJs("return 5+5",JsArray(JsNumber(999)))).mapTo[JsNumber]
) yield {
println(result)
try {
system.shutdown()
System.exit(0)
} catch {
case _: Throwable =>
}
}
}
}
I would suggest you to use already created web scrappers in Scala.
For example ScalaWebDcraper which has nicely writted DSL and scrapping feature.
https://github.com/Rovak/ScalaWebscraper
It can be combined with Goose, which is a web article extractor. You can use it to fetch article data from the links you visit with the previous library.
https://github.com/jiminoc/goose
Also, checkout Metascrapper, a Scala Library for Scraping Page Metadata
https://beachape.com/blog/2013/09/05/introducing-metascraper-a-scala-library-for-scraping-page-metadata/
And check this question, lot's of valuable info inside.