Need help in setup of Redshift and Scala using Play - scala

I am new to Scala and Redshift, I am trying to connect redshift with play framework. I have tried a couple of things but still not able to connect. i am using these configurations
db.default.driver=org.redshift.Driver
db.default.url="jdbc:redshift://url:5439/myDb?"
db.default.username="name"
db.default.password="password"
play.modules.enabled += "scalikejdbc.PlayModule"
# scalikejdbc.PlayModule doesn't depend on Play's DBModule
play.modules.disabled += "play.api.db.DBModule"
My SBT file looks like this
scalaVersion := "2.12.4"
libraryDependencies += guice
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % Test
// https://mvnrepository.com/artifact/com.typesafe.play/anorm
libraryDependencies ++= Seq(
"org.scalikejdbc" %% "scalikejdbc" % "3.2.1",
"com.h2database" % "h2" % "1.4.196",
"ch.qos.logback" % "logback-classic" % "1.2.3"
)
// https://mvnrepository.com/artifact/com.amazon/redshift-jdbc41
resolvers += "redshift" at "http://redshift-maven-repository.s3-website-us-east-1.amazonaws.com/release"
libraryDependencies += "com.amazon.redshift" % "redshift-jdbc4" % "1.2.10.1009
"
I am getting this error connecting DB

Setup Redshift with Play Scala
Application.conf
db.default.driver=com.amazon.redshift.jdbc4.Driver
db.default.url="jdbc:redshift://url:5439/db"
db.default.username="name"
db.default.password="password"
SBT file
libraryDependencies += jdbc
libraryDependencies ++= Seq(
"org.scalikejdbc" %% "scalikejdbc" % "3.2.0",
"org.scalikejdbc" %% "scalikejdbc-config" % "3.2.0",
"org.scalikejdbc" %% "scalikejdbc-play-initializer" % "2.6.0-scalikejdbc-3.2"
)
// https://mvnrepository.com/artifact/com.amazon/redshift-jdbc41
resolvers += "redshift" at "http://redshift-maven-repository.s3-website-us-east-1.amazonaws.com/release"
libraryDependencies += "com.amazon.redshift" % "redshift-jdbc4" % "1.2.10.1009"
Controller
package controllers
import javax.inject._
import play.api._
import play.api.mvc._
import play.api.db._
/**
* This controller creates an `Action` to handle HTTP requests to the
* application's home page.
*/
#Singleton
class HomeController #Inject()(db: Database, cc: ControllerComponents) extends AbstractController(cc) {
/**
* Create an Action to render an HTML page.
*
* The configuration in the `routes` file means that this method
* will be called when the application receives a `GET` request with
* a path of `/`.
*/
def index() = Action {
var outString = "Number is "
val conn = db.getConnection()
try {
val stmt = conn.createStatement
val rs = stmt.executeQuery("SELECT 9 as testkey ")
while (rs.next()) {
outString += rs.getString("testkey")
}
} finally {
conn.close()
}
Ok(outString)
// implicit request: Request[AnyContent] =>
//
// Ok(views.html.index())
}
}

Related

Trying to create Unit tests for Slick database for Play web application in Scala

I am struggling trying to configure unit tests for my Play web applications using Scala. I am using Play 2.6 and Scala 2.11.8. When I use the database configuration and execute sbt test I get the error No implementation for play.api.db.slick.DatabaseConfigProvider was bound on my console. So I am going to show how I set my web app and maybe someone could point what is wrong. Just for the sake, the web application is working well. It is just the unit test for the database that I cannot create.
build.sbt
import play.sbt.PlayImport._
name := """crypto-miners-demo"""
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.11.8"
libraryDependencies += guice
libraryDependencies += evolutions
libraryDependencies += jdbc
libraryDependencies += filters
libraryDependencies += ws
libraryDependencies += "com.h2database" % "h2" % "1.4.194"
libraryDependencies += "com.typesafe.play" %% "anorm" % "2.5.3"
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.0" % Test
libraryDependencies += "com.typesafe.play" %% "play-slick" % "3.0.0"
libraryDependencies += "com.typesafe.play" %% "play-slick-evolutions" % "3.0.0"
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.19.3"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.0"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.5"
application.conf:
play.application.loader = di.ApplicationLoader
play.filters.csrf.header.bypassHeaders {
X-Requested-With = "*"
Csrf-Token = "nocheck"
}
play.filters.csrf.bypassCorsTrustedOrigins = false
play.filters.disabled += play.filters.csrf.CSRFFilter
slick.dbs.default.profile = "slick.jdbc.SQLiteProfile$"
slick.dbs.default.db.driver = "org.sqlite.JDBC"
slick.dbs.default.db.url = "jdbc:sqlite:development.db"
slick.dbs.default.db.username = ""
slick.dbs.default.db.password = ""
db.default {
driver = org.sqlite.JDBC
url = "jdbc:sqlite:development.db"
username = ""
password = ""
}
play.modules.disabled += "play.api.db.DBModule"
RackRepositorySpec.scala:
import org.scalatestplus.play.PlaySpec
import org.scalatestplus.play.guice.GuiceOneAppPerTest
import play.api.db.evolutions._
import play.api.db.slick.DatabaseConfigProvider
import play.api.db.{Database, Databases}
import play.api.inject.bind
import play.api.inject.guice.GuiceInjectorBuilder
import play.api.test.Injecting
class RackRepositorySpec extends PlaySpec with GuiceOneAppPerTest with Injecting {
val database = Databases(
driver = "org.sqlite.JDBC",
url = "jdbc:sqlite:development.db",
name = "default",
config = Map(
"username" -> "",
"password" -> ""
)
)
val guice = new GuiceInjectorBuilder()
.overrides(bind[Database].toInstance(database))
.injector()
val defaultDbProvider = guice.instanceOf[DatabaseConfigProvider]
def beforeAll() = Evolutions.applyEvolutions(database)
def afterAll() = {
// Evolutions.cleanupEvolutions(database)
database.shutdown()
}
Evolution(
1,
"create table test (id bigint not null, name varchar(255));",
"drop table test;"
)
}
and I get this error when I execute sbt test:
[info] models.RackRepositorySpec *** ABORTED ***
[info] com.google.inject.ConfigurationException: Guice configuration errors:
[info]
[info] 1) No implementation for play.api.db.slick.DatabaseConfigProvider was bound.
[info] while locating play.api.db.slick.DatabaseConfigProvider
[info]
[info] 1 error
[info] at com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1045)
[info] at com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1004)
[info] at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054)
[info] at play.api.inject.guice.GuiceInjector.instanceOf(GuiceInjectorBuilder.scala:409)
[info] at play.api.inject.guice.GuiceInjector.instanceOf(GuiceInjectorBuilder.scala:404)
[info] at play.api.inject.ContextClassLoaderInjector$$anonfun$instanceOf$2.apply(Injector.scala:117)
[info] at play.api.inject.ContextClassLoaderInjector.withContext(Injector.scala:126)
[info] at play.api.inject.ContextClassLoaderInjector.instanceOf(Injector.scala:117)
[info] at models.RackRepositorySpec.<init>(RackRepositorySpec.scala:26)
[info] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
I solved using the libraryDependencies += specs2 % Test in the build.sbt. I hope this is a good practice for slick:
import org.specs2.mutable.Specification
import play.api.Application
import play.api.test.WithApplicationLoader
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.{Await, Future}
import scala.concurrent.duration.DurationInt
class RackRepositorySpec extends Specification {
"RackRepository" should {
"work as expected" in new WithApplicationLoader {
val app2dao = Application.instanceCache[RackRepository]
val rackRepository: RackRepository = app2dao(app)
Await.result(rackRepository.delete("r-1"), 3 seconds)
Await.result(rackRepository.delete("r-2"), 3 seconds)
Await.result(rackRepository.delete("r-3"), 3 seconds)
val testRacks = Set(
RackRow("r-1", 0.2F, System.currentTimeMillis()),
RackRow("r-2", 0.5F, System.currentTimeMillis()),
RackRow("r-3", 0.8F, System.currentTimeMillis())
)
Await.result(Future.sequence(testRacks.map(rackRepository.insert)), 3 seconds)
val storedRacks = Await.result(rackRepository.list(), 3 seconds)
storedRacks.toSet must contain(testRacks)
}
}
}

Failure retries with Finagle

I'm struggling with Finagle client retries.
For some reason, client doesn't retry failed requests in test, even though I use a custom classifier, which should mark any response code other than 200 as RetryableFailure. I have tried bulding client both with ClientBuilder and with Http.client.
I narrowed down my code and test to only the necessary bit.
Please take a look:
MyApi.scala:
import io.circe._, io.circe.generic.auto._, io.circe.jawn._, io.circe.syntax._
import com.twitter.finagle.http._
import com.twitter.finagle.Service
import com.twitter.finagle.Codec
import com.twitter.finagle.builder._
import scala.concurrent.ExecutionContext.Implicits.global
import com.twitter.conversions.time._
import com.twitter.finagle.http.service.HttpResponseClassifier
import com.twitter.finagle.service._
import com.twitter.util.Return
class MyApi(val host: String, val protocol: String = "https") {
// def request: Service[Request, Response] = {
// val clientBuilder = ClientBuilder()
// .codec(Http())
// .responseClassifier(classifier)
// .noFailureAccrual
// .requestTimeout(30.seconds)
// .hostConnectionLimit(5)
// .tcpConnectTimeout(5.seconds)
// .retryBudget(budget)
// .retries(5)
// protocol match {
// case "https" => clientBuilder.hosts(s"$host:443").tls(host).build()
// case _ => clientBuilder.hosts(host).build()
// }
// }
val budget: RetryBudget = RetryBudget(
ttl = 10.seconds,
minRetriesPerSec = 5,
percentCanRetry = 0.1
)
val classifier: ResponseClassifier = {
case ReqRep(_, Return(r: Response)) if r.statusCode == 200 =>
ResponseClass.Success
case _ => ResponseClass.RetryableFailure
}
def request: Service[Request, Response] = com.twitter.finagle.Http.client
.withResponseClassifier(classifier)
.withRetryBudget(budget)
.withRetryBackoff(Backoff.exponentialJittered(2.seconds, 32.seconds))
.newService(host)
def requestUsers = request(users(1))
val users: (Int => Request) = (n) => RequestBuilder()
.url(s"$protocol://$host/api?results=$n")
.buildGet()
}
object MyApi {
def apply(host: String, protocol: String) = new MyApi(host, protocol)
}
MyApiSpec.scala:
import com.twitter.util.Return
import com.twitter.finagle.http.service.HttpResponseClassifier
import org.scalatest._
import org.scalatest.concurrent._
import org.scalamock.proxy.ProxyMockFactory
import scala.language.postfixOps
import com.twitter.util.{Await}
import org.scalamock.scalatest.MockFactory
import com.twitter.finagle.http._
import org.scalamock.proxy.ProxyMockFactory
import io.finch._
import com.twitter.finagle.Http
import scala.io.Source
import com.twitter.io.{Reader, Buf}
import com.twitter.finagle.ListeningServer
class MyApiSpec extends FlatSpec with Matchers with MockFactory with ScalaFutures with ProxyMockFactory with BeforeAndAfter with BeforeAndAfterAll {
var server: ListeningServer = _
val ru = MyApi("localhost:1490", "http")
after {
server.close()
}
var attempts = 0
val failure: Endpoint[String] = get("api") {
println(attempts)
if (attempts > 1) {
Ok("test message")
}
else {
attempts += 1
BadGateway(new Exception("try again"))
}
}
it should "avoid network issues by retrying" in {
server = Http.server
.serve(":1490", failure.toServiceAs[Text.Plain])
val users = Await.result(ru.requestUsers).contentString
assert(users == "test message")
}
}
build.sbt:
name := "myapi"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6"
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % "test"
libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.0.2"
val circeVersion = "0.4.1"
libraryDependencies ++= Seq(
"io.circe" %% "circe-core",
"io.circe" %% "circe-generic",
"io.circe" %% "circe-parser"
).map(_ % circeVersion)
libraryDependencies ++= Seq(
"com.github.finagle" %% "finch-core" % "0.11.0-M2",
"com.github.finagle" %% "finch-circe" % "0.11.0-M2"
)
libraryDependencies += "com.typesafe" % "config" % "1.3.0"
libraryDependencies += "com.typesafe.slick" %% "slick" % "3.1.1"
libraryDependencies += "com.typesafe.slick" %% "slick-hikaricp" % "3.1.0"
libraryDependencies += "com.typesafe.slick" %% "slick-testkit" % "3.1.1" % "test"
libraryDependencies += "org.postgresql" % "postgresql" % "9.4-1206-jdbc4"
libraryDependencies += "junit" % "junit" % "4.8.1" % "test"
libraryDependencies += "org.scalamock" %% "scalamock-scalatest-support" % "3.2.2" % "test"
val scalazVersion = "7.2.4"
libraryDependencies ++= Seq(
"org.scalaz" %% "scalaz-core" % scalazVersion,
"org.scalaz" %% "scalaz-effect" % scalazVersion,
"org.scalaz" %% "scalaz-scalacheck-binding" % scalazVersion % "test"
)
scalacOptions += "-feature"
initialCommands in console := "import scalaz._, Scalaz._"
Sorry, it's so confusing. Finagle doesn't retry application-level failures (set via ResponseClassifiers) by default so you'd need to enable retries explicitly (see Retries section in the user guide).
Try building something between the lines:
import com.twitter.conversions.time._
import com.twitter.finagle.Http
import com.twitter.finagle.http.{Request, Response}
import com.twitter.finagle.util.DefaultTimer
import com.twitter.finagle.service.{Backoff, RetryFilter}
import com.twitter.finagle.stats.NullStatsReceiver
import com.twitter.util._
import com.twitter.finagle.param.HighResTimer
val twitter = Http.client.newService("twitter.com")
val shouldRetry: PartialFunction [(Request, Try[Response]), Boolean] = {
case (_, Return(rep)) => rep.status != 200
}
implicit val t = HighResTimer.Default
// 3 retries, backoff 1 second
val retry = RetryFilter(Backoff.const(1.second).take(3))(shouldRetry)
val retryTwitter = retry.andThen(twitter)

object mongodb is not a member of package com

After run i getting this object mongodb is not a member of package com in Play
I cant resolve that problem. Im trying to do Play Scala MongoDB and from client side Scala.js
my build.sbt is >
lazy val server = (project in file("server")).settings(
scalaVersion := scalaV,
scalaJSProjects := clients,
pipelineStages := Seq(scalaJSProd, gzip),
resolvers += "scalaz-bintray" at "https://dl.bintray.com/scalaz/releases",
libraryDependencies ++= Seq(
"com.vmunier" %% "play-scalajs-scripts" % "0.3.0",
"org.mongodb" %% "casbah" % "3.1.0",
"org.webjars" % "jquery" % "1.11.1",
"com.novus" %% "salat" % "1.9.9",
"se.radley" %% "play-plugins-salat" % "1.5.0",
"commons-codec" % "commons-codec" % "1.6",
specs2 % Test
object code
package model
import com.mongodb.DBObject
import com.mongodb.casbah.MongoConnection
import com.mongodb.casbah.commons.MongoDBObject
/**
* Created by qqQQaAs on 2016-01-04.
*/
object AllBooks {
def toJson() {
val collection = MongoConnection()("books")("books")
val query = "Krakow"
// simply convert the result to a string, separating items with a comma
// this string goes inside an "array", and it's ready to hit the road
val json = "[%s]".format(
collection.find(query).toList.mkString(",")
)
}
Controller
package controllers
import com.mongodb.casbah._
import model.AllBooks
import play.api.mvc._
import shared.SharedMessages
object Application extends Controller {
def index = Action {
Ok(views.html.index(SharedMessages.itWorks))
}
def service() = Action {
val json = AllBooks.toJson()
Ok(json).as("application/json")
}
}
Thanks for help guys
You should import the following pages as per the official documentation:
import org.mongodb.scala._
import org.bson._

spray Collection ToResponseMarshallable

I am trying to return a List from my complete directive in spray-routing.
complete {
List("hello")
}
However, I am getting an error -
Expression of type List[String] doesn't conform to expected type ToResponseMarshallable
I am getting the same error with Seq. I see that marshallers for List and Seq are not provided by default in spray-httpx documentation. However, spray-json provides marshalling support in its DefaultJsonProtocol. I have imported spray.httpx.SprayJsonSupport._ and spray.json.DefaultJsonProtocol._ in my code, but that too hasn't helped.
Any idea how can I marshal a List/Seq using spray-json? Or will I have to write my own Marshaller?
(My scala version is 2.11.4)
Using spray 1.3.2 and spray-json 1.3.2 it should be possible.
Make sure you import (although you say you do, but double check).
import spray.httpx.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
Consider following example:
import akka.actor.{ActorSystem, Props, Actor}
import akka.io.IO
import spray.can.Http
import spray.routing.HttpService
import akka.pattern.ask
import akka.util.Timeout
import scala.concurrent.duration._
import spray.httpx.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
object Boot extends App {
implicit val system = ActorSystem("so")
val testActor = system.actorOf(Props[TestActor])
implicit val timeout = Timeout(5.seconds)
IO(Http) ? Http.Bind(testActor, interface = "0.0.0.0", port = 5000)
}
class TestActor extends Actor with HttpService {
def receive = runRoute(route)
def actorRefFactory = context
val route = get{
path("ping") {
complete(List("OK"))
}
}
}
Requesting /ping returns ["OK"] which looks okay.
Just for reference build.sbt bellow.
build.sbt
val akkaVersion = "2.3.5"
val sprayVersion = "1.3.2"
resolvers += "spray" at "http://repo.spray.io/"
scalaVersion := "2.11.5"
scalacOptions := Seq("-feature", "-unchecked", "-deprecation", "-encoding", "utf8")
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"io.spray" %% "spray-can" % sprayVersion,
"io.spray" %% "spray-routing" % sprayVersion,
"io.spray" %% "spray-client" % sprayVersion,
"io.spray" %% "spray-json" % "1.3.1"
)
API for marshalling seems to have changed since akka 2.3.5. For akka-2.4.11.2, SprayJsonSupport needs import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
1) Following is working sbt,
name := "some-project"
version := "1.0"
scalaVersion := "2.11.5"
scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8")
val akkaVersion = "2.4.5"
libraryDependencies ++= {
Seq(
"com.typesafe.akka" %% "akka-http-experimental" % akkaVersion,
"com.typesafe.akka" % "akka-http-spray-json-experimental_2.11" % akkaVersion,
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion,
"org.mongodb" % "mongo-java-driver" % "3.4.1",
"org.apache.kafka" %% "kafka" % "0.10.1.1",
"org.apache.kafka" % "kafka-clients" % "0.10.1.1",
//test
"org.scalatest" %% "scalatest" % "2.2.5" % "test",
"com.typesafe.akka" %% "akka-http-testkit" % "10.0.9" % "test",
"de.flapdoodle.embed" % "de.flapdoodle.embed.mongo" % "2.0.0" % "test"
)
}
parallelExecution in Test := false
2) And imports are,
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
object GetMusicResources {
val music_get_api =
path("music") {
get {
complete {
List("my data1")
}
}
}
}
3) Tests
Get("/music") ~> GetMusicResources.music_get_api ~> check {
val response1: Future[ByteString] = response.entity.toStrict(300.millis).map {_.data }
response1.map(_.utf8String).map { responseString =>
responseString shouldBe """["my data1"]"""
}
}
In my case, I was missing the second import in :
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._ // <= this one
In case anyone stumble into the same issue

java.lang.NoSuchMethodError with Scala actors

I have a simple Scala application (taken from here) which I want to test. The whole project is compiled successfully with SBT. However, when I launch the tests with sbt test I get the following error message:
Could not run test ConnectionTest:java.lang.NoSuchMethodError: akka.actor.Props$.apply(Lscala/Function0;)Lakka/actor/Props;
From the internet search, I get the impression that some of my versioning is not compatible but that is merely a guess. What may be wrong?
[Test Case]
import akka.actor.{Props, Actor, ActorSystem, ActorRef}
import akka.testkit.{TestKit, TestActorRef, ImplicitSender}
import org.scalatest.{WordSpecLike, BeforeAndAfterAll}
import org.scalatest.matchers.MustMatchers
class ConnectionTest extends TestKit(ActorSystem("ConnectionSpec"))
with WordSpecLike
with MustMatchers
with BeforeAndAfterAll {
override def afterAll() { system.shutdown() }
"Server" must {
"bind to port " in {
// create server
val server = system.actorOf(Props[Server], name = "server")
expectMsg("Bound to /127.0.0.1:8888")
}
}
}
[build.sbt]
name := "MyApp"
version := "0.2"
scalaVersion := "2.10.4"
mainClass := Some("akka.Main")
resolvers += "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
libraryDependencies +=
"com.typesafe.akka" %% "akka-actor" % "2.3.2"
libraryDependencies +=
"com.typesafe.akka" %% "akka-testkit" % "2.1.4" % "test"
libraryDependencies +=
"org.scalatest" % "scalatest_2.10" % "2.0" % "test"
You need to use the same version of Akka for testkit.
libraryDependencies +=
"com.typesafe.akka" %% "akka-testkit" % "2.1.4" % "test"
should be
libraryDependencies +=
"com.typesafe.akka" %% "akka-testkit" % "2.3.2" % "test"