Submitting multipart/form-data in ScalaWS - scala

I'm trying to create a post with 2 parts: the first part is a json text and the second, is the binary audio content.
The code to send the post is actually:
val ref = UUID.randomUUID()
val requestJobData = s"""
{
"reference": "$ref",
"operating_mode": "accurate",
"callback_url": "http://localhost:8100/",
"model": { "name": "por-bra" },
"channels": {
"firstChannelLabel": {
"result_format": "transcript",
"format":"audio/x-wav",
"num_speakers":2,
}
}
}
"""
val jsonPart = DataPart("json", requestJobData)
val filePart = FilePart("firstChannelLabel", "audio(1).wav", Some("audio/x-wav"), FileIO.fromPath(Paths.get("C:\\Input\\_pending\\audio(1).wav")))
val body = Source(filePart :: jsonPart :: List())
val response =
client.url(s"$baseUrl")
.post(body)
.map {response => response.body[String]}
When I try this code I get these errors:
Error:(73, 14) Cannot find an instance of akka.stream.scaladsl.Source[Product with Serializable with play.api.mvc.MultipartFormData.Part[akka.stream.scaladsl.Source[akka.util.ByteString,scala.concurrent.Future[akka.stream.IOResult]]],akka.NotUsed] to WSBody. Define a BodyWritable[akka.stream.scaladsl.Source[Product with Serializable with play.api.mvc.MultipartFormData.Part[akka.stream.scaladsl.Source[akka.util.ByteString,scala.concurrent.Future[akka.stream.IOResult]]],akka.NotUsed]] or extend play.api.libs.ws.ahc.DefaultBodyWritables
.post(body)
Error:(73, 14) not enough arguments for method post: (implicit evidence$3: play.api.libs.ws.BodyWritable[akka.stream.scaladsl.Source[Product with Serializable with play.api.mvc.MultipartFormData.Part[akka.stream.scaladsl.Source[akka.util.ByteString,scala.concurrent.Future[akka.stream.IOResult]]],akka.NotUsed]])scala.concurrent.Future[play.api.libs.ws.StandaloneWSRequest#Response].
Unspecified value parameter evidence$3.
.post(body)
My dependencies are:
scalaVersion := "2.12.1"
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.0" % Test
libraryDependencies += ws
libraryDependencies += guice
libraryDependencies += "com.typesafe" % "config" % "1.3.1"
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.5.3",
"com.typesafe.akka" %% "akka-testkit" % "2.5.3" % Test
)
libraryDependencies += "com.typesafe.play" %% "play-ahc-ws-standalone" % "1.0.0"
libraryDependencies += "com.typesafe.play" %% "play-ws-standalone-json" % "1.0.0"
libraryDependencies ++= Seq(
"com.github.pureconfig" %% "pureconfig" % "0.7.2"
)
Can anyone help with this? Thanks in advance!

As I needed to accomplish this task faster, for this specific part I had to switch to another library called apache.http (http://hc.apache.org/httpcomponents-client-ga/)
These are my current settings:
build.sbt
libraryDependencies ++= Seq(
"org.apache.httpcomponents" % "httpclient" % "4.3.1",
"org.apache.httpcomponents" % "httpmime" % "4.3.1"
)
Complete POST example:
import org.apache.http.client.methods.{HttpGet, HttpPost}
import org.apache.http.entity.ContentType
import org.apache.http.entity.mime.MultipartEntityBuilder
import org.apache.http.impl.client.HttpClients
import org.apache.http.util.EntityUtils
val ref = UUID.randomUUID()
val requestJobData =
s"""
{
"reference": "$ref",
"operating_mode": "accurate",
"model": { "name": "por-bra" },
"channels": {
"firstChannelLabel": {
"result_format": "lattice",
"format":"audio/x-wav",
"num_speakers":2
}
}
}
"""
val entity = MultipartEntityBuilder.create()
entity.addTextBody("json", requestJobData, ContentType.APPLICATION_JSON)
val path = Paths.get(audioPath)
val fileName = path.getFileName().toString()
val fileStream = new FileInputStream(audioPath)
entity.addBinaryBody("firstChannelLabel", fileStream, ContentType.create("audio/x-wav"), fileName)
val client = HttpClients.createDefault()
try {
val post = new HttpPost(baseUrl)
post.setEntity(entity.build())
val response = client.execute(post)
}
catch {
case e: Throwable =>
log.error("erro ao agendar job: " + e.toString())
}
finally {
client.close()
}
It's really working. I omitted some parts of the code regarding communication with actors and other logging, but I hope this helps anyone.

Related

Need help in setup of Redshift and Scala using Play

I am new to Scala and Redshift, I am trying to connect redshift with play framework. I have tried a couple of things but still not able to connect. i am using these configurations
db.default.driver=org.redshift.Driver
db.default.url="jdbc:redshift://url:5439/myDb?"
db.default.username="name"
db.default.password="password"
play.modules.enabled += "scalikejdbc.PlayModule"
# scalikejdbc.PlayModule doesn't depend on Play's DBModule
play.modules.disabled += "play.api.db.DBModule"
My SBT file looks like this
scalaVersion := "2.12.4"
libraryDependencies += guice
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % Test
// https://mvnrepository.com/artifact/com.typesafe.play/anorm
libraryDependencies ++= Seq(
"org.scalikejdbc" %% "scalikejdbc" % "3.2.1",
"com.h2database" % "h2" % "1.4.196",
"ch.qos.logback" % "logback-classic" % "1.2.3"
)
// https://mvnrepository.com/artifact/com.amazon/redshift-jdbc41
resolvers += "redshift" at "http://redshift-maven-repository.s3-website-us-east-1.amazonaws.com/release"
libraryDependencies += "com.amazon.redshift" % "redshift-jdbc4" % "1.2.10.1009
"
I am getting this error connecting DB
Setup Redshift with Play Scala
Application.conf
db.default.driver=com.amazon.redshift.jdbc4.Driver
db.default.url="jdbc:redshift://url:5439/db"
db.default.username="name"
db.default.password="password"
SBT file
libraryDependencies += jdbc
libraryDependencies ++= Seq(
"org.scalikejdbc" %% "scalikejdbc" % "3.2.0",
"org.scalikejdbc" %% "scalikejdbc-config" % "3.2.0",
"org.scalikejdbc" %% "scalikejdbc-play-initializer" % "2.6.0-scalikejdbc-3.2"
)
// https://mvnrepository.com/artifact/com.amazon/redshift-jdbc41
resolvers += "redshift" at "http://redshift-maven-repository.s3-website-us-east-1.amazonaws.com/release"
libraryDependencies += "com.amazon.redshift" % "redshift-jdbc4" % "1.2.10.1009"
Controller
package controllers
import javax.inject._
import play.api._
import play.api.mvc._
import play.api.db._
/**
* This controller creates an `Action` to handle HTTP requests to the
* application's home page.
*/
#Singleton
class HomeController #Inject()(db: Database, cc: ControllerComponents) extends AbstractController(cc) {
/**
* Create an Action to render an HTML page.
*
* The configuration in the `routes` file means that this method
* will be called when the application receives a `GET` request with
* a path of `/`.
*/
def index() = Action {
var outString = "Number is "
val conn = db.getConnection()
try {
val stmt = conn.createStatement
val rs = stmt.executeQuery("SELECT 9 as testkey ")
while (rs.next()) {
outString += rs.getString("testkey")
}
} finally {
conn.close()
}
Ok(outString)
// implicit request: Request[AnyContent] =>
//
// Ok(views.html.index())
}
}

Scala Akka Streams project fails to compile, "value ~> is not a member of akka.streams.Outlet[In]"

I am using the Akka Streams API in a Scala project, working in Intellij IDEA with the SBT plugin. I have a worker pool Flow as described here: https://doc.akka.io/docs/akka/current/scala/stream/stream-cookbook.html. Here is my code:
package streams
import akka.NotUsed
import akka.stream.scaladsl.{Balance, Flow, GraphDSL, Merge}
import akka.stream.{FlowShape, Graph}
object WorkerPoolFlow {
def apply[In, Out](
worker: Flow[In, Out, Any],
workerCount: Int):
Graph[FlowShape[In, Out], NotUsed] = {
GraphDSL.create() { implicit b =>
val balance = b.add(Balance[In](workerCount, waitForAllDownstreams = true))
val merge = b.add(Merge[Out](workerCount))
for (i <- 0 until workerCount)
balance.out(i) ~> worker.async ~> merge.in(i)
FlowShape(
balance.in,
merge.out)
}
}
}
For some reason the project is now failing to compile, giving this error: value ~> is not a member of akka.stream.Outlet[In].
It compiled fine until today. The only change I am aware of making is installing a Scala linter plugin scalafmt, and importing a few new libraries in build.sbt. Here is my build.sbt:
name := "myProject"
version := "0.1"
scalaVersion := "2.11.11"
unmanagedJars in Compile += file("localDep1.jar")
unmanagedJars in Compile += file("localDep2.jar")
libraryDependencies += "io.spray" %% "spray-json" % "1.3.3"
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.5.6"
libraryDependencies += "com.typesafe.akka" %% "akka-stream" % "2.5.6"
libraryDependencies += "com.typesafe.akka" %% "akka-testkit" % "2.5.6" % Test
libraryDependencies += "com.typesafe.akka" %% "akka-stream-testkit" % "2.5.6" % Test
libraryDependencies += "com.47deg" %% "fetch" % "0.6.0"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.2.3"
I have tried reloading SBT, building from SBT outside of IDEA, removing and re-adding dependencies, and cleaning the project, with no luck.
Import GraphDSL.Implicits._:
object WorkerPoolFlow {
def apply[In, Out](
worker: Flow[In, Out, Any],
workerCount: Int):
Graph[FlowShape[In, Out], NotUsed] = {
import GraphDSL.Implicits._
GraphDSL.create() { implicit b =>
...
}
}
}

Failure retries with Finagle

I'm struggling with Finagle client retries.
For some reason, client doesn't retry failed requests in test, even though I use a custom classifier, which should mark any response code other than 200 as RetryableFailure. I have tried bulding client both with ClientBuilder and with Http.client.
I narrowed down my code and test to only the necessary bit.
Please take a look:
MyApi.scala:
import io.circe._, io.circe.generic.auto._, io.circe.jawn._, io.circe.syntax._
import com.twitter.finagle.http._
import com.twitter.finagle.Service
import com.twitter.finagle.Codec
import com.twitter.finagle.builder._
import scala.concurrent.ExecutionContext.Implicits.global
import com.twitter.conversions.time._
import com.twitter.finagle.http.service.HttpResponseClassifier
import com.twitter.finagle.service._
import com.twitter.util.Return
class MyApi(val host: String, val protocol: String = "https") {
// def request: Service[Request, Response] = {
// val clientBuilder = ClientBuilder()
// .codec(Http())
// .responseClassifier(classifier)
// .noFailureAccrual
// .requestTimeout(30.seconds)
// .hostConnectionLimit(5)
// .tcpConnectTimeout(5.seconds)
// .retryBudget(budget)
// .retries(5)
// protocol match {
// case "https" => clientBuilder.hosts(s"$host:443").tls(host).build()
// case _ => clientBuilder.hosts(host).build()
// }
// }
val budget: RetryBudget = RetryBudget(
ttl = 10.seconds,
minRetriesPerSec = 5,
percentCanRetry = 0.1
)
val classifier: ResponseClassifier = {
case ReqRep(_, Return(r: Response)) if r.statusCode == 200 =>
ResponseClass.Success
case _ => ResponseClass.RetryableFailure
}
def request: Service[Request, Response] = com.twitter.finagle.Http.client
.withResponseClassifier(classifier)
.withRetryBudget(budget)
.withRetryBackoff(Backoff.exponentialJittered(2.seconds, 32.seconds))
.newService(host)
def requestUsers = request(users(1))
val users: (Int => Request) = (n) => RequestBuilder()
.url(s"$protocol://$host/api?results=$n")
.buildGet()
}
object MyApi {
def apply(host: String, protocol: String) = new MyApi(host, protocol)
}
MyApiSpec.scala:
import com.twitter.util.Return
import com.twitter.finagle.http.service.HttpResponseClassifier
import org.scalatest._
import org.scalatest.concurrent._
import org.scalamock.proxy.ProxyMockFactory
import scala.language.postfixOps
import com.twitter.util.{Await}
import org.scalamock.scalatest.MockFactory
import com.twitter.finagle.http._
import org.scalamock.proxy.ProxyMockFactory
import io.finch._
import com.twitter.finagle.Http
import scala.io.Source
import com.twitter.io.{Reader, Buf}
import com.twitter.finagle.ListeningServer
class MyApiSpec extends FlatSpec with Matchers with MockFactory with ScalaFutures with ProxyMockFactory with BeforeAndAfter with BeforeAndAfterAll {
var server: ListeningServer = _
val ru = MyApi("localhost:1490", "http")
after {
server.close()
}
var attempts = 0
val failure: Endpoint[String] = get("api") {
println(attempts)
if (attempts > 1) {
Ok("test message")
}
else {
attempts += 1
BadGateway(new Exception("try again"))
}
}
it should "avoid network issues by retrying" in {
server = Http.server
.serve(":1490", failure.toServiceAs[Text.Plain])
val users = Await.result(ru.requestUsers).contentString
assert(users == "test message")
}
}
build.sbt:
name := "myapi"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies += "org.scalactic" %% "scalactic" % "2.2.6"
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % "test"
libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.0.2"
val circeVersion = "0.4.1"
libraryDependencies ++= Seq(
"io.circe" %% "circe-core",
"io.circe" %% "circe-generic",
"io.circe" %% "circe-parser"
).map(_ % circeVersion)
libraryDependencies ++= Seq(
"com.github.finagle" %% "finch-core" % "0.11.0-M2",
"com.github.finagle" %% "finch-circe" % "0.11.0-M2"
)
libraryDependencies += "com.typesafe" % "config" % "1.3.0"
libraryDependencies += "com.typesafe.slick" %% "slick" % "3.1.1"
libraryDependencies += "com.typesafe.slick" %% "slick-hikaricp" % "3.1.0"
libraryDependencies += "com.typesafe.slick" %% "slick-testkit" % "3.1.1" % "test"
libraryDependencies += "org.postgresql" % "postgresql" % "9.4-1206-jdbc4"
libraryDependencies += "junit" % "junit" % "4.8.1" % "test"
libraryDependencies += "org.scalamock" %% "scalamock-scalatest-support" % "3.2.2" % "test"
val scalazVersion = "7.2.4"
libraryDependencies ++= Seq(
"org.scalaz" %% "scalaz-core" % scalazVersion,
"org.scalaz" %% "scalaz-effect" % scalazVersion,
"org.scalaz" %% "scalaz-scalacheck-binding" % scalazVersion % "test"
)
scalacOptions += "-feature"
initialCommands in console := "import scalaz._, Scalaz._"
Sorry, it's so confusing. Finagle doesn't retry application-level failures (set via ResponseClassifiers) by default so you'd need to enable retries explicitly (see Retries section in the user guide).
Try building something between the lines:
import com.twitter.conversions.time._
import com.twitter.finagle.Http
import com.twitter.finagle.http.{Request, Response}
import com.twitter.finagle.util.DefaultTimer
import com.twitter.finagle.service.{Backoff, RetryFilter}
import com.twitter.finagle.stats.NullStatsReceiver
import com.twitter.util._
import com.twitter.finagle.param.HighResTimer
val twitter = Http.client.newService("twitter.com")
val shouldRetry: PartialFunction [(Request, Try[Response]), Boolean] = {
case (_, Return(rep)) => rep.status != 200
}
implicit val t = HighResTimer.Default
// 3 retries, backoff 1 second
val retry = RetryFilter(Backoff.const(1.second).take(3))(shouldRetry)
val retryTwitter = retry.andThen(twitter)

spray Collection ToResponseMarshallable

I am trying to return a List from my complete directive in spray-routing.
complete {
List("hello")
}
However, I am getting an error -
Expression of type List[String] doesn't conform to expected type ToResponseMarshallable
I am getting the same error with Seq. I see that marshallers for List and Seq are not provided by default in spray-httpx documentation. However, spray-json provides marshalling support in its DefaultJsonProtocol. I have imported spray.httpx.SprayJsonSupport._ and spray.json.DefaultJsonProtocol._ in my code, but that too hasn't helped.
Any idea how can I marshal a List/Seq using spray-json? Or will I have to write my own Marshaller?
(My scala version is 2.11.4)
Using spray 1.3.2 and spray-json 1.3.2 it should be possible.
Make sure you import (although you say you do, but double check).
import spray.httpx.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
Consider following example:
import akka.actor.{ActorSystem, Props, Actor}
import akka.io.IO
import spray.can.Http
import spray.routing.HttpService
import akka.pattern.ask
import akka.util.Timeout
import scala.concurrent.duration._
import spray.httpx.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
object Boot extends App {
implicit val system = ActorSystem("so")
val testActor = system.actorOf(Props[TestActor])
implicit val timeout = Timeout(5.seconds)
IO(Http) ? Http.Bind(testActor, interface = "0.0.0.0", port = 5000)
}
class TestActor extends Actor with HttpService {
def receive = runRoute(route)
def actorRefFactory = context
val route = get{
path("ping") {
complete(List("OK"))
}
}
}
Requesting /ping returns ["OK"] which looks okay.
Just for reference build.sbt bellow.
build.sbt
val akkaVersion = "2.3.5"
val sprayVersion = "1.3.2"
resolvers += "spray" at "http://repo.spray.io/"
scalaVersion := "2.11.5"
scalacOptions := Seq("-feature", "-unchecked", "-deprecation", "-encoding", "utf8")
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"io.spray" %% "spray-can" % sprayVersion,
"io.spray" %% "spray-routing" % sprayVersion,
"io.spray" %% "spray-client" % sprayVersion,
"io.spray" %% "spray-json" % "1.3.1"
)
API for marshalling seems to have changed since akka 2.3.5. For akka-2.4.11.2, SprayJsonSupport needs import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
1) Following is working sbt,
name := "some-project"
version := "1.0"
scalaVersion := "2.11.5"
scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8")
val akkaVersion = "2.4.5"
libraryDependencies ++= {
Seq(
"com.typesafe.akka" %% "akka-http-experimental" % akkaVersion,
"com.typesafe.akka" % "akka-http-spray-json-experimental_2.11" % akkaVersion,
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion,
"org.mongodb" % "mongo-java-driver" % "3.4.1",
"org.apache.kafka" %% "kafka" % "0.10.1.1",
"org.apache.kafka" % "kafka-clients" % "0.10.1.1",
//test
"org.scalatest" %% "scalatest" % "2.2.5" % "test",
"com.typesafe.akka" %% "akka-http-testkit" % "10.0.9" % "test",
"de.flapdoodle.embed" % "de.flapdoodle.embed.mongo" % "2.0.0" % "test"
)
}
parallelExecution in Test := false
2) And imports are,
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
object GetMusicResources {
val music_get_api =
path("music") {
get {
complete {
List("my data1")
}
}
}
}
3) Tests
Get("/music") ~> GetMusicResources.music_get_api ~> check {
val response1: Future[ByteString] = response.entity.toStrict(300.millis).map {_.data }
response1.map(_.utf8String).map { responseString =>
responseString shouldBe """["my data1"]"""
}
}
In my case, I was missing the second import in :
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._ // <= this one
In case anyone stumble into the same issue

Why does SBT compile just the last subproject?

I'm getting crazy, really :-(
Given the following SBT build file...
import sbt._
import sbt.Keys._
object BrixBuild extends Build {
lazy val buildSettings = Project.defaultSettings ++ Seq(
organization := "com.mycompany",
version := "0.1-SNAPSHOT",
scalaVersion := "2.10.0-RC3",
scalacOptions in Compile ++= Seq("-encoding", "UTF-8", "-deprecation", "-feature", "-unchecked"),
resolvers += Resolvers.typesafe,
target := file("target")
)
lazy val util = Project(
id = "brix-util",
base = file("brix-util"),
settings = buildSettings ++ Seq(
libraryDependencies ++= Dependencies.util
)
)
lazy val slick = Project(
id = "brix-slick",
base = file("brix-slick"),
settings = buildSettings ++ Seq(
libraryDependencies ++= Dependencies.slick
)
)
}
object Resolvers {
val typesafe = "Typesafe Releases" at "http://repo.typesafe.com/typesafe/releases"
}
object Dependencies {
private object Compile {
val commonsCodec = "commons-codec" % "commons-codec" % "1.7"
val slick = "com.typesafe" %% "slick" % "1.0.0-RC1"
}
private object Test {
val specs2 = "org.specs2" %% "specs2" % "1.12.3" % "test"
val slf4j = "org.slf4j" % "slf4j-nop" % "1.6.4" % "test"
val h2 = "com.h2database" % "h2" % "1.3.166" % "test"
}
val util = Seq(Compile.commonsCodec, Test.specs2, Test.slf4j)
val slick = Seq(Compile.slick, Test.specs2, Test.slf4j, Test.h2)
}
... only the last subproject get compiled. Why? Am I missing something?
Any help would be really appreciated. Tx.
You should define a "parent" project that aggregates the subprojects. For an example see https://github.com/typesafehub/scalalogging/blob/master/project/Build.scala.