Scala Akka Streams project fails to compile, "value ~> is not a member of akka.streams.Outlet[In]" - scala

I am using the Akka Streams API in a Scala project, working in Intellij IDEA with the SBT plugin. I have a worker pool Flow as described here: https://doc.akka.io/docs/akka/current/scala/stream/stream-cookbook.html. Here is my code:
package streams
import akka.NotUsed
import akka.stream.scaladsl.{Balance, Flow, GraphDSL, Merge}
import akka.stream.{FlowShape, Graph}
object WorkerPoolFlow {
def apply[In, Out](
worker: Flow[In, Out, Any],
workerCount: Int):
Graph[FlowShape[In, Out], NotUsed] = {
GraphDSL.create() { implicit b =>
val balance = b.add(Balance[In](workerCount, waitForAllDownstreams = true))
val merge = b.add(Merge[Out](workerCount))
for (i <- 0 until workerCount)
balance.out(i) ~> worker.async ~> merge.in(i)
FlowShape(
balance.in,
merge.out)
}
}
}
For some reason the project is now failing to compile, giving this error: value ~> is not a member of akka.stream.Outlet[In].
It compiled fine until today. The only change I am aware of making is installing a Scala linter plugin scalafmt, and importing a few new libraries in build.sbt. Here is my build.sbt:
name := "myProject"
version := "0.1"
scalaVersion := "2.11.11"
unmanagedJars in Compile += file("localDep1.jar")
unmanagedJars in Compile += file("localDep2.jar")
libraryDependencies += "io.spray" %% "spray-json" % "1.3.3"
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.5.6"
libraryDependencies += "com.typesafe.akka" %% "akka-stream" % "2.5.6"
libraryDependencies += "com.typesafe.akka" %% "akka-testkit" % "2.5.6" % Test
libraryDependencies += "com.typesafe.akka" %% "akka-stream-testkit" % "2.5.6" % Test
libraryDependencies += "com.47deg" %% "fetch" % "0.6.0"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.2.3"
I have tried reloading SBT, building from SBT outside of IDEA, removing and re-adding dependencies, and cleaning the project, with no luck.

Import GraphDSL.Implicits._:
object WorkerPoolFlow {
def apply[In, Out](
worker: Flow[In, Out, Any],
workerCount: Int):
Graph[FlowShape[In, Out], NotUsed] = {
import GraphDSL.Implicits._
GraphDSL.create() { implicit b =>
...
}
}
}

Related

Scala Flink get java.lang.NoClassDefFoundError: scala/Product$class after using case class for customized DeserializationSchema

It work fine when using generic class.
But get java.lang.NoClassDefFoundError: scala/Product$class error after change class to case class.
Not sure is sbt packaging problem or code problem.
When I'm using:
sbt
scala: 2.11.12
java: 8
sbt assembly to package
package example
import java.util.Properties
import java.nio.charset.StandardCharsets
import org.apache.flink.api.scala._
import org.apache.flink.streaming.util.serialization.{DeserializationSchema, SerializationSchema}
import org.apache.flink.streaming.api.scala.{DataStream, StreamExecutionEnvironment}
import org.apache.flink.streaming.connectors.kafka.{FlinkKafkaConsumer, FlinkKafkaProducer}
import org.apache.flink.streaming.api.watermark.Watermark
import org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks
import org.apache.flink.api.common.typeinfo.TypeInformation
import Config._
case class Record(
id: String,
startTime: Long
) {}
class RecordDeSerializer extends DeserializationSchema[Record] with SerializationSchema[Record] {
override def serialize(record: Record): Array[Byte] = {
return "123".getBytes(StandardCharsets.UTF_8)
}
override def deserialize(b: Array[Byte]): Record = {
Record("1", 123)
}
override def isEndOfStream(record: Record): Boolean = false
override def getProducedType: TypeInformation[Record] = {
createTypeInformation[Record]
}
}
object RecordConsumer {
def main(args: Array[String]): Unit = {
val config : Properties = {
var p = new Properties()
p.setProperty("zookeeper.connect", Config.KafkaZookeeperServers)
p.setProperty("bootstrap.servers", Config.KafkaBootstrapServers)
p.setProperty("group.id", Config.KafkaGroupID)
p
}
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.enableCheckpointing(1000)
var consumer = new FlinkKafkaConsumer[Record](
Config.KafkaTopic,
new RecordDeSerializer(),
config
)
consumer.setStartFromEarliest()
val stream = env.addSource(consumer).print
env.execute("record consumer")
}
}
Error
2020-08-05 04:07:33,963 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Discarding checkpoint 1670 of job 4de8831901fa72790d0a9a973cc17dde.
java.lang.NoClassDefFoundError: scala/Product$class
...
build.SBT
First idea is that maybe version is not right.
But every thing work fine if use normal class
Here is build.sbt
ThisBuild / resolvers ++= Seq(
"Apache Development Snapshot Repository" at "https://repository.apache.org/content/repositories/snapshots/",
Resolver.mavenLocal
)
name := "deedee"
version := "0.1-SNAPSHOT"
organization := "dexterlab"
ThisBuild / scalaVersion := "2.11.8"
val flinkVersion = "1.8.2"
val flinkDependencies = Seq(
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-java" % flinkVersion % "provided",
"org.apache.flink" %% "flink-connector-kafka" % flinkVersion,
)
val thirdPartyDependencies = Seq(
"com.github.nscala-time" %% "nscala-time" % "2.24.0",
"com.typesafe.play" %% "play-json" % "2.6.14",
)
lazy val root = (project in file(".")).
settings(
libraryDependencies ++= flinkDependencies,
libraryDependencies ++= thirdPartyDependencies,
libraryDependencies += "org.scala-lang" % "scala-compiler" % scalaVersion.value,
)
assembly / mainClass := Some("dexterlab.TelecoDataConsumer")
// make run command include the provided dependencies
Compile / run := Defaults.runTask(Compile / fullClasspath,
Compile / run / mainClass,
Compile / run / runner
).evaluated
// stays inside the sbt console when we press "ctrl-c" while a Flink programme executes with "run" or "runMain"
Compile / run / fork := true
Global / cancelable := true
// exclude Scala library from assembly
assembly / assemblyOption := (assembly / assemblyOption).value.copy(includeScala = false)
autoCompilerPlugins := true
Finally success after I add this line in build.sbt
assembly / assemblyOption := (assemblu / assemblyOption).value.copy(includeScala = true)
To include scala library when running sbt assembly

spray Collection ToResponseMarshallable

I am trying to return a List from my complete directive in spray-routing.
complete {
List("hello")
}
However, I am getting an error -
Expression of type List[String] doesn't conform to expected type ToResponseMarshallable
I am getting the same error with Seq. I see that marshallers for List and Seq are not provided by default in spray-httpx documentation. However, spray-json provides marshalling support in its DefaultJsonProtocol. I have imported spray.httpx.SprayJsonSupport._ and spray.json.DefaultJsonProtocol._ in my code, but that too hasn't helped.
Any idea how can I marshal a List/Seq using spray-json? Or will I have to write my own Marshaller?
(My scala version is 2.11.4)
Using spray 1.3.2 and spray-json 1.3.2 it should be possible.
Make sure you import (although you say you do, but double check).
import spray.httpx.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
Consider following example:
import akka.actor.{ActorSystem, Props, Actor}
import akka.io.IO
import spray.can.Http
import spray.routing.HttpService
import akka.pattern.ask
import akka.util.Timeout
import scala.concurrent.duration._
import spray.httpx.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
object Boot extends App {
implicit val system = ActorSystem("so")
val testActor = system.actorOf(Props[TestActor])
implicit val timeout = Timeout(5.seconds)
IO(Http) ? Http.Bind(testActor, interface = "0.0.0.0", port = 5000)
}
class TestActor extends Actor with HttpService {
def receive = runRoute(route)
def actorRefFactory = context
val route = get{
path("ping") {
complete(List("OK"))
}
}
}
Requesting /ping returns ["OK"] which looks okay.
Just for reference build.sbt bellow.
build.sbt
val akkaVersion = "2.3.5"
val sprayVersion = "1.3.2"
resolvers += "spray" at "http://repo.spray.io/"
scalaVersion := "2.11.5"
scalacOptions := Seq("-feature", "-unchecked", "-deprecation", "-encoding", "utf8")
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"io.spray" %% "spray-can" % sprayVersion,
"io.spray" %% "spray-routing" % sprayVersion,
"io.spray" %% "spray-client" % sprayVersion,
"io.spray" %% "spray-json" % "1.3.1"
)
API for marshalling seems to have changed since akka 2.3.5. For akka-2.4.11.2, SprayJsonSupport needs import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
1) Following is working sbt,
name := "some-project"
version := "1.0"
scalaVersion := "2.11.5"
scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8")
val akkaVersion = "2.4.5"
libraryDependencies ++= {
Seq(
"com.typesafe.akka" %% "akka-http-experimental" % akkaVersion,
"com.typesafe.akka" % "akka-http-spray-json-experimental_2.11" % akkaVersion,
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion,
"org.mongodb" % "mongo-java-driver" % "3.4.1",
"org.apache.kafka" %% "kafka" % "0.10.1.1",
"org.apache.kafka" % "kafka-clients" % "0.10.1.1",
//test
"org.scalatest" %% "scalatest" % "2.2.5" % "test",
"com.typesafe.akka" %% "akka-http-testkit" % "10.0.9" % "test",
"de.flapdoodle.embed" % "de.flapdoodle.embed.mongo" % "2.0.0" % "test"
)
}
parallelExecution in Test := false
2) And imports are,
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._
object GetMusicResources {
val music_get_api =
path("music") {
get {
complete {
List("my data1")
}
}
}
}
3) Tests
Get("/music") ~> GetMusicResources.music_get_api ~> check {
val response1: Future[ByteString] = response.entity.toStrict(300.millis).map {_.data }
response1.map(_.utf8String).map { responseString =>
responseString shouldBe """["my data1"]"""
}
}
In my case, I was missing the second import in :
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport._
import spray.json.DefaultJsonProtocol._ // <= this one
In case anyone stumble into the same issue

"object github is not a member of package com" for WeKanban2 example project in Scala in action book

I'm new in Scala so started from reading "Scala in Action" book 2013 year. But I cannot build plain sbt WeKanban2 project. I use newer version scala and sbt then in example.
I get error when run sbt:
error: object github is not a member of package com import com.github.siasia.WebPlugin._
my build.properties:
//was sbt.version=0.12.0
sbt.version=0.13.7
plugins.sbt
// was libraryDependencies <+= sbtVersion(v => "com.github.siasia" %% "xsbt-eb-plugin" (v+"-0.2.11.1"))
addSbtPlugin("com.earldouglas" % "xsbt-web-plugin" % "0.7.0")
build.sbt
import com.github.siasia.WebPlugin._
name := "weKanban"
organization := "scalainaction"
version := "0.2"
//was: scalaVersion := "2.10.0"
scalaVersion := "2.11.4"
scalacOptions ++= Seq("-unchecked", "-deprecation")
resolvers ++= Seq(
"Scala-Tools Maven2 Releases Repository" at "http://scala-tools.org/repo-releases",
"Scala-Tools Maven2 Snapshots Repository" at "http://scala-tools.org/repo-snapshots"
)
libraryDependencies ++= Seq(
"org.scalaz" %% "scalaz-core" % scalazVersion,
"org.scalaz" %% "scalaz-http" % scalazVersion,
"org.eclipse.jetty" % "jetty-servlet" % jettyVersion % "container",
"org.eclipse.jetty" % "jetty-webapp" % jettyVersion % "test, container",
"org.eclipse.jetty" % "jetty-server" % jettyVersion % "container",
"com.h2database" % "h2" % "1.2.137",
"org.squeryl" % "squeryl_2.10" % "0.9.5-6"
)
seq(webSettings :_*)
build.scala
import sbt._
import Keys._
object H2TaskManager {
var process: Option[Process] = None
lazy val H2 = config("h2") extend(Compile)
val startH2 = TaskKey[Unit]("start", "Starts H2 database")
val startH2Task = startH2 in H2 <<= (fullClasspath in Compile) map { cp =>
startDatabase(cp.map(_.data).map(_.getAbsolutePath()).filter(_.contains("h2database")))}
def startDatabase(paths: Seq[String]) = {
process match {
case None =>
val cp = paths.mkString(System.getProperty("path.seperator"))
val command = "java -cp " + cp + " org.h2.tools.Server"
println("Starting Database with command: " + command)
process = Some(Process(command).run())
println("Database started ! ")
case Some(_) =>
println("H2 Database already started")
}
}
val stopH2 = TaskKey[Unit]("stop", "Stops H2 database")
val stopH2Task = stopH2 in H2 :={
process match {
case None => println("Database already stopped")
case Some(_) =>
println("Stopping database...")
process.foreach{_.destroy()}
process = None
println("Database stopped...")
}
}
}
object MainBuild extends Build {
import H2TaskManager._
lazy val scalazVersion = "6.0.3"
lazy val jettyVersion = "7.3.0.v20110203"
lazy val wekanban = Project(
"wekanban",
file(".")) settings(startH2Task, stopH2Task)}
What should I do to fix this error? What should I do to start app from Idea 14 in debug mode?
Your import
import com.github.siasia.WebPlugin._
failed to find path github and below. In your plugins.sbt you wrote:
// was libraryDependencies <+= sbtVersion(v => "com.github.siasia" %% "xsbt-eb-plugin" (v+"-0.2.11.1"))
addSbtPlugin("com.earldouglas" % "xsbt-web-plugin" % "0.7.0")
The github-path probably existed in the outcommented com.github.siasia.xsbt-eb-plugin. However the com.earldouglas.xsbt-web-plugin have only the path com.earldouglas.xwp. You should check out if any of the classes there (https://github.com/earldouglas/xsbt-web-plugin/tree/master/src/main/scala) resembles what you need to import.
Probably not related exactly to this package but similar. The library was present in sbt and everything looked ok. But I still was unable to build the project so I went to my $HOME/.ivy2/cache located the dependency directory and deleted it, then went back the $PROJECT_HOME/ and executed sbt build

java.lang.NoSuchMethodError with Scala actors

I have a simple Scala application (taken from here) which I want to test. The whole project is compiled successfully with SBT. However, when I launch the tests with sbt test I get the following error message:
Could not run test ConnectionTest:java.lang.NoSuchMethodError: akka.actor.Props$.apply(Lscala/Function0;)Lakka/actor/Props;
From the internet search, I get the impression that some of my versioning is not compatible but that is merely a guess. What may be wrong?
[Test Case]
import akka.actor.{Props, Actor, ActorSystem, ActorRef}
import akka.testkit.{TestKit, TestActorRef, ImplicitSender}
import org.scalatest.{WordSpecLike, BeforeAndAfterAll}
import org.scalatest.matchers.MustMatchers
class ConnectionTest extends TestKit(ActorSystem("ConnectionSpec"))
with WordSpecLike
with MustMatchers
with BeforeAndAfterAll {
override def afterAll() { system.shutdown() }
"Server" must {
"bind to port " in {
// create server
val server = system.actorOf(Props[Server], name = "server")
expectMsg("Bound to /127.0.0.1:8888")
}
}
}
[build.sbt]
name := "MyApp"
version := "0.2"
scalaVersion := "2.10.4"
mainClass := Some("akka.Main")
resolvers += "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
libraryDependencies +=
"com.typesafe.akka" %% "akka-actor" % "2.3.2"
libraryDependencies +=
"com.typesafe.akka" %% "akka-testkit" % "2.1.4" % "test"
libraryDependencies +=
"org.scalatest" % "scalatest_2.10" % "2.0" % "test"
You need to use the same version of Akka for testkit.
libraryDependencies +=
"com.typesafe.akka" %% "akka-testkit" % "2.1.4" % "test"
should be
libraryDependencies +=
"com.typesafe.akka" %% "akka-testkit" % "2.3.2" % "test"

How can I add jars from more than one unmanaged directory in an SBT .scala project configuration

I'm trying to get SBT to build a project that could have more than one unmanaged directory. If I had a single directory, I could easily do it like this:
unmanagedBase := file( "custom-libs" ).getAbsoluteFile
But since I have two directories with unmanaged jars, I need to be able to add them all. I have found some information in here, but still doesn't seem useful for my full .scala file build.
I have created a simple project that shows the issue in here. And below is my Build.scala file.
UPDATE
I've got some help form the sbt-users list and have been able to define the unmanaged-jars correctly, but the code still doesn't compile (but sbt show unmanaged-jars shows the files correctly).
import sbt._
import com.github.siasia._
import PluginKeys._
import Keys._
object Build extends sbt.Build {
import Dependencies._
val unmanagedListing = unmanagedJars := {
Dependencies.listUnmanaged( file(".").getAbsoluteFile )
}
lazy val myProject = Project("spray-template", file("."))
.settings(WebPlugin.webSettings: _*)
.settings(port in config("container") := 8080)
.settings(
organization := "com.example",
version := "0.9.0-RC1",
scalaVersion := "2.9.1",
scalacOptions := Seq("-deprecation", "-encoding", "utf8"),
resolvers ++= Dependencies.resolutionRepos,
libraryDependencies ++= Seq(
Compile.akkaActor,
Compile.sprayServer,
Test.specs2,
Container.jettyWebApp,
Container.akkaSlf4j,
Container.slf4j,
Container.logback
),
unmanagedListing
)
}
object Dependencies {
val resolutionRepos = Seq(
ScalaToolsSnapshots,
"Typesafe repo" at "http://repo.typesafe.com/typesafe/releases/",
"spray repo" at "http://repo.spray.cc/"
)
def listUnmanaged( base : RichFile ) : Keys.Classpath = {
val baseDirectories = (base / "custom-libs") +++ ( base / "custom-libs2" )
(baseDirectories ** "*.jar").classpath
}
object V {
val akka = "1.3"
val spray = "0.9.0-RC1"
val specs2 = "1.7.1"
val jetty = "8.1.0.v20120127"
val slf4j = "1.6.4"
val logback = "1.0.0"
}
object Compile {
val akkaActor = "se.scalablesolutions.akka" % "akka-actor" % V.akka % "compile"
val sprayServer = "cc.spray" % "spray-server" % V.spray % "compile"
}
object Test {
val specs2 = "org.specs2" %% "specs2" % V.specs2 % "test"
}
object Container {
val jettyWebApp = "org.eclipse.jetty" % "jetty-webapp" % V.jetty % "container"
val akkaSlf4j = "se.scalablesolutions.akka" % "akka-slf4j" % V.akka
val slf4j = "org.slf4j" % "slf4j-api" % V.slf4j
val logback = "ch.qos.logback" % "logback-classic" % V.logback
}
}
I just post the fragment from my build.sbt file, using sbt 0.11.x. It could probably be refactored a bit.
unmanagedJars in Compile <++= baseDirectory map { base =>
val libs = base / "lib"
val dirs = (libs / "batik") +++ (libs / "libtw") +++ (libs / "kiama")
(dirs ** "*.jar").classpath
}
You can add additional paths to the list of folders to scan for unmanaged dependencies. For example, to look in a folder called "config" in addition to "lib" for the run task, you can add the following. For the compile task change Runtime to Compile.
unmanagedClasspath in Runtime <+= (baseDirectory) map {
bd => Attributed.blank(bd / "config")
}
As answered by Eugene Vigdorchik, what made it work as the following code:
import sbt._
import com.github.siasia._
import PluginKeys._
import Keys._
object Build extends sbt.Build {
import Dependencies._
var unmanagedListing = unmanagedJars in Compile := {
Dependencies.listUnmanaged( file(".").getAbsoluteFile )
}
lazy val myProject = Project("spray-template", file("."))
.settings(WebPlugin.webSettings: _*)
.settings(port in config("container") := 8080)
.settings(
organization := "com.example",
version := "0.9.0-RC1",
scalaVersion := "2.9.1",
scalacOptions := Seq("-deprecation", "-encoding", "utf8"),
resolvers ++= Dependencies.resolutionRepos,
libraryDependencies ++= Seq(
C.akkaActor,
C.sprayServer,
Test.specs2,
Container.jettyWebApp,
Container.akkaSlf4j,
Container.slf4j,
Container.logback
),
unmanagedListing
)
}
object Dependencies {
val resolutionRepos = Seq(
ScalaToolsSnapshots,
"Typesafe repo" at "http://repo.typesafe.com/typesafe/releases/",
"spray repo" at "http://repo.spray.cc/"
)
def listUnmanaged( base : RichFile ) : Keys.Classpath = {
val baseDirectories = (base / "custom-libs") +++ ( base / "custom-libs2" )
(baseDirectories ** "*.jar").classpath
}
object V {
val akka = "1.3"
val spray = "0.9.0-RC1"
val specs2 = "1.7.1"
val jetty = "8.1.0.v20120127"
val slf4j = "1.6.4"
val logback = "1.0.0"
}
object C {
val akkaActor = "se.scalablesolutions.akka" % "akka-actor" % V.akka % "compile"
val sprayServer = "cc.spray" % "spray-server" % V.spray % "compile"
}
object Test {
val specs2 = "org.specs2" %% "specs2" % V.specs2 % "test"
}
object Container {
val jettyWebApp = "org.eclipse.jetty" % "jetty-webapp" % V.jetty % "container"
val akkaSlf4j = "se.scalablesolutions.akka" % "akka-slf4j" % V.akka
val slf4j = "org.slf4j" % "slf4j-api" % V.slf4j
val logback = "ch.qos.logback" % "logback-classic" % V.logback
}
}
Source repo with the full example available on Github.
Here's a general solution to recursive loading of unmanaged JARs (for sbt 0.13.x): https://stackoverflow.com/a/29357699/1348306