I made some first experiments with Akka Streams and Scala 3. Although I know that Akka has only experimental support for Scala 3, I am a little bit surprised that the following simple program does not work:
object HelloAkka extends App {
implicit val actorSystem: ActorSystem = ActorSystem();
implicit val executionContext: ExecutionContext = actorSystem.dispatcher
val done = Source(1 to 5)
.mapAsync(2)(n => Future {
println(s"processing item $n in thread ${Thread.currentThread().getId}")
})
.runWith(Sink.ignore)
Await.result(done, Duration.Inf)
println("stream processing finished.")
Await.result(actorSystem.terminate(), Duration.Inf)
println("main thread terminated.")
}
When I build this program with Scala 3.2.0 using the following build.sbt the program hangs when mapAsync is invoked (no future is ever executed):
ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / scalaVersion := "3.2.0" // fails
// ThisBuild / scalaVersion := "2.13.8" // works
lazy val root = (project in file("."))
.settings(
name := "hello-akka",
libraryDependencies += "com.typesafe.akka" %% "akka-stream" % "2.6.20"
)
When changing the Scala version to 2.13.8 the program behaves as expected. Also version 2.7.0-M1 of the Akka Streams library shows the same behavior.
Since mapAsync is elementary to Akka Streams programming, I suspect that I miss an obvious error.
Full running example: https://scastie.scala-lang.org/jheinzel/PHzMfGNDSmaSbssTX5YhWw.
I suspect that you're encountering weirdness with extends App which relies on DelayedInit being treated specially by the compiler. Scala 3 no longer gives DelayedInit special treatment, instead the code in the body of HelloAkka becomes part of the static initializer of the object, which means it executes before the HelloAkka object is considered to be a real object by the JVM. This means that any thread outside the thread initializing HelloAkka (i.e. the equivalent of the "main thread") will block until such time as the initializing thread completes, which in this case entails a deadlock thanks to the Awaits in the body of HelloAkka.
Moving your code into a main method (which is not typically necessary with Scastie) causes your program to run as expected.
Removing the Awaits to allow the static initializer to return causes the program to run as expected
Awaiting in an App which is intended to be compiled for Scala 3 will deadlock if some asynchronous task (in this case the Akka Stream's mapAsync stage) reads a val/var from the App.
The Scala 3 docs recommend either #main annotations or an explicit def main.
Related
The error message
`value` can only be used within a task or setting macro, such as :=, +=, ++=, Def.task, or Def.setting.
val x = version.value
^
clearly indicates how to fix the problem, for example, using :=
val x = settingKey[String]("")
x := version.value
The explanation in sbt uses macros heavily states
The value method itself is in fact a macro, one that if you invoke it
outside of the context of another macro, will result in a compile time
error, the exact error message being...
And you can see why, since sbt settings are entirely declarative, you
can’t access the value of a task from the key, it doesn’t make sense
to do that.
however I am confused what is meant by declarative nature of sbt being the reason. For example, intuitively I would think the following vanilla Scala snippet is semantically similar to sbt's
def version: String = ???
lazy val x = s"Hello $version" // ok
trait Foo {
def version: String
val x = version // ok
}
As this is legal, clearly the Scala snippet is not semantically equivalent to the sbt one. I was wondering if someone could elaborate on why value cannot be used outside macros? Is the reason purely syntactic related to macro syntax or am I missing something fundamental about sbt's nature?
As another sentence there says
Defining sbt’s task engine is done by giving sbt a series of settings, each setting declaring a task implementation. sbt then executes those settings in order. Tasks can be declared multiple times by multiple settings, the last one to execute wins.
So at the moment when the line
val x = version.value
would be executed (if it were allowed!), that whole program is still being set up and SBT doesn't know the final definition of version.
In what sense is the program "still being set up"?
SBT's order of actions is, basically (maybe missing something):
All your Scala build code is run.
It contains some settings and tasks definitions, SBT collects those when it encounters (along with ones from core, plugins, etc.)
They are topologically sorted into the task graph, deduplicated ("the last one to execute wins"), etc.
Settings are evaluated.
Now you are allowed to actually run tasks (e.g. from SBT console).
version.value is only available after step 4, but val x = version.value runs on step 1.
Would not lazy evaluation take care of that?
Well, when you write val x = ... there is no lazy evaluation. But lazy val x = ... runs on step 1 too.
The scala code is using cats and works well:
import cats.implicits._
import cats.Functor
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future
object Hello extends App {
Functor[Future].map(Future("hello"))(_ + "!")
}
But if I add this import:
import cats.instances.future._
It will report such compilation errors:
Error:(18, 10) could not find implicit value for parameter instance: cats.Functor[scala.concurrent.Future]
Functor[Future].map(Future("hello"))(_ + "!")
Why it happens, and how can I debug it to find reason? I used all kinds of ways I know, but can't find anything.
The build.sbt file is:
name := "Cats Implicit Functor of Future Compliation Error Demo"
version := "0.1"
organization := "org.my"
scalaVersion := "2.12.4"
sbtVersion := "1.0.4"
libraryDependencies ++= Seq(
"org.typelevel" %% "cats-core" % "1.0.1"
)
The object cats.implicits has the FutureInstances trait as a linear supertype. The FutureInstances has an implicit catsStdInstancesForFuture method, which produces a Monad[Future], which in turn is a Functor[Future].
On the other hand, the object cats.instances.future also mixes in FutureInstances, so it again provides an implicit method catsStdInstancesForFuture, but through another pathway.
Now the compiler has two possibilities to generate a Functor[Future]:
by invoking cats.instances.future.catsStdInstancesForFuture
by invoking cats.implicits.catsStdInstancesForFuture
Since it cannot decide which one to take, it exits with an error message.
To avoid that, don't use cats.implicits._ together with cats.instances.future._. Either omit one of the imports, or use the
`import packagename.objectname.{member1name, member2name}`
to select only those implicits that you need.
Adding "-print" to scalacOptions could help when debugging implicits:
scalacOptions ++= Seq(
...
"-print",
...
)
It will print out the desugared code with cats.implicits. and cats.instances.-pieces added everywhere. Unfortunately, it tends to produce quite a lot of noise.
The more fundamental reason why this happens is that there is no way to define higher-dimensional-cells (kind-of "homotopies") between the two (equivalent) pathways that lead to a Functor[Future]. If we had a possibility to tell the compiler that it doesn't matter which path to take, then everything would be much nicer. Since we can't do it, we have to make sure that there is always only one way to generate an implicit Functor[Future].
The problem is that the instances are imported twice, meaning scalac cannot disambiguate between them and doesn't know which one to use and then fails.
So either you use the implicits._ import or you import specific instances with instances.<datatype>._, but never both!
You can look at a more in depth look of cats imports here: https://typelevel.org/cats/typeclasses/imports.html
I'm new to Scala and SBT. I noticed an unfamiliar operator in the build.sbt of an open source project:
:=
Here are a couple examples of how it's used:
lazy val akkaApp = Project(id = "akka-app", base = file("akka-app"))
.settings(description := "Common Akka application stack: metrics, tracing, logging, and more.")
and it's used a few times in this larger code snippet:
lazy val jobServer = Project(id = "job-server", base = file("job-server"))
.settings(commonSettings)
.settings(revolverSettings)
.settings(assembly := null.asInstanceOf[File])
.settings(
description := "Spark as a Service: a RESTful job server for Apache Spark",
libraryDependencies ++= sparkDeps ++ slickDeps ++ cassandraDeps ++ securityDeps ++ coreTestDeps,
test in Test <<= (test in Test).dependsOn(packageBin in Compile in jobServerTestJar)
.dependsOn(clean in Compile in jobServerTestJar)
.dependsOn(buildPython in jobServerPython)
.dependsOn(clean in Compile in jobServerPython),
testOnly in Test <<= (testOnly in Test).dependsOn(packageBin in Compile in jobServerTestJar)
.dependsOn(clean in Compile in jobServerTestJar)
.dependsOn(buildPython in jobServerPython)
.dependsOn(clean in Compile in jobServerPython),
console in Compile <<= Defaults.consoleTask(fullClasspath in Compile, console in Compile),
fullClasspath in Compile <<= (fullClasspath in Compile).map { classpath =>
extraJarPaths ++ classpath
},
fork in Test := true
)
.settings(publishSettings)
.dependsOn(akkaApp, jobServerApi)
.disablePlugins(SbtScalariform)
My best guess is that it means "declare if not already declared".
The := has essentially nothing to do with the ordinary assignment operator =. It's not a built-in scala operator, but rather a family of methods/macros called :=. These methods (or macros) are members of classes such as SettingKey[T] (similarly for TaskKey[T] and InputKey[T]). They consume the right hand side of the key := value expression, and return instances of type Def.Setting[T] (or similarly, Tasks), where T is the type of the value represented by the key. They are usually written in infix notation. Without syntactic sugar, the invocations of these methods/macros would look as follows:
key.:=(value)
The constructed Settings and Tasks are in turn the basic building blocks of the build definition.
The important thing to understand here is that the keys on the left hand side are not some variables in a code block. Instead of merely representing a memory position in an active stack frame of a function call (as a simple variable would do), the keys on the left hand side are rather complex objects which can be inspected and passed around during the build process.
Is there any working solution for dropping into REPL console with for Scala 2.10?
This is mainly for debugging purpose - I want to pause in the middle of execution, and have a REPL console where I can inspect values and test the program's logic using complex expressions within my program at the current state of execution. Those who have programmed in Ruby might know similar function: the binding.pry.
AFAIK, Scala 2.9 and under used to have breakIf but it has been removed from the later versions. Using ILoop seems to be the new way but introduced issues due to sbt not adding scala-library to the classpath.
Several solutions such as this and this seem to offer a good workaround but my point is there must be a solution where I don't have to spend hours or even days just to make the REPL working.
In short, there's a lot more boilerplate steps involved - this is in contrast with binding.pry which is just a line of code with no additional boilerplate.
I am not aware if there's an issue introduced in executing the program as an sbt task as opposed to if running the program executable directly, but for development purpose I am currently running and testing my program using sbt task.
You could easily reimplement the breakIf method in your code. I don't think there is much cleaner way of doing that.
First you have to add a scala compiler library to your build.sbt
libraryDependencies += "org.scala-lang" % "scala-compiler" % scalaVersion.value
Once that's done you can implement breakIf
import scala.reflect.ClassTag
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter.{ILoop, NamedParam}
def breakIf[T](assertion: => Boolean, args: NamedParam*)(implicit tag: ClassTag[T]) = {
val repl = new ILoop()
repl.settings = new Settings()
repl.settings.embeddedDefaults[T]
repl.settings.Yreplsync.value = true
repl.in = repl.chooseReader(repl.settings)
repl.createInterpreter()
args.foreach(p => repl.bind(p.name, p.tpe, p.value))
repl.loop()
repl.closeInterpreter()
}
I think it's pretty straight forward, the only tricky part is that you have to set-up the classpath properly. You need to call embeddedDefaults with a class from your project (see my answer to another question).
You can use the new breakIf as follows:
val x = 10
breakIf[X](assertion = true, NamedParam("x", "Int", x))
Where X is just some of your classes.
I don't know if this answers your question, because it's hard to measure what is easy and what is hard.
Additionally just as a side note - if you want to use it for debugging purposes, why not use a debugger. I guess most of the debuggers can connect to a program, stop at a breakpoint and evaluate expressions in that context.
Edit
Seems like it doesn't work on current release of Scala 2.10, the working code seems to be:
import scala.reflect.ClassTag
import scala.tools.nsc.Settings
import scala.tools.nsc.interpreter.{ILoop, NamedParam}
def breakIf[T](assertion: => Boolean, args: NamedParam*)(implicit tag: ClassTag[T]) = {
val repl = new ILoop() {
override protected def postInitialization(): Unit = {
addThunk(args.foreach(p => intp.bind(p)))
super.postInitialization()
}
}
val settings = new Settings()
settings.Yreplsync.value = true
settings.usejavacp.value = true
settings.embeddedDefaults[T]
args.foreach(repl.intp.rebind)
repl.process(settings)
}
and usage is like
val x = 10
breakIf[X](assertion = true, NamedParam("x", x))
I was looking at this recently and found Ammonite to be a sufficient solution for my needs.
Add Ammonite to your library dependencies:
libraryDependencies += "com.lihaoyi" % "ammonite" % "1.6.0" cross CrossVersion.full
Invoke Ammonite where you want to drop to the REPL shell:
ammonite.Main().run()
Note that you have to pass any variables you want bound inside of run, e.g. run("var1" -> var1). Have a look at their example - Instantiating Ammonite.
Scopes matters in sbt. And I'm completely OK with it. But there are also delegating rules that allows you build a hierarchical structure of settings. I'd like to use it to bring extra settings to more specific rules.
import sbt._
import Keys._
object TestBuild extends Build {
val sourceExample = settingKey[Seq[String]]("example source for setting dependency")
val targetExample = settingKey[Seq[String]]("example of a dependent setting")
override lazy val settings = super.settings ++ Seq (
sourceExample := Seq("base"),
targetExample := "extended" +: sourceExample.value,
sourceExample in Test += "testing"
)
}
The example gives me unexpected output:
> show compile:sourceExample
[info] List(base)
> show test:sourceExample
[info] List(base, testing)
> show compile:targetExample
[info] List(extended, base)
> show test:targetExample
[info] List(extended, base)
I expect test:targetExample be List(extended, base, testing) not List(extended, base). Once I've get the result I immediately figure out why exactly it works as shown. test:targetExample delegates from *:targetExample the calculated value but not the rule for calculating it in nested scope.
This behavior brings two difficulties for me writing my own plugin. I have extra work to define same rules in every scope as a plugin developer. And I have to memorize scope definitions of internal tasks to use it correctly as user.
How can I overcome this inconvenience? I'd like to introduce settings in call-by-name semantic instead of call-by-value. What tricks may work for it?
P.S. libraryDependencies in Test looks much more concise that using % test.
I should make clear that I perfectly understand that the sbt derives values just as it is described in documentation. It works as the creator intended it to work.
But why should I obey to the rules? I see them completely counter-intuitive. Sbt introduces inheritance semantic that actually works unlike how inheritance used to be defined. When you write
trait A { lazy val x : Int = 5 }
trait B extends A { lazy val y : Int = x * 2}
trait C extends A { override lazy val x : Int = 3 }
you expect (new B with C).y be 6, not 10. Knowing that it would be actually 10 allows you to use this kind of inheritance correctly but leaves your with desire to find more conventional means for implementing inheritance. You may even write your own implementation based on name->value dictionary. And you may proceed further according to the tenth rule of programming.
So I'm searching for a hack that would bring inheritance semantic in accordance with common one. As a start point I may suggest writing command to scan all settings and push them from parents to children explicitly. And than invoke this command automatically each time sbt runs.
But it seems too dirty for me, so I'm curios if there is more graceful way to achieve similar semantic.
The reason for the "incorrect" value is that targetExample depends on sourceExample in Compile scope as in:
targetExample := "extended" +: sourceExample.value
Should it use sourceExample value from Test scope, use in Test to be explicit about your wish as follows:
targetExample := "extended" +: (sourceExample in Test).value
Use inspect to know the dependency chain.
BTW, I strongly advise using build.sbt file for such a simple build definition.
You could have default settings, and reuse it in different configurations as described in Plugins Best Practices - Playing nice with configurations. I believe, this should give you a semantic similar to what you're looking for.
You can define your base settings and reuse it in different configurations.
import sbt._
import Keys._
object TestBuild extends Build {
val sourceExample = settingKey[Seq[String]]("example source for setting dependency")
val targetExample = settingKey[Seq[String]]("example of a dependent setting")
override lazy val settings = super.settings ++
inConfig(Compile)(basePluginSettings) ++
inConfig(Test)(basePluginSettings ++ Seq(
sourceExample += "testing" // we are already "in Test" here
))
lazy val basePluginSettings: Seq[Setting[_]] = Seq (
sourceExample := Seq("base"),
targetExample := "extended" +: sourceExample.value
)
}
PS. Since you're talking about writing your plugin, you may also want to look at the new way of writing sbt plugins, namely AutoPlugin, as the old mechanism is now deprecated.