I have the following ZIO project
object MinimalCounterexample extends ZIOAppDefault {
val fetch = {
for {
result <- Client.request("http://localhost:9123/whatever")
data <- result.body.asString
_ <- zio.Console.printLine(data)
} yield ()
}
override def run = fetch.provide(
EventLoopGroup.auto(),
ChannelFactory.auto,
)
}
When I compile this I get this error:
io.netty.channel.ChannelFactory[io.netty.channel.Channel] does not conform to bound <: AnyKind
What gives? How do I fix this?
The problem is due to passing -Yno-kind-polymorphism to the scala compiler.
Removing this flag from scalacOptions in build.sbt fixes the problem.
Related
Compilation error occurs for the below snippet only when it is method level implementation and error does not occur when it is defined in main. What is the difference ? Scala version used is 2.13.5.
class FibonacciGenerator {
def generate(total: Int): List[Int] = {
val fibSeries: LazyList[Int] = LazyList.cons(1, LazyList.cons(1, fibSeries.zip(fibSeries.tail).map { t => t._1 + t._2 }))
fibSeries.take(total).toList
}
}
Same implementation does not fail below.
object Misc extends App {
val x: LazyList[Int] = LazyList.cons(1, LazyList.cons(1, x.zip(x.tail).map{t => t._1 + t._2}))
println(x)
println(x.take(10).toList)
The difference is that in the latter case you are using objects which are created lazily
An object... is created lazily when it is referenced, like a lazy val.
In the first case it should work if you declare lazy val fibSeries.
Code below is a simplified version of the real code. We "inherited" the domain model case object FutTest and case class FutTest, which we can't modify. The actual domain models are served from a Database, so I believe the Future approach is valid, but it causes problems which I don't understand.
import org.scalatest.FunSpec
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
case object FutTest {
def create(sz: Int) = { FutTest(sz) }
}
case class FutTest(size: Int)
class FutureTest extends FunSpec {
def one(v: Int): Future[FutTest] = {
Future { FutTest.create(v) }
}
def two(t: FutTest) = {
Future { FutTest.create(t.size) }
}
def compileError1: Future[FutTest] = {
one(10).map(f => two(f))
}
def compileError2: Future[FutTest] = {
for { o <- one(10) } yield (two(o))
}
}
The error messages:
[INFO] Using incremental compilation
[INFO] Compiling 7 Scala sources and 5 .. target/test-classes...
[ERROR] domain.FutureTest.scala:25: type mismatch;
found : scala.concurrent.Future[domain.FutTest]
required: domain.FutTest
[ERROR] one(10).map(f => two(f))
[ERROR] ^
[ERROR] domain/FutureTest.scala:29: type mismatch;
found : scala.concurrent.Future[domain.FutTest]
required: domain.FutTest
[ERROR] for { o <- one(10) } yield (two(o))
I tried the above code with plain Int instead of FutTest and all is fine. Why is the compiler complaining and how can we solve this without touching the existing domain.
flatMap is what you want.
one(10).flatMap(f => two(f))
or
one(10).flatMap(two)
Using for comprehension,
for { o <- one(10); t <- two(o) } yield t
One() returns a Future and two() also returns a Future so you need to flatMap instead of map. When you map to two(), your result is Future[Future[FutTest]] and needs to be flattened.
Doing
one(10).flatMap(f => two(f))
should do the trick.
According to the Scala Language Specification (ยง6.19), "An enumerator sequence always starts with a generator". Why?
I sometimes find this restriction to be a hindrance when using for-comprehensions with monads, because it means you can't do things like this:
def getFooValue(): Future[Int] = {
for {
manager = Manager.getManager() // could throw an exception
foo <- manager.makeFoo() // method call returns a Future
value = foo.getValue()
} yield value
}
Indeed, scalac rejects this with the error message '<-' expected but '=' found.
If this was valid syntax in Scala, one advantage would be that any exception thrown by Manager.getManager() would be caught by the Future monad used within the for-comprehension, and would cause it to yield a failed Future, which is what I want. The workaround of moving the call to Manager.getManager() outside the for-comprehension doesn't have this advantage:
def getFooValue(): Future[Int] = {
val manager = Manager.getManager()
for {
foo <- manager.makeFoo()
value = foo.getValue()
} yield value
}
In this case, an exception thrown by foo.getValue() will yield a failed Future (which is what I want), but an exception thrown by Manager.getManager() will be thrown back to the caller of getFooValue() (which is not what I want). Other possible ways of handling the exception are more verbose.
I find this restriction especially puzzling because in Haskell's otherwise similar do notation, there is no requirement that a do block should begin with a statement containing <-. Can anyone explain this difference between Scala and Haskell?
Here's a complete working example showing how exceptions are caught by the Future monad in for-comprehensions:
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Try, Success, Failure}
class Foo(val value: Int) {
def getValue(crash: Boolean): Int = {
if (crash) {
throw new Exception("failed to get value")
} else {
value
}
}
}
class Manager {
def makeFoo(crash: Boolean): Future[Foo] = {
if (crash) {
throw new Exception("failed to make Foo")
} else {
Future(new Foo(10))
}
}
}
object Manager {
def getManager(crash: Boolean): Manager = {
if (crash) {
throw new Exception("failed to get manager")
} else {
new Manager()
}
}
}
object Main extends App {
def getFooValue(crashGetManager: Boolean,
crashMakeFoo: Boolean,
crashGetValue: Boolean): Future[Int] = {
for {
manager <- Future(Manager.getManager(crashGetManager))
foo <- manager.makeFoo(crashMakeFoo)
value = foo.getValue(crashGetValue)
} yield value
}
def waitForValue(future: Future[Int]): Unit = {
val result = Try(Await.result(future, Duration("10 seconds")))
result match {
case Success(value) => println(s"Got value: $value")
case Failure(e) => println(s"Got error: $e")
}
}
val future1 = getFooValue(false, false, false)
waitForValue(future1)
val future2 = getFooValue(true, false, false)
waitForValue(future2)
val future3 = getFooValue(false, true, false)
waitForValue(future3)
val future4 = getFooValue(false, false, true)
waitForValue(future4)
}
Here's the output:
Got value: 10
Got error: java.lang.Exception: failed to get manager
Got error: java.lang.Exception: failed to make Foo
Got error: java.lang.Exception: failed to get value
This is a trivial example, but I'm working on a project in which we have a lot of non-trivial code that depends on this behaviour. As far as I understand, this is one of the main advantages of using Future (or Try) as a monad. What I find strange is that I have to write
manager <- Future(Manager.getManager(crashGetManager))
instead of
manager = Manager.getManager(crashGetManager)
(Edited to reflect #RexKerr's point that the monad is doing the work of catching the exceptions.)
for comprehensions do not catch exceptions. Try does, and it has the appropriate methods to participate in for-comprehensions, so you can
for {
manager <- Try { Manager.getManager() }
...
}
But then it's expecting Try all the way down unless you manually or implicitly have a way to switch container types (e.g. something that converts Try to a List).
So I'm not sure your premises are right. Any assignment you made in a for-comprehension can just be made early.
(Also, there is no point doing an assignment inside a for comprehension just to yield that exact value. Just do the computation in the yield block.)
(Also, just to illustrate that multiple types can play a role in for comprehensions so there's not a super-obvious correct answer for how to wrap an early assignment in terms of later types:
// List and Option, via implicit conversion
for {i <- List(1,2,3); j <- Option(i).filter(_ <2)} yield j
// Custom compatible types with map/flatMap
// Use :paste in the REPL to define A and B together
class A[X] { def flatMap[Y](f: X => B[Y]): A[Y] = new A[Y] }
class B[X](x: X) { def map[Y](f: X => Y): B[Y] = new B(f(x)) }
for{ i <- (new A[Int]); j <- (new B(i)) } yield j.toString
Even if you take the first type you still have the problem of whether there is a unique "bind" (way to wrap) and whether to doubly-wrap things that are already the correct type. There could be rules for all these things, but for-comprehensions are already hard enough to learn, no?)
Haskell translates the equivalent of for { manager = Manager.getManager(); ... } to the equivalent of lazy val manager = Manager.getManager(); for { ... }. This seems to work:
scala> lazy val x: Int = throw new Exception("")
x: Int = <lazy>
scala> for { y <- Future(x + 1) } yield y
res8: scala.concurrent.Future[Int] = scala.concurrent.impl.Promise$DefaultPromise#fedb05d
scala> Try(Await.result(res1, Duration("10 seconds")))
res9: scala.util.Try[Int] = Failure(java.lang.Exception: )
I think the reason this can't be done is because for-loops are syntactic sugar for flatMap and map methods (except if you are using a condition in the for-loop, in that case it's desugared with the method withFilter). When you are storing in a immutable variable, you can't use these methods. That's the reason you would be ok using Try as pointed out by Rex Kerr. In that case, you should be able to use map and flatMap methods.
I would like to troubleshoot the issue but couldn't move further. Can anyone please help
import org.apache.hadoop.mapred.lib.MultipleTextOutputFormat
class KeyBasedOutput[T >: Null, V <: AnyRef] extends MultipleTextOutputFormat[T , V] {
override def generateFileNameForKeyValue(key: T, value: V, leaf: String) = {
key.toString
}
override def generateActualKey(key: T, value: V) = {
null
}
}
val cp1 =sqlContext.sql("select * from d_prev_fact").map(t => t.mkString("\t")).map{x => val parts = x.split("\t")
val partition_key = parts(3)
val rows = parts.slice(0, parts.length).mkString("\t")
("date=" + partition_key.toString, rows.toString)}
cp1.saveAsHadoopFile(FACT_CP)
I have got an error as below and not able to debug
scala> cp1.saveAsHadoopFile(FACT_CP,classOf[String],classOf[String],classOf[KeyBasedOutput[String, String]])
java.lang.RuntimeException: java.lang.NoSuchMethodException: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$KeyBasedOutput.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at org.apache.hadoop.mapred.JobConf.getOutputFormat(JobConf.java:709)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:742)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:674)
The idea is to write the values into multiple folder based on the Key
Put KeyBasedOutput to a jar and start spark-shell --jars /path/to/the/jar
I'm not certain, but I think type erasure combined with reflection may be causing this problem for you. Try defining a non generic subclass of KeyBasedOutput that hard codes the type parameters and use that.
class StringKeyBasedOutput extends KeyBasedOutput[String, String]
I was wondering if scala had an equivalent to java's #SuppressWarnings that can be applied to a function or whatever to ignore any deprecation warnings[1] that function emits?
1: Relevant warning in my case is: method stop in class Thread is deprecated: see corresponding Javadoc for more information. I am aware of the problems with stop however there are still some cases where due to legacy code we have to use it.
No, and an enhancement request [1] for such a feature was closed as wontfix.
I agree it would be useful. I expect that the Scala core team aren't against the idea, but they have finite resources and many higher priorities.
update: this feature was eventually implemented in scala 2.13.2 release on 2020-04-22, see this answer
[1] https://issues.scala-lang.org/browse/SI-1781
EDIT: You should use #nowarn
There is a simple compiler plugin for this: silencer (a bit shameless plug)
Scala 2.13.2 provides #nowarn annotation developed on the basis of ghik's silencer, for example
import scala.annotation.nowarn
def t = { 0: #nowarn; 1 }
raises no warnings, whilst
def t = { 0; 1 }
gives
warning: a pure expression does nothing in statement position; multiline expressions might require enclosing parentheses
def t = { 0; 1 }
^
Here is how to suppress all warnings in sbt:
import sbt._
import Keys._
import KeyRanks.DTask
import xsbti.{Reporter, Problem, Position, Severity}
private lazy val compilerReporter = TaskKey[xsbti.Reporter](
"compilerReporter",
"Experimental hook to listen (or send) compilation failure messages.",
DTask
)
val ignoreWarnings = Seq(
compilerReporter in (Compile, compile) :=
new xsbti.Reporter {
private val buffer = collection.mutable.ArrayBuffer.empty[Problem]
def reset(): Unit = buffer.clear()
def hasErrors: Boolean = buffer.exists(_.severity == Severity.Error)
def hasWarnings: Boolean = buffer.exists(_.severity == Severity.Warn)
def printSummary(): Unit = {
print("\033c")
if (problems.nonEmpty) {
problems.foreach{ p =>
println("=====================================================")
println(p.position)
println(p.message)
println()
println()
}
}
}
def problems: Array[Problem] = buffer.toArray
def log(problem: Problem): Unit = {
if (problem.severity == Severity.Error) {
buffer.append(problem)
}
}
def log(pos: Position, msg: String, sev: Severity): Unit = {
log(new Problem {
def category: String = "foo"
def severity: Severity = sev
def message: String = msg
def position: Position = pos
})
}
def comment(pos: xsbti.Position, msg: String): Unit = ()
}
)