If I have a plain scala List[A] (i.e. scala.collection.immutable.List) and I have some state monad State[S,B]. I'd like to use Scalaz's specialized traverseS ala:
val stuff: scala.collection.immutable.List[Int] = List(1,2,3)
val state: State[S,A]
stuff.traverseS(i => modify { ... })
What do I need to import to convert scala.collection.immutable.List to Scalaz's List? I couldn't find searching through GitHub and I wasn't able to get it working with trying to import carte blanche. Perhaps I'm missing something obvious.
Thanks for your help.
scalaz doens't have its own list implementation. Likely you just need to import the list typeclass instances with import scalaz.std.list._ and the traverse syntax with import scalaz.syntax.traverse._
Related
I'm a scala newbie, using pyspark extensively (on DataBricks, FWIW). I'm finding that Protobuf deserialization is too slow for me in python, so I'm porting my deserialization udf to scala.
I've compiled my .proto files to scala and then a JAR using scalapb as described here
When I try to use these instructions to create a UDF like this:
import gnmi.gnmi._
import org.apache.spark.sql.{Dataset, DataFrame, functions => F}
import spark.implicits.StringToColumn
import scalapb.spark.ProtoSQL
// import scalapb.spark.ProtoSQL.implicits._
import scalapb.spark.Implicits._
val deserialize_proto_udf = ProtoSQL.udf { bytes: Array[Byte] => SubscribeResponse.parseFrom(bytes) }
I get the following error:
command-4409173194576223:9: error: could not find implicit value for evidence parameter of type frameless.TypedEncoder[Array[Byte]]
val deserialize_proto_udf = ProtoSQL.udf { bytes: Array[Byte] => SubscribeResponse.parseFrom(bytes) }
I've double checked that I'm importing the correct implicits, to no avail. I'm pretty fuzzy on implicits, evidence parameters and scala in general.
I would really appreciate it if someone would point me in the right direction. I don't even know how to start diagnosing!!!
Update
It seems like frameless doesn't include an implicit encoder for Array[Byte]???
This works:
frameless.TypedEncoder[Byte]
this does not:
frameless.TypedEncoder[Array[Byte]]
The code for frameless.TypedEncoder seems to include a generic Array encoder, but I'm not sure I'm reading it correctly.
#Dymtro, Thanks for the suggestion. That helped.
Does anyone have ideas about what is going on here?
Update
Ok, progress - this looks like a DataBricks issue. I think that the notebook does something like the following on startup:
import spark.implicits._
I'm using scalapb, which requires that you don't do that
I'm hunting for a way to disable that automatic import now, or "unimport" or "shadow" those modules after they get imported.
If spark.implicits._ are already imported then a way to "unimport" (hide or shadow them) is to create a duplicate object and import it too
object implicitShadowing extends SQLImplicits with Serializable {
protected override def _sqlContext: SQLContext = ???
}
import implicitShadowing._
Testing for case class Person(id: Long, name: String)
// no import
List(Person(1, "a")).toDS() // doesn't compile, value toDS is not a member of List[Person]
import spark.implicits._
List(Person(1, "a")).toDS() // compiles
import spark.implicits._
import implicitShadowing._
List(Person(1, "a")).toDS() // doesn't compile, value toDS is not a member of List[Person]
How to override an implicit value?
Wildcard Import, then Hide Particular Implicit?
How to override an implicit value, that is imported?
How can an implicit be unimported from the Scala repl?
Not able to hide Scala Class from Import
NullPointerException on implicit resolution
Constructing an overridable implicit
Caching the circe implicitly resolved Encoder/Decoder instances
Scala implicit def do not work if the def name is toString
Is there a workaround for this format parameter in Scala?
Please check whether this helps.
Possible problem can be that you don't want just to unimport spark.implicits._ (scalapb.spark.Implicits._), you probably want to import scalapb.spark.ProtoSQL.implicits._ too. And I don't know whether implicitShadowing._ shadow some of them too.
Another possible workaround is to resolve implicits manually and use them explicitly.
I am learning Scala here https://scastie.scala-lang.org/iRJ8VOw7TySZ4KQ5qlWkdw.
import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer
import scala.collection.mutable.SortedSet
// import scala.collection.immutable.Map // import not required to use immutable.Map
val m = Map((1 -> "a"), (2 -> "bb"))
val r = m.filter(_._1 > 1)
r // Map(2 -> bb): scala.collection.immutable.Map
println(r)
In the above codes, r is of type scala.collection.immutable.Map, even if scala.collection.immutable.Map not imported but scala.collection.mutable imported.
if import scala.collection.immutable.Map and scala.collection.mutable._, r is immutable.Map. if import scala.collection.mutable._, r is mutable.Map.
I am a little confused. Anyone can help explain it? Thanks
The first problem is that Map is not imported from scale.collection.mutable because there is a problem with your import statement.
Change
import scala.collection.mutable
to
import scala.collection.mutable._
or
import scala.collection.mutable.Map
and Map will be mutable.Map.
When you import both scale.collection.immutable.Map and scala.collection.mutable._ the immutable Map wins because it is a more specific. So the most specific import wins and in the case of a tie you will get a compiler error.
There are like five different questions in here, let's deal with them one by one.
Why import collection.mutable seemingly does nothing?
If we omit the unnecessary details from your code snippet, we obtain:
import scala.collection.mutable
val r = Map(2 -> 3)
You asked why r is still scala.collection.immutable.Map.
First, notice that
import scala.collection.mutable
is not the same as
import scala.collection.mutable._
The latter is a wildcard import (which would import mutable Map on demand), whereas the former only imports the symbol mutable. Therefore, in your code, only the symbol mutable is made available, but this does not influence the meaning of Map in any way.
Where does the scala.collection.immutable.Map come from even if nothing is imported?
The Predef is always imported implicitly (unless you deactivate it with a compiler flag). There is a type alias definition
type Map[A, +B] = collection.immutable.Map[A, B]
in Predef, and there is also the direct shortcut to the companion object of immutable.Map:
val Map: collection.immutable.Map.type
which allows you to construct maps using the Map(...)-syntax without new-keyword.
Why does import collection.immutable.Map take precedence over import collection.mutable._?
The specification states about the precedence of imports:
Definitions and declarations that are local, inherited, or made available by a package clause in the same compilation unit where the definition occurs have highest precedence.
Explicit imports have next highest precedence.
Wildcard imports have next highest precedence.
Definitions made available by a package clause not in the compilation unit where the definition occurs have lowest precedence.
Therefore, because of points 2 and 3, an explicit import
import scala.collection.mutable.Map
has precedence over the wildcard import
import scala.collection.immutable._
Why import collection.mutable._ makes Map mutable?
Now, one last thing is left to clarify: why does the mutable._ wildcard import take precedence over the definition in Predef? Here is a short example that demonstrates it:
import scala.collection.mutable._
val m = Map()
println(m.getClass)
The output is (maybe somewhat surprisingly):
class scala.collection.mutable.HashMap
The specification seems to be a bit vague on this point, at least I could not find where it says that ordinary wildcard imports overshadow the definitions in Predef. However, here it says:
Every compilation unit implicitly imports the following packages, in the given order:
the package java.lang,
the package scala, and
the object scala.Predef, unless there is an explicit top-level import that references scala.Predef.
Members of a later import in that order hide members of an earlier import.
I think the last sentence also covers the imports that come after scala.Predef that is, also the explicit wildcard imports. This is probably why import scala.collection.mutable._ overshadows the definitions in Predef.
What's going on in the REPL?
When you are experimenting with imports, you must keep in mind that the REPL evaluates every line as if it were in a new nested block. This means that, for example, the script
import scala.collection.mutable._
import scala.collection.immutable._
Map()
will result in
error: reference to Map is ambiguous
But if you type the same lines in a REPL one by one, then no error will occur, and you will get an immutable Map. This is because in the REPL, the above lines are interpreted as
import scala.collection.mutable._
{
import scala.collection.immutable._
{
Map()
}
}
so that the last import takes precedence. I'm not sure how Scastie handles it, probably it's the same there, so it is one additional complication that you have to keep in mind.
By default a Map in Scala is immutable.
You don´t import mutable.Map, so it is still immutable.
If you import it like this it should work:
import scala.collection.mutable.Map
if import scala.collection.immutable.Map and
scala.collection.mutable., r is immutable.Map. if import
scala.collection.mutable., r is mutable.Map.
The most qualified import statement wins.
By default Scala gives us immutable Map. There are two ways to create mutable Map:
**1.** Use import statement - *scala.collection.mutable._* to bring mutable map it into scope
**2.** Specify the full path to *scala.collection.mutable.Map*
With the import "import scala.collection.mutable._", you are importing the all the mutable collection available along with mutable map.
If you are adding scala.collection.immutable.Map along with scala.collection.mutable._, you are telling the scala compiler to use all mutable collection, but in case of Map, use immutable.
tl;dr use import scala.collection.mutable.Map
This is because m is an instance of scala.collection.immutable.Map (the default behavior). Similar behavior in java everything in java.lang package is implicitly imported.
You imported the package immutable, this does not automatically bring into scope scala.collection.immutable.Map to hide scala.collection.mutable.Map
For this you should import scala.collection.mutable.Map
import scala.collection.mutable
val m = Map((1 -> "a"), (2 -> "bb"))
println(m.getClass) // class scala.collection.immutable.Map$Map2
val mutableMap = mutable.Map((1->"A"), (2->"BB"))
println(mutableMap.getClass) // class scala.collection.mutable.HashMap
Alternatively
import scala.collection.mutable.Map
val m = Map((1 -> "a"), (2 -> "bb"))
println(m.getClass) // class scala.collection.mutable.HashMap
Given:
def convert[T](list: List[Either[String, T]]): Validated[NonEmptyList[String], NonEmptyList[T]] =
NonEmptyList.fromList(list)
.toRight("list is empty")
.flatMap(...
How do I flat map the NonEmptyList[Either[String, T]] so ultimately I end up with my Validated return value?
Is there anything in the cats library to account for this scenario? Or do I need to do this manually following something like: Best way to turn a Lists of Eithers into an Either of Lists?
I'd write this as follows:
import cats.data.{ NonEmptyList, Validated, ValidatedNel }
import cats.instances.list._, cats.syntax.list._
import cats.syntax.either._
import cats.syntax.option._
import cats.syntax.traverse._
def convert[T](list: List[Either[String, T]]): ValidatedNel[String, NonEmptyList[T]] =
list.traverse(_.toValidatedNel).andThen(_.toNel.toValidNel("list is empty"))
First we flip the whole thing inside out while transforming the Eithers to Validateds (with traverse and toValidatedNel), to get a ValidatedNel[String, List[T]], and then we handle the case where the result is empty (with andThen and toNel).
The andThen is probably one of the pieces you're missing—it's essentially flatMap for Validated (but without the implications and syntactic sugar baggage that flatMap brings). If you wanted you could probably pretty easily change my version to do the empty list check first, as in your sketch, but the way I've written it feels a little more natural to me.
Footnote: I have no idea why the enrichment method for Option is named toValidNel while the one for Either is toValidatedNel—I hadn't noticed this before, probably because I hadn't used them in the same line before. This seems unfortunate, especially since we're stuck with it for a while now that Cats 1.0 is out.
Another footnote: note that you'll need the -Ypartial-unification compiler option enabled for traverse to work without type parameters if you're on 2.11.
I am using scalaz7 in a project and sometimes I run into issues with imports. The simplest way get started is
import scalaz._
import Scalaz._
but sometimes this can lead to conflicts. What I have been doing until now the following slightly painful process:
work out a minimal example that needs the same imports as my actual code
copy that example in a separate project
compile it with the option -Xprint:typer to find out how the code looks after implicit resolution
import the needed implicits in the original project.
Although this works, I would like to streamline it. I see that scalaz7 has much more fine-grained imports, but I do not fully understand how they are organized. For instance, I see one can do
import scalaz.std.option._
import scalaz.std.AllInstances._
import scalaz.std.AllFunctions._
import scalaz.syntax.monad._
import scalaz.syntax.all._
import scalaz.syntax.std.boolean._
import scalaz.syntax.std.all._
and so on.
How are these sub-imports organized?
As an example, say I want to work with validations. What would I need, for instance to inject validation implicits and make the following compile?
3.fail[String]
What about making ValidationNEL[A, B] an instance of Applicative?
This blog post explains the package structure and imports a la carte in scalaz7 in detail: http://eed3si9n.com/learning-scalaz-day13
For your specific examples, for 3.failure[String] you'd need:
import scalaz.syntax.validation._
Validation already has a method ap:
scala> "hello".successNel[Int] ap ((s: String) => "x"+s).successNel[Int]
res1: scalaz.Validation[scalaz.NonEmptyList[Int],java.lang.String] = Success(xhello)
To get the <*> operator, you need this import:
import scalaz.syntax.applicative._
Then you can do:
"hello".successNel[Int] <*> ((s: String) => "x"+s).successNel[Int]
I see a couple of questions which make use of the scalaz Monad for what looks like a scala Future.
Here and here. I havent seen a satisfactory way of resolving this as an implicit type class without using a global execution context, however I feel that the import of these type classes shouldnt have to have static knowledge of the context.
Is there something I am missing here?
(Im assuming they are not using scalaz.concurrent.Future)
The ExecutionContext just needs to be implicitly available at the call site where your Monad is known to be Future. I agree there is some awkwardness surrounding potentially multiple different definitions of your type class existing in your program, but there is no need to depend on an implementation of it statically.
import scala.concurrent.Future
import scalaz._
import Scalaz._
def foo[A, T[_]: Traverse, M[_]: Monad](t: T[M[A]]): M[T[A]] =
implicitly[Traverse[T]].sequence(t)
def bar(l: List[Future[Int]])(implicit ctx: ExecutionContext): Future[List[Int]] =
foo(l)
https://github.com/scalaz/scalaz/blob/v7.1.0/core/src/main/scala/scalaz/std/Future.scala#L8