I am a Scala noob reading through a parsing library, and have reached some syntax I do not understand:
def parseA[_: P] = P("a")
val Parsed.Success(value, successIndex) = parse("a", parseA(_))
I want to be able to combine these lines into one, ie
val Parsed.Success(value, successIndex) = parse("a", P("a"))
but this gives a compile error:
Error:(8, 61) overloaded method value P with alternatives:
[T](t: fastparse.P[T])(implicit name: sourcecode.Name, implicit ctx: fastparse.P[_])fastparse.P[T] <and>
=> fastparse.ParsingRun.type
cannot be applied to (String)
Error occurred in an application involving default arguments.
val Parsed.Success(value, successIndex) = parse(source, P("a"))
How should this line be written? And can you name the syntax concepts involved to maximise my learning?
_: P is the same as (implicit ctx: P[_]), that means that method is asking for an implicit parameter of type P[_] (the underscore means that it does not care for the inner type. See What are all the uses of an underscore in Scala?).
P("a") is calling this method, which requires such implicit in scope, and that is why in your second example it fails to compile, because it did not find the implicit parameter.
The features sued here are implicits, existential types & macros...
All of them are very advanced techniques. If you are just starting, I would suggest to leave them for latter.
Implicits are very important and useful, I would start from there, but first make sure you feel comfortable with "normal" Scala (whatever that means).
For the second question, I think this should work.
def program[_: P] = parse("a", P("a"))
val Parsed.Success(value, successIndex) = program
Related
Some time ago I've started using Cats and found OptionT very useful to work with Future[Option[T]] in most cases. But I faced with one drawback, to use AplicativeError I need to define type alias type FutureOption[T] = OptionT[Future, X] to matching F[_] required by AplicativeError and explicitly specify the type of my expression as FutureOption[T].
type FutureOption[T] = OptionT[Future, T] // definition to match F[_] kind
val x = OptionT.liftF(Future.failed(new Exception("error"))) : FutureOption[String] // need to specify type explicitly
x.recover {
case NonFatal(e) => "fixed"
}
If I remove type definition and explicit type specification of my expression the recover will not be available because OptionT[Future, T] don't match F[_], so it can't be converted implicitly to AplicativeErrorOps.
Unfortunately, the example below won't work because there is no recover method.
val x = OptionT.liftF(Future.failed(new Exception("error")))
x.recover {
case NonFatal(e) => "fixed"
}
Is there any way to avoid such kind of boilerplate code? At least I want to avoid specifying expression types as FutureOption[T] explicitly.
In addition to the other answer, I would like to suggest that you make sure you have -Ypartial-unification enabled for your build.
This is a fix for partial unification of type constructors. You can find a more detailed explanation about the fix here.
With partial unification enabled the code you provided in your question compiles fine. Please note that if you're using an IDE (e.g. Intellij) you might get "false negatives" (the code is underlined as incorrect and code completion doesn't work), but the scalac/sbt/gradle will compile it just fine.
Yes, there are at least two ways to cope with the type ascription.
using type lambdas (this can be intimidating):
val a: { type λ[A] = OptionT[Future, A] }#λ
using a compiler plugin like kind-projector, example usage:
val a: Lambda[A => OptionT[Future, A]]
But if you wanted to call Future's recover, you can always do:
val x = OptionT.liftF(Future.failed(new Exception("error")))
x.value.recover ...
I've looked around and found several other examples of this, but I don't really understand from those answers what's actually going on.
I'd like to understand why the following code fails to compile:
val df = readFiles(sqlContext).
withColumn("timestamp", udf(UDFs.parseDate _)($"timestamp"))
Giving the error:
Error:(29, 58) not enough arguments for method udf: (implicit evidence$2: reflect.runtime.universe.TypeTag[java.sql.Date], implicit evidence$3: reflect.runtime.universe.TypeTag[String])org.apache.spark.sql.UserDefinedFunction.
Unspecified value parameter evidence$3.
withColumn("timestamp", udf(UDFs.parseDate _)($"timestamp")).
^
Whereas this code does compile:
val parseDate = udf(UDFs.parseDate _)
val df = readFiles(sqlContext).
withColumn("timestamp", parseDate($"timestamp"))
Obviously I've found a "workaround" but I'd really like to understand:
What this error really means. The info I have found on TypeTags and ClassTags has been really difficult to understand. I don't come from a Java background, which perhaps doesn't help, but I think I should be able to grasp it…
If I can achieve what I want without a separate function definition
The error message is a bit mis-leading indeed; the reason for it is that the function udf takes an implicit parameter list but you are passing an actual parameter. Since I don't know much about spark and since the udf signature is a bit convoluted I'll try to explain what is going on with a simplified example.
In practice udf is a function that given some explicit parameters and an implicit parameter list gives you another function; let's define the following function that given a pivot of type T for which we have an implicit Ordering will give as a function that allows us to split a sequence in two, one containing elements smaller than pivot and the other containing elements that are bigger:
def buildFn[T](pivot: T)(implicit ev: Ordering[T]): Seq[T] => (Seq[T], Seq[T]) = ???
Let's leave out the implementation as it's not important. Now, if I do the following:
val elements: Seq[Int] = ???
val (small, big) = buildFn(10)(elements)
I will make the same kind of mistake that you are showing in your code, i.e. the compiler will think that I am explicitly passing elements as the implicit parameter list and this won't compile. The error message of my example will be somewhat different from the one you have because in my case the number of parameters I am mistakenly passing for the implicit parameter list matches the expected one and then the error will be about types not lining up.
Instead, if I write it as:
val elements: Seq[Int] = ???
val fn = buildFn(10)
val (small, big) = fn(elements)
In this case the compiler will correctly pass the implicit parameters to the function. I don't know of any way to circumvent this problem, unless you want to pass the actual implicit parameters explicitly but I find it quite ugly and not always practical; for reference this is what I mean:
val elements: Seq[Int] = ???
val (small, big) = buildFn(10)(implicitly[Ordering[Int]])(elements)
I am new to Scala, and I hope this question is not too basic. I couldn't find the answer to this question on the web (which might be because I don't know the relevant keywords).
I am trying to understand the following definition:
def functionName[T <: AnyRef](name: Symbol)(range: String*)(f: T => String)(implicit tag: ClassTag[T]): DiscreteAttribute[T] = {
val r = ....
new anotherFunctionName[T](name.toString, f, Some(r))
}
First , why is it defined as def functionName[...](...)(...)(...)(...)? Can't we define it as def functionName[...](..., ..., ..., ...)?
Second, how does range: String* from range: String?
Third, would it be a problem if implicit tag: ClassTag[T] did not exist?
First , why is it defined as def functionName...(...)(...)(...)? Can't we define it as def functionName[...](..., ..., ..., ...)?
One good reason to use currying is to support type inference. Consider these two functions:
def pred1[A](x: A, f: A => Boolean): Boolean = f(x)
def pred2[A](x: A)(f: A => Boolean): Boolean = f(x)
Since type information flows from left to right if you try to call pred1 like this:
pred1(1, x => x > 0)
type of the x => x > 0 cannot be determined yet and you'll get an error:
<console>:22: error: missing parameter type
pred1(1, x => x > 0)
^
To make it work you have to specify argument type of the anonymous function:
pred1(1, (x: Int) => x > 0)
pred2 from the other hand can be used without specifying argument type:
pred2(1)(x => x > 0)
or simply:
pred2(1)(_ > 0)
Second, how does range: String* from range: String?
It is a syntax for defining Repeated Parameters a.k.a varargs. Ignoring other differences it can be used only on the last position and is available as a scala.Seq (here scala.Seq[String]). Typical usage is apply method of the collections types which allows for syntax like SomeDummyCollection(1, 2, 3). For more see:
What does `:_*` (colon underscore star) do in Scala?
Scala variadic functions and Seq
Is there a difference in Scala between Seq[T] and T*?
Third, would it be a problem if implicit tag: ClassTag[T] did not exist?
As already stated by Aivean it shouldn't be the case here. ClassTags are automatically generated by the compiler and should be accessible as long as the class exists. In general case if implicit argument cannot be accessed you'll get an error:
scala> import scala.concurrent._
import scala.concurrent._
scala> val answer: Future[Int] = Future(42)
<console>:13: error: Cannot find an implicit ExecutionContext. You might pass
an (implicit ec: ExecutionContext) parameter to your method
or import scala.concurrent.ExecutionContext.Implicits.global.
val answer: Future[Int] = Future(42)
Multiple argument lists: this is called "currying", and enables you to call a function with only some of the arguments, yielding a function that takes the rest of the arguments and produces the result type (partial function application). Here is a link to Scala documentation that gives an example of using this. Further, any implicit arguments to a function must be specified together in one argument list, coming after any other argument lists. While defining functions this way is not necessary (apart from any implicit arguments), this style of function definition can sometimes make it clearer how the function is expected to be used, and/or make the syntax for partial application look more natural (f(x) rather than f(x, _)).
Arguments with an asterisk: "varargs". This syntax denotes that rather than a single argument being expected, a variable number of arguments can be passed in, which will be handled as (in this case) a Seq[String]. It is the equivalent of specifying (String... range) in Java.
the implicit ClassTag: this is often needed to ensure proper typing of the function result, where the type (T here) cannot be determined at compile time. Since Scala runs on the JVM, which does not retain type information beyond compile time, this is a work-around used in Scala to ensure information about the type(s) involved is still available at runtime.
Check currying:Methods may define multiple parameter lists. When a method is called with a fewer number of parameter lists, then this will yield a function taking the missing parameter lists as its arguments.
range:String* is the syntax for varargs
implicit TypeTag parameter in Scala is the alternative for Class<T> clazzparameter in Java. It will be always available if your class is defined in scope. Read more about type tags.
I have the following method:
def lift[P <: Product, L <: HList](params: P)(implicit hl: Generic.Aux[P, L]) = {
directive[L](_(hl to params))
}
and it perfectly works if i pass more then two arguments:
val result = lift("string", 'a', 10) // compiles
val result2 = list(true, 5) // compiles
But when i'm passing a single argument it can't resolve implicit:
val failes = lift("string")
It can't find Generic implicit for [String, Nothing], why does it work in other cases?
You're seeing the result of auto-tupling, which is a Scala (mis-)feature that causes lift(true, 5) to be parsed as lift((true, 5)) when there's no lift method with the appropriate number of values (two, in this case). The compiler won't automatically wrap a single value in a Tuple1, however—you just get a compiler error.
See for example this answer for more details about auto-tupling, and this thread for some reasons auto-tupling is a terrible thing to include in your language.
There are a couple of possible workarounds. The first would be to create an implicit conversion from values to Tuple1, as suggested in this answer. I wouldn't personally recommend this approach—every implicit conversion you introduce into your code is another mine in the minefield.
Instead I'd suggest avoiding autotupling altogether. Write out list((true, 5)) explicitly—you get a lot of extra clarity at the cost of only a couple of extra characters. Unfortunately there's no comparable literal support for Tuple1, so you have to write out lift(Tuple1("string")), but even that's not too bad, and if you really wanted you could define a new liftOne method that would do it for you.
Can anyone explain the compile error below? Interestingly, if I change the return type of the get() method to String, the code compiles just fine. Note that the thenReturn method has two overloads: a unary method and a varargs method that takes at least one argument. It seems to me that if the invocation is ambiguous here, then it would always be ambiguous.
More importantly, is there any way to resolve the ambiguity?
import org.scalatest.mock.MockitoSugar
import org.mockito.Mockito._
trait Thing {
def get(): java.lang.Object
}
new MockitoSugar {
val t = mock[Thing]
when(t.get()).thenReturn("a")
}
error: ambiguous reference to overloaded definition,
both method thenReturn in trait OngoingStubbing of type
java.lang.Object,java.lang.Object*)org.mockito.stubbing.OngoingStubbing[java.lang.Object]
and method thenReturn in trait OngoingStubbing of type
(java.lang.Object)org.mockito.stubbing.OngoingStubbing[java.lang.Object]
match argument types (java.lang.String)
when(t.get()).thenReturn("a")
Well, it is ambiguous. I suppose Java semantics allow for it, and it might merit a ticket asking for Java semantics to be applied in Scala.
The source of the ambiguitity is this: a vararg parameter may receive any number of arguments, including 0. So, when you write thenReturn("a"), do you mean to call the thenReturn which receives a single argument, or do you mean to call the thenReturn that receives one object plus a vararg, passing 0 arguments to the vararg?
Now, what this kind of thing happens, Scala tries to find which method is "more specific". Anyone interested in the details should look up that in Scala's specification, but here is the explanation of what happens in this particular case:
object t {
def f(x: AnyRef) = 1 // A
def f(x: AnyRef, xs: AnyRef*) = 2 // B
}
if you call f("foo"), both A and B
are applicable. Which one is more
specific?
it is possible to call B with parameters of type (AnyRef), so A is
as specific as B.
it is possible to call A with parameters of type (AnyRef,
Seq[AnyRef]) thanks to tuple
conversion, Tuple2[AnyRef,
Seq[AnyRef]] conforms to AnyRef. So
B is as specific as A. Since both are
as specific as the other, the
reference to f is ambiguous.
As to the "tuple conversion" thing, it is one of the most obscure syntactic sugars of Scala. If you make a call f(a, b), where a and b have types A and B, and there is no f accepting (A, B) but there is an f which accepts (Tuple2(A, B)), then the parameters (a, b) will be converted into a tuple.
For example:
scala> def f(t: Tuple2[Int, Int]) = t._1 + t._2
f: (t: (Int, Int))Int
scala> f(1,2)
res0: Int = 3
Now, there is no tuple conversion going on when thenReturn("a") is called. That is not the problem. The problem is that, given that tuple conversion is possible, neither version of thenReturn is more specific, because any parameter passed to one could be passed to the other as well.
In the specific case of Mockito, it's possible to use the alternate API methods designed for use with void methods:
doReturn("a").when(t).get()
Clunky, but it'll have to do, as Martin et al don't seem likely to compromise Scala in order to support Java's varargs.
Well, I figured out how to resolve the ambiguity (seems kind of obvious in retrospect):
when(t.get()).thenReturn("a", Array[Object](): _*)
As Andreas noted, if the ambiguous method requires a null reference rather than an empty array, you can use something like
v.overloadedMethod(arg0, null.asInstanceOf[Array[Object]]: _*)
to resolve the ambiguity.
If you look at the standard library APIs you'll see this issue handled like this:
def meth(t1: Thing): OtherThing = { ... }
def meth(t1: Thing, t2: Thing, ts: Thing*): OtherThing = { ... }
By doing this, no call (with at least one Thing parameter) is ambiguous without extra fluff like Array[Thing](): _*.
I had a similar problem using Oval (oval.sf.net) trying to call it's validate()-method.
Oval defines 2 validate() methods:
public List<ConstraintViolation> validate(final Object validatedObject)
public List<ConstraintViolation> validate(final Object validatedObject, final String... profiles)
Trying this from Scala:
validator.validate(value)
produces the following compiler-error:
both method validate in class Validator of type (x$1: Any,x$2: <repeated...>[java.lang.String])java.util.List[net.sf.oval.ConstraintViolation]
and method validate in class Validator of type (x$1: Any)java.util.List[net.sf.oval.ConstraintViolation]
match argument types (T)
var violations = validator.validate(entity);
Oval needs the varargs-parameter to be null, not an empty-array, so I finally got it to work with this:
validator.validate(value, null.asInstanceOf[Array[String]]: _*)