Scala type programming resources - scala

According to this question, Scala's type system is Turing complete. What resources are available that enable a newcomer to take advantage of the power of type-level programming?
Here are the resources I've found so far:
Daniel Spiewak's High Wizardry in the Land of Scala
Apocalisp's Type-Level Programming in Scala
Jesper's HList
These resources are great, but I feel like I'm missing the basics, and so do not have a solid foundation on which to build. For instance, where is there an introduction to type definitions? What operations can I perform on types?
Are there any good introductory resources?

Overview
Type-level programming has many similarities with traditional, value-level programming. However, unlike value-level programming, where the computation occurs at runtime, in type-level programming, the computation occurs at compile time. I will try to draw parallels between programming at the value-level and programming at the type-level.
Paradigms
There are two main paradigms in type-level programming: "object-oriented" and "functional". Most examples linked to from here follow the object-oriented paradigm.
A good, fairly simple example of type-level programming in the object-oriented paradigm can be found in apocalisp's implementation of the lambda calculus, replicated here:
// Abstract trait
trait Lambda {
type subst[U <: Lambda] <: Lambda
type apply[U <: Lambda] <: Lambda
type eval <: Lambda
}
// Implementations
trait App[S <: Lambda, T <: Lambda] extends Lambda {
type subst[U <: Lambda] = App[S#subst[U], T#subst[U]]
type apply[U] = Nothing
type eval = S#eval#apply[T]
}
trait Lam[T <: Lambda] extends Lambda {
type subst[U <: Lambda] = Lam[T]
type apply[U <: Lambda] = T#subst[U]#eval
type eval = Lam[T]
}
trait X extends Lambda {
type subst[U <: Lambda] = U
type apply[U] = Lambda
type eval = X
}
As can be seen in the example, the object-oriented paradigm for type-level programming proceeds as follows:
First: define an abstract trait with various abstract type fields (see below for what an abstract field is). This is a template for guaranteeing that certain types fields exist in all implementations without forcing an implementation. In the lambda calculus example, this corresponds to trait Lambda that guarantees that the following types exist: subst, apply, and eval.
Next: define subtraits that extend the abstract trait and implement the various abstract type fields
Often, these subtraits will be parameterized with arguments. In the lambda calculus example, the subtypes are trait App extends Lambda which is parameterized with two types (S and T, both must be subtypes of Lambda), trait Lam extends Lambda parameterized with one type (T), and trait X extends Lambda (which is not parameterized).
the type fields are often implemented by referring to the type parameters of the subtrait and sometimes referencing their type fields via the hash operator: # (which is very similar to the dot operator: . for values). In trait App of the lambda calculus example, the type eval is implemented as follows: type eval = S#eval#apply[T]. This is essentially calling the eval type of the trait's parameter S, and calling apply with parameter T on the result. Note, S is guaranteed to have an eval type because the parameter specifies it to be a subtype of Lambda. Similarly, the result of eval must have an apply type, since it is specified to be a subtype of Lambda, as specified in the abstract trait Lambda.
The Functional paradigm consists of defining lots of parameterized type constructors that are not grouped together in traits.
Comparison between value-level programming and type-level programming
abstract class
value-level: abstract class C { val x }
type-level: trait C { type X }
path dependent types
C.x (referencing field value/function x in object C)
C#x (referencing field type x in trait C)
function signature (no implementation)
value-level: def f(x:X) : Y
type-level: type f[x <: X] <: Y (this is called a "type constructor" and usually occurs in the abstract trait)
function implementation
value-level: def f(x:X) : Y = x
type-level: type f[x <: X] = x
conditionals
see here
checking equality
value-level: a:A == b:B
type-level: implicitly[A =:= B]
value-level: Happens in the JVM via a unit test at runtime (i.e. no runtime errors):
in essense is an assert: assert(a == b)
type-level: Happens in the compiler via a typecheck (i.e. no compiler errors):
in essence is a type comparison: e.g. implicitly[A =:= B]
A <:< B, compiles only if A is a subtype of B
A =:= B, compiles only if A is a subtype of B and B is a subtype of A
A <%< B, ("viewable as") compiles only if A is viewable as B (i.e. there is an implicit conversion from A to a subtype of B)
an example
more comparison operators
Converting between types and values
In many of the examples, types defined via traits are often both abstract and sealed, and therefore can neither be instantiated directly nor via anonymous subclass. So it is common to use null as a placeholder value when doing a value-level computation using some type of interest:
e.g. val x:A = null, where A is the type you care about
Due to type-erasure, parameterized types all look the same. Furthermore, (as mentioned above) the values you're working with tend to all be null, and so conditioning on the object type (e.g. via a match statement) is ineffective.
The trick is to use implicit functions and values. The base case is usually an implicit value and the recursive case is usually an implicit function. Indeed, type-level programming makes heavy use of implicits.
Consider this example (taken from metascala and apocalisp):
sealed trait Nat
sealed trait _0 extends Nat
sealed trait Succ[N <: Nat] extends Nat
Here you have a peano encoding of the natural numbers. That is, you have a type for each non-negative integer: a special type for 0, namely _0; and each integer greater than zero has a type of the form Succ[A], where A is the type representing a smaller integer. For instance, the type representing 2 would be: Succ[Succ[_0]] (successor applied twice to the type representing zero).
We can alias various natural numbers for more convenient reference. Example:
type _3 = Succ[Succ[Succ[_0]]]
(This is a lot like defining a val to be the result of a function.)
Now, suppose we want to define a value-level function def toInt[T <: Nat](v : T) which takes in an argument value, v, that conforms to Nat and returns an integer representing the natural number encoded in v's type. For example, if we have the value val x:_3 = null (null of type Succ[Succ[Succ[_0]]]), we would want toInt(x) to return 3.
To implement toInt, we're going to make use of the following class:
class TypeToValue[T, VT](value : VT) { def getValue() = value }
As we will see below, there will be an object constructed from class TypeToValue for each Nat from _0 up to (e.g.) _3, and each will store the value representation of the corresponding type (i.e. TypeToValue[_0, Int] will store the value 0, TypeToValue[Succ[_0], Int] will store the value 1, etc.). Note, TypeToValue is parameterized by two types: T and VT. T corresponds to the type we're trying to assign values to (in our example, Nat) and VT corresponds to the type of value we're assigning to it (in our example, Int).
Now we make the following two implicit definitions:
implicit val _0ToInt = new TypeToValue[_0, Int](0)
implicit def succToInt[P <: Nat](implicit v : TypeToValue[P, Int]) =
new TypeToValue[Succ[P], Int](1 + v.getValue())
And we implement toInt as follows:
def toInt[T <: Nat](v : T)(implicit ttv : TypeToValue[T, Int]) : Int = ttv.getValue()
To understand how toInt works, let's consider what it does on a couple of inputs:
val z:_0 = null
val y:Succ[_0] = null
When we call toInt(z), the compiler looks for an implicit argument ttv of type TypeToValue[_0, Int] (since z is of type _0). It finds the object _0ToInt, it calls the getValue method of this object and gets back 0. The important point to note is that we did not specify to the program which object to use, the compiler found it implicitly.
Now let's consider toInt(y). This time, the compiler looks for an implicit argument ttv of type TypeToValue[Succ[_0], Int] (since y is of type Succ[_0]). It finds the function succToInt, which can return an object of the appropriate type (TypeToValue[Succ[_0], Int]) and evaluates it. This function itself takes an implicit argument (v) of type TypeToValue[_0, Int] (that is, a TypeToValue where the first type parameter is has one fewer Succ[_]). The compiler supplies _0ToInt (as was done in the evaluation of toInt(z) above), and succToInt constructs a new TypeToValue object with value 1. Again, it is important to note that the compiler is providing all of these values implicitly, since we do not have access to them explicitly.
Checking your work
There are several ways to verify that your type-level computations are doing what you expect. Here are a few approaches. Make two types A and B, that you want to verify are equal. Then check that the following compile:
Equal[A, B]
with: trait Equal[T1 >: T2 <: T2, T2] (taken from apocolisp)
implicitly[A =:= B]
Alternatively, you can convert the type to a value (as shown above) and do a runtime check of the values. E.g. assert(toInt(a) == toInt(b)), where a is of type A and b is of type B.
Additional Resources
The complete set of available constructs can be found in the types section of the scala reference manual (pdf).
Adriaan Moors has several academic papers about type constructors and related topics with examples from scala:
Generics of a higher kind (pdf)
Type Constructor Polymorphism for Scala: Theory and Practice (pdf) (PhD thesis, which includes the previous paper by Moors)
Type Constructor Polymorphism Inference
Apocalisp is a blog with many examples of type-level programming in scala.
Type-Level Programming in Scala is a fantastic guided tour of some type-level programming which includes booleans, natural numbers (as above), binary numbers, heterogeneous lists, and more.
More Scala Typehackery is the lambda calculus implementation above.
ScalaZ is a very active project that is providing functionality that extends the Scala API using various type-level programming features. It is a very interesting project that has a big following.
MetaScala is a type-level library for Scala, including meta types for natural numbers, booleans, units, HList, etc. It is a project by Jesper Nordenberg (his blog).
The Michid (blog) has some awesome examples of type-level programming in Scala (from other answer):
Meta-Programming with Scala Part I: Addition
Meta-Programming with Scala Part II: Multiplication
Meta-Programming with Scala Part III: Partial function application
Meta-Programming with Scala: Conditional Compilation and Loop Unrolling
Scala type level encoding of the SKI calculus
Debasish Ghosh (blog) has some relevant posts as well:
Higher order abstractions in scala
Static typing gives you a head start
Scala implicits type classes, here I come
Refactoring into scala type-classes
Using generalized type constraints
How scalas type system words for you
Choosing between abstract type members
(I've been doing some research on this subject and here's what I've learned. I'm still new to it, so please point out any inaccuracies in this answer.)

In addition to the other links here, there are also my blog posts on type level meta programming in Scala:
Meta-Programming with Scala Part I: Addition
Meta-Programming with Scala Part II: Multiplication
Meta-Programming with Scala Part III: Partial function application
Meta-Programming with Scala: Conditional Compilation and Loop Unrolling
Scala type level encoding of the SKI calculus

As suggested on Twitter: Shapeless: An exploration of generic/polytypic programming in Scala by Miles Sabin.

Sing, a type-level metaprogramming library in Scala.
The Beginning of Type-level Programming in Scala

Scalaz has source code, a wiki and examples.

Related

Why Scala Infer the Bottom Type when the type parameter is not specified?

I wonder if anyone could explain the inferencing rule in this particular case below, and most importantly it's rational/implication ?
case class E[A, B](a: A) // class E
E(2) // E[Int,Nothing] = E(2)
Note that I could have wrote E[Int](2). What matter to me is why is the second parameter type inferred to be Nothing (i.e. Bottom type) instead of let say Any for instance ? Why is that and What's the rational/Implication ?
Just to give some context, this is related to the definition of Either and how it works for Left and Right. Both are defined according to the pattern
final case class X[+A, +B](value: A) extends Either[A, B]
Where you instantiate it let say as Right[Int](2) and the type inferred is Right[Nothing, Int] and by extension Either[Nothing, Int]
EDIT1
There is consistency here, but i still can figure out the rational. Below is the same definition with a contra-variant paramete:
case class E[A, -B](a: A)// class E
E(2) // E[Int, Any] = E(2)
Hence we do have the same thing the other way around when it is contra-variant, and that make the all behavior or inference rule, coherent. However the rational for this i am not sure ....
Why not the opposite rule i.e. infer Any when Co-Variant/Invariant and Nothing when Contra-Variant ?
EDIT2
In the light of #slouc Answer, which make good sense, i'm left with still understanding what and why the compiler is doing what it is doing. The example below illustrate my confusion
val myleft = Left("Error") // Left[String,Nothing] = Left(Error)
myleft map { (e:Int) => e * 4} // Either[String,Int] = Left(Error)
First the compiler fix the type to something that "for sure work" to reuse the conclusion of #slouc (albeit make more sense in the context of a Function) Left[String,Nothing]
Next the compile infer myleft to be of type Either[String,Int]
given map definition def map[B](f: A => B): Either[E, B], (e:Int) => e * 4 can only be supplied if myleft is actually Left[String,Int] or Either[String,Int]
So in other words, my question is, what is the point of fixing the type to Nothing if it is to change it later.
Indeed the following does not compile
val aleft: Left[String, Nothing] = Left[String, Int]("Error")
type mismatch;
found : scala.util.Left[String,Int]
required: Left[String,Nothing]
val aleft: Left[String, Nothing] = Left[String, Int]("Error")
So why would I infer to a type, that normally would block me to do anything else over variable of that type (but for sure works in term of inference), to ultimately change that type, so i can do something with a variable of that inferred type.
EDIT3
Edit2 is a bit misunderstanding and everything is clarified in #slouc answer and comments.
Covariance:
Given type F[+A] and relation A <: B, then the following holds: F[A] <: F[B]
Contravariance:
Given type F[-A] and relation A <: B, then the following holds: F[A] >: F[B]
If the compiler cannot infer the exact type, it will resolve the lowest possible type in case of covariance and highest possible type in case of contravariance.
Why?
This is a very important rule when it comes to variance in subtyping. It can be shown on the example of the following data type from Scala:
trait Function1[Input-, Output+]
Generally speaking, when a type is placed in the function/method parameters, it means it's in the so-called "contravariant position". If it's used in function/method return values, it's in the so-called "covariant position". If it's in both, then it's invariant.
Now, given the rules from the beginning of this post, we conclude that, given:
trait Food
trait Fruit extends Food
trait Apple extends Fruit
def foo(someFunction: Fruit => Fruit) = ???
we can supply
val f: Food => Apple = ???
foo(f)
Function f is a valid substitute for someFunction because:
Food is a supertype of Fruit (contravariance of input)
Apple is a subtype of Fruit (covariance of output)
We can explain this in natural language like this:
"Method foo needs a function that can take a Fruit and produce a
Fruit. This means foo will have some Fruit and will need a
function it can feed it to, and expect some Fruit back. If it gets a
function Food => Apple, everything is fine - it can still feed it
Fruit (because the function takes any food), and it can receive
Fruit (apples are fruit, so the contract is respected).
Coming back to your initial dilemma, hopefully this explains why, without any extra information, compiler will resort to lowest possible type for covariant types and highest possible type for contravariant ones. If we want to supply a function to foo, there's one that we know surely works: Any => Nothing.
Variance in general.
Variance in Scala documentation.
Article about variance in Scala (full disclosure: I wrote it).
EDIT:
I think I know what's confusing you.
When you instantiate a Left[String, Nothing], you're allowed to later map it with a function Int => Whatever, or String => Whatever, or Any => Whatever. This is precisly because of the contravariance of function input explained earlier. That's why your map works.
"what is the point of fixing the type to Nothing if it is to change it
later?"
I think it's a bit hard to wrap your head around compiler fixing the unknown type to Nothing in case of contravariance. When it fixes the unknown type to Any in case of covariance, it feels more natural (it can be "Anything"). Because of the duality of covariance and contravariance explained earlier, same reasoning applies for contravariant Nothing and covariant Any.
This is a quote from
Unification of Compile-Time and Runtime Metaprogramming in Scala
by Eugene Burmako
https://infoscience.epfl.ch/record/226166 (p. 95-96)
During type inference, the typechecker collects constraints on missing
type arguments from bounds of type parameters, from types of term
arguments, and even from results of implicit search (type inference
works together with implicit search because Scala supports an analogue
of functional dependencies). One can view these constraints as a
system of inequalities where unknown type arguments are represented as
type variables and order is imposed by the subtyping relation.
After collecting constraints, the typechecker starts a step-by-step
process that, on each step, tries to apply a certain transformation to
inequalities, creating an equivalent, yet supposedly simpler system of
inequalities. The goal of type inference is to transform the original
inequalities to equalities that represent a unique solution of the
original system.
Most of the time, type inference succeeds. In that
case, missing type arguments are inferred to the types represented by
the solution.
However, sometimes type inference fails. For example,
when a type parameter T is phantom, i.e. unused in the term parameters
of the method, its only entry in the system of inequalities will be
L <: T <: U, where L and U are its lower and upper bound respectively.
If L != U, this inequality does not have a unique solution, and that
means a failure of type inference.
When type inference fails, i.e.
when it is unable to take any more transformation steps and its
working state still contains some inequalities, the typechecker breaks
the stalemate. It takes all yet uninferred type arguments, i.e. those
whose variables are still represented by inequalities, and forcibly
minimizes them, i.e. equates them to their lower bounds. This produces
a result where some type arguments are inferred precisely, and some
are replaced with seemingly arbitrary types. For instance,
unconstrained type parameters are inferred to Nothing, which is a
common source of confusion for Scala beginners.
You can learn more about type inference in Scala:
Hubert Plociniczak Decrypting Local Type Inference https://infoscience.epfl.ch/record/214757
Guillaume Martres Scala 3, Type Inference and You! https://www.youtube.com/watch?v=lMvOykNQ4zs
Guillaume Martres Dotty and types: the story so far https://www.youtube.com/watch?v=YIQjfCKDR5A
Slides http://guillaume.martres.me/talks/
Aleksander Boruch-Gruszecki GADTs in Dotty https://www.youtube.com/watch?v=VV9lPg3fNl8

What is a kind projector

I've been digging into FP and everything that surrounds it, and I found the concept of kind projector written somewhere, without details nor explanations.
The only thing I found was this github project, and I'm starting to think if it was referring to this particular project, or to some generic concept in FP?
So, what is a kind projector? Why is it useful? (if possible, can you provide examples, resources, etc?)
This is indeed just a slightly awkward name for the specific plugin for the Scala compiler you linked to. I don't think it has any significance to itself, but it kind of fits its purpose.
What the plugin does is to provide an alternative syntax to Scala's usual workaround for type lambdas, which uses a language feature called type projections.
Say you wanted to implement Functor for Either. Now, Functor requires kind * -> *, whereas Either has kind * -> * -> *. So we need to first fix the first argument, and can then provide the implementation for the partially applied type constructor. The only way you can do this in "regular" Scala is this:
implicit def eitherIsFunctor[A]: Functor[{type λ[X] = Either[A, X]}#λ] = { ... }
where {type λ[X] = Either[A, X]} is an anonymous structural type, which is only immediately used to "project out" λ, the type we actually want. In Haskell, you could just say
instance Functor (Either a) where ...
where Either is partially applied (and a is quantified over automatically).
The plugin allows one to replace the projection with something that looks more like a usual partial application in Scala, namely Either[A, ?], instead of the hardly understandable {type λ[X] = Either[A, X]}#λ (and also provides general type lambdas, I think, always by converting them down to anonymous types and projections).
Scala 3 provides native type lambdas which are no longer based on type projection
A type lambda such as [X] =>> F[X] defines a function from types to
types.
For example,
trait Functor[F[_]]
new Functor[Either[String, Int]] {} // error
new Functor[({ type λ[X] = Either[String, X] })#λ] {} // Scala 2 type lambda based on type projection
new Functor[λ[X => Either[String, X]]] {} // kind projector type lambda
new Functor[Either[String, *]] {} // kind projector type lambda
new Functor[[X] =>> Either[String, X]] {} // Scala 3 type lambda
Also, there exists a proposal SIP: Underscore Syntax for Type Lambdas #5379 such that
Functor[Either[String, _]] // equivalent to Functor[[X] =>> Either[String, X]]

Scala: Function definition of PartialFunction is ambiguous

Function definition of the PartialFunction is following:
trait PartialFunction[-A, +B] extends (A) ⇒ B
PartialFunction would allow us to filter by using case with collect on collection.
For example when you have list of integers and PartialFunction isEven[Int, String] which will convert to String if the value in the collection is even number. So it returns a new collection with the return type +Bin the definition.
My question is, why contravariant -A and covariance +B? It basically has ability to accept any input to any output. Why do we need to indicate the input should be a any type or super type of a type A and return type should be B or its subclass? Can't we just say:
trait PartialFunction[A, B]
No, it does not "basically allow it to accept any input to any output". Co- and contravariance are not trivial (in the mathematical sense) relations, they are strictly defined by the bound type - as you probably realize.
This specific form of type bounds, i.e. contravariance of arguments and covariance of the return type is a general phenomenon, not limited to Scala, and comes from the notion of function types from formal type theory, specifically:
If T1 → T2 is a function type then a subtype of it is any function S1 → S2 with the property that T1 <: S1 and S2 <: T2.
allowing for a well-defined subtyping relation of said function types.

Any reason why scala does not explicitly support dependent types?

There are path dependent types and I think it is possible to express almost all the features of such languages as Epigram or Agda in Scala, but I'm wondering why Scala does not support this more explicitly like it does very nicely in other areas (say, DSLs) ?
Anything I'm missing like "it is not necessary" ?
Syntactic convenience aside, the combination of singleton types, path-dependent types and implicit values means that Scala has surprisingly good support for dependent typing, as I've tried to demonstrate in shapeless.
Scala's intrinsic support for dependent types is via path-dependent types. These allow a type to depend on a selector path through an object- (ie. value-) graph like so,
scala> class Foo { class Bar }
defined class Foo
scala> val foo1 = new Foo
foo1: Foo = Foo#24bc0658
scala> val foo2 = new Foo
foo2: Foo = Foo#6f7f757
scala> implicitly[foo1.Bar =:= foo1.Bar] // OK: equal types
res0: =:=[foo1.Bar,foo1.Bar] = <function1>
scala> implicitly[foo1.Bar =:= foo2.Bar] // Not OK: unequal types
<console>:11: error: Cannot prove that foo1.Bar =:= foo2.Bar.
implicitly[foo1.Bar =:= foo2.Bar]
In my view, the above should be enough to answer the question "Is Scala a dependently typed language?" in the positive: it's clear that here we have types which are distinguished by the values which are their prefixes.
However, it's often objected that Scala isn't a "fully" dependently type language because it doesn't have dependent sum and product types as found in Agda or Coq or Idris as intrinsics. I think this reflects a fixation on form over fundamentals to some extent, nevertheless, I'll try and show that Scala is a lot closer to these other languages than is typically acknowledged.
Despite the terminology, dependent sum types (also known as Sigma types) are simply a pair of values where the type of the second value is dependent on the first value. This is directly representable in Scala,
scala> trait Sigma {
| val foo: Foo
| val bar: foo.Bar
| }
defined trait Sigma
scala> val sigma = new Sigma {
| val foo = foo1
| val bar = new foo.Bar
| }
sigma: java.lang.Object with Sigma{val bar: this.foo.Bar} = $anon$1#e3fabd8
and in fact, this is a crucial part of the encoding of dependent method types which is needed to escape from the 'Bakery of Doom' in Scala prior to 2.10 (or earlier via the experimental -Ydependent-method types Scala compiler option).
Dependent product types (aka Pi types) are essentially functions from values to types. They are key to the representation of statically sized vectors and the other poster children for dependently typed programming languages. We can encode Pi types in Scala using a combination of path dependent types, singleton types and implicit parameters. First we define a trait which is going to represent a function from a value of type T to a type U,
scala> trait Pi[T] { type U }
defined trait Pi
We can than define a polymorphic method which uses this type,
scala> def depList[T](t: T)(implicit pi: Pi[T]): List[pi.U] = Nil
depList: [T](t: T)(implicit pi: Pi[T])List[pi.U]
(note the use of the path-dependent type pi.U in the result type List[pi.U]). Given a value of type T, this function will return a(n empty) list of values of the type corresponding to that particular T value.
Now let's define some suitable values and implicit witnesses for the functional relationships we want to hold,
scala> object Foo
defined module Foo
scala> object Bar
defined module Bar
scala> implicit val fooInt = new Pi[Foo.type] { type U = Int }
fooInt: java.lang.Object with Pi[Foo.type]{type U = Int} = $anon$1#60681a11
scala> implicit val barString = new Pi[Bar.type] { type U = String }
barString: java.lang.Object with Pi[Bar.type]{type U = String} = $anon$1#187602ae
And now here is our Pi-type-using function in action,
scala> depList(Foo)
res2: List[fooInt.U] = List()
scala> depList(Bar)
res3: List[barString.U] = List()
scala> implicitly[res2.type <:< List[Int]]
res4: <:<[res2.type,List[Int]] = <function1>
scala> implicitly[res2.type <:< List[String]]
<console>:19: error: Cannot prove that res2.type <:< List[String].
implicitly[res2.type <:< List[String]]
^
scala> implicitly[res3.type <:< List[String]]
res6: <:<[res3.type,List[String]] = <function1>
scala> implicitly[res3.type <:< List[Int]]
<console>:19: error: Cannot prove that res3.type <:< List[Int].
implicitly[res3.type <:< List[Int]]
(note that here we use Scala's <:< subtype-witnessing operator rather than =:= because res2.type and res3.type are singleton types and hence more precise than the types we are verifying on the RHS).
In practice, however, in Scala we wouldn't start by encoding Sigma and Pi types and then proceeding from there as we would in Agda or Idris. Instead we would use path-dependent types, singleton types and implicits directly. You can find numerous examples of how this plays out in shapeless: sized types, extensible records, comprehensive HLists, scrap your boilerplate, generic Zippers etc. etc.
The only remaining objection I can see is that in the above encoding of Pi types we require the singleton types of the depended-on values to be expressible. Unfortunately in Scala this is only possible for values of reference types and not for values of non-reference types (esp. eg. Int). This is a shame, but not an intrinsic difficulty: Scala's type checker represents the singleton types of non-reference values internally, and there have been a couple of experiments in making them directly expressible. In practice we can work around the problem with a fairly standard type-level encoding of the natural numbers.
In any case, I don't think this slight domain restriction can be used as an objection to Scala's status as a dependently typed language. If it is, then the same could be said for Dependent ML (which only allows dependencies on natural number values) which would be a bizarre conclusion.
I would assume it is because (as I know from experience, having used dependent types in the Coq proof assistant, which fully supports them but still not in a very convenient way) dependent types are a very advanced programming language feature which is really hard to get right - and can cause an exponential blowup in complexity in practice. They're still a topic of computer science research.
I believe that Scala's path-dependent types can only represent Σ-types, but not Π-types. This:
trait Pi[T] { type U }
is not exactly a Π-type. By definition, Π-type, or dependent product, is a function which result type depends on argument value, representing universal quantifier, i.e. ∀x: A, B(x). In the case above, however, it depends only on type T, but not on some value of this type. Pi trait itself is a Σ-type, an existential quantifier, i.e. ∃x: A, B(x). Object's self-reference in this case is acting as quantified variable. When passed in as implicit parameter, however, it reduces to an ordinary type function, since it is resolved type-wise. Encoding for dependent product in Scala may look like the following:
trait Sigma[T] {
val x: T
type U //can depend on x
}
// (t: T) => (∃ mapping(x, U), x == t) => (u: U); sadly, refinement won't compile
def pi[T](t: T)(implicit mapping: Sigma[T] { val x = t }): mapping.U
The missing piece here is an ability to statically constraint field x to expected value t, effectively forming an equation representing the property of all values inhabiting type T. Together with our Σ-types, used to express the existence of object with given property, the logic is formed, in which our equation is a theorem to be proven.
On a side note, in real case theorem may be highly nontrivial, up to the point where it cannot be automatically derived from code or solved without significant amount of effort. One can even formulate Riemann Hypothesis this way, only to find the signature impossible to implement without actually proving it, looping forever or throwing an exception.
The question was about using dependently typed feature more directly and, in my opinion,
there would be a benefit in having a more direct dependent typing approach than what Scala offers.
Current answers try to argue the question on type theoretical level.
I want to put a more pragmatic spin on it.
This may explain why people are divided on the level of support of dependent types in the Scala language. We may have somewhat different definitions in mind. (not to say one is right and one is wrong).
This is not an attempt to answer the question how easy would it be to turn
Scala into something like Idris (I imagine very hard) or to write a library
offering more direct support for Idris like capabilities (like singletons tries to be in Haskell).
Instead, I want to emphasize the pragmatic difference between Scala and a language like Idris.
What are code bits for value and type level expressions?
Idris uses the same code, Scala uses very different code.
Scala (similarly to Haskell) may be able to encode lots of type level calculations.
This is shown by libraries like shapeless.
These libraries do it using some really impressive and clever tricks.
However, their type level code is (currently) quite different from value level expressions
(I find that gap to be somewhat closer in Haskell). Idris allows to use value level expression on the type level AS IS.
The obvious benefit is code reuse (you do not need to code type level expressions
separately from value level if you need them in both places). It should be way easier to
write value level code. It should be easier to not have to deal with hacks like singletons (not to mention performance cost). You do not need to learn two things you learn one thing.
On a pragmatic level, we end up needing fewer concepts. Type synonyms, type families, functions, ... how about just functions? In my opinion, this unifying benefits go much deeper and are more than syntactic convenience.
Consider verified code. See:
https://github.com/idris-lang/Idris-dev/blob/v1.3.0/libs/contrib/Interfaces/Verified.idr
Type checker verifies proofs of monadic/functor/applicative laws and the
proofs are about actual implementations of monad/functor/applicative and not some encoded
type level equivalent that may be the same or not the same.
The big question is what are we proving?
The same can me done using clever encoding tricks (see the following for Haskell version, I have not seen one for Scala)
https://blog.jle.im/entry/verified-instances-in-haskell.html
https://github.com/rpeszek/IdrisTddNotes/wiki/Play_FunctorLaws
except the types are so complicated that it is hard to see the laws, the value
level expressions are converted (automatically but still) to type level things and
you need to trust that conversion as well.
There is room for error in all of this which kinda defies the purpose of compiler acting as
a proof assistant.
(EDITED 2018.8.10) Talking about proof assistance, here is another big difference between Idris and Scala. There is nothing in Scala (or Haskell) that can prevent from writing diverging proofs:
case class Void(underlying: Nothing) extends AnyVal //should be uninhabited
def impossible() : Void = impossible()
while Idris has total keyword preventing code like this from compiling.
A Scala library that tries to unify value and type level code (like Haskell singletons) would be an interesting test for Scala's support of dependent types. Can such library be done much better in Scala because of path-dependent types?
I am too new to Scala to answer that question myself.

What is a higher kinded type in Scala?

You can find the following on the web:
Higher kinded type == type constructor?
class AClass[T]{...} // For example, class List[T]
Some say this is a higher kinded type, because it
abstracts over types which would be compliant with the definition.
Higher kinded types are types which take other types and construct a new type
These though are also known as type constructor. (For example, in Programming in Scala).
Higher kinded type == type constructor which takes type constructor as a type parameter?
In the paper Generics of a Higher Kind, you can read
... types that abstract over types that abstract over types
('higher-kinded types') ..."
which suggests that
class XClass[M[T]]{...} // or
trait YTrait[N[_]]{...} // e.g. trait Functor[F[_]]
is a higher kinded type.
So with this in mind, it is difficult to distinguish between type constructor, higher kinded type and type constructor which takes type constructors as type parameter, therefore the question above.
Let me make up for starting some of this confusion by pitching in with some disambiguation. I like to use the analogy to the value level to explain this, as people tend to be more familiar with it.
A type constructor is a type that you can apply to type arguments to "construct" a type.
A value constructor is a value that you can apply to value arguments to "construct" a value.
Value constructors are usually called "functions" or "methods". These "constructors" are also said to be "polymorphic" (because they can be used to construct "stuff" of varying "shape"), or "abstractions" (since they abstract over what varies between different polymorphic instantiations).
In the context of abstraction/polymorphism, first-order refers to "single use" of abstraction: you abstract over a type once, but that type itself cannot abstract over anything. Java 5 generics are first-order.
The first-order interpretation of the above characterizations of abstractions are:
A type constructor is a type that you can apply to proper type arguments to "construct" a proper type.
A value constructor is a value that you can apply to proper value arguments to "construct" a proper value.
To emphasize there's no abstraction involved (I guess you could call this "zero-order", but I have not seen this used anywhere), such as the value 1 or the type String, we usually say something is a "proper" value or type.
A proper value is "immediately usable" in the sense that it is not waiting for arguments (it does not abstract over them). Think of them as values that you can easily print/inspect (serializing a function is cheating!).
A proper type is a type that classifies values (including value constructors), type constructors do not classify any values (they first need to be applied to the right type arguments to yield a proper type). To instantiate a type, it's necessary (but not sufficient) that it be a proper type. (It might be an abstract class, or a class that you don't have access to.)
"Higher-order" is simply a generic term that means repeated use of polymorphism/abstraction. It means the same thing for polymorphic types and values. Concretely, a higher-order abstraction abstracts over something that abstracts over something. For types, the term "higher-kinded" is a special-purpose version of the more general "higher-order".
Thus, the higher-order version of our characterization becomes:
A type constructor is a type that you can apply to type arguments (proper types or type constructors) to "construct" a proper type (constructor).
A value constructor is a value that you can apply to value arguments (proper values or value constructors) to "construct" a proper value (constructor).
Thus, "higher-order" simply means that when you say "abstracting over X", you really mean it! The X that is abstracted over does not lose its own "abstraction rights": it can abstract all it wants. (By the way, I use the verb "abstract" here to mean: to leave out something that is not essential for the definition of a value or type, so that it can be varied/provided by the user of the abstraction as an argument.)
Here are some examples (inspired by Lutz's questions by email) of proper, first-order and higher-order values and types:
proper first-order higher-order
values 10 (x: Int) => x (f: (Int => Int)) => f(10)
types (classes) String List Functor
types String ({type λ[x] = x})#λ ({type λ[F[x]] = F[String]})#λ
Where the used classes were defined as:
class String
class List[T]
class Functor[F[_]]
To avoid the indirection through defining classes, you need to somehow express anonymous type functions, which are not expressible directly in Scala, but you can use structural types without too much syntactic overhead (the #λ-style is due to https://stackoverflow.com/users/160378/retronym afaik):
In some hypothetical future version of Scala that supports anonymous type functions, you could shorten that last line from the examples to:
types (informally) String [x] => x [F[x]] => F[String]) // I repeat, this is not valid Scala, and might never be
(On a personal note, I regret ever having talked about "higher-kinded types", they're just types after all! When you absolutely need to disambiguate, I suggest saying things like "type constructor parameter", "type constructor member", or "type constructor alias", to emphasize that you're not talking about just proper types.)
ps: To complicate matters further, "polymorphic" is ambiguous in a different way, since a polymorphic type sometimes means a universally quantified type, such as Forall T, T => T, which is a proper type, since it classifies polymorphic values (in Scala, this value can be written as the structural type {def apply[T](x: T): T = x})
(This answer is an attempt to decorate Adriaan Moors answer by some graphical and historical information.)
Higher kinded types are part of Scala since 2.5.
Before that Scala, as Java till now,
did not allow to use type constructor
("generics" in Java) to be used as
type parameter to a type constructor. e.g.
trait Monad [M[_]]
was not possible.
In Scala 2.5 the type system had been extended by the ability to classify
types on a higher level (known as type constructor polymorphism). These
classifications are known as kinds.
(Picture derived from Generics of a Higher Kind)
The consequence is, that type constructor (e.g. List) could be used
just as other types in type parameter position of type constructors and so
they became first class types since Scala 2.5. (Similar to functions which are first class values in Scala).
In the context of a type system supporting higher kinds, we can
distinguish proper types, types like Int or List[Int] from first-order types like List and types of a higher kind like Functor or Monad (types which abstract over types which abstract over types).
The type system of Java on the other side does not support kinds and therefore has no types
of a "higher kind".
So this has to be seen against the background of the supporting type system.
In the case of Scala you often see examples of a type constructor like
trait Iterable[A, Container[_]]
with the headline "Higher kinded types", e.g. in Scala for generic programmers, section 4.3
This is sometimes missleading, because many refer to Container as higher kinded type and not Iterable, but what is more precise is,
the use of Container as type constructor parameter of a higher kinded (higher-order) type here Iterable.
The kind of ordinary types like Int and Char, whose instances are values, is *. The kind of unary type constructors like Maybe is * -> *; binary type constructors like Either have (curried) kind * -> * -> *, and so on. You can view types like Maybe and Either as type-level functions: they take one or more types, and return a type.
A function is higher-order if it has an order greater than 1, where the order is (informally) the nesting depth, to the left, of function arrows:
Order 0: 1 :: Int
Order 1: chr :: Int -> Char
Order 2: fix :: (a -> a) -> a, map :: (a -> b) -> [a] -> [b]
Order 3: ((A -> B) -> C) -> D
Order 4: (((A -> B) -> C) -> D) -> E
So, long story short, a higher-kinded type is just a type-level higher-order function which abstracts over type constructors:
Order 0: Int :: *
Order 1: Maybe :: * -> *
Order 2: Functor :: (* -> *) -> Constraint—higher-kinded: converts unary type constructors to typeclass constraints
I would say: A higher kinded type abstracts over a type constructor. E.g. consider
trait Functor [F[_]] {
def map[A,B] (fn: A=>B)(fa: F[A]): F[B]
}
Here Functor is a "higher kinded type" (as used in the "Generics of a Higher Kind" paper). It is not a concrete ("first-order") type constructor like List (which abstracts over proper types only). It abstracts over all unary ("first-order") type constructors (as denoted with F[_]).
Or to put it in another way: In Java, we have clearly type constructors (e.g. List<T>), but we have no "higher kinded types", because we can't abstract over them (e.g. we can't write the Functor interface defined above - at least not directly).
The term "higher order (type constructor) polymorphism" is used to describe systems that support "higher kinded types".
Scala REPL provides :kind command which
scala> :help kind
:kind [-v] <type>
Displays the kind of a given type.
For example,
scala> trait Foo[A]
trait Foo
scala> trait Bar[F[_]]
trait Bar
scala> :kind -v Foo
Foo's kind is F[A]
* -> *
This is a type constructor: a 1st-order-kinded type.
scala> :kind -v Foo[Int]
Foo[Int]'s kind is A
*
This is a proper type.
scala> :kind -v Bar
Bar's kind is X[F[A]]
(* -> *) -> *
This is a type constructor that takes type constructor(s): a higher-kinded type.
scala> :kind -v Bar[Foo]
Bar[Foo]'s kind is A
*
This is a proper type.
The :help provides clear definitions so I think it is worth posting it here in its entirety (Scala 2.13.2)
scala> :help kind
:kind [-v] <type>
Displays the kind of a given type.
-v Displays verbose info.
"Kind" is a word used to classify types and type constructors
according to their level of abstractness.
Concrete, fully specified types such as `Int` and `Option[Int]`
are called "proper types" and denoted as `A` using Scala
notation, or with the `*` symbol.
scala> :kind Option[Int]
Option[Int]'s kind is A
In the above, `Option` is an example of a first-order type
constructor, which is denoted as `F[A]` using Scala notation, or
* -> * using the star notation. `:kind` also includes variance
information in its output, so if we ask for the kind of `Option`,
we actually see `F[+A]`:
scala> :k -v Option
Option's kind is F[+A]
* -(+)-> *
This is a type constructor: a 1st-order-kinded type.
When you have more complicated types, `:kind` can be used to find
out what you need to pass in.
scala> trait ~>[-F1[_], +F2[_]] {}
scala> :kind ~>
~>'s kind is X[-F1[A1],+F2[A2]]
This shows that `~>` accepts something of `F[A]` kind, such as
`List` or `Vector`. It's an example of a type constructor that
abstracts over type constructors, also known as a higher-order
type constructor or a higher-kinded type.