In this question from 2013, Mr. Odersky notes that "it's too early to tell" whether libraries like Scalaz will be able to exist (at least in their current state) under Dotty, due to the castration of higher-kinded and existential types.
In the time passed, has Dotty's implications for Scalaz & Cats been elucidated? Will proposed features like built-in Effects and Records change the scope of these projects?
I understand that Dotty is still a ways off from replacing scalac, but as I am considering investing time applying purely functional constructs and methodologies to my work, I believe it important to consider the future of its flagship libraries.
One example of the latest on Dotty is "Scaling Scala" By Chris McKinlay (December 15, 2016) (the same article also mention the Scalaz and Cats situation)
Martin Odersky has been leading work on Dotty, a novel research compiler based on the Dependent Object Types (DOT) calculus (basically a simplified version of Scala) and ideas from the functional programming (FP) database community.
The team working on Dotty development has shown some remarkable improvements over the state of the art, most notably with respect to compilation times. I asked Odersky what he thought was novel about the Dotty architecture and would help end users. Here’s what he said:
Two things come to mind:
first, it's closely related to formal foundations, giving us better guidance on how to design a sound typesystem. This will lead to fewer surprises for users down the road.
Second, it has an essentially functional architecture. This makes it easier to extend, easier to get correct, and will lead to more robust APIs where the compiler is used as a service for IDEs and meta programming.
Although Dotty opens up a number of interesting language possibilities (notably full-spectrum dependent types, a la Agda and Idris), Odersky has chosen to prioritize making it immediately useful to the community. Language differences are fairly small, and most of them are in order to either simplify the language (like removing procedure syntax) or fix bugs (unsound pattern matching) or both (early initializers).
Still, I couldn’t resist asking him if there is any chance of full-spectrum dependent types ending up in Scala at some point. Here is what he said:
Never say never :-). In fact, we are currently working with Viktor Kuncak on integrating the Leon program prover with Scala, which demands richer dependent types than we have now. But it's currently strictly research, with a completely open outcome.
The Scala and Dotty teams are working closely toward convergence for Scala 2.x and Dotty, and they’ve indicated that they take continuity very seriously. Scala 2.12 and 2.13 have language flags that unlock features being incubated in Dotty (e.g., existential types), and the Dotty compiler has a Scala 2 compatibility mode. There’s even a migration tool.
Related
I attended the following keynote on the future of Scala by Martin Odersky:
https://skillsmatter.com/skillscasts/8866-from-dot-to-dotty
At 1:01:00 an answer to an audience question seems to say that future Scala will not be Turing complete.
Did I understand this correctly? Will Scala 3 no longer be Turing complete? If so, what practical impact will this have on someone like me who uses Scala daily at work to solve practical problems? In other words, what do industrial Scala programmers loose and what do they gain by removing Turing completeness?
At 1:01:00 an answer to an audience question seems to say that future Scala will not be Turing complete.
Did I understand this correctly? Will Scala 3 no longer be Turing complete?
No, that's neither the question being asked nor the answer being given.
Firstly: The question being asked is not whether Scala will not be Turing-complete, the question being asked is whether the Scala Type System will not be Turing-complete.
Secondly: The answer being given is not that the Type System of future Scala will not be Turing-complete. Martin Odersky clearly says that with implicits, the Type System will definitely be Turing-complete, and without implicits, he doesn't want to make a prediction as to whether or not it will be Turing-complete.
So, to answer your question:
Scala will definitely still be Turing-complete.
The question you linked to is not about Scala, it is about Scala's Type System.
Scala's Type System will also still be Turing-complete because of implicits.
Scala's Type System without implicits may or may not be Turing-complete, we don't know yet, and Martin Odersky doesn't want to make any predictions.
If so, what practical impact will this have on someone like me who uses Scala daily at work to solve practical problems?
None whatsoever. First off, again, the Type System will still be Turing-complete because of implicits. And secondly, even it weren't, AFAIK, the Turing-completeness of Scala's Type System has not been used for anything pragmatically interesting. There are libraries which do perform sophisticated type-level computations, but those computations always terminate. Nobody has written a library which performs arbitrary Turing-complete computation at the type-level. (And in fact, it isn't even possible, because even though Scala's type system is Turing-complete, all currently existing implementations of Scala (there is only one anyway) have a strict limit on the recursion depth of the type checker).
In other words, what do industrial Scala programmers loose and what do they gain by removing Turing completeness?
Let's first talk about the type system: they don't lose anything. What they gain is the fact that compilation is guaranteed to terminate, and the fact that this means that the compiler can prove stuff about the program it couldn't prove otherwise.
Let's also answer the hypothetical question: what if Scala weren't Turing-complete? Well, we could no longer write infinite loops. That's pretty much it. Note, however, that lots of things that are typically modeled as infinite loops (or infinite recursion) over data can still be modeled as finite co-recursion over co-data! (For example, an event loop in an Operating System, a Web Server, or a GUI.)
OTOH, lots of things compilers can't do are "because it's equivalent to solving the Halting Problem". Well, in a language that isn't Turing-complete, the Halting Problem doesn't exist! So, the compiler can prove many more things about programs than it could in a Turing-complete language.
But, to re-iterate: there are no plans of making Scala not Turing-complete. There are no plans of making implicits not Turing-complete. There are restrictions to the type system which may or may not make the type system not Turing-complete.
Is my understanding above actually correct, that is, will Scala 3 no longer be Turing complete?
No, Scala 3 is still a Turing-complete language. You can already experiment with it by trying out Dotty, the current prototype for what will become Scala 3.
If you can give a link to the particular slide of the particular talk you're referring to, we can help you figure out what it was actually trying to express.
In hist recent talk at Strange Loop Martin Odersky shed the light on his vision of Scala's future version called Dotty. I understand this is work-in-progress and it even may not flow into Scala (at least not very fast) due to many possible backward-compatibility issues. But if it happens, how should we program in Scala today to be forward-compatible with Dotty? I didn't get all the ideas from the talk so I'd like someone more profound to summarize the changes and describe how can we prepare to them.
The only other advice I would give is to also stay away from structural types containing a type members. Those type members are the elements that make type-checking in Scala turing complete, and I am not yet sure whether we want to support that in Dotty. Structural types that contain only vals and defs will likely stay supported.
Towards the end of the talk, he says "if your program is compiled without any language feature imports for higher-kinded and existentials then you should be ok".
So if I understand it correctly the only thing you would need to be aware of is any possible usage of existential types or higher-kinded types in your code.
I've heard people claim that:
Scala's type system is amazing (existential types, variant, co-variant)
Because of the power of macros, everything is a library in Clojure: (pattern matching, logic programming, non-determinism, ..)
Question:
If both assertions are true, why is Scala's type system not a library in Clojure? Is it because:
types are one of these things that do not work well as a library? [i.e. the changes would somehow have to threaded through every existing clojure library, including clojure.core?]
is Scala's notion of types fundamentally incompatible with clojure protocol / records?
... ?
It's an interesting question.
You are certainly right about Scala having an amazing type system, and about Clojure being phenomenal for meta-programming and extension of the language (although that is about more than just macros....).
A few reasons I can think of:
Clojure is a dynamically typed language while Scala is a statically typed language. Having powerful type inference isn't so much use in a language where you can assume relatively little about the types of your inputs.
Clojure already has a very interesting project to add typing as a library (Typed Clojure) which looks very promising - however it's very different in approach to Scala as it is designed for a dynamic language from the start (inspired more by Typed Racket, I believe).
Clojure philosophy actually discourages certain OOP concepts (particularly implementation inheritance, mutable objects, and data encapsulation). A type system that supports these things (as Scala does) wouldn't be a good fit for Clojure idioms - at best they would be ignored, but they could easily encourage a style of development that would cause people to run into severe problems later.
Clojure already provides tools that solve many of the problems you would typically solve with types in other languages - e.g. the use of protocols for polymorphism.
There's a strong focus in the Clojure community on simplicity (in the sense of the excellent video "Simple Made Easy" - see particularly the slide at 39:30). While Scala's type system is certainly amazing, I think it's a stretch to describe it as "Simple"
Putting in a Scala-style type system would probably require a complete rewrite of the Clojure compiler and make it substantially more complex. Nobody seems to have signed up so far to take on that particular challenge... and there's a risk that even if someone were willing and able to do this then the changes could be rejected for the various cultural / technical reasons covered above.
In the absence of a major change to Clojure itself (which I think would be unlikely) then one interesting possibility would be to create a DSL within Clojure that provided Scala-style type inference for a specific domain and compiled this DSL direct to optimised Java bytecode. I could see that being a useful approach for specific problem domains (large scale numerical data crunching with big matrices, for example).
To simply answer your question "... why is Scala's type system not a library in Clojure?":
Because the type system is part of the scala compiler and not of the scala library. The whole power of scalas type system only exists at compile time. The JVM has no support for things like that, because of type erasure and also, because it would simply slow down execution. And also there is no need for it. If you have a statically typed language, you don't need type information at runtime, unless you want to do dirty stuff.
edit:
#mikera the jvm is sure capable of running the scala compiler, I did not say anything like that. I just said, that the jvm has no support for type systems like that. It does not even support generics. At runtime all these types are gone. The compiler checks for the correctness of a program and removes all the higher kinded types / generics.
example:
val xs: List[Int] = List(1,2,3,4)
val x1: Int = xs.head
will at runtime look like this:
val xs: List = List.apply(1,2,3,4)
val x1: Int = xs.head.asInstanceOf[Int]
But it doesn't matter, because the compiler checked it before. You can only get in trouble here, when you use reflection, because you could put any value in the list and it would break at runtime exactly where the value is casted to Int.
And this is one of the reasons, why the scala type system is not part of the scala library, but built into the compiler.
And also the question of the OP was "... why is Scala's type system not a library in Clojure?" and not "Is it possible to create a type system such as scalas for clojure?" and I perfectly answered that question.
I've been hearing a lot about different JVM languages, still in vaporware mode, that propose to implement reification somehow. I have this nagging half-remembered (or wholly imagined, don't know which) thought that somewhere I read that Scala somehow took advantage of the JVM's type erasure to do things that it wouldn't be able to do with reification. Which doesn't really make sense to me since Scala is implemented on the CLR as well as on the JVM, so if reification caused some kind of limitation it would show up in the CLR implementation (unless Scala on the CLR is just ignoring reification).
So, is there a good side to type erasure for Scala, or is reification an unmitigated good thing?
See Ola Bini's blog. As we all know, Java has use-site covariance, implemented by having little question marks wherever you think variance is appropriate. Scala has definition-site covariance, implemented by the class designer. He says:
Generics is a complicated language feature. It becomes even more
complicated when added to an existing language that already has
subtyping. These two features don’t play very well together in the
general case, and great care has to be taken when adding them to a
language. Adding them to a virtual machine is simple if that machine
only has to serve one language - and that language uses the same
generics. But generics isn’t done. It isn’t completely understood how
to handle correctly and new breakthroughs are happening (Scala is a
good example of this). At this point, generics can’t be considered
“done right”. There isn’t only one type of generics - they vary in
implementation strategies, feature and corner cases.
...
What this all means is that if you want to add reified generics to the
JVM, you should be very certain that that implementation can encompass
both all static languages that want to do innovation in their own
version of generics, and all dynamic languages that want to create a
good implementation and a nice interfacing facility with Java
libraries. Because if you add reified generics that doesn’t fulfill
these criteria, you will stifle innovation and make it that much
harder to use the JVM as a multi language VM.
i.e. If we had reified generics in the JVM, most likely those reified generics wouldn't be suitable for the features we really like about Scala, and we'd be stuck with something suboptimal.
In Java and C++ designing program's objects hierarchy is pretty obvious. But beginning Scala I found myself difficult to decide what classes to define to better employ Scala's syntactic sugar facilities (an even idealess about how should I design for better performance). Any good readings on this question?
I have read 4 books on Scala, but I have not found what you are asking for. I guess you have read "Programming in Scala" by Odersky (Artima) already. If not, this is a link to the on-line version:
http://www.docstoc.com/docs/8692868/Programming-In-Scala
This book gives many examples how to construct object-oriented models in Scala, but all examples are very small in number of classes. I do not know of any book that will teach you how to structure large scale systems using Scala.
Imperative object-orientation has
been around since Smalltalk, so we
know a lot about this paradigm.
Functional object-orientation on the
other hand, is a rather new concept,
so in a few years I expect books
describing large scale FOO systems to
appear. Anyway, I think that the PiS
book gives you a pretty good picture
how you can put together the basic
building blocks of a system, like
Factory pattern, how to replace the
Strategy pattern with function
literals and so on.
One thing that Viktor Klang once told me (and something I really agree upon) is that one difference between C++/Java and Scala OO is that you define a lot more (smaller) classes when you use Scala. Why? Because you can! The syntactic sugar for the case class result in a very small penalty for defining a class, both in typing and in readability of the code. And as you know, many small classes usually means better OO (fewer bugs) but worse performance.
One other thing I have noticed is that I use the factory pattern a lot more when dealing with immutable objects, since all "changes" of an instance results in creating a new instance. Thank God for the copy() method on the case class. This method makes the factory methods a lot shorter.
I do not know if this helped you at all, but I think this subject is very interesting myself, and I too await more literature on this subject.
Cheers!
This is still an evolving matter. For instance, the just released Scala 2.8.0 brought support of type constructor inference, which enabled a pattern of type classes in Scala. The Scala library itself has just began using this pattern. Just yesterday I heard of a new Lift module in which they are going to try to avoid inheritance in favor of type classes.
Scala 2.8.0 also introduced lower priority implicits, plus default and named parameters, both of which can be used, separately or together, to produce very different designs than what was possible before.
And if we go back in time, we note that other important features are not that old either:
Extractor methods on case classes object companions where introduced February 2008 (before that, the only way to do extraction on case classes was through pattern matching).
Lazy values and Structural types where introduced July 2007.
Abstract types support for type constructors was introduced in May 2007.
Extractors for non-case classes was introduced in January 2007.
It seems that implicit parameters were only introduced in March 2006, when they replaced the way views were implemented.
All that means we are all learning how to design Scala software. Be sure to rely on tested designs of functional and object oriented paradigms, to see how new features in Scala are used in other languages, like Haskell and type classes or Python and default (optional) and named parameters.
Some people dislike this aspect of Scala, others love it. But other languages share it. C# is adding features as fast as Scala. Java is slower, but it goes through changes too. It added generics in 2004, and the next version should bring some changes to better support concurrent and parallel programming.
I don't think that there are much tutorials for this. I'd suggest to stay with the way you do it now, but to look through "idiomatic" Scala code as well and to pay special attention in the following cases:
use case classes or case objects instead of enums or "value objects"
use objects for singletons
if you need behavior "depending on the context" or dependency-injection-like functionality, use implicits
when designing a type hierarchy or if you can factor things out of a concrete class, use traits when possible
Fine grained inheritance hierarchies are OK. Keep in mind that you have pattern matching
Know the "pimp my library" pattern
And ask as many questions as you feel you need to understand a certain point. The Scala community is very friendly and helpful. I'd suggest the Scala mailing list, Scala IRC or scala-forum.org
I've just accidentally googled to a file called "ScalaStyleGuide.pdf". Going to read...