Are there any documented anti-patterns for functional programming? [closed] - scala

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
Next month I'm going to work on a new R&D project that will adopt a functional programming language (I voted for Haskell, but right now F# got more consensus). Now, I've played with such languages for a while and developed a few command line tools with them, but this is a quite bigger project and I'm trying to improve my functional programming knowledge and technique. I've also read a lot on the topic, but I can't find any books or resources that document anti-patterns in the functional programming world.
Now, learning about anti-patterns means learning about other smart people failures: in OOP I know a few of them, and I'm experienced enough to choose wisely when something that generally is an anti-pattern, perfectly fit my needs. But I can choose this because I know the lesson learned by other smart guys.
Thus, my question is: are there any documented anti-patterns in functional programming? Till now, all of my collegues told me that they do not know any, but they can't state why.
If yes, please include one single link to an authoritative source (a catalogue, an essay, a book or equivalent).
If no, please support your answer by a proper theorem.
Please don't turn this question in a list: it is a boolean question that just requires a proof to evaluate the answer. For example, if you are Oleg Kiselyov, "Yes" is enough, since everybody will be able to find your essay on the topic. Still, please be generous.
Note that I am looking for formal anti-patterns, not simple bad habits or bad practices.
From the linked wikipedia article on Anti-Patterns:
... there must be at least two key elements present to formally distinguish an actual anti-pattern from a simple bad habit, bad practice, or bad idea:
some repeated pattern of action, process or structure that initially appears to be beneficial, but ultimately produces more bad consequences than beneficial results, and
an alternative solution exists that is clearly documented, proven in actual practice and repeatable.
Moreover by "documented" I mean something from authoritative authors or well known sources.
The languages that I'm used to are:
Haskell (where I'm really starting to think that if code compiles, it works!)
Scala
F#
but I can also adapt knowledge about anti-patterns documented in other functional languages.
I searched a lot in the web, but all the resources I've found are either related to OOP or to function layout (define variable at the beginning of the function, and the like...).

The only anti-pattern I've seen is over-monadization, and since monads can be incredibly useful this falls somewhere in between a bad practice and an anti-pattern.
Suppose you have some property P that you want to be true of some of your objects. You could decorate your objects with a P monad (here in Scala, use paste in the REPL to get the object and its companion to stick together):
class P[A](val value: A) {
def flatMap[B](f: A => P[B]): P[B] = f(value) // AKA bind, >>=
def map[B](f: A => B) = flatMap(f andThen P.pure) // (to keep `for` happy)
}
object P {
def pure[A](a: A) = new P(a) // AKA unit, return
}
Okay, so far so good; we cheated a little bit by making value a val rather than making this a comonad (if that's what we wanted), but we now have a handy wrapper in which we can wrap anything. Now let's suppose we also have properties Q and R.
class Q[A](val value: A) {
def flatMap[B](f: A => Q[B]): Q[B] = f(value)
def map[B](f: A => B) = flatMap(f andThen Q.pure)
}
object Q {
def pure[A](a: A) = new Q(a)
}
class R[A](val value: A) {
def flatMap[B](f: A => R[B]): R[B] = f(value)
def map[B](f: A => B) = flatMap(f andThen R.pure)
}
object R {
def pure[A](a: A) = new R(a)
}
So we decorate our object:
class Foo { override def toString = "foo" }
val bippy = R.pure( Q.pure( P.pure( new Foo ) ) )
Now we are suddenly faced with a host of problems. If we have a method that requires property Q, how do we get to it?
def bar(qf: Q[Foo]) = qf.value.toString + "bar"
Well, clearly bar(bippy) isn't going to work. There are traverse or swap operations that effectively flip monads, so we could, if we'd defined swap in an appropriate way, do something like
bippy.map(_.swap).map(_.map(bar))
to get our string back (actually, a R[P[String]]). But we've now committed ourselves to doing something like this for every method that we call.
This is usually the wrong thing to do. When possible, you should use some other abstraction mechanism that is equally safe. For instance, in Scala you could also create marker traits
trait X
trait Y
trait Z
val tweel = new Foo with X with Y with Z
def baz(yf: Foo with Y) = yf.toString + "baz"
baz(tweel)
Whew! So much easier. Now it is very important to point out that not everything is easier. For example, with this method if you start manipulating Foo you will have to keep track of all the decorators yourself instead of letting the monadic map/flatMap do it for you. But very often you don't need to do a bunch of in-kind manipulations, and then the deeply nested monads are an anti-pattern.
(Note: monadic nesting has a stack structure while traits have a set structure; there is no inherent reason why a compiler could not allow set-like monads, but it's not a natural construct for typical formulations of type theory. The anti-pattern is a simple consequence of the fact that deep stacks are tricky to work with. They can be somewhat easier if you implement all the Forth stack operations for your monads (or the standard set of Monad transformers in Haskell).)

Related

Understanding GenericTraversableTemplate and other Scala collection internals

I was exchanging emails with an acquaintance that is a big Kotlin, Clojure and Java8 fan and asked him why not Scala. He provided many reasons (Scala is too academic, too many features, not the first time I hear this and I think this is very subjective)
but his biggest pain point was as an example, that he doesn't like a language where he can't understand the implementation of basic data structures, and he gave LinkedList as an example.
I took a look at scala.collection.LinkedList and counted the things I either understand or somewhat understand.
CanBuildFrom - after some effort, I get it, type classes, not the longest suicide note
in history [1]
LinkedListLike - I can't remember where I read it, but I got convinced this is there for a good reason
But then I started to stare at these
GenericTraversableTemplate - now I'm scratching my head as well...
SeqFactory, GenericCompanion - OK, now you lost me, I start to understand his point
Can someone who understand this well please explain GenericTraversableTemplate SeqFactory and GenericCompanion in the context of LinkedList? What they are for, what impact on the end user they have (e.g. I'm sure they are there for a good reason, what is that reason?)
Are they there for a practical reason? or is it a level of abstraction that could have been simplified?
I like Scala collections because I don't have to understand the internals to be able to effectively use them. I don't mind a complex implementation if it helps me to keep my usage simpler. e.g. I don't mind paying the price of a complex library if I get the ability to write cleaner more elegant code using it in return. but it will sure be nice to better understand it.
[1] - Is the Scala 2.8 collections library a case of "the longest suicide note in history"?
I will try to describe the concepts from the point of view of a random pedestrian (I've never contributed a single line to the Scala collection library, so don't hit me too hard if I'm wrong).
Since LinkedList is now deprecated, and because Maps provide a better example, I will use TreeMap as example.
CanBuildFrom
The motivation is this: If we take a TreeMap[Int, Int] and map it with
case (x, y) => (2 * x, y * y * 0.3d)
we get TreeMap[Int, Double]. This type safety alone would already explain the necessity for
simple genericBuilder[X] constructs.
However, if we map it with
case (x, y) => x
we obtain an Iterable[Int] (more precisely: a List[Int]), this is no longer a Map, the type of the container has changed. This is where CBF's come into play:
CanBuildFrom[This, X, That]
can be seen as a kind of "type-level function" that tells us: if we map a collection of type
This with a function that returns values of type X, we can build a That. The most specific
CBF is provided at compile time, in the first case it will be something like
CanBuildFrom[TreeMap[_,_], (X,Y), TreeMap[X,Y]]
in the second case it will be something like
CanBuildFrom[TreeMap[_,_], X, Iterable[X]]
and so we always get the right type of the container. The pattern is pretty general.
Every time you have a generic function
foo[X1, ..., Xn](x1: X1, ..., xn: Xn): Y
where the result type Y depends on X1, ..., Xn, you can introduce an implicit parameter as
follows:
foo[X1, ...., Xn, Y](x1: X1, ..., xn: Xn)(implicit CanFooFrom[X1, ..., Xn, Y]): Y
and then define the type-level function X1, ..., Xn -> Y piecewise by providing multiple
implicit CanFooFrom's.
LinkedListLike
In the class definition, we see something like this:
TreeMap[A, B] extends SortedMap[A, B] with SortedMapLike[A, B, TreeMap[A, B]]
This is Scala's way to express the so-called F-bounded polymorphism.
The motivation is as follows: Suppose we have a dozen (or at least two...) implementations of the trait SortedMap[A, B]. Now we want to implement a method withoutHead, it could look
somewhat like this:
def withoutHead = this.remove(this.head)
If we move the implementation into SortedMap[A, B] itself, the best we can do is this:
def withoutHead: SortedMap[A, B] = this.remove(this.head)
But is this the most specific result type we can get? No, that's too vague.
We would like to return TreeMap[A, B] if the original map is a TreeMap, and
CrazySortedLinkedHashMap (or whatever...) if the original was a CrazySortedLinkedHashMap.
This is why we move the implementation into SortedMapLike, and give the following signature to the withoutHead method:
trait SortedMapLike[A, B, Repr <: SortedMap[A, B]] {
...
def withoutHead: Repr = this.remove(this.head)
}
now because TreeMap[A, B] extends SortedMapLike[A, B, TreeMap[A, B]], the result type of
withoutHead is TreeMap[A,B]. The same holds for CrazySortedLinkedHashMap: we get the exact type back. In Java, you would either have to return SortedMap[A, B] or override the method in each subclass (which turned out to be a maintenance nightmare for the feature-rich traits in Scala)
GenericTraversableTemplate
The type is: GenericTraversableTemplate[+A, +CC[X] <: GenTraversable[X]]
As far as i can tell, this is just a trait that provides implementations of
methods that somehow return regular collections with same container type but
possibly different content type (stuff like flatten, transpose, unzip).
Stuff like foldLeft, reduce, exists are not here because these methods care only about content type, not container type.
Stuff like flatMap is not here, because the container type can change (again, CBF's).
Why is it a separate trait, is there a fundamental reason why it exists?
I don't think so... It probably would be possible to group the godzillion of methods somewhat differently. But this is just what happens naturally: you start to implement a trait, and it turns out that it has very many methods. So instead you group loosely related methods, and put them into 10 different traits with awkward names like "GenTraversableTemplate", and them mix them all into traits/classes where you need them...
GenericCompanion
This is just an abstract class that implements some basic functionality which is common
for companion objects of most collection classes (essentially, it just implements very
simple factory methods apply(varargs) and empty).
For example there is method apply that takes varargs of some type A and returns a collection of type CC[A]:
Array(1, 2, 3, 4) // calls Array.apply[A](elems: A*) on the companion object
List(1, 2, 3, 4) // same for List
The implementation is very simple, it's something like this:
def apply[A](varargs: A*): CC[A] = {
val builder = newBuilder[A]
for (arg <- varargs) builder += arg
builder.result()
}
This is obviously the same for Arrays and Lists and TreeMaps and almost everything else, except 'constrained irregular Collections' like Bitset. So this is just common functionality in a common ancestor class of most companion objects. Nothing special about that.
SeqFactory
Similar to GenericCompanion, but this time more specifically for Sequences.
Adds some common factory methods like fill() and iterate() and tabulate() etc.
Again, nothing particularly rocket-scientific here...
Few general remarks
In general: I don't think that one should attempt to understand every single trait in this library. Rather, one should try to look at the library as a whole. As a whole, it has a very interesting architecture. And in my personal opinion, it's actually a very aesthetic piece of software, but one has to stare at it for quite a while (and try to re-implement the whole architectural pattern several times) to grasp it. On the other hand: for example CBF's are kind of "design pattern" that clearly should be eliminated in successors of this language. The whole story with the scope of implicit CBF's still seems like a total nightmare to me. But many things seemed completely inscrutable at first, and almost always, it ended with an epiphany (which is very specific for Scala: for the majority of other languages, such struggles usually end with the thought "Author of this is a complete idiot").

Why does Scalaz use complex symbols and no in-code documentation?

I'm sometimes looking at Scalaz and find it pretty hard to understand for a beginner Scala programmer.
implicit def KleisliCategory[M[_]: Monad]: Category[({type λ[α, β]=Kleisli[M, α, β]})#λ] = new Category[({type λ[α, β]=Kleisli[M, α, β]})#λ] {
def id[A] = ☆(_ η)
def compose[X, Y, Z](f: Kleisli[M, Y, Z], g: Kleisli[M, X, Y]) = f <=< g
}
implicit def CokleisliCategory[M[_]: Comonad]: Category[({type λ[α, β]=Cokleisli[M, α, β]})#λ] = new Category[({type λ[α, β]=Cokleisli[M, α, β]})#λ] {
def id[A] = ★(_ copure)
def compose[X, Y, Z](f: Cokleisli[M, Y, Z], g: Cokleisli[M, X, Y]) = f =<= g
}
Scalaz methods may seem obvious for experienced functional programmers, but for anyone else it's hard to understand.
Why is there so few documentation in Scalaz code?
Why do they use so many operators that are unreadable for most people?
I don't even know how to type ★ or ☆ without copy/pasting. And it's just an example because there are many.
Some people say that Scalaz was unreadable at the beginning, but 2 years later they find it great.
I wonder where to start with Scalaz. The Scala validation seems the easiest part, but after that?
I agree that Scalaz is mostly undocumented. The problem is that it collects a lot of advanced concepts from Haskell (and the underlying mathematics) and documenting them all in detail would become writing a whole book about functional programming (and mathematics). So I believe Scalaz's approach is:
If you know and need some concept from functional programming prepared for Scala, you'll most likely find it here.
If you don't know it, you'll have to learn it elsewhere.
Let's have a look at your example: If you know Kleisli categories and how every monads gives rise to one, the definition is quite self-contained. If you don't, then KleisliCategory has no use for you anyway.
(In my experience, Haskell is better for learning advanced concepts from functional programming. While Scala is far better than Java, it still drags around Java's OO/imperative heritage that clutters things a bit.)
Considering the Unicode symbols. Looking at sources it seems that they're used only as a syntactic sugar or at least they have a non-Unicode counterpart:
def ☆[M[_], A, B](f: A => M[B]): Kleisli[M, A, B] = kleisli(f)
def η[F[_]](implicit p: Pure[F]): F[A] = pure
def cokleisli[W[_], A, B](f: W[A] => B): Cokleisli[W, A, B] = ★(f)
So you can go without them, if you want.
Still, I'm not sure if having them in Scalaz is a good idea. It could make the code impossible to read for someone who lacks the proper fonts. I'd rather prefer pure ASCII symbols.
A place to start is to read Learn You a Haskell which covers many concepts.
Watch Chris Marshall's (#oxbow_lakes) scalaz talks: http://skillsmatter.com/expert/scala/chris-marshall
Get a copy of Functional Programming in Scala from Manning written by some of the authors of scalaz.
I have a couple of small examples on my blog http://www.casualmiracles.com/blog/
I would say there is even easier places to start with scalaz than validation, which is the various enrichments on Option like ~foo which returns the value contained in the option or the 'zero' for the options's type (0 for numbers, empty string for String etc).
I forgot about a very detailed series of articles called Learning Scalaz at http://eed3si9n.com/
As with any open-source project, the only really true and acceptable answer to "Why isn't their better documentation?" is "Because no one has written it yet. You are free to volunteer."
(I honestly have no idea whether this answer will result in upvotes or downvotes. Interesting experiment.)

What are the exact applications of Functor, PointedFunctor, ApplicativeFunctor and Monad? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I have been studying Scala (and Haskell by extension) for some time now and I'm totally captured by their type system and functional paradigm. Quite recently I stumbled upon "Type Level Programming" and got dragged into things like Functors and other things I haven't heard about (except for Monad, which I knew was something of a mystic nature but had no idea of what to make use of it!).
I studied the concepts in Haskell (and got bewitched by its type system and type inference capabilities by the way) and I kinda have a firm grasp of what it means for a type to be Functor, PointedFunctor, ApplicativeFunctor or Monoid in a purely technical level (I still don't know what a Monad is even in technical level) but I'm feeling like an idiot since I see no uses for all this except that perhaps a good categorization of some concepts is acquired (?). What are these things useful for ? Why make life so complex? Why study these stuff and categorize them into various classes?
Why make life so complex?
They are all there to make life more simple!
Firstly, they make the code we write cleaner and clearer.
Secondly, they increase our expressiveness without adding genuinely new language features.
For example, Monads let you use a standard syntax to express complex computational contexts, Functors let you think and program in standard ways about data structures, and Applicative Functors let you treat effectful or complex computational contexts as simply as you treat straightforward data, letting you use the functional paradigm cleanly outside pure data.
They all help code reuse and help us understand each others' code because they give us a standard way of thinking about things.
Once you're used to them, you'll not want to do without them!
They are all abstractions. Eg. a monoid is something that supports a zero element and addition (must be associative). Examples would be integers, lists, strings etc. I think its rather nice to have a single common "interface" for all those different types.
So why are they useful? You can write a generic sum function for all monoids for example. Instead of writing one for string, integers and so on you only write one generic function. I think that's quite useful.
We should first ask: What are functors, monads, .... Unless we know what these concept are (mean) it's difficult to talk about their uses.
These concept come from category theory.
They arise from the fact that many many objects in mathematics (and consequently in functional programming) share some common, abstract properties. Once we know and understand these properties, we can use them to write very generic code that is reusable for very large amount of tasks.
To give an example: Everybody knows function
map :: (a -> b) -> ([a] -> [b])
Having a function from a to b we can create a function that works on lists [a]. Functors are generalization of this concept. Anything that can be mapped over in this way (and preserves functor laws) is called a functor. So Functor declars
fmap :: Functor f => (a -> b) -> (f a -> f b)
In the case of lists f becomes []. With fmap we can map over lists (there it's equal to map) but also over Maybes, various collections, trees, even over functions.
See also
https://math.stackexchange.com/questions/370/good-book-lecture-notes-about-category-theory
https://math.stackexchange.com/questions/21128/when-to-learn-category-theory
Additionally I want to point out that monads are not "mystic". Especially container-like monads (list, Maybe, Identity) are quite easy to understand. They are similiar to functors, but with a twist: Using fmap the "shape" (e.g. number of elements in a list) of the original functor is preserved, e.g. you can't use fmap to implement something like filter. That's why monads have a function called "bind" (in Haskell it's (>>=)) which allows this kind of thing, but they aren't magic either (e.g. for lists, it's the same as good old concatMap). Additionally monads have a function return to wrap a single value.
Now a lot of other, not "container-like" things are monads. There are monads that can operate on "stored computations" (Cont for continuation monad). They can provide (Reader), collect (Writer) or hold (State) some kind of "additional context". A very useful "context" is the "state of the rest of the world", better known as IO. In that case the type system (especially the restrictions imposed by polymorphism and type classes) can shield against unwanted interactions, and force a certain computation order (which is not trivial in a lazy language), so we need no dirty hacks or language back-doors in order to do IO in a pure language. Some people think this is kind of magic, but it's just clever use of the type system, and monads are not the only solution for this problem (e.g. the language Clean uses "uniqueness types" for this).
"What are these thigs useful for?"
Ah! you could use them to write FizzBuzz:
http://dave.fayr.am/posts/2012-10-4-finding-fizzbuzz.html

What are practical uses of applicative style?

I am a Scala programmer, learning Haskell now. It's easy to find practical use cases and real world examples for OO concepts, such as decorators, strategy pattern etc. Books and interwebs are filled with it.
I came to the realization that this somehow is not the case for functional concepts. Case in point: applicatives.
I am struggling to find practical use cases for applicatives. Almost all of the tutorials and books I have come across so far provide the examples of [] and Maybe. I expected applicatives to be more applicable than that, seeing all the attention they get in the FP community.
I think I understand the conceptual basis for applicatives (maybe I am wrong), and I have waited long for my moment of enlightenment. But it doesn't seem to be happening. Never while programming, have I had a moment when I would shout with a joy, "Eureka! I can use applicative here!" (except again, for [] and Maybe).
Can someone please guide me how applicatives can be used in a day-to-day programming? How do I start spotting the pattern? Thanks!
Applicatives are great when you've got a plain old function of several variables, and you have the arguments but they're wrapped up in some kind of context. For instance, you have the plain old concatenate function (++) but you want to apply it to 2 strings which were acquired through I/O. Then the fact that IO is an applicative functor comes to the rescue:
Prelude Control.Applicative> (++) <$> getLine <*> getLine
hi
there
"hithere"
Even though you explicitly asked for non-Maybe examples, it seems like a great use case to me, so I'll give an example. You have a regular function of several variables, but you don't know if you have all the values you need (some of them may have failed to compute, yielding Nothing). So essentially because you have "partial values", you want to turn your function into a partial function, which is undefined if any of its inputs is undefined. Then
Prelude Control.Applicative> (+) <$> Just 3 <*> Just 5
Just 8
but
Prelude Control.Applicative> (+) <$> Just 3 <*> Nothing
Nothing
which is exactly what you want.
The basic idea is that you're "lifting" a regular function into a context where it can be applied to as many arguments as you like. The extra power of Applicative over just a basic Functor is that it can lift functions of arbitrary arity, whereas fmap can only lift a unary function.
Since many applicatives are also monads, I feel there's really two sides to this question.
Why would I want to use the applicative interface instead of the monadic one when both are available?
This is mostly a matter of style. Although monads have the syntactic sugar of do-notation, using applicative style frequently leads to more compact code.
In this example, we have a type Foo and we want to construct random values of this type. Using the monad instance for IO, we might write
data Foo = Foo Int Double
randomFoo = do
x <- randomIO
y <- randomIO
return $ Foo x y
The applicative variant is quite a bit shorter.
randomFoo = Foo <$> randomIO <*> randomIO
Of course, we could use liftM2 to get similar brevity, however the applicative style is neater than having to rely on arity-specific lifting functions.
In practice, I mostly find myself using applicatives much in the same way like I use point-free style: To avoid naming intermediate values when an operation is more clearly expressed as a composition of other operations.
Why would I want to use an applicative that is not a monad?
Since applicatives are more restricted than monads, this means that you can extract more useful static information about them.
An example of this is applicative parsers. Whereas monadic parsers support sequential composition using (>>=) :: Monad m => m a -> (a -> m b) -> m b, applicative parsers only use (<*>) :: Applicative f => f (a -> b) -> f a -> f b. The types make the difference obvious: In monadic parsers the grammar can change depending on the input, whereas in an applicative parser the grammar is fixed.
By limiting the interface in this way, we can for example determine whether a parser will accept the empty string without running it. We can also determine the first and follow sets, which can be used for optimization, or, as I've been playing with recently, constructing parsers that support better error recovery.
I think of Functor, Applicative and Monad as design patterns.
Imagine you want to write a Future[T] class. That is, a class that holds values that are to be calculated.
In a Java mindset, you might create it like
trait Future[T] {
def get: T
}
Where 'get' blocks until the value is available.
You might realize this, and rewrite it to take a callback:
trait Future[T] {
def foreach(f: T => Unit): Unit
}
But then what happens if there are two uses for the future? It means you need to keep a list of callbacks. Also, what happens if a method receives a Future[Int] and needs to return a calculation based on the Int inside? Or what do you do if you have two futures and you need to calculate something based on the values they will provide?
But if you know of FP concepts, you know that instead of working directly on T, you can manipulate the Future instance.
trait Future[T] {
def map[U](f: T => U): Future[U]
}
Now your application changes so that each time you need to work on the contained value, you just return a new Future.
Once you start in this path, you can't stop there. You realize that in order to manipulate two futures, you just need to model as an applicative, in order to create futures, you need a monad definition for future, etc.
UPDATE: As suggested by #Eric, I've written a blog post: http://www.tikalk.com/incubator/blog/functional-programming-scala-rest-us
I finally understood how applicatives can help in day-to-day programming with that presentation:
https://web.archive.org/web/20100818221025/http://applicative-errors-scala.googlecode.com/svn/artifacts/0.6/chunk-html/index.html
The autor shows how applicatives can help for combining validations and handling failures.
The presentation is in Scala, but the author also provides the full code example for Haskell, Java and C#.
Warning: my answer is rather preachy/apologetic. So sue me.
Well, how often in your day-to-day Haskell programming do you create new data types? Sounds like you want to know when to make your own Applicative instance, and in all honesty unless you are rolling your own parser, you probably won't need to do it very much. Using applicative instances, on the other hand, you should learn to do frequently.
Applicative is not a "design pattern" like decorators or strategies. It is an abstraction, which makes it much more pervasive and generally useful, but much less tangible. The reason you have a hard time finding "practical uses" is because the example uses for it are almost too simple. You use decorators to put scrollbars on windows. You use strategies to unify the interface for both aggressive and defensive moves for your chess bot. But what are applicatives for? Well, they're a lot more generalized, so it's hard to say what they are for, and that's OK. Applicatives are handy as parsing combinators; the Yesod web framework uses Applicative to help set up and extract information from forms. If you look, you'll find a million and one uses for Applicative; it's all over the place. But since it's so abstract, you just need to get the feel for it in order to recognize the many places where it can help make your life easier.
I think Applicatives ease the general usage of monadic code. How many times have you had the situation that you wanted to apply a function but the function was not monadic and the value you want to apply it to is monadic? For me: quite a lot of times!
Here is an example that I just wrote yesterday:
ghci> import Data.Time.Clock
ghci> import Data.Time.Calendar
ghci> getCurrentTime >>= return . toGregorian . utctDay
in comparison to this using Applicative:
ghci> import Control.Applicative
ghci> toGregorian . utctDay <$> getCurrentTime
This form looks "more natural" (at least to my eyes :)
Coming at Applicative from "Functor" it generalizes "fmap" to easily express acting on several arguments (liftA2) or a sequence of arguments (using <*>).
Coming at Applicative from "Monad" it does not let the computation depend on the value that is computed. Specifically you cannot pattern match and branch on a returned value, typically all you can do is pass it to another constructor or function.
Thus I see Applicative as sandwiched in between Functor and Monad. Recognizing when you are not branching on the values from a monadic computation is one way to see when to switch to Applicative.
Here is an example taken from the aeson package:
data Coord = Coord { x :: Double, y :: Double }
instance FromJSON Coord where
parseJSON (Object v) =
Coord <$>
v .: "x" <*>
v .: "y"
There are some ADTs like ZipList that can have applicative instances, but not monadic instances. This was a very helpful example for me when understanding the difference between applicatives and monads. Since so many applicatives are also monads, it's easy to not see the difference between the two without a concrete example like ZipList.
I think it might be worthwhile to browse the sources of packages on Hackage, and see first-handedly how applicative functors and the like are used in existing Haskell code.
I described an example of practical use of the applicative functor in a discussion, which I quote below.
Note the code examples are pseudo-code for my hypothetical language which would hide the type classes in a conceptual form of subtyping, so if you see a method call for apply just translate into your type class model, e.g. <*> in Scalaz or Haskell.
If we mark elements of an array or hashmap with null or none to
indicate their index or key is valid yet valueless, the Applicative
enables without any boilerplate skipping the valueless elements while
applying operations to the elements that have a value. And more
importantly it can automatically handle any Wrapped semantics that
are unknown a priori, i.e. operations on T over
Hashmap[Wrapped[T]] (any over any level of composition, e.g. Hashmap[Wrapped[Wrapped2[T]]] because applicative is composable but monad is not).
I can already picture how it will make my code easier to
understand. I can focus on the semantics, not on all the
cruft to get me there and my semantics will be open under extension of
Wrapped whereas all your example code isn’t.
Significantly, I forgot to point out before that your prior examples
do not emulate the return value of the Applicative, which will be a
List, not a Nullable, Option, or Maybe. So even my attempts to
repair your examples were not emulating Applicative.apply.
Remember the functionToApply is the input to the
Applicative.apply, so the container maintains control.
list1.apply( list2.apply( ... listN.apply( List.lift(functionToApply) ) ... ) )
Equivalently.
list1.apply( list2.apply( ... listN.map(functionToApply) ... ) )
And my proposed syntactical sugar which the compiler would translate
to the above.
funcToApply(list1, list2, ... list N)
It is useful to read that interactive discussion, because I can't copy it all here. I expect that url to not break, given who the owner of that blog is. For example, I quote from further down the discussion.
the conflation of out-of-statement control flow with assignment is probably not desired by most programmers
Applicative.apply is for generalizing the partial application of functions to parameterized types (a.k.a. generics) at any level of nesting (composition) of the type parameter. This is all about making more generalized composition possible. The generality can’t be accomplished by pulling it outside the completed evaluation (i.e. return value) of the function, analogous to the onion can’t be peeled from the inside-out.
Thus it isn’t conflation, it is a new degree-of-freedom that is not currently available to you. Per our discussion up thread, this is why you must throw exceptions or stored them in a global variable, because your language doesn’t have this degree-of-freedom. And that is not the only application of these category theory functors (expounded in my comment in moderator queue).
I provided a link to an example abstracting validation in Scala, F#, and C#, which is currently stuck in moderator queue. Compare the obnoxious C# version of the code. And the reason is because the C# is not generalized. I intuitively expect that C# case-specific boilerplate will explode geometrically as the program grows.

Is the Scala 2.8 collections library a case of "the longest suicide note in history"? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
I have just started to look at the Scala collections library re-implementation which is coming in the imminent 2.8 release. Those familiar with the library from 2.7 will notice that the library, from a usage perspective, has changed little. For example...
> List("Paris", "London").map(_.length)
res0: List[Int] List(5, 6)
...would work in either versions. The library is eminently useable: in fact it's fantastic. However, those previously unfamiliar with Scala and poking around to get a feel for the language now have to make sense of method signatures like:
def map[B, That](f: A => B)(implicit bf: CanBuildFrom[Repr, B, That]): That
For such simple functionality, this is a daunting signature and one which I find myself struggling to understand. Not that I think Scala was ever likely to be the next Java (or /C/C++/C#) - I don't believe its creators were aiming it at that market - but I think it is/was certainly feasible for Scala to become the next Ruby or Python (i.e. to gain a significant commercial user-base)
Is this going to put people off coming to Scala?
Is this going to give Scala a bad name in the commercial world as an academic plaything that only dedicated PhD students can understand? Are CTOs and heads of software going to get scared off?
Was the library re-design a sensible idea?
If you're using Scala commercially, are you worried about this? Are you planning to adopt 2.8 immediately or wait to see what happens?
Steve Yegge once attacked Scala (mistakenly in my opinion) for what he saw as its overcomplicated type-system. I worry that someone is going to have a field day spreading FUD with this API (similarly to how Josh Bloch scared the JCP out of adding closures to Java).
Note - I should be clear that, whilst I believe that Joshua Bloch was influential in the rejection of the BGGA closures proposal, I don't ascribe this to anything other than his honestly-held beliefs that the proposal represented a mistake.
Despite whatever my wife and coworkers keep telling me, I don't think I'm an idiot: I have a good degree in mathematics from the University of Oxford, and I've been programming commercially for almost 12 years and in Scala for about a year (also commercially).
Note the inflammatory subject title is a quotation made about the manifesto of a UK political party in the early 1980s. This question is subjective but it is a genuine question, I've made it CW and I'd like some opinions on the matter.
I hope it's not a "suicide note", but I can see your point. You hit on what is at the same time both a strength and a problem of Scala: its extensibility. This lets us implement most major functionality in libraries. In some other languages, sequences with something like map or collect would be built in, and nobody has to see all the hoops the compiler has to go through to make them work smoothly. In Scala, it's all in a library, and therefore out in the open.
In fact the functionality of map that's supported by its complicated type is pretty advanced. Consider this:
scala> import collection.immutable.BitSet
import collection.immutable.BitSet
scala> val bits = BitSet(1, 2, 3)
bits: scala.collection.immutable.BitSet = BitSet(1, 2, 3)
scala> val shifted = bits map { _ + 1 }
shifted: scala.collection.immutable.BitSet = BitSet(2, 3, 4)
scala> val displayed = bits map { _.toString + "!" }
displayed: scala.collection.immutable.Set[java.lang.String] = Set(1!, 2!, 3!)
See how you always get the best possible type? If you map Ints to Ints you get again a BitSet, but if you map Ints to Strings, you get a general Set. Both the static type and the runtime representation of map's result depend on the result type of the function that's passed to it. And this works even if the set is empty, so the function is never applied! As far as I know there is no other collection framework with an equivalent functionality. Yet from a user perspective this is how things are supposed to work.
The problem we have is that all the clever technology that makes this happen leaks into the type signatures which become large and scary. But maybe a user should not be shown by default the full type signature of map? How about if she looked up map in BitSet she got:
map(f: Int => Int): BitSet (click here for more general type)
The docs would not lie in that case, because from a user perspective indeed map has the type (Int => Int) => BitSet. But map also has a more general type which can be inspected by clicking on another link.
We have not yet implemented functionality like this in our tools. But I believe we need to do this, to avoid scaring people off and to give more useful info. With tools like that, hopefully smart frameworks and libraries will not become suicide notes.
I do not have a PhD, nor any other kind of degree neither in CS nor math nor indeed any other field. I have no prior experience with Scala nor any other similar language. I have no experience with even remotely comparable type systems. In fact, the only language that I have more than just a superficial knowledge of which even has a type system is Pascal, not exactly known for its sophisticated type system. (Although it does have range types, which AFAIK pretty much no other language has, but that isn't really relevant here.) The other three languages I know are BASIC, Smalltalk and Ruby, none of which even have a type system.
And yet, I have no trouble at all understanding the signature of the map function you posted. It looks to me like pretty much the same signature that map has in every other language I have ever seen. The difference is that this version is more generic. It looks more like a C++ STL thing than, say, Haskell. In particular, it abstracts away from the concrete collection type by only requiring that the argument is IterableLike, and also abstracts away from the concrete return type by only requiring that an implicit conversion function exists which can build something out of that collection of result values. Yes, that is quite complex, but it really is only an expression of the general paradigm of generic programming: do not assume anything that you don't actually have to.
In this case, map does not actually need the collection to be a list, or being ordered or being sortable or anything like that. The only thing that map cares about is that it can get access to all elements of the collection, one after the other, but in no particular order. And it does not need to know what the resulting collection is, it only needs to know how to build it. So, that is what its type signature requires.
So, instead of
map :: (a → b) → [a] → [b]
which is the traditional type signature for map, it is generalized to not require a concrete List but rather just an IterableLike data structure
map :: (IterableLike i, IterableLike j) ⇒ (a → b) → i → j
which is then further generalized by only requiring that a function exists that can convert the result to whatever data structure the user wants:
map :: IterableLike i ⇒ (a → b) → i → ([b] → c) → c
I admit that the syntax is a bit clunkier, but the semantics are the same. Basically, it starts from
def map[B](f: (A) ⇒ B): List[B]
which is the traditional signature for map. (Note how due to the object-oriented nature of Scala, the input list parameter vanishes, because it is now the implicit receiver parameter that every method in a single-dispatch OO system has.) Then it generalized from a concrete List to a more general IterableLike
def map[B](f: (A) ⇒ B): IterableLike[B]
Now, it replaces the IterableLike result collection with a function that produces, well, really just about anything.
def map[B, That](f: A ⇒ B)(implicit bf: CanBuildFrom[Repr, B, That]): That
Which I really believe is not that hard to understand. There's really only a couple of intellectual tools you need:
You need to know (roughly) what map is. If you gave only the type signature without the name of the method, I admit, it would be a lot harder to figure out what is going on. But since you already know what map is supposed to do, and you know what its type signature is supposed to be, you can quickly scan the signature and focus on the anomalies, like "why does this map take two functions as arguments, not one?"
You need to be able to actually read the type signature. But even if you have never seen Scala before, this should be quite easy, since it really is just a mixture of type syntaxes you already know from other languages: VB.NET uses square brackets for parametric polymorphism, and using an arrow to denote the return type and a colon to separate name and type, is actually the norm.
You need to know roughly what generic programming is about. (Which isn't that hard to figure out, since it's basically all spelled out in the name: it's literally just programming in a generic fashion).
None of these three should give any professional or even hobbyist programmer a serious headache. map has been a standard function in pretty much every language designed in the last 50 years, the fact that different languages have different syntax should be obvious to anyone who has designed a website with HTML and CSS and you can't subscribe to an even remotely programming related mailinglist without some annoying C++ fanboy from the church of St. Stepanov explaining the virtues of generic programming.
Yes, Scala is complex. Yes, Scala has one of the most sophisticated type systems known to man, rivaling and even surpassing languages like Haskell, Miranda, Clean or Cyclone. But if complexity were an argument against success of a programming language, C++ would have died long ago and we would all be writing Scheme. There are lots of reasons why Scala will very likely not be successful, but the fact that programmers can't be bothered to turn on their brains before sitting down in front of the keyboard is probably not going to be the main one.
Same thing in C++:
template <template <class, class> class C,
class T,
class A,
class T_return,
class T_arg
>
C<T_return, typename A::rebind<T_return>::other>
map(C<T, A> &c,T_return(*func)(T_arg) )
{
C<T_return, typename A::rebind<T_return>::other> res;
for ( C<T,A>::iterator it=c.begin() ; it != c.end(); it++ ){
res.push_back(func(*it));
}
return res;
}
Well, I can understand your pain, but, quite frankly, people like you and I -- or pretty much any regular Stack Overflow user -- are not the rule.
What I mean by that is that... most programmers won't care about that type signature, because they'll never see them! They don't read documentation.
As long as they saw some example of how the code works, and the code doesn't fail them in producing the result they expect, they won't ever look at the documentation. When that fails, they'll look at the documentation and expect to see usage examples at the top.
With these things in mind, I think that:
Anyone (as in, most people) who ever comes across that type signature will mock Scala to no end if they are pre-disposed against it, and will consider it a symbol of Scala's power if they like Scala.
If the documentation isn't enhanced to provide usage examples and explain clearly what a method is for and how to use it, it can detract from Scala adoption a bit.
In the long run, it won't matter. That Scala can do stuff like that will make libraries written for Scala much more powerful and safer to use. These libraries and frameworks will attract programmers atracted to powerful tools.
Programmers who like simplicity and directness will continue to use PHP, or similar languages.
Alas, Java programmers are much into power tools, so, in answering that, I have just revised my expectation of mainstream Scala adoption. I have no doubt at all that Scala will become a mainstream language. Not C-mainstream, but perhaps Perl-mainstream or PHP-mainstream.
Speaking of Java, did you ever replace the class loader? Have you ever looked into what that involves? Java can be scary, if you look at the places framework writers do. It's just that most people don't. The same thing applies to Scala, IMHO, but early adopters have a tendency to look under each rock they encounter, to see if there's something hiding there.
Is this going to put people off coming to Scala?
Yes, but it will also prevent people from being put off. I've considered the lack of collections that use higher-kinded types to be a major weakness ever since Scala gained support for higher-kinded types. It make the API docs more complicated, but it really makes usage more natural.
Is this going to give scala a bad name in the commercial world as an academic plaything that only dedicated PhD students can understand? Are CTOs and heads of software going to get scared off?
Some probably will. I don't think Scala is accessible to many "professional" developers, partially due to the complexity of Scala and partly due to the unwillingness of many developers to learn. The CTOs who employ such developers will rightly be scared off.
Was the library re-design a sensible idea?
Absolutely. It makes collections fit much better with the rest of the language and the type system, even if it still has some rough edges.
If you're using scala commercially, are you worried about this? Are you planning to adopt 2.8 immediately or wait to see what happens?
I'm not using it commercially. I'll probably wait until at least a couple revs into the 2.8.x series before even trying to introduce it so that the bugs can be flushed out. I'll also wait to see how much success EPFL has in improving its development a release processes. What I'm seeing looks hopeful, but I work for a conservative company.
One the more general topic of "is Scala too complicated for mainstream developers?"...
Most developers, mainstream or otherwise, are maintaining or extending existing systems. This means that most of what they use is dictated by decisions made long ago. There are still plenty of people writing COBOL.
Tomorrow's mainstream developer will work maintaining and extending the applications that are being built today. Many of these applications are not being built by mainstream developers. Tomorrow's mainstream developers will use the language that is being used by today's most successful developers of new applications.
One way that the Scala community can help ease the fear of programmers new to Scala is to focus on practice and to teach by example--a lot of examples that start small and grow gradually larger. Here are a few sites that take this approach:
Daily Scala
Learning Scala in small bites
Simply Scala
After spending some time on these sites, one quickly realizes that Scala and its libraries, though perhaps difficult to design and implement, are not so difficult to use, especially in the common cases.
I have an undergraduate degree from a cheap "mass market" US university, so I'd say I fall into the middle of the user intelligence (or at least education) scale :) I've been dabbling with Scala for just a few months and have worked on two or three non-trivial apps.
Especially now that IntelliJ has released their fine IDE with what IMHO is currently the best Scala plugin, Scala development is relatively painless:
I find I can use Scala as a "Java without semicolons," i.e. I write similar-looking code to what I'd do in Java, and benefit a little from syntactic brevity such as that gained by type inference. Exception handling, when I do it at all, is more convenient. Class definition is much less verbose without the getter/setter boilerplate.
Once in a while I manage to write a single line to accomplish the equivalent of multiple lines of Java. Where applicable, chains of functional methods like map, fold, collect, filter etc. are fun to compose and elegant to behold.
Only rarely do I find myself benefitting from Scala's more high-powered features: Closures and partial (or curried) functions, pattern matching... that kinda thing.
As a newbie, I continue to struggle with the terse and idiomatic syntax. Method calls without parameters don't need parentheses except where they do; cases in the match statement need a fat arrow ( => ), but there are also places where you need a thin arrow ( -> ). Many methods have short but rather cryptic names like /: or \: - I can get my stuff done if I flip enough manual pages, but some of my code ends up looking like Perl or line noise. Ironically, one of the most popular bits of syntactic shorthand is missing in action: I keep getting bitten by the fact that Int doesn't define a ++ method.
This is just my opinion: I feel like Scala has the power of C++ combined with the complexity and readability of C++. The syntactic complexity of the language also makes the API documentation hard to read.
Scala is very well thought out and brilliant in many respects. I suspect many an academic would love to program in it. However, it's also full of cleverness and gotchas, it has a much higher learning curve than Java and is harder to read. If I scan the fora and see how many developers are still struggling with the finer points of Java, I cannot conceive of Scala ever becoming a mainstream language. No company will be able to justify sending its developers on a 3 week Scala course when formerly they only needed a 1 week Java course.
I think primary problem with that method is that the (implicit bf : CanBuildFrom[Repr, B, That]) goes without any explanation. Even though I know what implicit arguments are there's nothing indicating how this affects the call. Chasing through the scaladoc only leaves me more confused (few of the classes related to CanBuildFrom even have documentation).
I think a simple "there must be an implicit object in scope for bf that provides a builder for objects of type B into the return type That" would help somewhat, but it's kind of a heady concept when all you really want to do is map A's to B's. In fact, I'm not sure that's right, because I don't know what the type Repr means, and the documentation for Traversable certainly gives no clue at all.
So, I'm left with two options, neither of them pleasant:
Assume it will just work how the old map works and how map works in most other languages
Dig into the source code some more
I get that Scala is essentially exposing the guts of how these things work and that ultimately this is provide a way to do what oxbow_lakes is describing. But it's a distraction in the signature.
I'm a Scala beginner and I honestly don't see a problem with that type signature. The parameter is the function to map and the implicit parameter the builder to return the correct collection. Clear and readable.
The whole thing's quite elegant, actually. The builder type parameters let the compiler choose the correct return type while the implicit parameter mechanism hides this extra parameter from the class user. I tried this:
Map(1 -> "a", 2 -> "b").map((t) => (t._2) -> (t._1)) // returns Map("a" -> 1, "b" -> 2)
Map(1 -> "a", 2 -> "b").map((t) => t._2) // returns List("a", "b")
That's polymorphism done right.
Now, granted, it's not a mainstream paradigm and it will scare away many. But, it will also attract many who value its expressiveness and elegance.
Unfortunately the signature for map that you gave is an incorrect one for map and there is indeed legitimate criticism.
The first criticism is that by subverting the signature for map, we have something that is more general. It is a common error to believe that this is a virtue by default. It isn't. The map function is very well defined as a covariant functor Fx -> (x -> y) -> Fy with adherence to the two laws of composition and identity. Anything else attributed to "map" is a travesty.
The given signature is something else, but it is not map. What I suspect it is trying to be is a specialised and slightly altered version of the "traverse" signature from the paper, The Essence of the Iterator Pattern. Here is its signature:
traverse :: (Traversable t, Applicative f) => (a -> f b) -> t a -> f (t b)
I shall convert it to Scala:
def traverse[A, B](f: A => F[B], a: T[A])(implicit t: Traversable[T], ap: Applicative[F]): F[T[B]
Of course it fails -- it is not general enough! Also, it is slightly different (note that you can get map by running traverse through the Identity functor). However, I suspect that if the library writers were more aware of library generalisations that are well documented (Applicative Programming with Effects precedes the aforementioned), then we wouldn't see this error.
Second, the map function is a special-case in Scala because of its use in for-comprehensions. This unfortunately means that a library designer who is better equipped cannot ignore this error without also sacrificing the syntactic sugar of comprehensions. In other words, if the Scala library designers were to destroy a method, then this is easily ignored, but please not map!
I hope someone speaks up about it, because as it is, it will become harder to workaround the errors that Scala insists on making, apparently for reasons that I have strong objections to. That is, the solution to "the irresponsible objections from the average programmer (i.e. too hard!)" is not "appease them to make it easier for them" but instead, provide pointers and assistance to become better programmers. Myself and Scala's objectives are in contention on this issue, but back to your point.
You were probably making your point, predicting specific responses from "the average programmer." That is, the people who will claim "but it is too complicated!" or some such. These are the Yegges or Blochs that you refer to. My response to these people of the anti-intellectualism/pragmatism movement is quite harsh and I'm already anticipating a barrage of responses, so I will omit it.
I truly hope the Scala libraries improve, or at least, the errors can be safely tucked away in a corner. Java is a language where "trying to do anything useful" is so incredibly costly, that it is often not worth it because the overwhelming amount of errors simply cannot be avoided. I implore Scala to not go down the same path.
I totally agree with both the question and Martin's answer :). Even in Java, reading javadoc with generics is much harder than it should be due to the extra noise. This is compounded in Scala where implicit parameters are used as in the questions's example code (while the implicits do very useful collection-morphing stuff).
I don't think its a problem with the language per se - I think its more a tooling issue. And while I agree with what Jörg W Mittag says, I think looking at scaladoc (or the documentation of a type in your IDE) - it should require as little brain power as possible to grok what a method is, what it takes and returns. There shouldn't be a need to hack up a bit of algebra on a bit of paper to get it :)
For sure IDEs need a nice way to show all the methods for any variable/expression/type (which as with Martin's example can have all the generics inlined so its nice and easy to grok). I like Martin's idea of hiding the implicits by default too.
To take the example in scaladoc...
def map[B, That](f: A => B)(implicit bf: CanBuildFrom[Repr, B, That]): That
When looking at this in scaladoc I'd like the generic block [B, That] to be hidden by default as well as the implicit parameter (maybe they show if you hover a little icon with the mouse) - as its extra stuff to grok reading it which usually isn't that relevant. e.g. imagine if this looked like...
def map(f: A => B): That
nice and clear and obvious what it does. You might wonder what 'That' is, if you mouse over or click it it could expand the [B, That] text highlighting the 'That' for example.
Maybe a little icon could be used for the [] declaration and (implicit...) block so its clear there are little bits of the statement collapsed? Its hard to use a token for it, but I'll use a . for now...
def map.(f: A => B).: That
So by default the 'noise' of the type system is hidden from the main 80% of what folks need to look at - the method name, its parameter types and its return type in nice simple concise way - with little expandable links to the detail if you really care that much.
Mostly folks are reading scaladoc to find out what methods they can call on a type and what parameters they can pass. We're kinda overloading users with way too much detail right how IMHO.
Here's another example...
def orElse[A1 <: A, B1 >: B](that: PartialFunction[A1, B1]): PartialFunction[A1, B1]
Now if we hid the generics declaration its easier to read
def orElse(that: PartialFunction[A1, B1]): PartialFunction[A1, B1]
Then if folks hover over, say, A1 we could show the declaration of A1 being A1 <: A. Covariant and contravariant types in generics add lots of noise too which can be rendered in a much easier to grok way to users I think.
I don't know how to break it to you, but I have a PhD from Cambridge, and I'm using 2.8 just fine.
More seriously, I hardly spent any time with 2.7 (it won't inter-op with a Java library I am using) and started using Scala just over a month ago. I have some experience with Haskell (not much), but just ignored the stuff you're worried about and looked for methods that matched my experience with Java (which I use for a living).
So: I am a "new user" and I wasn't put off - the fact that it works like Java gave me enough confidence to ignore the bits I didn't understand.
(However, the reason I was looking at Scala was partly to see whether to push it at work, and I am not going to do so yet. Making the documentation less intimidating would certainly help, but what surprised me is how much it is still changing and being developed (to be fair what surprised me most was how awesome it is, but the changes came a close second). So I guess what I am saying is that I'd rather prefer the limited resources were put into getting it into a final state - I don't think they were expecting to be this popular this soon.)
Don't know Scala at all, however a few weeks ago I could not read Clojure. Now I can read most of it, but can not write anything yet beyond the most simplistic examples. I suspect Scala is no different. You need a good book or course depending on how you learn. Just reading the map declaration above, I got maybe 1/3 of it.
I believe the bigger problems are not the syntax of these languages, but adopting and internalizing the paradigms that make them usable in everyday production code. For me Java was not a huge leap from C++, which was not a huge leap from C, which was not a leap at all from Pascal, nor Basic etc... But coding in a functional language like Clojure is a huge leap (for me anyway). I guess in Scala you can code in Java style or Scala style. But in Clojure you will create quite the mess trying to keep your imperative habits from Java.
Scala has a lot of crazy features (particularly where implicit parameters are concerned) that look very complicated and academic, but are designed to make things easy to use. The most useful ones get syntactic sugar (like [A <% B] which means that an object of type A has an implicit conversion to an object of type B) and a well-documented explanation of what they do. But most of the time, as a client of these libraries you can ignore the implicit parameters and trust them to do the right thing.
Is this going to put people off coming to Scala?
I don't think it is the main factor that will affect how popular Scala will become, because Scala has a lot of power and its syntax is not as foreign to a Java/C++/PHP programmer as Haskell, OCaml, SML, Lisps, etc..
But I do think Scala's popularity will plateau at less than where Java is today, because I also think the next mainstream language must be much simplified, and the only way I see to get there is pure immutability, i.e. declarative like HTML, but Turing complete. However, I am biased because I am developing such a language, but I only did so after ruling out over a several month study that Scala could not suffice for what I needed.
Is this going to give Scala a bad name in the commercial world as an academic plaything that only dedicated PhD students can understand? Are CTOs and heads of software going to get scared off?
I don't think Scala's reputation will suffer from the Haskell complex. But I think that some will put off learning it, because for most programmers, I don't yet see a use case that forces them to use Scala, and they will procrastinate learning about it. Perhaps the highly-scalable server side is the most compelling use case.
And, for the mainstream market, first learning Scala is not a "breath of fresh air", where one is writing programs immediately, such as first using HTML or Python. Scala tends to grow on you, after one learns all the details that one stumbles on from the start. However, maybe if I had read Programming in Scala from the start, my experience and opinion of the learning curve would have been different.
Was the library re-design a sensible idea?
Definitely.
If you're using Scala commercially, are you worried about this? Are you planning to adopt 2.8 immediately or wait to see what happens?
I am using Scala as the initial platform of my new language. I probably wouldn't be building code on Scala's collection library if I was using Scala commercially otherwise. I would create my own category theory based library, since the one time I looked, I found Scalaz's type signatures even more verbose and unwieldy than Scala's collection library. Part of that problem perhaps is Scala's way of implementing type classes, and that is a minor reason I am creating my own language.
I decided to write this answer, because I wanted to force myself to research and compare Scala's collection class design to the one I am doing for my language. Might as well share my thought process.
The 2.8 Scala collections use of a builder abstraction is a sound design principle. I want to explore two design tradeoffs below.
WRITE-ONLY CODE: After writing this section, I read Carl Smotricz's comment which agrees with what I expect to be the tradeoff. James Strachan and davetron5000's comments concur that the meaning of That (it is not even That[B]) and the mechanism of the implicit is not easy to grasp intuitively. See my use of monoid in issue #2 below, which I think is much more explicit. Derek Mahar's comment is about writing Scala, but what about reading the Scala of others that is not "in the common cases".
One criticism I have read about Scala, is that it is easier to write it, than read the code that others have written. And I find this to be occasionally true for various reasons (e.g. many ways to write a function, automatic closures, Unit for DSLs, etc), but I am undecided if this is major factor. Here the use of implicit function parameters has pluses and minuses. On the plus side, it reduces verbosity and automates selection of the builder object. In Odersky's example the conversion from a BitSet, i.e. Set[Int], to a Set[String] is implicit. The unfamiliar reader of the code might not readily know what the type of collection is, unless they can reason well about the all the potential invisible implicit builder candidates which might exist in the current package scope. Of course, the experienced programmer and the writer of the code will know that BitSet is limited to Int, thus a map to String has to convert to a different collection type. But which collection type? It isn't specified explicitly.
AD-HOC COLLECTION DESIGN: After writing this section, I read Tony Morris's comment and realized I am making nearly the same point. Perhaps my more verbose exposition will make the point more clear.
In "Fighting Bit Rot with Types" Odersky & Moors, two use cases are presented. They are the restriction of BitSet to Int elements, and Map to pair tuple elements, and are provided as the reason that the general element mapping function, A => B, must be able to build alternative destination collection types. However, afaik this is flawed from a category theory perspective. To be consistent in category theory and thus avoid corner cases, these collection types are functors, in which each morphism, A => B, must map between objects in the same functor category, List[A] => List[B], BitSet[A] => BitSet[B]. For example, an Option is a functor that can be viewed as a collection of sets of one Some( object ) and the None. There is no general map from Option's None, or List's Nil, to other functors which don't have an "empty" state.
There is a tradeoff design choice made here. In the design for collections library of my new language, I chose to make everything a functor, which means if I implement a BitSet, it needs to support all element types, by using a non-bit field internal representation when presented with a non-integer type parameter, and that functionality is already in the Set which it inherits from in Scala. And Map in my design needs to map only its values, and it can provide a separate non-functor method for mapping its (key,value) pair tuples. One advantage is that each functor is then usually also an applicative and perhaps a monad too. Thus all functions between element types, e.g. A => B => C => D => ..., are automatically lifted to the functions between lifted applicative types, e.g. List[A] => List[B] => List[C] => List[D] => .... For mapping from a functor to another collection class, I offer a map overload which takes a monoid, e.g. Nil, None, 0, "", Array(), etc.. So the builder abstraction function is the append method of a monoid and is supplied explicitly as a necessary input parameter, thus with no invisible implicit conversions. (Tangent: this input parameter also enables appending to non-empty monoids, which Scala's map design can't do.) Such conversions are a map and a fold in the same iteration pass. Also I provide a traversable, in the category sense, "Applicative programming with effects" McBride & Patterson, which also enables map + fold in a single iteration pass from any traversable to any applicative, where most every collection class is both. Also the state monad is an applicative and thus is a fully generalized builder abstraction from any traversable.
So afaics the Scala collections is "ad-hoc" in the sense that it is not grounded in category theory, and category theory is the essense of higher-level denotational semantics. Although Scala's implicit builders are at first appearance "more generalized" than a functor model + monoid builder + traversable -> applicative, they are afaik not proven to be consistent with any category, and thus we don't know what rules they follow in the most general sense and what the corner cases will be given they may not obey any category model. It is simply not true that adding more variables makes something more general, and this was one of huge benefits of category theory is it provides rules by which to maintain generality while lifting to higher-level semantics. A collection is a category.
I read somewhere, I think it was Odersky, as another justification for the library design, is that programming in a pure functional style has the cost of limited recursion and speed where tail recursion isn't used. I haven't found it difficult to employ tail recursion in every case that I have encountered so far.
Additionally I am carrying in my mind an incomplete idea that some of Scala's tradeoffs are due to trying to be both an mutable and immutable language, unlike for example Haskell or the language I am developing. This concurs with Tony Morris's comment about for comprehensions. In my language, there are no loops and no mutable constructs. My language will sit on top of Scala (for now) and owes much to it, and this wouldn't be possible if Scala didn't have the general type system and mutability. That might not be true though, because I think Odersky & Moors ("Fighting Bit Rot with Types") are incorrect to state that Scala is the only OOP language with higher-kinds, because I verified (myself and via Bob Harper) that Standard ML has them. Also appears SML's type system may be equivalently flexible (since 1980s), which may not be readily appreciated because the syntax is not so much similar to Java (and C++/PHP) as Scala. In any case, this isn't a criticism of Scala, but rather an attempt to present an incomplete analysis of tradeoffs, which is I hope germane to the question. Scala and SML don't suffer from Haskell's inability to do diamond multiple inheritance, which is critical and I understand is why so many functions in the Haskell Prelude are repeated for different types.
It seems necessary to state ones degree here: B.A. in Political Science and B.ed in Computer Science.
To the point:
Is this going to put people off coming to Scala?
Scala is difficult, because its underlying programming paradigm is difficult. Functional programming scares a lot of people. It is possible to build closures in PHP but people rarely do. So no, not this signature but all the rest will put people off, if they do not have the specific education to make them value the power of the underlying paradigm.
If this education is available, everyone can do it. Last year I build a chess computer with a bunch of school kids in SCALA! They had their problems but they did fine in the end.
If you're using Scala commercially, are you worried about this? Are you planning to adopt 2.8 immediately or wait to see what happens?
I would not be worried.
I have a maths degree from Oxford too! It took me a while to 'get' the new collections stuff. But I like it a lot now that I do. In fact, the typing of 'map' was one of the first big things that bugged me in 2.7 (perhaps since the first thing I did was subclass one of the collection classes).
Reading Martin's paper on the new 2.8 collections really helped explain the use of implicits, but yes the documentation itself definitely needs to do a better job of explaining the role of different kind of implicits within method signatures of core APIs.
My main concern is more this: when is 2.8 going to be released? When will the bug reports stop coming in for it? have scala team bitten off more than they can chew with 2.8 / tried to change too much at once?
I'd really like to see 2.8 stabilised for release as a priority before adding anything else new at all, and wonder (while watching from the sidelines) if some improvements could be made to the way the development roadmap for the scala compiler is managed.
What about error messages in use site?
And what about when comes the use case one needs to integrate existing types with a custom one that fits a DSL. One have to be well educated on matters of association, precedence, implicit conversions, implicit parameters, higher kinds, and maybe existential types.
It's very good to know that mostly it's simple but it's not necessarily enough. At least there must be one guy who knows this stuff if widespread library is to be designed.