I am new to spray.
I have work with several strange (for me) programming languages like python, JQuery, etc... With them I could at least understand what some code segment do.
Unfortunately with Spray, I can not read and understand even a simple code.
Could some one please help me to read (describe with word, what the code do) the following simple code block.
Note: Very high level I know, this will pick the url parameter and add them together. But what I want is, understand this code block crystal clearly, as I could teach to someone else. HNil, Directive1, Directive1, :: are some what strange for me.
val twoIntParameters: Directive[Int :: Int :: HNil] =
parameters('a.as[Int], 'b.as[Int])
val myDirective: Directive1[String] =
twoIntParameters.hmap {
case a :: b :: HNil => (a + b).toString
}
// test `myDirective` using the testkit DSL
Get("/?a=2&b=5") ~> myDirective(x => complete(x)) ~> check {
responseAs[String] === "7"
}
spray-routing is build around the concept of Directive.
You can think of a Directive as a transformation over an HTTP request.
The cardinality associated with a directive is the number of arguments is passes down the transform chain after performing the transformation.
Directive0 is a directive that doesn't provide (or extract) any argument.
Directive1[A] provides one argument of type A.
Directive[A :: B :: HNil] provides 2 arguments of types A and B, or - to be more precise - provides an heterogeneous list made of A and B (the implementation is a shapeless's HList).
Let's take the examples in your code
val twoIntParameters: Directive[Int :: Int :: HNil] =
parameters('a.as[Int], 'b.as[Int])
You're defining a new directive that extracts two integers from the HTTP request, i.e. has the type Directive[Int :: Int :: HNil].
The implementation simply leverages a directive provided already by spray, i.e. parameters.
parameters is a directive that allows to extract the query parameters from a HTTP request and convert them to a specific type, in this case Int for both parameters.
val myDirective: Directive1[String] =
twoIntParameters.hmap {
case a :: b :: HNil => (a + b).toString
}
myDirective is a directive that extracts one parameter of type String.
Its implementation uses the previously defined twoIntParameters directive and maps over its result, applying a transformation to it.
In this case we're taking the two Int, summing them and turning the result into a String.
So, what's with the hmap? That's just a way provided by spray of working with Directives that return a shapeless HList. hmap requires a function that HList to anything, in this case a String.
HLists can be pattern matched over, just like a normal scala List, and that's what you're seeing in the example.
Finally, this is just an idea of how directives work from a functional point of view.
If you want to understand the fine details of the DSL syntax, you will have to dig a little further and read about the Magnet Pattern.
Here I found a very good workshop.
https://www.youtube.com/watch?v=XPuOlpWEvmw
Related
I'm working on some fairly generic code (yet another CSV file reader) and I've run into some trouble with contravariance. I've stripped the code down to something that I think demonstrates the problem here (I copied this from my Scala worksheet):
def f1(x: Int)(s: String) = Try(x+s.toInt)
val f1a: Int=>String=>Try[Int] = f1 _
val r1 = f1a(3)("2")
def f2(x: String)(s: String) = Try(x.toInt+s.toInt)
val f2a: String=>String=>Try[Int] = f2 _
val r2 = f2a("3")("2")
val fs: Seq[String with Int=>String=>Try[Int]] = Seq(f1a, f2a)
val xs: Seq[Any] = Seq(3,"3")
val r3 = for {(f,x) <- fs zip xs} yield f(x)("2")
Last line edited to avoid irrelevant issues after #Lee and #badcook commented.
I've tried to simplify things for the reader by providing the types for the vals although of course that's not required by the compiler. Each of the expressions r1 and r2 evaluates to Success(5) as expected. The expression for r3 does not compile. The error is:
type mismatch; found : x.type (with underlying type Any) required: String with Int
The problem essentially lies with the type of x, the parameter of f(x) in the for-comprehension. This is a contravariant position (the argument to a function) but xs and fs are covariant (the element type of a Seq). Thus x has type Any, but f requires a type String with Int.
This is an uninhabited compound type and I have not been able to find a clean way to solve this problem. I can cast the values f1a and f2a to Any=>Int=>String but that's not the way we like to do things in Scala!
Based on what I assume you are trying to do, the Scala compiler is saving you from one mistake. Your for comprehension is a Cartesian product of your elements and your functions. This means your function expecting an Int is going to get called once with a String and your function expecting a String is going to get called once with an Int.
What you probably want is to zip your two Seqs together and then map over the resulting pairs, applying each function to its paired element. Unfortunately, as you've noted, contravariance will prevent this from typechecking. This is fundamentally because all elements of a Seq must be the same type, whereas you want to preserve the fact that your elements have different types and that those types match the different types of your functions.
In short, if you want to go down this path, you're looking for heterogeneous lists, also known as HLists, rather than ordinary Scala collections. shapeless has an implementation of this and provides map and zip methods to go along with it. There's a bit of a learning curve there (I would recommend getting familiar with higher-rank types and how shapeless implements them) and HLists are a bit unwieldy to use compared to ordinary collections, but this will solve this particular kind of problem.
I have been looking into FP languages (off and on) for some time and have played with Scala, Haskell, F#, and some others. I like what I see and understand some of the fundamental concepts of FP (with absolutely no background in Category Theory - so don't talk Math, please).
So, given a type M[A] we have map which takes a function A=>B and returns a M[B]. But we also have flatMap which takes a function A=>M[B] and returns a M[B]. We also have flatten which takes a M[M[A]] and returns a M[A].
In addition, many of the sources I have read describe flatMap as map followed by flatten.
So, given that flatMap seems to be equivalent to flatten compose map, what is its purpose? Please don't say it is to support 'for comprehensions' as this question really isn't Scala-specific. And I am less concerned with the syntactic sugar than I am in the concept behind it. The same question arises with Haskell's bind operator (>>=). I believe they both are related to some Category Theory concept but I don't speak that language.
I have watched Brian Beckman's great video Don't Fear the Monad more than once and I think I see that flatMap is the monadic composition operator but I have never really seen it used the way he describes this operator. Does it perform this function? If so, how do I map that concept to flatMap?
BTW, I had a long writeup on this question with lots of listings showing experiments I ran trying to get to the bottom of the meaning of flatMap and then ran into this question which answered some of my questions. Sometimes I hate Scala implicits. They can really muddy the waters. :)
FlatMap, known as "bind" in some other languages, is as you said yourself for function composition.
Imagine for a moment that you have some functions like these:
def foo(x: Int): Option[Int] = Some(x + 2)
def bar(x: Int): Option[Int] = Some(x * 3)
The functions work great, calling foo(3) returns Some(5), and calling bar(3) returns Some(9), and we're all happy.
But now you've run into the situation that requires you to do the operation more than once.
foo(3).map(x => foo(x)) // or just foo(3).map(foo) for short
Job done, right?
Except not really. The output of the expression above is Some(Some(7)), not Some(7), and if you now want to chain another map on the end you can't because foo and bar take an Int, and not an Option[Int].
Enter flatMap
foo(3).flatMap(foo)
Will return Some(7), and
foo(3).flatMap(foo).flatMap(bar)
Returns Some(15).
This is great! Using flatMap lets you chain functions of the shape A => M[B] to oblivion (in the previous example A and B are Int, and M is Option).
More technically speaking; flatMap and bind have the signature M[A] => (A => M[B]) => M[B], meaning they take a "wrapped" value, such as Some(3), Right('foo), or List(1,2,3) and shove it through a function that would normally take an unwrapped value, such as the aforementioned foo and bar. It does this by first "unwrapping" the value, and then passing it through the function.
I've seen the box analogy being used for this, so observe my expertly drawn MSPaint illustration:
This unwrapping and re-wrapping behavior means that if I were to introduce a third function that doesn't return an Option[Int] and tried to flatMap it to the sequence, it wouldn't work because flatMap expects you to return a monad (in this case an Option)
def baz(x: Int): String = x + " is a number"
foo(3).flatMap(foo).flatMap(bar).flatMap(baz) // <<< ERROR
To get around this, if your function doesn't return a monad, you'd just have to use the regular map function
foo(3).flatMap(foo).flatMap(bar).map(baz)
Which would then return Some("15 is a number")
It's the same reason you provide more than one way to do anything: it's a common enough operation that you may want to wrap it.
You could ask the opposite question: why have map and flatten when you already have flatMap and a way to store a single element inside your collection? That is,
x map f
x filter p
can be replaced by
x flatMap ( xi => x.take(0) :+ f(xi) )
x flatMap ( xi => if (p(xi)) x.take(0) :+ xi else x.take(0) )
so why bother with map and filter?
In fact, there are various minimal sets of operations you need to reconstruct many of the others (flatMap is a good choice because of its flexibility).
Pragmatically, it's better to have the tool you need. Same reason why there are non-adjustable wrenches.
The simplest reason is to compose an output set where each entry in the input set may produce more than one (or zero!) outputs.
For example, consider a program which outputs addresses for people to generate mailers. Most people have one address. Some have two or more. Some people, unfortunately, have none. Flatmap is a generalized algorithm to take a list of these people and return all of the addresses, regardless of how many come from each person.
The zero output case is particularly useful for monads, which often (always?) return exactly zero or one results (think Maybe- returns zero results if the computation fails, or one if it succeeds). In that case you want to perform an operation on "all of the results", which it just so happens may be one or many.
The "flatMap", or "bind", method, provides an invaluable way to chain together methods that provide their output wrapped in a Monadic construct (like List, Option, or Future). For example, suppose you have two methods that produce a Future of a result (eg. they make long-running calls to databases or web service calls or the like, and should be used asynchronously):
def fn1(input1: A): Future[B] // (for some types A and B)
def fn2(input2: B): Future[C] // (for some types B and C)
How to combine these? With flatMap, we can do this as simply as:
def fn3(input3: A): Future[C] = fn1(a).flatMap(b => fn2(b))
In this sense, we have "composed" a function fn3 out of fn1 and fn2 using flatMap, which has the same general structure (and so can be composed in turn with further similar functions).
The map method would give us a not-so-convenient - and not readily chainable - Future[Future[C]]. Certainly we can then use flatten to reduce this, but the flatMap method does it in one call, and can be chained as far as we wish.
This is so useful a way of working, in fact, that Scala provides the for-comprehension as essentially a short-cut for this (Haskell, too, provides a short-hand way of writing a chain of bind operations - I'm not a Haskell expert, though, and don't recall the details) - hence the talk you will have come across about for-comprehensions being "de-sugared" into a chain of flatMap calls (along with possible filter calls and a final map call for the yield).
Well, one could argue, you don't need .flatten either. Why not just do something like
#tailrec
def flatten[T](in: Seq[Seq[T], out: Seq[T] = Nil): Seq[T] = in match {
case Nil => out
case head ::tail => flatten(tail, out ++ head)
}
Same can be said about map:
#tailrec
def map[A,B](in: Seq[A], out: Seq[B] = Nil)(f: A => B): Seq[B] = in match {
case Nil => out
case head :: tail => map(tail, out :+ f(head))(f)
}
So, why are .flatten and .map provided by the library? Same reason .flatMap is: convenience.
There is also .collect, which is really just
list.filter(f.isDefinedAt _).map(f)
.reduce is actually nothing more then list.foldLeft(list.head)(f),
.headOption is
list match {
case Nil => None
case head :: _ => Some(head)
}
Etc ...
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
Can someone explain to me in simple terms what the Shapeless library is for?
Scala has generics and inheritance functionality so I'm a bit confused what Shapeless is for.
Maybe a use case to clarify things would be helpful.
It's a little hard to explain, as shapeless has a wide range of features; I'd probably find it easier to "explain, in simple terms, what variables are for". You definitely want to start with the feature overview.
Broadly speaking, shapeless is about programming with types. Doing things at compile-time that would more commonly be done at runtime, keeping precise track of the type of each element in a list, being able to translate from tuples to HLists to case classes, creating polymorphic functions (as opposed to methods), etc.
A typical usage scenario would go something like:
Read a bunch of values from somewhere into a List
perform a type-safe cast of that List into an HList
map over that HList with a polymorphic function that e.g. normalises values
convert the 3rd element (which is statically known to be an Int) into a 0-padded string
construct a case class using values from the HList
For reference, an HList will have a precise type, such as Int :: String :: Boolean :: HNil (yes, that really is a single type) where everything is pinned down and the size is fixed. So you either need to know at compile time exactly what will be going into your HList, or you need the type-safe cast.
If you take the tail of such an HList, you get a String :: Boolean :: HNil, and a compile-time guarantee that the head of this will be a String. Prepending a value to the head will similarly preserve all types involved.
Shapeless also comes with the Generic type class, allowing you to use HList operations on tuples and case classes as well.
The other features I tend to use are:
Coproducts, which allow you to statically type a value as being e.g. "a String, Double or Int, but nothing else" (much like Either, but not limited to just two possibilities)
Lenses, which simplify working with nested case classes.
Looking at an HList is something that might seem baffling until you try to work with types and delegate or switch on types. Take a look at the following:
val myList = 1 :: 2 :: "3" :: fred :: Nil
What is the type of myList here? If you were to inspect it, you'd see it was of type List[Any]. That's not very helpful. What's even less helpful is if I tried to use the following PartialFunction[Any] to map over it:
myList.map{
case x: Int => x
case x: String => Int.parseInt(x)
}
At runtime, this might throw a MatchError because I haven't actually told you what type fred is. It could be of type Fred.
With an HList you can know right at compile time if you've failed to capture one of the types of that list. In the above, if I had defined myList = 1 :: 2 :: "3" :: fred :: HNil when I accessed the 3rd element, it's type would be String and this would be known at compile time.
As #KevinWright states, there's more to it than that with Shapeless but HList is one of the defining features of the library.
Everything in Shapeless has two things in common:
First, it isn't in the Scala standard library, but arguably should be. Therefore, asking what Shapeless is for is a bit like asking with the Scala standard library is for! It's for everything. It's a grab bag.
(but it isn't a totally arbitrary grab bag, because:)
Second, everything in Shapeless provides increased checking and safety at compile time. Nothing in Shapeless (that I can think of?) actually “does” anything at runtime. All of the interesting action happens when your code is compiled. The goal is always increased confidence that if your code compiles at all, it won't crash or do the wrong thing at runtime. (Hence this notable quip: https://twitter.com/mergeconflict/status/304090286659866624 "Does
Miles Sabin even exist at runtime?")
There is a nice introduction to what type-level programming is all about, with links to further resources, at https://stackoverflow.com/a/4443972/86485.
I've been learning Scala and decided to play with JSON parsing using json4s. I decided to use XPath syntax for deserializing and came across this strange bit of syntax I've never seen before.
val json = JsonMethods.parse("""{"meaningOfLife": 42}""")
val JInt(x) = json\"meaningOfLife"
The part confusing me is this bit right here
val JInt(x) = ...
I can't wrap my mind around what's happening there, I don't even know how to search this syntax or what it's called. Can anyone help me out? Scala is an amazing language with a lot of neat features that I'm not used to in other languages like C++ and Java.
Edit
To clarify, I'm confused because x isn't defined, but it's somehow being passed into a function or constructor then being assigned to the result of json\"meaningOfLife" which returns a JValue.
Edit 2
After some research and playing around, I figured out that this has something to do with case classes. I was able to run the following code.
case class MyCaseClass (x: Int)
val MyCaseClass(x) = new MyCaseClass(5)
println(x, x.getClass) // prints (5,int)
Which, after looking at some of the code, gives me a good understanding at what's happening.
val MyCaseClass(x) = MyCaseClass(5)
Is extracting (for lack of a better term) the Int value 5 from the instantiated MyCaseClass and storing that into x, meaning x will be of type Int.
In the code for json4s a JInt is a JValue which the \ operator returns. So the JInt(x) is taking out a BigInt (stored in the class JInt) and putting that into the value x from what I gather.
But I still have a question. What is this process called? Is there any documentation on it?
It's called "irrefutable pattern matching" and it's essentially equivalent to this bit of code:
val json = JsonMethods.parse("""{"meaningOfLife": 42}""")
val x = json match {
case JInt(xMatched) => xMatched
}
In other words, any case class or any extractor that fits the pattern of the declaration's left-hand-side can be used in this way.
Addendum:
The "irrefutable" means that a MatchError will be thrown if the pattern cannot be satisfied.
Since Scala 2.7.2 there is something called Manifest which is a workaround for Java's type erasure. But how does Manifest work exactly and why / when do you need to use it?
The blog post Manifests: Reified Types by Jorge Ortiz explains some of it, but it doesn't explain how to use it together with context bounds.
Also, what is ClassManifest, what's the difference with Manifest?
I have some code (part of a larger program, can't easily include it here) that has some warnings with regard to type erasure; I suspect I can solve these by using manifests, but I'm not sure exactly how.
The compiler knows more information about types than the JVM runtime can easily represent. A Manifest is a way for the compiler to send an inter-dimensional message to the code at runtime about the type information that was lost.
It isn't clear if a Manifest would benefit the errors you are seeing without knowing more detail.
One common use of Manifests is to have your code behave differently based on the static type of a collection. For example, what if you wanted to treat a List[String] differently from other types of a List:
def foo[T](x: List[T])(implicit m: Manifest[T]) = {
if (m <:< manifest[String])
println("Hey, this list is full of strings")
else
println("Non-stringy list")
}
foo(List("one", "two")) // Hey, this list is full of strings
foo(List(1, 2)) // Non-stringy list
foo(List("one", 2)) // Non-stringy list
A reflection-based solution to this would probably involve inspecting each element of the list.
A context bound seems most suited to using type-classes in scala, and is well explained here by Debasish Ghosh:
http://debasishg.blogspot.com/2010/06/scala-implicits-type-classes-here-i.html
Context bounds can also just make the method signatures more readable. For example, the above function could be re-written using context bounds like so:
def foo[T: Manifest](x: List[T]) = {
if (manifest[T] <:< manifest[String])
println("Hey, this list is full of strings")
else
println("Non-stringy list")
}
A Manifest was intended to reify generic types that get type-erased to run on the JVM (which does not support generics). However, they had some serious issues: they were too simplistic, and were unable to fully support Scala's type system. They were thus deprecated in Scala 2.10, and are replaced with TypeTags (which are essentially what the Scala compiler itself uses to represent types, and therefore fully support Scala types). For more details on the difference, see:
Scala: What is a TypeTag and how do I use it?
How do the new Scala TypeTags improve the (deprecated) Manifests?
In other words
when do you need it?
Before 2013-01-04, when Scala 2.10 was released.
Not a complete answer, but regarding the difference between Manifest and ClassManifest, you can find an example in the Scala 2.8 Array paper:
The only remaining question is how to implement generic array creation. Unlike Java, Scala allows an instance creation new Array[T] where T is a type parameter. How can this be implemented, given the fact that there does not exist a uniform array representation in Java?
The only way to do this is to require additional runtime information which describes the type T. Scala 2.8 has a new mechanism for this, which is called a Manifest. An object of type Manifest[T] provides complete information about the type T.
Manifest values are typically passed in implicit parameters; and the compiler knows how to construct them for statically known types T.
There exists also a weaker form named ClassManifest which can be constructed from knowing just the top-level class of a type, without necessarily knowing all its argument types.
It is this type of runtime information that’s required for array creation.
Example:
One needs to provide this information by passing a ClassManifest[T] into the
method as an implicit parameter:
def tabulate[T](len:Int, f:Int=>T)(implicit m:ClassManifest[T]) = {
val xs = new Array[T](len)
for (i <- 0 until len) xs(i) = f(i)
xs
}
As a shorthand form, a context bound1 can be used on the type parameter T instead,
(See this SO question for illustration)
, giving:
def tabulate[T: ClassManifest](len:Int, f:Int=>T) = {
val xs = new Array[T](len)
for (i <- 0 until len) xs(i) = f(i)
xs
}
When calling tabulate on a type such as Int, or String, or List[T], the Scala compiler can create a class manifest to pass as implicit argument to tabulate.
Let's also chck out manifest in scala sources (Manifest.scala), we see:
Manifest.scala:
def manifest[T](implicit m: Manifest[T]) = m
So with regards to following example code:
def foo[A](somelist: List[A])(implicit m: Manifest[A]): String = {
if (m <:< manifest[String]) {
"its a string"
} else {
"its not a string"
}
}
we can see that the manifest function searches for an implicit m: Manifest[T] which satisfies the type parameter you provide in our example code it was manifest[String]. So when you call something like:
if (m <:< manifest[String]) {
you are checking if the current implicit m which you defined in your function is of type manifest[String] and as the manifest is a function of type manifest[T] it would search for a specific manifest[String] and it would find if there is such an implicit.