Is there a standard way of combining existing Scala collection functions to achieve the following? Or is this already defined in some popular extension library like Scalaz?
def partialReduceLeft[T](elements: List[T], nextReduction: List[T] => (T, List[T])): List[T] =
if (elements == Nil)
Nil
else {
val (reduction, residual) = nextReduction(elements)
if (residual.length >= elements.length)
throw new Exception("Residual collection from nextReduction function must be smaller than its input collection.")
if (residual == Nil)
List(reduction)
else
reduction :: partialReduceLeft(residual, nextReduction)
}
The function takes a collection and applies a user-defined function which returns the first reduction, which may consume one or more elements. The method keeps going until all elements are consumed.
The resulting collection may have an equal or lower size to the input collection (I rather unscientifically call this a 'partial reduce left' - for want of knowing the exact term for this type of standard function :)).
My implementation is not tail-recursive, and to be honest, I'd much rather use someone else's code!!
There is similar method in scalaz: unfold.
You could implement your method using unfold this way:
def partialReduceLeft[T](elements: List[T],
nextReduction: List[T] => (T, List[T])): Stream[T] =
unfold(elements){ es => es.nonEmpty option nextReduction(es) }
Related
I had spent weeks on trying to understand the idea behind "lifting" in scala.
Originally, it was from the example related to Chapter 4 of book "Functional Programming in Scala"
Then I found below topic "How map work on Options in Scala?"
The selected answer specify that:
def map[B](f: A => B): Option[B] = this match (Let's considered this as (*) )
So, from above code, I assume that function "map" is derived from function match. Hence, the mechanism behind "map"
is a kind of pattern matching to provide a case selection between Some, and None
Then, I created below examples by using function map for Seq, Option, and Map (Let's considered below examples as (**) )
Example 1: map for Seq
val xs = Seq(1, 2, 3)
xs.map(println)
Example 2: map for Option
val a:Option[Int] = Some(5)
a.map(println)
val b:Option[Int] = None
b.map(println)
Example 3: map for Map
val capitals = Map("France" -> "Paris", "Japan" -> "Tokyo")
capitals.map(println)
From (*) and (**), I could not know whether "map" is a pattern matching or an iteration, or both.
Thank you for helping me to understand this.
#Jwvh provided a more programming based answer but I want to dig a little bit deeper.
I certainly appreciate you trying to understand how things work in Scala, however if you really want to dig that deep, I am afraid you will need to obtain some basic knowledge of Category Theory since there is no "idea behind lifting in scala" but just the "idea behind lifting"
This is also why functions like "map" can be very confusing. Inherently, programmers are taught map etc. as operations on collections, where as they are actually operations that come with Functors and Natural Transformations (this is normally referred to as fmap in Category Theory and also Haskell).
Before I move on, the short answer is it is a pattern matching in the examples you gave and in some of them it is both. Map is defined specifically to the case, the only condition is that it maintains functoriality
Attention: I will not be defining every single term below, since I would need to write a book to build up to some of the following definitions, interested readers are welcome to research them on their own. You should be able to get some basic understanding by following the types
Let's consider these as Functors, the definition will be something along the lines of this:
In (very very) short, we consider types as objects in the category of our language. The functions between these types (type constructors) are morphisms between types in this category. The set of these transformations are called Endo-Functors (take us from the category of Scala and drop us back in the category of Scala). Functors have to have a polymorphic (which actually has a whole different (extra) definition in category theory) map function, that will take some object A, through some type constructor turn it into object B.
implicit val option: Functor[Option] = new Functor[Option] {
override def map[A,B](optA: Option[A])(f: (A) => B): Option[B] = optA match{
case Some(a) => Some(f(a))
case _ => None
}
}
implicit val seq: Functor[Seq[_]] = new Functor[Seq[_]] {
override def map[A,B](sA: Seq[A])(f: (A) => B): Seq[B] = sA match{
case a :: tail => Seq(f(a), map(tail)(f))
case Nil => Nil
}
}
As you can see in the second case, there is a little bit of both (more of a recursion than iteration but still).
Now before the internet blows up on me, I will say you cant pattern match on Seq in Scala. It works here because the default Seq is also a List. I just provided this example because it is simpler to understand. The underlying definition something along the lines of that.
Now hold on a second. If you look at these types, you see that they also have flatMap defined on them. This means they are something more special than plain Functors. They are Monads. So beyond satisfying functoriality, they obey the monadic laws.
Turns out Monad has a different kind of meaning in the core scala, more on that here: What exactly makes Option a monad in Scala?
But again very very short, this means that we are now in a category where the endofunctors from our previous category are the objects and the mappings between them are morphisms (natural transformations), this is slightly more accurate because if you think about it when you take a type and transform it, you take (carry over) all of it's internal type constructors (2-cell or internal morphisms) with it, you do not only take this sole idea of a type without it's functions.
implicit val optionMonad: Monad[Option] = new Monad[Option] {
override def flatMap[A, B](optA: Option[A])(f: (A) => Option[B]): Option[B] = optA match{
case Some(a) => f(a)
case _ => None
}
def pure[A](a: A): Option[A] = Some(a)
//You can define map using pure and flatmap
}
implicit val seqMonad: Monad[Seq[_]] = new Monad[Seq[_]] {
override def flatMap[A, B](sA: Seq[A])(f: (A) => Seq[B]): Seq[B] = sA match{
case x :: xs => f(a).append(flatMap(tail)(f))
case Nil => Nil
}
override def pure[A](a: A): Seq[A] = Seq(a)
//Same warning as above, also you can implement map with the above 2 funcs
}
One thing you can always count on is map being having pattern match (or some if statement). Why?
In order to satisfy the identity laws, we need to have some sort of a "base case", a unit object and in many cases (such as Lists) those types are gonna be what we call either a product or coproduct.
Hopefully, this did not confuse you further. I wish I could get into every detail of this but it would simply take pages, I highly recommend getting into categories to fully understand where these come from.
From the ScalaDocs page we can see that the type profile for the Standard Library map() method is a little different.
def map[B](f: (A) => B): Seq[B]
So the Standard Library map() is the means to transition from a collection of elements of type A to the same collection but the elements are type B. (A and B might be the same type. They aren't required to be different.)
So, yes, it iterates through the collection applying function f() to each element A to create each new element B. And function f() might use pattern matching in its code, but it doesn't have to.
Now consider a.map(println). Every element of a is sent to println which returns Unit. So if a is List[Int] then the result of a.map(println) is List[Unit], which isn't terribly useful.
When all we want is the side effect of sending information to StdOut then we use foreach() which doesn't create a new collection: a.foreach(println)
Function map for Option isn't about pattern matching. The match/case used in your referred link is just one of the many ways to define the function. It could've been defined using if/else. In fact, that's how it's defined in Scala 2.13 source of class Option:
sealed abstract class Option[+A] extends IterableOnce[A] with Product with Serializable {
self =>
...
final def map[B](f: A => B): Option[B] =
if (isEmpty) None else Some(f(this.get))
...
}
If you view Option like a "collection" of either one element (Some(x)) or no elements (None), it might be easier to see the resemblance of how map transforms an Option versus, say, a List:
val f: Int => Int = _ + 1
List(42).map(f)
// res1: List[Int] = List(43)
List.empty[Int].map(f)
// res2: List[Int] = List()
Some(42).map(f)
// res3: Option[Int] = Some(43)
None.map(f)
// res4: Option[Int] = None
I have this two functions
def pattern(s: String): Option[Pattern] =
try {
Some(Pattern.compile(s))
} catch {
case e: PatternSyntaxException => None
}
and
def mkMatcher(pat: String): Option[String => Boolean] =
pattern(pat) map (p => (s: String) => p.matcher(s).matches)
Map is the higher-order function that applies a given function to each element of a list.
Now I am not getting that how map is working here as per above statement.
Map is the higher-order function that applies a given function to each element of a list.
This is an uncommonly restrictive definition of map.
At any rate, it works because it was defined by someone who did not hold to that.
For example, that someone wrote something akin to
sealed trait Option[+A] {
def map[B](f: A => B): Option[B] = this match {
case Some(value) => Some(f(value))
case None => None
}
}
as part of the standard library. This makes map applicable to Option[A]
It was defined because it makes sense to map many kinds of data structures not just lists.
Mapping is a transformation applied to the elements held by the data structure.
It applies a function to each element.
Option[A] can be thought of as a trivial sequence. It either has zero or one elements. To map it means to apply the function on its element if it has one.
Now it may not make much sense to use this facility all of the time, but there are cases where it is useful.
For example, it is one of a few distinct methods that, when present enable enable For Expressions to operate on a type. Option[A] can be used in for expressions which can be convenient.
For example
val option: Option[Int] = Some(2)
val squared: Option[Int] = for {
n <- option
if n % 2 == 0
} yield n * n
Interestingly, this implies that filter is also defined on Option[A].
If you just have a simple value it may well be clearer to use a less general construct.
Map is working the same way that it does with other collections types like List and Vector. It applies your function to the contents of the collection, potentially changing the type but keeping the collection type the same.
In many cases you can treat an Option just like a collection with either 0 or 1 elements. You can do a lot of the same operations on Option that you can on other collections.
You can modify the value
var opt = Option(1)
opt.map(_ + 3)
opt.map(_ * math.Pi)
opt.filter(_ == 1)
opt.collect({case i if i > 0 => i.toString })
opt.foreach(println)
and you can test the value
opt.contains(3)
opt.forall(_ > 0)
opt.exists(_ > 0)
opt.isEmpty
Using these methods you rarely need to use a match statement to unpick an Option.
Now I have some Scala code similar to the following:
def foo(x: Int, fn: (Int, Int) => Boolean): Boolean = {
for {
i <- 0 until x
j <- i + 1 until x
if fn(i, j)
} return true
false
}
But I get the feeling that return true is not so functional (or maybe it is?). Is there a way to rewrite this piece of code in a more elegant way?
In general, what is the more functional (if any) way to write the return-early-from-a-loop kind of code?
There are several methods can help, such as find, exists, etc. For your case, try this:
def foo2(x: Int, fn: (Int, Int) => Boolean): Boolean = {
(0 until x).exists(i =>
(i+1 until x).exists(j=>fn(i, j)))
}
Since all you are checking for is existence, you can just compose 2 uses of exists:
(0 until x).exists(i => (i + 1 until x).exists(fn(i, _)))
More generally, if you are concerned with more than just determining if a certain element exists, you can convert your comprehension to a series of Streams, Iterators, or views, you can use exists and it will evaluate lazily, avoiding unnecessary executions of the loop:
def foo(x: Int, fn: (Int, Int) => Boolean): Boolean = {
(for {
i <- (0 until x).iterator
j <- (i + 1 until x).iterator
} yield(i, j)).exists(fn.tupled)
}
You can also use map and flatMap instead of a for, and toStream or view instead of iterator:
(0 until x).view.flatMap(i => (i + 1 until x).toStream.map(j => i -> j)).exists(fn.tupled)
You can also use view on any collection to get a collection where all the transformers are performed lazily. This is possibly the most idiomatic way to short-circuit a collection traversal. From the docs on views:
Scala collections are by default strict in all their transformers, except for Stream, which implements all its transformer methods lazily. However, there is a systematic way to turn every collection into a lazy one and vice versa, which is based on collection views. A view is a special kind of collection that represents some base collection, but implements all transformers lazily.
As far as overhead is concerned, it really depends on the specifics! Different collections have different implementations of view, toStream, and iterator that may vary in amount of overhead. If fn is very expensive to compute, this overhead is probably worth it, and keeping a consistent, idiomatic, functional style to your code makes it more maintainable, debuggable, and readable. If you are in a situation that calls for extreme optimization, you may want to fall back to the lower-level constructs like return (which isn't without it's own overhead!).
take this as a follow up to this SO question
I'm new to scala and working through the 99 problems. The given solution to p9 is:
object P09 {
def pack[A](ls: List[A]): List[List[A]] = {
if (ls.isEmpty) List(List())
else {
val (packed, next) = ls span { _ == ls.head }
if (next == Nil) List(packed)
else packed :: pack(next)
}
}
}
The span function is doing all the work here. As you can see from the API doc (it's the link) span returns a Tuple2 (actually the doc says it returns a pair - but that's been deprecated in favor or Tuple2). I was trying to figure out why you don't get something back like a list-of-lists or some such thing and stumbled across the SO link above. As I understand it, the reason for the Tuple2 has to do with increasing performance by not having to deal with Java's 'boxing/unboxing' of things like ints into objects like Integers. My question is
1) is that an accurate statement?
2) are there other reasons for something like span to return a Tuple2?
thx!
A TupleN object has at least two major differences when compared to a "standard" List+:
(less importantly) the size of the tuple is known beforehand, allowing to better reason about it (by the programmer and the compiler).
(more importantly) a tuple preserves the type information for each of its elements/"slots".
Note that, as alluded, the type Tuple2 is a part of the TupleN family, all utilizing the same concept. For example:
scala> ("1",2,3l)
res0: (String, Int, Long) = (1,2,3)
scala> res0.getClass
res1: Class[_ <: (String, Int, Long)] = class scala.Tuple3
As you can see here, each of the elements in the 3-tuple has a distinct type, allowing for better pattern matching, stricter type protection etc.
+heterogeneous lists are also possible in Scala, but, so far, they're not part of the standard library, and arguably harder to understand, especially for newcomers.
span returns exactly two values. A Tuple2 can hold exactly two values. A list can contain arbitrarily many values. Therefore Tuple2 is just a better fit than using a list.
I am a bit new to Scala, so apologies if this is something a bit trivial.
I have a list of items which I want to iterate through. I to execute a check on each of the items and if just one of them fails I want the whole function to return false. So you can see this as an AND condition. I want it to be evaluated lazily, i.e. the moment I encounter the first false return false.
I am used to the for - yield syntax which filters items generated through some generator (list of items, sequence etc.). In my case however I just want to break out and return false without executing the rest of the loop. In normal Java one would just do a return false; within the loop.
In an inefficient way (i.e. not stopping when I encounter the first false item), I could do it:
(for {
item <- items
if !satisfiesCondition(item)
} yield item).isEmpty
Which is essentially saying that if no items make it through the filter all of them satisfy the condition. But this seems a bit convoluted and inefficient (consider you have 1 million items and the first one already did not satisfy the condition).
What is the best and most elegant way to do this in Scala?
Stopping early at the first false for a condition is done using forall in Scala. (A related question)
Your solution rewritten:
items.forall(satisfiesCondition)
To demonstrate short-circuiting:
List(1,2,3,4,5,6).forall { x => println(x); x < 3 }
1
2
3
res1: Boolean = false
The opposite of forall is exists which stops as soon as a condition is met:
List(1,2,3,4,5,6).exists{ x => println(x); x > 3 }
1
2
3
4
res2: Boolean = true
Scala's for comprehensions are not general iterations. That means they cannot produce every possible result that one can produce out of an iteration, as, for example, the very thing you want to do.
There are three things that a Scala for comprehension can do, when you are returning a value (that is, using yield). In the most basic case, it can do this:
Given an object of type M[A], and a function A => B (that is, which returns an object of type B when given an object of type A), return an object of type M[B];
For example, given a sequence of characters, Seq[Char], get UTF-16 integer for that character:
val codes = for (char <- "A String") yield char.toInt
The expression char.toInt converts a Char into an Int, so the String -- which is implicitly converted into a Seq[Char] in Scala --, becomes a Seq[Int] (actually, an IndexedSeq[Int], through some Scala collection magic).
The second thing it can do is this:
Given objects of type M[A], M[B], M[C], etc, and a function of A, B, C, etc into D, return an object of type M[D];
You can think of this as a generalization of the previous transformation, though not everything that could support the previous transformation can necessarily support this transformation. For example, we could produce coordinates for all coordinates of a battleship game like this:
val coords = for {
column <- 'A' to 'L'
row <- 1 to 10
} yield s"$column$row"
In this case, we have objects of the types Seq[Char] and Seq[Int], and a function (Char, Int) => String, so we get back a Seq[String].
The third, and final, thing a for comprehension can do is this:
Given an object of type M[A], such that the type M[T] has a zero value for any type T, a function A => B, and a condition A => Boolean, return either the zero or an object of type M[B], depending on the condition;
This one is harder to understand, though it may look simple at first. Let's look at something that looks simple first, say, finding all vowels in a sequence of characters:
def vowels(s: String) = for {
letter <- s
if Set('a', 'e', 'i', 'o', 'u') contains letter.toLower
} yield letter.toLower
val aStringVowels = vowels("A String")
It looks simple: we have a condition, we have a function Char => Char, and we get a result, and there doesn't seem to be any need for a "zero" of any kind. In this case, the zero would be the empty sequence, but it hardly seems worth mentioning it.
To explain it better, I'll switch from Seq to Option. An Option[A] has two sub-types: Some[A] and None. The zero, evidently, is the None. It is used when you need to represent the possible absence of a value, or the value itself.
Now, let's say we have a web server where users who are logged in and are administrators get extra javascript on their web pages for administration tasks (like wordpress does). First, we need to get the user, if there's a user logged in, let's say this is done by this method:
def getUser(req: HttpRequest): Option[User]
If the user is not logged in, we get None, otherwise we get Some(user), where user is the data structure with information about the user that made the request. We can then model that operation like this:
def adminJs(req; HttpRequest): Option[String] = for {
user <- getUser(req)
if user.isAdmin
} yield adminScriptForUser(user)
Here it is easier to see the point of the zero. When the condition is false, adminScriptForUser(user) cannot be executed, so the for comprehension needs something to return instead, and that something is the "zero": None.
In technical terms, Scala's for comprehensions provides syntactic sugars for operations on monads, with an extra operation for monads with zero (see list comprehensions in the same article).
What you actually want to accomplish is called a catamorphism, usually represented as a fold method, which can be thought of as a function of M[A] => B. You can write it with fold, foldLeft or foldRight in a sequence, but none of them would actually short-circuit the iteration.
Short-circuiting arises naturally out of non-strict evaluation, which is the default in Haskell, in which most of these papers are written. Scala, as most other languages, is by default strict.
There are three solutions to your problem:
Use the special methods forall or exists, which target your precise use case, though they don't solve the generic problem;
Use a non-strict collection; there's Scala's Stream, but it has problems that prevents its effective use. The Scalaz library can help you there;
Use an early return, which is how Scala library solves this problem in the general case (in specific cases, it uses better optimizations).
As an example of the third option, you could write this:
def hasEven(xs: List[Int]): Boolean = {
for (x <- xs) if (x % 2 == 0) return true
false
}
Note as well that this is called a "for loop", not a "for comprehension", because it doesn't return a value (well, it returns Unit), since it doesn't have the yield keyword.
You can read more about real generic iteration in the article The Essence of The Iterator Pattern, which is a Scala experiment with the concepts described in the paper by the same name.
forall is definitely the best choice for the specific scenario but for illustration here's good old recursion:
#tailrec def hasEven(xs: List[Int]): Boolean = xs match {
case head :: tail if head % 2 == 0 => true
case Nil => false
case _ => hasEven(xs.tail)
}
I tend to use recursion a lot for loops w/short circuit use cases that don't involve collections.
UPDATE:
DO NOT USE THE CODE IN MY ANSWER BELOW!
Shortly after I posted the answer below (after misinterpreting the original poster's question), I have discovered a way superior generic answer (to the listing of requirements below) here: https://stackoverflow.com/a/60177908/501113
It appears you have several requirements:
Iterate through a (possibly large) list of items doing some (possibly expensive) work
The work done to an item could return an error
At the first item that returns an error, short circuit the iteration, throw away the work already done, and return the item's error
A for comprehension isn't designed for this (as is detailed in the other answers).
And I was unable to find another Scala collections pre-built iterator that provided the requirements above.
While the code below is based on a contrived example (transforming a String of digits into a BigInt), it is the general pattern I prefer to use; i.e. process a collection and transform it into something else.
def getDigits(shouldOnlyBeDigits: String): Either[IllegalArgumentException, BigInt] = {
#scala.annotation.tailrec
def recursive(
charactersRemaining: String = shouldOnlyBeDigits
, accumulator: List[Int] = Nil
): Either[IllegalArgumentException, List[Int]] =
if (charactersRemaining.isEmpty)
Right(accumulator) //All work completed without error
else {
val item = charactersRemaining.head
val isSuccess =
item.isDigit //Work the item
if (isSuccess)
//This item's work completed without error, so keep iterating
recursive(charactersRemaining.tail, (item - 48) :: accumulator)
else {
//This item hit an error, so short circuit
Left(new IllegalArgumentException(s"item [$item] is not a digit"))
}
}
recursive().map(digits => BigInt(digits.reverse.mkString))
}
When it is called as getDigits("1234") in a REPL (or Scala Worksheet), it returns:
val res0: Either[IllegalArgumentException,BigInt] = Right(1234)
And when called as getDigits("12A34") in a REPL (or Scala Worksheet), it returns:
val res1: Either[IllegalArgumentException,BigInt] = Left(java.lang.IllegalArgumentException: item [A] is not digit)
You can play with this in Scastie here:
https://scastie.scala-lang.org/7ddVynRITIOqUflQybfXUA