Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 1 year ago.
Improve this question
I am trying to pass a list to the scala function, transform it and return it.
val fun=(lst:List[Int]):List[Int]=>lst+10
error: identifier expected but integer literal found.
There are several issues with your code.
You are mixing the definition of the type of fun with its implementation.
You probably want something like this:
val fun: List[Int] => List[Int] = (lst: List[Int]) => lst + 10
Then the type of the parameter is actually redundant with the type definition of fun, so you can remove it:
val fun: List[Int] => List[Int] = lst => lst + 10
You are trying to use the + operation/method on a List[Int] and a Int. There's no such method. If you want to append an item into a list you can use :+ instead:
val fun: List[Int] => List[Int] = lst => lst :+ 10
More about the available methods on List in the official documentation: https://www.scala-lang.org/api/2.13.x/scala/collection/immutable/List.html
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
The code below isn't giving me the desired output. I am getting the output of finallist as individual characters separated by commas; I was expecting lists with two values only (filename, sizeofcolumn).
val pathurl="adl://*****.azuredatalakestore.net/<folder>/<sub_folder>"
val filelist=dbutils.fs.ls(pathurl)
val newdf = df.select("path").rdd.map(r => r(0)).collect.toList
var finallist = scala.collection.mutable.ListBuffer.empty[Any]
newdf.foreach(f => {
val MasterPq = spark.read.option("header","true").option("inferSchema","true").parquet(f.toString())
val size = MasterPq.columns.length
val mergedlist = List(f.toString(), size.toString())
mergedlist.map((x => {finallist = finallist ++ x}))
})
println(finallist)
The bug in your code is that you're using the ++ method to append values to your list. This method is used to append two list.
scala> List(1, 2) ++ List(3, 4)
res0: List[Int] = List(1, 2, 3, 4)
In scala strings are viewn as a list of characters, so your appending each individual character to your list.
scala> List(1, 2) ++ "Hello"
res3: List[AnyVal] = List(1, 2, H, e, l, l, o)
Since you're using a mutable list, you can append values with the '+=' method. If you just want to get your code working, than the following should be enough, but it is not a good solution.
// mergedlist.map((x => {finallist = finallist ++ x}))
mergedlist.map((x => finallist += x}))
You're probably new to scala, coming from a imperative language like Java. Scala collections do not work as you're known from such programming languages. Scala's collections are immutable by default. Instead of modifying collections, you're using using functions such as map to build new lists based on the old list.
The map function is one of the most used functions on lists. It takes an anonymous function as parameter that takes one element and transforms it to another value. This function is applied onto all methods of the list thereby build a new list. Here's an example:
scala> val list = List(1, 2, 3).map(i => i * 2)
list: List[Int] = List(2, 4, 6)
In this example, a function that multiplies integers by two is applied onto each element in the list. The results are put into the new list. Maybe this illustration helps to comprehend the process:
List(1, 2, 3)
| | |
* 2 * 2 * 2
↓ ↓ ↓
List(2, 4, 6)
We could use the map function to solve your task.
We can use it to map each element in the newdf list into a tuple with the corresponding (filename, filesize).
val finallist = newdf.map { f =>
val masterPq = spark.read.option("header","true").option("inferSchema","true").parquet(f.toString())
val size = masterPq.columns.length
(f.toString(), size.toString())
}
I think this code is shorter, simpler, easier to read and just way more beautiful. I will definitely recommend you to learn more about Scala's collections and immutable collections in general. Once you understand them, you'll just love them!
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
Please help me understand what map(_(0)) means here:
scala> a.collect
res97: Array[org.apache.spark.sql.Row] = Array([1039], [1010], [1002], [926])
scala> a.collect.map(_(0))
res98: Array[Any] = Array(1039, 1010, 1002, 926)
1. .map in functional programming applies the function you want to each element of your collection.
Say, you want to add some data to each element in an array you have, which can be done as below,
scala> val data = Array("a", "b", "c")
data: Array[String] = Array(a, b, c)
scala> data.map(element => element+"-add something")
res10: Array[String] = Array(a-add something, b-add something, c-add something)
Here, I'm saying, on each element add something, but element is unnecessary because you are adding on every element anyway. So, _ is what represents any element here.
So, same map can be done in following way.
scala> data.map(_+"-add something")
res9: Array[String] = Array(a-add something, b-add something, c-add something)
Also, note that _ is used when you have one line mapping function.
2. collection(index) is the way to access nth element in a collection.
eg.
scala> val collection = Array(Vector(1039), Vector(1010), Vector(1002), Vector(926))
collection: Array[scala.collection.immutable.Vector[Int]] = Array(Vector(1039), Vector(1010), Vector(1002), Vector(926))
scala> collection(0)
res13: scala.collection.immutable.Vector[Int] = Vector(1039)
So, combining #1 and #2, in your case you are mapping the original collection and getting the first element.
scala> collection.map(_.head)
res17: Array[Int] = Array(1039, 1010, 1002, 926)
Refs
https://twitter.github.io/scala_school/collections.html#map
Map, Map and flatMap in Scala
You are accessing the zeroth element of the items in the collection a. _ is a common placeholder in Scala when working with the items in a collection.
More concretely, your code is equivalent to
a.collect.map(item => item(0))
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
In scala shell, are the res variables val or var?
thanks.
res in scala shell are val.
you can verify this by trying to reassign a value to res.
e.g. - scala> List(1) res1: List[Int] = List(1)
scala> res1=List(2) console>:8: error: reassignment to val
res1=List(2)
Showing that the res variable (varName) is only used in val res = expr:
https://github.com/scala/scala/blob/v2.11.5/src/repl/scala/tools/nsc/interpreter/IMain.scala#L495
(The variant is that x ; y is rewritten to x ; val res = y or similar.)
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I find myself wanting to avoid tuples just because the access syntax is ugly. Given that the arity is limited, why does Scala not support a nicer syntax the looks better and is easier to type? This is my suggestion:
val t = (1,2,3)
// Proposed equivalent reference syntax.
assert(t.a == t._1)
assert(t.b == t._2)
assert(t.c == t._3)
So, am I trying to be elegant, or just stupid?
Usually you don't need to use the _1 etc. methods because you can do pattern matching:
val t = (1,2,3)
val (a, b, c) = t
val t2 = ((1, 2), (3, 4))
t2 match {
case ((a, b), (c, d)) if a > c && b > d => true
case _ => false
}
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking for code must demonstrate a minimal understanding of the problem being solved. Include attempted solutions, why they didn't work, and the expected results. See also: Stack Overflow question checklist
Closed 9 years ago.
Improve this question
Is there any function in Scala that interchanges an integer with a letter of the alphabet? For example whenever there is a '1' in a list it is interchanged with an 'a'? I have to search through a list, if I find a '1' I have to change It to 'a' ,else print list as it is. Thanks
def alphabet(i: Int): Char = ('a' to 'z')(i - 1)
alphabet: (i: Int)Char
scala> alphabet(1)
res0: Char = a
scala> alphabet(14)
res2: Char = n
scala> alphabet(30)
java.lang.IndexOutOfBoundsException: 29
scala> List[Any](2,1,5,6) map { case 1 => 'a'; case x => x }
res0: List[Any] = List(2, a, 5, 6)