I've grabbed some Scala CSV parsing code from here:
Use Scala parser combinator to parse CSV files
And then I tried to write a basic test for it:
assertEquals(List(List()), CSV.parse(""))
And this fails, with message:
java.lang.AssertionError: expected: scala.collection.immutable.$colon$colon but was: scala.collection.immutable.$colon$colon
Any ideas? The output from CSV.parse is an empty List[List[String]] but seems to have a different hashCode than List(Nil) or ListList[String] etc. I can't seem to find any way to compose a list which is equal to the output of CSV.parse("").
UPDATE:
Here is the failure using REPL:
scala> assertEquals(List(Nil), CSV.parse(""))
java.lang.AssertionError: expected: scala.collection.immutable.$colon$colon<List(List())> but was: scala.collection.immutable.$colon$colon<List(List())>
Edited: I tried the parser you supplied in the link:
scala> CSV.parse("")
res7: List[List[String]] = List(List(""))
So apparently, it doesn't return a List with an empty List, but a List with a List with the empty string. So your test should fail.
Related
What is the propper way of read a list input using the readf function?
The Scala Standard Library shows that the sintax is:
def readf(format: String): List[Any]
but I'm not finding to introduce on the Format field and always get the error:
Solution.scala:11: error: not found: value format
val arr = scala.io.StdIn.readf(format:String)
when try to store in arr list of Int.
the scaladoc said :
def readf(format: String): List[Any] Reads in some structured input
(from the default input), specified by a format specifier. See class
java.text.MessageFormat for details of the format specification.
format the format of the input.
returns a list of all extracted values.
Definition Classes StdIn Exceptions thrown java.io.EOFException if the
end of the input stream has been reached.
So, go to your browser, search "java.text.MessageFormat", and find it :
https://docs.oracle.com/javase/7/docs/api/java/text/MessageFormat.html
Following code converts Scala List into java.util.List (Tested in Scala 2.11)
import scala.collection.JavaConverters._
val a = List(1, 2, 3)
val b = a.asJava
However, the conversion result seems incomplete. Because some methods in java.util.List do not work.
scala> b.remove(2)
java.lang.UnsupportedOperationException
at java.util.AbstractList.remove(AbstractList.java:161)
... 29 elided
My workaround is as follows:
val c = new java.util.ArrayList(a.asJava)
This works but seems redundant in API-design perspective.
Is this the correct way of using asJava method?
Why does Scala's JavaConverters produce incomplete result?
Because some methods in java.util.List do not work.
These methods are explicitly optional, because java.util.List covers both mutable and immutable lists, and some of implementations in Java standard library don't support them either:
Removes the first occurrence of the specified element from this list, if it is present (optional operation)...
Throws: ... UnsupportedOperationException - if the remove operation is not supported by this list
Same for other Java collection interfaces. So the result does completely satisfy the interface.
It appears that what you're getting from the asJava is an immutable list, as you've started with an immutable Scala list. Try the following
val a = List(1,2,3).to[scala.collection.mutable.ListBuffer].asJava
Wrong syntax problem in recursively deleting scala files
Files.walk(path, FileVisitOption.FOLLOW_LINKS)
.sorted(Comparator.reverseOrder())
.forEach(Files.deleteIfExists)
The issue is that you're trying to pass a scala-style function to a method expecting a java-8-style function. There's a couple libraries out there that can do the conversion, or you could write it yourself (it's not complicated), or probably the simplest is to just convert the java collection to a scala collection that has a foreach method expecting a scala-style function as an argument:
import scala.collection.JavaConverters._
Files.walk(path, FileVisitOption.FOLLOW_LINKS)
.sorted(Comparator.reverseOrder())
.iterator().asScala
.foreach(Files.deleteIfExists)
In Scala 2.12 I expect this should work:
...forEach(Files.deleteIfExists(_: Path))
The reason you need to specify argument type is because expected type is Consumer[_ >: Path], not Consumer[Path] as it would be in Scala.
If it doesn't work (can't test at the moment), try
val deleteIfExists: Consumer[Path] = Files.deleteIfExists(_)
...forEach(deleteIfExists)
Before Scala 2.12, Joe K's answer is the correct one.
I am new to Spark and Scala as well, so this might be a very basic question.
I created a text file with 4 lines of some words. The rest of the code is as below:
val data = sc.textFile("file:///home//test.txt").map(x=> x.split(" "))
println(data.collect)
println(data.take(2))
println(data.collect.foreach(println))
All the above "println" commands are producing output as: [Ljava.lang.String;#1ebec410
Any idea how do I display the actual contents of the rdd, I have even tried "saveAstextfile", it also save the same line as java...
I am using Intellij IDE for spark scala and yes, I have gone through other posts related to this, but no help. Thanking you in advance
The final return type of RDD is RDD[Array[String]] Previously you were printing the Array[String] that prints something like this [Ljava.lang.String;#1ebec410) Because the toString() method of Array is not overridden so it is just printing the HASHCODE of object
You can try casting Array[String] to List[String] by using implicit method toList now you will be able to see the content inside the list because toString() method of list in scala in overridden and shows the content
That Means if you try
data.collect.foreach(arr => println(arr.toList))
this will show you the content or as #Raphael has suggested
data.collect().foreach(arr => println(arr.mkString(", ")))
this will also work because arr.mkString(", ")will convert the array into String and Each element Seperated by ,
Hope this clears you doubt
Thanks
data is of type RDD[Array[String]], what you print is the toString of the Array[String] ( [Ljava.lang.String;#1ebec410), try this:
data.collect().foreach(arr => println(arr.mkString(", ")))
val label = Try("here_a").getOrElse("here_b")
In the case where here_a is not found, this is not falling back to here_b. Why is the .getOrElse not functioning?
Thanks #jwvh. These values are sting files paths and thus the exception is the following Exception in thread "main" java.io.FileNotFoundException:
As per Andrew James Ramirez's comment I tried this but issue persists.
Try(throw new Exception("FAIL here_a")).getOrElse("here_b")
I have also tried
Try(throw new Exception("FileNotFoundException here_a")).getOrElse("here_b")
Edit
It seems I may have oversimplified this question for SO. Some more context. The string is actually a file path. Perhaps this is making a difference?
Effectively, a json file may be found in one of two possible locations. I thus wish to try the first location and if a java.io.FileNotFoundException is returned, fall back on the second location. This is what I presently have:
val input_file = Try(throw new Exception("FAIL location_a/file_a.json")).getOrElse("location_b/file_a.json")
Edit V2
I am embarrassed to say that I found the simple error. I am running this scala code on spark and I forgot to repackage in between testing. sbt package was all that was required. :-/
I think you misunderstood Try. and .getOrElse
Definition of Try :
The Try type represents a computation that may either result in an exception, or return a successfully computed value. It's similar to, but semantically different from the scala.util.Either type.
scala> Try("here_a")
res1: scala.util.Try[String] = Success(here_a)
scala> Try("here_a").get
res2: String = here_a
scala> Try(throw new Exception("FAIL here_a")).getOrElse("here_b")
res3: String = here_b
scala>
It only fails if you throw Exception. null is still a value.
Try() returns either a Success(x) or a Failure(e), where x is the successful value returned and e is the exception that was thrown. getOrElse() unwraps the Success or supplies a default value.
If you're not getting the "OrElse" then your code isn't throwing a catchable exception.
It works as you expect it should.
$ scala
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_40).
Type in expressions for evaluation. Or try :help.
scala> val label = Try(throw new RuntimeException("here_a")).getOrElse("here_b")
label: String = here_b
Please provide more context in your question if you feel this answer is insufficient.