spark dataframe handle option some none - scala

Question

Similar to Apache Spark: dealing with Option/Some/None in RDDs I have a function which is applied via df.mapPartitions
def mapToTopics(iterator: Iterator[RawRecords]): Iterator[TopicContent] = {
iterator.map(k => {
browser.parseString(k.content) >> elementList("doc").map(d => {
TopicContent((d >> text("docno")).head, (d >> text("text")).head, k.path)
})
})
}
The following is also defined:
#transient lazy val browser = JsoupBrowser()
case class TopicContent(topic: String, content: String, filepath: String)
case class RawRecords(path: String, content: String)
Above will throw an error (NoSuchElementException) if no xml tags with text exist (which happens for some malformed documents)
How can I correct and simplify this code to properly handle the options?
When trying to use a util.Try as outlined by the link above and applying a flatMap my code would fail, as instead of Element it was using Char
edit
try {
Some(TopicContent((d >> text("docno")).head, (d >> text("text")).head, k.path))
} catch {
case noelem: NoSuchElementException => {
println(d.head)
None
}
}
})
val flattended = results.flatten
Will unfortunately only return a Option[Nothing]
edit4
https://gist.github.com/geoHeil/bfb01427b88cf58ea755f912ce539712 a minimal sample without spark (and full code below as well)
import net.ruippeixotog.scalascraper.browser.JsoupBrowser
import net.ruippeixotog.scalascraper.dsl.DSL.Extract._
import net.ruippeixotog.scalascraper.dsl.DSL._
import net.ruippeixotog.scalascraper.scraper.ContentExtractors.elementList
#transient lazy val browser = JsoupBrowser()
val broken =
"""
|<docno>
| LA051089-0001
| </docno>
| <docid>
| 54901
| </docid>
| <date>
| <p> May 10, 1989, Wednesday, Home Edition </p>
| </date>
| <section>
| <p> Metro; Part 2; Page 2; Column 2 </p>
| </section>
| <graphic>
| <p> Photo, Cloudy and Clear A stormy afternoon provides a clear view of Los Angeles' skyline, with the still-emerging Library Tower rising above its companion buildings. KEN LUBAS / Los Angeles Times </p>
| </graphic>
| <type>
| <p> Wild Art </p>
| </type>
""".stripMargin
val correct =
"""
|<DOC>
|<DOCNO> FR940104-0-00001 </DOCNO>
|<PARENT> FR940104-0-00001 </PARENT>
|<TEXT>
|
|<!-- PJG FTAG 4700 -->
|
|<!-- PJG STAG 4700 -->
|
|<!-- PJG ITAG l=90 g=1 f=1 -->
|
|<!-- PJG /ITAG -->
|
|<!-- PJG ITAG l=90 g=1 f=4 -->
|Federal Register
|<!-- PJG /ITAG -->
|
|<!-- PJG ITAG l=90 g=1 f=1 -->
|&blank;/&blank;Vol. 59, No. 2&blank;/&blank;Tuesday, January 4, 1994&blank;/&blank;Rules and Regulations
|
|<!-- PJG 0012 frnewline -->
|
|<!-- PJG /ITAG -->
|
|<!-- PJG ITAG l=01 g=1 f=1 -->
|Vol. 59, No. 2
|<!-- PJG 0012 frnewline -->
|
|<!-- PJG /ITAG -->
|
|<!-- PJG ITAG l=02 g=1 f=1 -->
|Tuesday, January 4, 1994
|<!-- PJG 0012 frnewline -->
|
|<!-- PJG 0012 frnewline -->
|
|<!-- PJG /ITAG -->
|
|<!-- PJG /STAG -->
|
|<!-- PJG /FTAG -->
|</TEXT>
|</DOC>
""".stripMargin
case class RawRecords(path: String, content: String)
case class TopicContent(topic: String, content: String, filepath: String)
val raw = Seq(RawRecords("first", correct), RawRecords("second", broken))
val result = mapToTopics(raw.iterator)
// Variant 1
def mapToTopics(iterator: Iterator[RawRecords]): Iterator[TopicContent] = {
iterator.flatMap(k => {
val documents = browser.parseString(k.content) >> elementList("doc")
documents.map(d => {
val docno = d >> text("docno")
// try {
val textContent = d >> text("text")
TopicContent(docno, textContent, k.path)
// } catch {
// case _:NoSuchElementException => TopicContent(docno, None, k.path)
// }
}) //.filter(_.content !=None)
})
}
// When broken down even further you see the following will produce Options of strings
browser.parseString(raw(0).content) >> elementList("doc").map(d => {
val docno = d >> text("docno")
val textContent = d >> text("text")
(docno.headOption, textContent.headOption)
})
// while below will now map to characters. What is wrong here?
val documents = browser.parseString(raw(0).content) >> elementList("doc")
documents.map(d => {
val docno = d >> text("docno")
val textContent = d >> text("text")
(docno.headOption, textContent.headOption)
})

Answer 1

The difference between the two examples lies in the precedence of the operators. When you're doing browser.parseString(raw(0).content) >> elementList("doc").map(...), you're calling map on elementList("doc"), and not on the whole expression. In order for the first example to behave the same as the second one, you need to write either (browser.parseString(raw(0).content) >> elementList("doc")).map(...) (recommended) or browser.parseString(raw(0).content) >> elementList("doc") map(...).
In the context of scala-scraper, the library you're using, the two expressions mean very different things. With browser.parseString(raw(0).content) >> elementList("doc") you're extracting a List[Element] from a document, and calling map on that does just what you'd expect from a collection. On the other hand, elementList("doc") is an HtmlExtractor[List[Element]] and calling map on an extractor creates a new HtmlExtractor with the results of the original extractor transformed. That's the reason why you end up with two very different results.

Answer 2

I am unfamiliar with the API you are using, but using headOpton in a for comprehension might help you:
import net.ruippeixotog.scalascraper.dsl.DSL._
import net.ruippeixotog.scalascraper.dsl.DSL.Extract._
import net.ruippeixotog.scalascraper.dsl.DSL.Parse._
iterator.map(k => {
browser.parseString(k.content) >> elementList("doc").flatMap(d => {
for {
docno <- text("docno")).headOption
text <- (d >> text("text")).headOption
} yield TopicContent(docno, text, k.path)
})
})
This way you only construct TopicContent, really a Some(TopicContent), when both docno and text are present--and None otherwise. Then the flatMap removes all the Nones and extracts the content in Somes leaving you with a collection of TopicContent instances created for all valid XML.