Use case for WebDriver (https://github.com/typesafehub/webdriver) to execute js - scala

Question

I'm working on a web scraper using this project (based on Scala, Spray, Akka and PhantomJS)
The problem is that I can't find a more specific example of how to use it, and the documentation is missing a lot of details
1- I would like to know how to give an specific URL so I can get data from it
2- How can I excecute, or pass a javascript file or function so that phantom can run and do some stuff(return specific data or whatever, from the site in point 1- )
Here is my Main.scala file: (Is almost the same as the one in the project)
package com.typesafe.webdriver.tester
import akka.actor.{ActorRef, ActorSystem}
import akka.pattern.ask
import com.typesafe.webdriver.{Session, PhantomJs, LocalBrowser}
import akka.util.Timeout
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import spray.json._
import spray.http._
object Main {
def main(args: Array[String]) {
implicit val system = ActorSystem("webdriver-system")
implicit val timeout = Timeout(5.seconds)
system.scheduler.scheduleOnce(7.seconds) {
system.shutdown()
System.exit(1)
}
val browser = system.actorOf(PhantomJs.props(system), "localBrowser")
browser ! LocalBrowser.Startup
for (
session <- (browser ? LocalBrowser.CreateSession).mapTo[ActorRef];
result <- (session ? Session.ExecuteNativeJs("return 5+5",JsArray(JsNumber(999)))).mapTo[JsNumber]
) yield {
println(result)
try {
system.shutdown()
System.exit(0)
} catch {
case _: Throwable =>
}
}
}
}

Answer 1

I would suggest you to use already created web scrappers in Scala.
For example ScalaWebDcraper which has nicely writted DSL and scrapping feature.
https://github.com/Rovak/ScalaWebscraper
It can be combined with Goose, which is a web article extractor. You can use it to fetch article data from the links you visit with the previous library.
https://github.com/jiminoc/goose
Also, checkout Metascrapper, a Scala Library for Scraping Page Metadata
https://beachape.com/blog/2013/09/05/introducing-metascraper-a-scala-library-for-scraping-page-metadata/
And check this question, lot's of valuable info inside.