Run Scales XML's pullXml in weblogic server - scala

I used pullXml due to the file size is quite large, > 100M, I wrote a sample program as follows:
object TestXml {
val mc = new java.math.MathContext(1024)
val zero = BigDecimal(0, mc)
def calculate(infile: String, encoding: String): BigDecimal = {
val inStream = new FileInputStream(infile)
val pull = pullXml(new InputSource(new InputStreamReader(inStream, encoding)))
val ns = Namespace("urn:abcaus.onair.sintecmedia.com")
val qnames = List(ns("ePGResp"), "EPGResponse"l, "Event"l)
def eventStream = iterate(qnames, pull).toStream
var count = zero
eventStream foreach { event => count += eventId(event) }
inStream.close
count
}
def eventId(event: XmlPath): Long =
text(event.\*("EventID")).toLong
def main(args: Array[String]): Unit = args.toList match {
case infile :: encoding :: Nil => println(calculate(infile, encoding))
case _ => println("usage scala -cp classpath au.net.abc.epg.TestLoadXml infile encoding")
}
}
I run the program on a command line as follows:
$JAVA_HOME/bin/java -cp JarContainsSampleProgram.jar:scala-library-2.10.2.jar:scala-reflect-2.10.2.jar:scalalogging-slf4j_2.10-1.0.1.jar:scalaz-core_2.10-7.0.0.jar:scalaz-effect_2.10-7.0.0.jar:scalaz-iterv_2.10-7.0.0.jar:scales-xml_2.10-0.6.0-M1.jar:slf4j-api-1.6.4.jar TestLoadXml /home/wonga4d/EPG/Huge.xml utf-16
It runs successfully returning a value, say, 879452677392.
However, when I deploy it as a oracle service bus Java callout (which is ok because Scala is JVM lang) to be used by an OSB proxy, still use the same input file and encoding, I got the following error
Callout to java method "public static scala.math.BigDecimal au.net.abc.epg.TestLoadXml.calculate(java.lang.String,java.lang.String)" resulted in exception: Got an event (Text()) that should not be in the prolog java.lang.RuntimeException: Got an event (Text()) that should not be in the prolog
at scala.sys.package$.error(package.scala:27)
at scala.Predef$.error(Predef.scala:142)
at scales.utils.package$.error(package.scala:19)
at scales.xml.parser.pull.PullUtils$$anonfun$getMisc$1.apply(PullIterator.scala:144)
at scales.xml.parser.pull.PullUtils$$anonfun$getMisc$1.apply(PullIterator.scala:141)
at scala.util.Either.fold(Either.scala:97)
at scales.xml.parser.pull.PullUtils$.getMisc(PullIterator.scala:141)
at scales.xml.parser.pull.XmlPull$class.start(PullIterator.scala:89)
at scales.xml.parser.pull.XmlPulls$$anon$1.start(XmlPull.scala:134)
at scales.xml.parser.pull.XmlPulls$$anon$1.<init>(XmlPull.scala:156)
at scales.xml.parser.pull.XmlPulls$class.pullXml(XmlPull.scala:134)
at scales.xml.package$.pullXml(package.scala:7)
at TestXml$.calculate(TestLoadXml.scala:23)
at TestXml.calculate(TestLoadXml.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at stages.transform.runtime.JavaCalloutRuntimeStep$1.run(JavaCalloutRuntimeStep.java:173)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
at weblogic.security.Security.runAs(Security.java:61)
at stages.transform.runtime.JavaCalloutRuntimeStep.processMessage(JavaCalloutRuntimeStep.java:195)
at com.bea.wli.sb.pipeline.debug.DebuggerRuntimeStep.processMessage(DebuggerRuntimeStep.java:74)
at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
at com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
It fails at pullXml. But it always succeeds when running on a commandline which I showed before. If I don't use pullXml but loadXml, it will always succeed even when running in a weblogic server. But loadXml will get a problem if loading a huge xml file. Both pullXml & loadXml methods are located in the same jar, scales-xml_2.10-0.6.0-M1.jar.
Just wonder if anyone ever used scales xml in weblogic server. Sounds like I got to give up using scales xml if weblogic server is the execution environment.
Thanks

that your code runs outside of weblogic shows its a javax.xml implementation issue. Its possible that this only needs you to supply an alternate implementation (e.g. aalto-xml) with a "child first" or app first classloader setting.
If you could let me know (answering here is cool) if that sorts things out that would be great, I'll add it to the docs.
Outside of this issue I hope Scales is working out for you :)
Cheers,
Chris

Related

Strange timeout with ScalaTest's Selenium DSL

I'm writing Selenium tests with ScalaTest's Selenium DSL and I'm running into timeouts I can't explain. To make matters more complicated, they only seem to happen some of the time.
The problem occurs whenever I access an Element after a page load or some Javascript rendering. It looks like this:
click on "editEmployee"
eventually {
textField(name("firstName")).value = "Steve"
}
My PatienceConfig is configured like this:
override implicit val patienceConfig: PatienceConfig =
PatienceConfig(timeout = Span(5, Seconds), interval = Span(50, Millis))
The test fails with the following error:
- should not display the old data after an employee was edited *** FAILED ***
The code passed to eventually never returned normally. Attempted 1 times over 10.023253653000001 seconds.
Last failure message: WebElement 'firstName' not found.. (EditOwnerTest.scala:24)
It makes sense that it doesn't succeed immediately, because the click causes some rendering, and the textfield may not be available right away. However, it shouldn't take 10 seconds to make an attempt to find it, right?
Also, I find it very interesting that the eventually block tried it only once, and that it took almost precisely 10 seconds. This smells like a timeout occurred somewhere, and it's not my PatienceConfig, because that was set to time out after 5 seconds.
With this workaround, it does work:
click on "editEmployee"
eventually {
find(name("firstName")).value // from ScalaTest's `OptionValues`
}
textField(name("firstName")).value = "Steve"
I did some digging in the ScalaTest source, and I've noticed that all calls that have this problem (it's not just textField), eventually call webElement at some point. The reason why the workaround works, is because it doesn't call webElement. webElement is defined like this:
def webElement(implicit driver: WebDriver, pos: source.Position = implicitly[source.Position]): WebElement = {
try {
driver.findElement(by)
}
catch {
case e: org.openqa.selenium.NoSuchElementException =>
// the following is avoid the suite instance to be bound/dragged into the messageFun, which can cause serialization problem.
val queryStringValue = queryString
throw new TestFailedException(
(_: StackDepthException) => Some("WebElement '" + queryStringValue + "' not found."),
Some(e),
pos
)
}
}
I've copied that code into my project and played around with it, and it looks like constructing and/or throwing the exception is where most of the 10 seconds are spent.
(EDIT Clarification: I've actually seen the code actually spend its 10 seconds inside the catch block. The implicit wait is set to 0, and besides, if I remove the catch block everything simply works as expected.)
So my question is, what can I do to avoid this strange behaviour? I don't want to have to insert superfluous calls to find all the time, because it's easily forgotten, especially since, as I said, the error occurs only some of the time. (I haven't been able to determine when the behaviour occurs and when it doesn't.)
It is clear that the textField(name("firstName")).value = "Steve" ends up calling the WebElement as you have found out.
Since the issue in the op is happening where ever web elements are involved (which in turn implies that webdriver is involved), I think it is safe to assume that the issue is related to the implicit wait on the Web driver.
implicitlyWait(Span(0, Seconds))
The above should ideally fix the issue. Also, making implicit wait to be 0 is a bad practice. Any web page might have some loading issues. The page load is handled by Selenium outside its wait conditions. But slow element load (may be due to ajax calls) could result in failure. I usually keep 10 seconds as my standard implicit wait. For scenarios which require more wait, explicit waits can be used.
def implicitlyWait(timeout: Span)(implicit driver: WebDriver): Unit = {
driver.manage.timeouts.implicitlyWait(timeout.totalNanos, TimeUnit.NANOSECONDS)
}
Execution Flow:
name("firstName") ends up having value as Query {Val by = By.className("firstName") }.
def name(elementName: String): NameQuery = new NameQuery(elementName)
case class NameQuery(queryString: String) extends Query { val by = By.name(queryString) }
Query is fed to the textField method which calls the Query.webElement as below.
def textField(query: Query)(implicit driver: WebDriver, pos: source.Position): TextField = new TextField(query.webElement)(pos)
sealed trait Query extends Product with Serializable {
val by: By
val queryString: String
def webElement(implicit driver: WebDriver, pos: source.Position = implicitly[source.Position]): WebElement = {
try {
driver.findElement(by)
}
catch {
case e: org.openqa.selenium.NoSuchElementException =>
// the following is avoid the suite instance to be bound/dragged into the messageFun, which can cause serialization problem.
val queryStringValue = queryString
throw new TestFailedException(
(_: StackDepthException) => Some("WebElement '" + queryStringValue + "' not found."),
Some(e),
pos
)
}
}
}
I don't know ScalaTest's specifics, but such strange timeouts usually occur when you're mixing up implicit and explicit waits together.
driver.findElement uses implicit waits internally. And depending on specified explicit waits timeout, you may face with summing both together.
Ideally, implicit waits should be set to 0 to avoid such issues.

Unable to use WURFL with Scala

When I run WURFL demo app for scala:
object Demo {
def main(args: Array[String]) {
// Create WURFL passing a GeneralWURFLEngine object with a wurfl xml
val wurflWrapper = new Wurfl(new GeneralWURFLEngine("classpath:/resources/wurfl.zip"))
// Set cache provider
wurflWrapper.setCacheProvider(new LRUMapCacheProvider)
// Set Performance/Accuracy Mode
wurflWrapper.setTargetAccuracy
// Set Capability Filter
wurflWrapper.setFilter(
"can_assign_phone_number",
"marketing_name",
"brand_name",
"model_name",
"is_smarttv",
"is_wireless_device",
"device_os",
"device_os_version",
"is_tablet",
"ux_full_desktop",
"pointing_method",
"preferred_markup",
"resolution_height",
"resolution_width",
"xhtml_support_level")
// User-Agent here
var userAgent = ""
// Defining headers
var headers = Map("Accept-Datetime"->"Thu, 31 May 2007 20:35:00 GMT")
headers += ("Content-Type"-> "application/x-www-form-urlencoded")
var device = wurflWrapper.deviceForHeaders(userAgent, headers)
val matchType = device.matchType
if (matchType == MatchType.conclusive)
{
println("Match Type is conclusive")
}
val wireless = device.capability("is_wireless_device")
println("Is wireless: " + wireless)
}
}
I get this exception:
[main] ERROR net.sourceforge.wurfl.core.GeneralWURFLEngine - cannot initialize: java.lang.NullPointerException: in is null
java.lang.NullPointerException: in is null
at java.util.zip.ZipInputStream.<init>(ZipInputStream.java:101)
at java.util.zip.ZipInputStream.<init>(ZipInputStream.java:80)
at net.sourceforge.wurfl.core.resource.FileLoader.fromZipFile(FileLoader.java:248)
at net.sourceforge.wurfl.core.resource.FileLoader.openInputStream(FileLoader.java:230)
at net.sourceforge.wurfl.core.resource.FileLoader.getStream(FileLoader.java:288)
at net.sourceforge.wurfl.core.resource.XMLResource.getData(XMLResource.java:163)
at net.sourceforge.wurfl.core.resource.DefaultWURFLModel.init(DefaultWURFLModel.java:115)
at net.sourceforge.wurfl.core.resource.DefaultWURFLModel.<init>(DefaultWURFLModel.java:107)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.init(GeneralWURFLEngine.java:340)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.initIfNeeded(GeneralWURFLEngine.java:319)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.getDeviceForRequest(GeneralWURFLEngine.java:451)
at com.scientiamobile.wurfl.Wurfl.deviceForHeaders(Wurfl.scala:77)
at com.Demo$.main(Demo.scala:49)
at com.Demo.main(Demo.scala)
Exception in thread "main" net.sourceforge.wurfl.core.exc.WURFLRuntimeException: WURFL unexpected exception
at net.sourceforge.wurfl.core.GeneralWURFLEngine.initIfNeeded(GeneralWURFLEngine.java:322)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.getDeviceForRequest(GeneralWURFLEngine.java:451)
at com.scientiamobile.wurfl.Wurfl.deviceForHeaders(Wurfl.scala:77)
at com.Demo$.main(Demo.scala:49)
at com.Demo.main(Demo.scala)
Caused by: java.lang.NullPointerException: in is null
at java.util.zip.ZipInputStream.<init>(ZipInputStream.java:101)
at java.util.zip.ZipInputStream.<init>(ZipInputStream.java:80)
at net.sourceforge.wurfl.core.resource.FileLoader.fromZipFile(FileLoader.java:248)
at net.sourceforge.wurfl.core.resource.FileLoader.openInputStream(FileLoader.java:230)
at net.sourceforge.wurfl.core.resource.FileLoader.getStream(FileLoader.java:288)
at net.sourceforge.wurfl.core.resource.XMLResource.getData(XMLResource.java:163)
at net.sourceforge.wurfl.core.resource.DefaultWURFLModel.init(DefaultWURFLModel.java:115)
at net.sourceforge.wurfl.core.resource.DefaultWURFLModel.<init>(DefaultWURFLModel.java:107)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.init(GeneralWURFLEngine.java:340)
at net.sourceforge.wurfl.core.GeneralWURFLEngine.initIfNeeded(GeneralWURFLEngine.java:319)
... 4 more
The "wurfl.zip" is well located under "resources".
I also tried adding it to main Scala classes path, but still not luck.
From a code perspective
val wurflWrapper = new Wurfl(new GeneralWURFLEngine("classpath:/resources/wurfl.zip"))
is a proper way to initialize your WURFL engine.
You may want to provide information about how you're running the demo, if you are running it inside an IDE (IDEA, Eclipse or Netbeans), or using command line, or other ways. It can also be useful to tell whether you're using Maven or not.
In case you are running it using command line, please provide a sample of how you launch the Scala app and how you set the classpath.
Assuming a scenario where you are compiling with maven and executing the project directly into the target dir using -cp classes, execution will result in your classpath error because resource files are not included in the classes directory.
Make sure that wurfl-scala-example-.jar is included your classpath.
If you are using the Demo project inside IntelliJ IDEA, please make sure that the resource directory is marked as "resource", otherwise IDEA run tool will not include the wurfl.zip file as a resource.
Hope this helps.

Spray, Slick, Spark - OutOfMemoryError: PermGen space

I have successfully implemented a simple web service using Spray and Slick that passes an incoming request through a Spark ML Prediction Pipeline. Everything was working fine until I tried to add a data layer. I have chosen Slick it seems to be popular.
However, I can't quite get it to work right. I have been basing most of my code on the Hello-Slick Activator Template. I use a DAO object like so:
object dataDAO {
val datum = TableQuery[Datum]
def dbInit = {
val db = Database.forConfig("h2mem1")
try {
Await.result(db.run(DBIO.seq(
datum.schema.create
)), Duration.Inf)
} finally db.close
}
def insertData(data: Data) = {
val db = Database.forConfig("h2mem1")
try {
Await.result(db.run(DBIO.seq(
datum += data,
datum.result.map(println)
)), Duration.Inf)
} finally db.close
}
}
case class Data(data1: String, data2: String)
class Datum(tag: Tag) extends Table[Data](tag, "DATUM") {
def data1 = column[String]("DATA_ONE", O.PrimaryKey)
def data2 = column[String]("DATA_TWO")
def * = (data1, data2) <> (Data.tupled, Data.unapply)
}
I initialize my database in my Boot object
object Boot extends App {
implicit val system = ActorSystem("raatl-demo")
Classifier.initializeData
PredictionDAO.dbInit
// More service initialization code ...
}
I try to add a record to my database before completing the service request
val predictionRoute = {
path("data") {
get {
parameter('q) { query =>
// do Spark stuff to get prediction
DataDAO.insertData(data)
respondWithMediaType(`application/json`) {
complete {
DataJson(data1, data2)
}
}
}
}
}
When I send a request to my service my application crashes
java.lang.OutOfMemoryError: PermGen space
I suspect I'm implementing the Slick API incorrectly. its hard to tell from the documentation, because it stuffs all the operations into a main method.
Finally, my conf is the same as the activator ui
h2mem1 = {
url = "jdbc:h2:mem:raatl"
driver = org.h2.Driver
connectionPool = disabled
keepAliveConnection = true
}
Has anyone encountered this before? I'm using Slick 3.1
java.lang.OutOfMemoryError: PermGen space is normally not a problem with your usage, here is what oracle says about this.
The detail message PermGen space indicates that the permanent generation is full. The permanent generation is the area of the heap where class and method objects are stored. If an application loads a very large number of classes, then the size of the permanent generation might need to be increased using the -XX:MaxPermSize option.
I do not think this is because of incorrect implementation of the Slick API. This probably happens because you are using multiple frameworks that loads many classes.
Your options are:
Increase the size of perm gen size -XX:MaxPermSize
Upgrade to Java 8. The perm gen space is now replaced with MetaSpace which is tuned automatically

Jetty works for HTTP but not HTTPS

I am trying to create a jetty consumer. I am able to get it successfully running using the endpoint uri:
jetty:http://0.0.0.0:8080
However, when I modify the endpoint uri for https:
jetty:https://0.0.0.0:8443
The page times out trying to load. This seems odd because the camel documentation states it should function right out of the box.
I have since loaded a signed SSL into java's default keystore, with my attempted implementation to load it below:http://camel.apache.org/jetty.html
I have a basic Jetty instance using the akka-camel library with akka and scala. ex:
class RestActor extends Actor with Consumer {
val ksp: KeyStoreParameters = new KeyStoreParameters();
ksp.setPassword("...");
val kmp: KeyManagersParameters = new KeyManagersParameters();
kmp.setKeyStore(ksp);
val scp: SSLContextParameters = new SSLContextParameters();
scp.setKeyManagers(kmp);
val jettyComponent: JettyHttpComponent = CamelExtension(context.system).context.getComponent("jetty", classOf[JettyHttpComponent])
jettyComponent.setSslContextParameters(scp);
def endpointUri = "jetty:https://0.0.0.0:8443/"
def receive = {
case msg: CamelMessage => {
...
}
...
}
...
}
This resulted in some progress, because the page does not timeout anymore, but instead gives a "The connection was interrupted" error. I am not sure where to go from here because camel is not throwing any Exceptions, but rather failing silently somewhere (apparently).
Does anybody know what would cause this behavior?
When using java's "keytool" I did not specify an output file. It didn't throw back an error, so it probably went somewhere. I created a new keystore and explicitly imported my crt into the keyfile. I then explicitly added the filepath to that keystore I created, and everything works now!
If I had to speculate, it is possible things failed silently because I was adding the certs to jetty's general bank of certs to use if eligible, instead of explicitly binding it as the SSL for the endpoint.
class RestActor extends Actor with Consumer {
val ksp: KeyStoreParameters = new KeyStoreParameters();
ksp.setResource("/path/to/keystore");
ksp.setPassword("...");
val kmp: KeyManagersParameters = new KeyManagersParameters();
kmp.setKeyStore(ksp);
val scp: SSLContextParameters = new SSLContextParameters();
scp.setKeyManagers(kmp);
val jettyComponent: JettyHttpComponent = CamelExtension(context.system).context.getComponent("jetty", classOf[JettyHttpComponent])
jettyComponent.setSslContextParameters(scp);
def endpointUri = "jetty:https://0.0.0.0:8443/"
def receive = {
case msg: CamelMessage => {
...
}
...
}
...
}
Hopefully somebody in the future can find use for this code as a template in implementing Jetty over SSL with akka-camel (surprisingly no examples seem to exist)

How can I access a resource when running an SBT runTask?

I've got an XML file that I need to read from the classpath in order to load some test data for my project with DBUnit when running a custom runTask in SBT.
The XML file is located in /src/main/resources and is copied properly to the /target/scala_2.8.1/classes during the build, but I get a MalformedURLException when trying to access it.
The weird thing is, I can access the file when this data loading functionality was part of my Scala specs unit tests.
Any ideas?
In my case the problem was that I used getClass.getResourceAsStream() in early initialiser. Had to specify the class explicitly with Class.forName() to solve it: Class.forName(<class name>).getResourceAsStream("/data.xml")
If the error is saying that the URL is malformed, it's probably true.
Here's a code I use to grab file from resource during test:
def copyFileFromResource(source: String, dest: File) {
val in = getClass.getResourceAsStream(source)
val reader = new java.io.BufferedReader(new java.io.InputStreamReader(in))
val out = new java.io.PrintWriter(new java.io.FileWriter(dest))
var line: String = null
line = reader.readLine
while (line != null) {
out.println(line)
line = reader.readLine
}
in.close
out.flush
}