So for spark sql you have do something like this:
val query = s"select *from table...."
val temp = sqlContext.sql(query)
Is there anything similar to this for just regular spark code? Like in python they have this exec(string) function that take string and run it as code.
This is the link: Click Me
Gatling application is doing a similar trick, they allow you to write an arbitrary Scala code for load testing and then run it with their framework.
As far as I remember they basically write Scala code into a file, prepend/append it with something like:
class MyClass {
def foo() = {
// Code goes here
}
}
and execute Scala compiler to compile this code and then load this code as a regular class.
Related
I have an object in Scala that returns a Set[String]:
object MyObject {
val setOfStrings = Set[String]("string1","string2")
}
I can successfully refer to that val from my Python code (using Py4J) with this:
jvm.com.package.MyObject.setOfStrings()
however I can't figure out how to do anything useful with it. I want to convert it into a python set.
I managed to do it using this:
eval(
str(
jvm.com.package.MyObject.setOfStrings()
)
.replace("Set(", "{\"")
.replace(")", "\"}")
.replace(", ", "\", \"")
)
but that's a horrifically brittle way of achieving it and I'm assuming Py4J provides a much better way.
I'm aware that SetConverter can convert a python set to a Java set, but I want to do the opposite. Anyone know how?
You could use a java.util.Set on the Scala side, which will be converted by Py4J into the Python equivalent MutableSet.
If you don't want to use Java collections in your Scala code base, then you might want to convert a scala.collection.immutable.Set into its Java equivalent when you want to communicate with Python.
I didn't run this code, but you could probably do this:
object YourObject {
val setOfStrings = Set[String]("string1","string2")
}
object ViewObject {
import scala.jdk.CollectionConverters._
def setOfStrings: java.util.Set[String] = YourObject.setOfStrings.asJava
}
and use it in Python:
jvm.com.package.ViewObject.setOfStrings()
Note that in this case, you won't be able to modify the collection. If you want to be able to update the view in Python and have the changes reflected back in Scala, then you'll have to use a scala.collection.mutable.Set in your Scala code.
Using the scalatags script tag I know I can define the following:
script("console.log('Running javascript code')")
But is there a way I can rather pass pure scala code instead? So something like the following:
script(println("Running scala code now"))
No, that is not possible.
You can achieve the same effect in a different way, by exporting a top-level function with the Scala code you want to run, then generate a script that calls that function:
object Exports {
#JSExportTopLevel("dynamicScriptCode")
def code(): Unit =
println("Running Scala code now")
}
...
script("dynamicScriptCode();")
Basic question, I want to set the standard input to be a specific string. Currently I am trying it with this:
import java.nio.charset.StandardCharsets
import java.io.ByteArrayInputStream
// Let's say we are inside a method now
val str = "textinputgoeshere"
System.setIn(new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8)))
Because that's similar to how I'd do it in Java, however str.getBytes seems to work differently in Scala as System in is set to a memory address when I check it with println....
I've looked at the Scala API: http://www.scala-lang.org/api/current/scala/Console$.html#setIn(in:java.io.InputStream):Unit
and I've found
def withIn[T](in: InputStream)(thunk: ⇒ T): T
But this seems to only set the input stream for a specific chunk of code, I'd like this to be a feature in a Setup method in my JUnit tests.
My problem ended up being something related to my code, not this specific concept. The correct way to override Standard In / System In to a String in Scala is the following:
val str = "your string here"
val in: InputStream = new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8))
Console.withIn(in)(yourMethod())"
My tests run correctly now.
I woud like my application to execute external scripts written in a Scala-based DSL i've developed. That's why I need something like this to work:
...
val a = evaluate("~/myextcode.scala")
...
myextcode.scala:
return 123
Can I reach some thing like this in Scala?
You can do that with Eval from twitter util. Check its Scaladocs here.
I want to create a class at run-time in Scala. For now, just consider a simple case where I want to make the equivalent of a java bean with some attributes, I only know these attributes at run time.
How can I create the scala class? I am willing to create from scala source file if there is a way to compile it and load it at run time, I may want to as I sometimes have some complex function I want to add to the class. How can I do it?
I worry that the scala interpreter which I read about is sandboxing the interpreted code that it loads so that it won't be available to the general application hosting the interpreter? If this is the case, then I wouldn't be able to use the dynamically loaded scala class.
Anyway, the question is, how can I dynamically create a scala class at run time and use it in my application, best case is to load it from a scala source file at run time, something like interpreterSource("file.scala") and its loaded into my current runtime, second best case is some creation by calling methods ie. createClass(...) to create it at runtime.
Thanks, Phil
There's not enough information to know the best answer, but do remember that you're running on the JVM, so any techniques or bytecode engineering libraries valid for Java should also be valid here.
There are hundreds of techniques you might use, but the best choice depends totally on your exact use case, as many aren't general purpose. Here's a couple of ideas though:
For a simple bean, you may as well
just use a map, or look into the
DynaBean class from apache commons.
For more advanced behaviour you could
invoke the compiler explicitly and
then grab the resulting .class file
via a classloader (this is largely
how JSPs do it)
A parser and custom DSL fit well in
some cases. As does bean shell
scripting.
Check out the ScalaDays video here: http://days2010.scala-lang.org/node/138/146
which demonstrates the use of Scala as a JSR-223 compliant scripting language.
This should cover most scenarios where you'd want to evaluate Scala at runtime.
You'll also want to look at the email thread here: http://scala-programming-language.1934581.n4.nabble.com/Compiler-API-td1992165.html#a1992165
This contains the following sample code:
// We currently call the compiler directly
// To reduce coupling, we could instead use ant and the scalac ant task
import scala.tools.nsc.{Global, Settings}
import scala.tools.nsc.reporters.ConsoleReporter
{
// called in the event of a compilation error
def error(message: String): Nothing = ...
val settings = new Settings(error)
settings.outdir.value = classesDir.getPath
settings.deprecation.value = true // enable detailed deprecation warnings
settings.unchecked.value = true // enable detailed unchecked warnings
val reporter = new ConsoleReporter(settings)
val compiler = new Global(settings, reporter)
(new compiler.Run).compile(filenames)
reporter.printSummary
if (reporter.hasErrors || reporter.WARNING.count > 0)
{
...
}
}
val mainMethod: Method = {
val urls = Array[URL]( classesDir.toURL )
val loader = new URLClassLoader(urls)
try {
val clazz: Class = loader.loadClass(...)
val method: Method = clazz.getMethod("main", Array[Class]( classOf[Array[String]] ))
if (Modifier.isStatic(method.getModifiers)) {
method
} else {
...
}
} catch {
case cnf: ClassNotFoundException => ...
case nsm: NoSuchMethodException => ...
}
}
mainMethod.invoke(null, Array[Object]( args ))