quick question on calling an object and running it from main just need the last command val op = df.coalesce(1).write.mode("overwrite").format("csv").save("report")to be able called from a main class
how do we intilize an instance of object and run to get to write to csv?
def run(implicit context: Context): Unit = {
val timer = Timer.start()
//not working
val newRep = Report_Adhoc
val d = newRep.tab.toDF()
val op = d.coalesce(1).write.mode("overwrite").format("csv").save("report")
println(s"pipeline complete in [${timer.elapsedTime()}]")
}
Main class is this ^ but this throws null point exception
object Report_Adhoc extends App with TransientLogger{
// code not including too verbose
val tab =
counts
.filter(c => c._1.id.nonEmpty && c._2.id.nonEmpty)
.map(c => (c._1, c._2, c._3, c._3.values.sum))
.sort($"_4".desc)
.map(count =>
row(
count._1.id, count._1.label,
count._2.id, count._2.label,
count._3(CITE), count._3(CROSS), count._3(MANUAL),
count._3(RECIPROCAL), count._3(TRANSITIVE), count._3(FAMILY),
count._4
)
)
val df = tab.toDF()
val op = df.coalesce(1).write.mode("overwrite").format("csv").save("report")
}
take a look at this:
https://www.scala-lang.org/api/2.12.3/scala/DelayedInit.html
https://www.scala-lang.org/api/2.12.3/scala/App.html
Also, don't put underscore in names. don't reuse main class/object. Last but not least, make a minimum reproduction instead of pasting big block of code like this.
Related
I'm writing a Scala library to operate upon Spark DataFrames. I have a bunch of classes, each of which contain a function that operates upon the supplied DataFrame:
class Foo(){val func = SomeFunction(,,,)}
class Bar(){val func = SomeFunction(,,,)}
class Baz(){val func = SomeFunction(,,,)}
The user of my library passes a parameter operation: String to indicate class to instantiate, the value passed has to be the name of one of those classes hence I have code that looks something like this:
operation match {
case Foo => new Foo().SomeFunction
case Bar => new Bar().SomeFunction
case Baz => new Baz().SomeFunction
}
I'm a novice Scala developer but this seems rather like a clunky way of achieving this. I'm hoping there is a simpler way to instantiate the desired class based on the value of operation given that it will be the same as the name of the desired class.
The reason I want to do this is that I want external contributors to contribute their own classes and I want to make it at easy as possible for them to do that, I don't want them to have to know they also need to go and change a pattern match.
For
case class SomeFunction(s: String)
class Foo(){val func = SomeFunction("Foo#func")}
class Bar(){val func = SomeFunction("Bar#func")}
class Baz(){val func = SomeFunction("Baz#func")}
//...
reflection-based version of
def foo(operation: String) = operation match {
case "Foo" => new Foo().func
case "Bar" => new Bar().func
case "Baz" => new Baz().func
// ...
}
is
import scala.reflect.runtime.universe
import scala.reflect.runtime.universe._
def foo(operation: String): SomeFunction = {
val runtimeMirror = universe.runtimeMirror(getClass.getClassLoader)
val classSymbol = runtimeMirror.staticClass(operation)
val constructorSymbol = classSymbol.primaryConstructor.asMethod
val classMirror = runtimeMirror.reflectClass(classSymbol)
val classType = classSymbol.toType
val constructorMirror = classMirror.reflectConstructor(constructorSymbol)
val instance = constructorMirror()
val fieldSymbol = classType.decl(TermName("func")).asTerm
val instanceMirror = runtimeMirror.reflect(instance)
val fieldMirror = instanceMirror.reflectField(fieldSymbol)
fieldMirror.get.asInstanceOf[SomeFunction]
}
Testing:
foo("Foo") //SomeFunction(Foo#func)
foo("Bar") //SomeFunction(Bar#func)
foo("Baz") //SomeFunction(Baz#func)
I'm looking to create a way to dynamically call logic depending on template id within scala. So template id 1 calls logic a, template id 2 call logic b, etc. The logic will be diverse but will have the same inputs/outputs. Also the number of different template ids will get into the thousands and will not be known ahead of time, so a loose coupling feels the way to go.
I've started looking at reflection to do this using scala 2.11.1 and can statically use reflection when I know the logic to be used ahead of time but have not found the correct way to dynamically use reflection, so for example passing in template id 2 will call logic b.
Below is a cut down example showing how the static version works and the skeleton I have so far for the dynamic version.
package thePackage
import scala.reflect.runtime.{universe => ru}
trait theTrait { def theMethod(x: String): Unit }
// the different logic held in different objects
object object1 extends theTrait {
def theMethod(x: String) = { println("a " + x ) }
}
object object2 extends theTrait {
def theMethod(x: String) = { println("b " + x ) }
}
object object3 extends theTrait {
def theMethod(x: String) = { println("c " + x ) }
}
// run static/dynamic reflection methods
object ReflectionTest {
// "static" invocation calling object1.theMethod
def staticInvocation() = {
val m = ru.runtimeMirror(getClass.getClassLoader)
val im = m.reflect(thePackage.object1)
val method = ru.typeOf[thePackage.object1.type]
.decl(ru.TermName("theMethod")).asMethod
val methodRun = im.reflectMethod(method)
methodRun("test")
}
staticInvocation
// "dynamic" invocation using integer to call different methods
def dynamicInvocation( y: Integer) = {
val m = ru.runtimeMirror(getClass.getClassLoader)
val module = m.staticModule("thePackage.object" + y)
val im = m.reflectModule(module)
// stuck... static approach does not work here
}
dynamicInvocation(1)
dynamicInvocation(2)
dynamicInvocation(3)
}
What needs to be added/changed to the dynamicInvocation method to make this work, or should I be using a different approach?
You need to get an instance mirror for your module, on which you can reflect the method.
def dynamicInvocation( y: Integer) = {
val m = ru.runtimeMirror(getClass.getClassLoader)
val module = m.staticModule("thePackage.object" + y)
val im = m.reflectModule(module)
val method = im.symbol.info.decl(ru.TermName("theMethod")).asMethod
val objMirror = m.reflect(im.instance)
objMirror.reflectMethod(method)("test")
}
It seems that TermName method in above solution has been replaced by newTermName and also the info.decl seems to not work. Below line worked for me
val method = im.symbol.typeSignature.member(ru.newTermName("testMethod")).asMethod
EDIT2:
So another heads up on this:
I still have no idea why this happens, but I have now a similar problem with jOOQ and the Dialect I have to it. My code here looks like this:
object MyDB {
private lazy val dialect = SQLDialect.POSTGRES
def withSession[T](f: DSLContext => T) = f(DSL.using(getConnectionPool, dialect))
}
if I remove the "lazy" it blows up when I try to execute jOOQ queries in line 552 of https://github.com/jOOQ/jOOQ/blob/version-3.2.0/jOOQ/src/main/java/org/jooq/impl/DefaultRenderContext.java
That happens to be a line where the dialect is evaluated. After I added the lazy everything works as expected.
Maybe this is an issue with the threading of LiftWeb and the executing thread does not see the correct value of the val? I have no idea...
EDIT:
I have found a way to do what I want simply by adding a lazy to the values in the first, broken version. So with lazy vals it all works well.
However I'll let this stay open, as I have no idea how to explain this behavior.
Original Post:
So I am trying to use Parameterized Queries in Slick.
My code is below, my problem is that I get an NPE (see comments) when I try to run this from within the webapplication (liftweb, container started with sbt) (the application creates an object of the class PlayerListCollector that is given the string "cola")
When I execute the object as App from within Eclipse the println at the bottom works just fine.
class PlayerListCollector(term: String) {
import PlayerListCollector._
val searchResult = executeSearch(term)
}
object PlayerListCollector extends Loggable with App{
private val searchNameCurrent = Parameters[String].flatMap {
case (term) => {
for {
p <- Players if p.uberName.isNotNull
n <- p.displayName if (n.displayName.toLowerCase.like(term))
} yield (p.id, n.displayName)
}
}
private def executeSearch(term: String) = {
val lowerTerm = "%"+term.toLowerCase()+"%"
logger info "HELLO " +lowerTerm // prints HELLO %cola%
val foo = searchNameCurrent(lowerTerm) // NPE right in this line
logger info foo // never executed from here on ...
val byCurrent = foo.list
logger info byCurrent
[...]
}
// this works if run directly from within eclipse!
println(DB withSession {
searchNameCurrent("%cola%").list
})
}
The problem vanishes when I change the code to look like this:
[...]
object PlayerListCollector extends Loggable with App{
private def executeSearch(term: String) = {
val searchNameCurrent = Parameters[String].flatMap {
case (term) => {
for {
p <- Players if p.uberName.isNotNull
n <- p.displayName if (n.displayName.toLowerCase.like(term))
} yield (p.id, n.displayName)
}
}
val lowerTerm = "%"+term.toLowerCase()+"%"
logger info "HELLO " +lowerTerm // prints HELLO %cola%
val foo = searchNameCurrent(lowerTerm) // executes just fine when the query is in a local val
logger info foo
val byCurrent = foo.list
logger info byCurrent // prints expected output
[...]
}
[...]
}
I have no idea whatsoever why this happens.
Isn't the whole point of a paramterized query to put it in a val that is only once filled with a value so it does not need to be compiled multiple times?
So it turns out I used the App-Trait (http://www.scala-lang.org/api/current/index.html#scala.App) on these objects.
Reading the big fat caveat tells us what is happening I guess.
I'm working through ScalaInAction (book is still a MEAP, but code is public on github)
Right now I'm in chapter 2 looking at this restClient: : https://github.com/nraychaudhuri/scalainaction/blob/master/chap02/RestClient.scala
First, I setup intelliJ with scala extensions and created a HelloWorld with main():
<ALL the imports>
object HelloWorld {
def main(args: Array[String]) {
<ALL the rest code from RestClient.scala>
}
}
I get the following error when compiling:
scala: forward reference extends over defintion of value command
val httppost = new HttpPost(url)
^
I can fix this by moving the following lines around until the ordering is correct with relation to the def's
require( args.size >= 2, "You need at least two arguments to make a get, post, or delete request")
val command = args.head
val params = parseArgs(args)
val url = args.last
command match {
case "post" => handlePostRequest
case "get" => handleGetRequest
case "delete" => handleDeleteRequest
case "options" => handleOptionsRequest
}
While browsing the github page, I found this: https://github.com/nraychaudhuri/scalainaction/tree/master/chap02/restclient
Which uses implements RestClient.scala using extends App instead of a main() method:
<All the imports>
object RestClient extends App {
<All the rest of the code from RestClient.scala>
}
I then changed my object HelloWorld to just use extends App instead of implementing a main() method and it works without errors
Why does the main() method way of doing this generate the error but the extends App does not?
Because main() is a method, and variable's in method could not be forward reference.
For example:
object Test {
// x, y is object's instance variable, it could be forward referenced
def sum = x + y // This is ok
val y = 10
val x = 10
}
But code in a method could not forward referenced.
object Test {
def sum = {
val t = x + y // This is not ok, you don't have x, y at this point
val x = 10
val y = 20
val z = x + y // This is ok
}
}
In your case, if you copy paste all codes from RestClient.scala to main(), you will have the same issue because var url is declared after its usage in handlePostRequest.
I have an external process that I would like to treat as a
function from String=>String. Given a line of input, it will respond with a single line of output. It seems that I should use
scala.sys.process, which is clearly an elegant library that makes many
shell operations easily accessible from within scala. However, I
can't figure out how to perform this simple use case.
If I write a single line to the process' stdin, it prints the result
in a single line. How can I use sys.process to create a wrapper so I
can use the process interactively? For example, if I had an
implementation for ProcessWrapper, here is a program and it's output:
// abstract definition
class ProcessWrapper(executable: String) {
def apply(line: String): String
}
// program using an implementation
val process = new ProcessWrapper("cat -b")
println(process("foo"))
println(process("bar"))
println(process("baz"))
Output:
1 foo
2 bar
3 baz
It is important that the process is not reloaded for each call to process because there is a significant initialization step.
So - after my comment - this would be my solution
import java.io.BufferedReader
import java.io.File
import java.io.InputStream
import java.io.InputStreamReader
import scala.annotation.tailrec
class ProcessWrapper(cmdLine: String, lineListenerOut: String => Unit, lineListenerErr: String => Unit,
finishHandler: => Unit,
lineMode: Boolean = true, envp: Array[String] = null, dir: File = null) {
class StreamRunnable(val stream: InputStream, listener: String => Unit) extends Runnable {
def run() {
try {
val in = new BufferedReader(new InputStreamReader(this.stream));
#tailrec
def readLines {
val line = in.readLine
if (line != null) {
listener(line)
readLines
}
}
readLines
}
finally {
this.stream.close
finishHandler
}
}
}
val process = Runtime.getRuntime().exec(cmdLine, envp, dir);
val outThread = new Thread(new StreamRunnable(process.getInputStream, lineListenerOut), "StreamHandlerOut")
val errThread = new Thread(new StreamRunnable(process.getErrorStream, lineListenerErr), "StreamHandlerErr")
val sendToProcess = process.getOutputStream
outThread.start
errThread.start
def apply(txt: String) {
sendToProcess.write(txt.getBytes)
if (lineMode)
sendToProcess.write('\n')
sendToProcess.flush
}
}
object ProcessWrapper {
def main(args: Array[String]) {
val process = new ProcessWrapper("python -i", txt => println("py> " + txt),
err => System.err.println("py err> " + err), System.exit(0))
while (true) {
process(readLine)
}
}
}
The main part is the StreamRunnable, where the process is read in a thread and the received line is passed on to a "LineListener" (a simple String => Unit - function).
The main is just a sample implementation - calling python ;)
I'm not sure, but you want somethings like that ?
case class ProcessWrapper(executable: String) {
import java.io.ByteArrayOutputStream
import scala.concurrent.duration.Duration
import java.util.concurrent.TimeUnit
lazy val process = sys.runtime.exec(executable)
def apply(line: String, blockedRead: Boolean = true): String = {
process.getOutputStream().write(line.getBytes())
process.getOutputStream().flush()
val r = new ByteArrayOutputStream
if (blockedRead) {
r.write(process.getInputStream().read())
}
while (process.getInputStream().available() > 0) {
r.write(process.getInputStream().read())
}
r.toString()
}
def close() = process.destroy()
}
val process = ProcessWrapper("cat -b")
println(process("foo\n"))
println(process("bar\n"))
println(process("baz\n"))
println(process("buz\n"))
println(process("puz\n"))
process.close
Result :
1 foo
2 bar
3 baz
4 buz
5 puz
I think that PlayCLI is a better way.
http://blog.greweb.fr/2013/01/playcli-play-iteratees-unix-pipe/ came across this today and looks exactly like what you want
How about using an Akka actor. The actor can have state and thus a reference to an open program (in a thread). You can send messages to that actor.
ProcessWrapper might be a typed actor itself or just something that converts the calls of a function to a call of an actor. If you only have 'process' as method name, then wrapper ! "message" would be enough.
Having a program open and ready to receive commands sounds like an actor that receives messages.
Edit: Probably I got the requirements wrong. You want to send multiple lines to the same process. That's not possible with the below solution.
One possibility would be to add an extension method to the ProcessBuilder that allows for taking the input from a string:
implicit class ProcessBuilderWithStringInput(val builder: ProcessBuilder) extends AnyVal {
// TODO: could use an implicit for the character set
def #<<(s: String) = builder.#<(new ByteArrayInputStream(s.getBytes))
}
You can now use the method like this:
scala> ("bc":ProcessBuilder).#<<("3 + 4\n").!!
res9: String =
"7
"
Note that the type annotation is necessary, because we need two conversions (String -> ProcessBuilder -> ProcessBuilderWithStringInput, and Scala will only apply one conversion automatically.