I have file in with an SQL query, which I need to run using SparkSQL. Since the file is too big, I don't want to copy it straight into the code.
.
├──_main.scala
├──_resources
│ └── query.sql
Howewer when I compile it
scalac -cp "<path-to-resources>/*:<jars path>/*" -d "main.jar" main.scala
and try to run, it throws NullPointerException when I try to call "mkString"
val queryIt = scala.io.Source.fromInputStream(getClass.getClassLoader.getResourceAsStream("query.sql"))
val query = try queryIt.mkString finally queryIt.close()
Error message
Related
I have a structure like this
src
└── main
├── resources
│ └── inserts.sql
└── my.package
└── Main.scala
In Main.scala I want to take the file inserts.sql and use Slick 3.0.0 to execute it on my db.
A SQL string can be executed directly by using the SQLActionBuilder class.
Also, since the BufferedSource object we get from Source.fromResource is Closable, we should wrap it with a Using block.
import slick.jdbc.SetParameter.SetUnit
import slick.jdbc.SQLActionBuilder
import scala.io.Source
import scala.util.Using
// ...
Using(Source.fromResource("inserts.sql")) { insertsSqlSource =>
val sqlActionBuilder = SQLActionBuilder(insertsSqlSource.mkString, SetUnit)
database.run(sqlActionBuilder.asUpdate)
}
You can read file content:
val query = scala.io.Source.fromResource("inserts.sql").mkString
and then create query using sql or sqlu interpolators:
//https://scala-slick.org/doc/3.0.0/sql.html
sql"$query".as[ExpectedType]
and run it as always :)
PS: Not tested. Don't have prepared env now.
Looks like there is no way to execute a sql file with Slick other then loading it into memory as a String and executing it with sql, sqlu or tsql.
Beware that in this case the $ interpolation is meant to insert bind variables into the query. To splice literal values into the query you must use #$ instead. Since in this case the variable is the whole query, we have to do
val inserts_sql = Source.fromResource("inserts.sql").mkString
db.run(sqlu"#$query")
I have an Ammonite Script that I want to deliver in a JAR.
In another project I want to use this Script - but so far with no success.
I tried according to the documentation (sol_local_build.sc):
import $ivy.`mycompany:myproject_2.12:2.1.0-SNAPSHOT`, local_build
#main
def doit():Unit =
println(local_build.curl("http://localhost:8080"))
local_build.sc is in the Script I want to use.
This is the exception I get:
sol_local_build.sc:2: '.' expected but eof found.
^
The script must be compiled on the fly.
Put your script in a standard sbt project
inside a directory, example directory name: "test1"
Put your external script (example name: "script.sc")
// script.sc
println("Hello world!")
into the resource directory ("test1\src\main\resources\script.sc") of the test1 project
Publish the projekt local, i.e. sbt publishLocal
It is published to ".ivy2\local\default\test1_2.12\0.1-SNAPSHOT\ ... " directory.
Now you can use the following ammonite script "test.sc".
It reads the "script.sc" from the jar in the local ivy repository
and writes it to the local directory (must have read/write access) and then executes an external process,
which calls the scala "interpreter" and executes the script.
// test.sc
import $ivy.`default:test1_2.12:0.1-SNAPSHOT`
val scriptCode = scala.util.Try {scala.io.Source.fromResource("script.sc").mkString} getOrElse """Println("Script-file not found!")"""
println("*" * 30)
println(scriptCode)
println("*" * 30)
println()
java.nio.file.Files.write(java.nio.file.Paths.get("script.sc"), scriptCode.getBytes(java.nio.charset.StandardCharsets.UTF_8))
val cmd = Seq("cmd.exe", "/c", "scala", "script.sc")
val output = sys.process.Process(cmd).!!
println(output)
Executing the script the Ammonite REPL, you get:
******************************
// script.sc
println("Hello world!")
******************************
Hello world!
The script has no error handling and leaves the file in the running directory.
You can speed up the execution with the "-savecompiled" compiler switch, i.e
val cmd = Seq("cmd.exe", "/c", "scala", "-savecompiled", "script.sc")
An additional .jar file is created then in the running directory.
Scala Scripts are not really interpreted, but are compiled "under the hood"
as every normal Scala programm.
Therefor all code must be reachable during compile time
and you cannot call a function inside the other script from the jar-file!
But Ammonite has a buid in multi-stage feature.
It compiles one part, executes it and then compiles the next part!
Little improved ammonite-script.
It's not error free but runs.
Maybe there is better way to get the script out of the jar.
You should ask Li Haoyi!
// test_ammo.sc
// using ammonite ops
// in subdirectoy /test1
// Ammonite REPL:
// import $exec.test1.test_ammo
// # Ammonite-multi-stage
import $ivy.`default::test1:0.1-SNAPSHOT`
//import scala.util.Properties
import scala.sys.process.Process
val scriptFileName = "script.sc"
write.over(pwd/"test1"/scriptFileName, read(resource(getClass.getClassLoader)/scriptFileName))
val cmd = Seq("cmd.exe", "/c", "scala", scriptFileName)
val output = Process(cmd).!!
println(output)
#
import $exec.script // no .sc suffix
ppp() // is a function inside script.sc
script.sc inside resources folder of project
published local with "sbt publishLocal":
// script.sc
println("Hello world!")
def ppp() = println("Hello world from ppp!")
For completeness, I could solve my problem as follows:
Just create a Scala File in this project.
Copy the Script content in
an Object.
package mycompany.myproject
object LocalBuild {
def curl(..)...
}
Add the dependencies to your sbt file (e.g. ammonite.ops)
Use it like:
$ivy.`mycompany:myproject_2.12:2.1.0-SNAPSHOT`, mycompany.myproject.LocalBuild
#main
def doit():Unit =
println(LocalBuild.curl("http://localhost:8080"))
I have a subproject in my build.sbt with a rather long setting for initialCommands, comprising a list of imports and some definitions. I'd like to test this as part of regular CI, because otherwise I won't notice breaking changes after refactoring code. It is not clear to me how to do so.
Just running sbt console doesn't seem to cut it, because there is always a "successful" exit code even when code doesn't compile.
Moving the code out into an object defined in a special source file won't help because I need the list of imports to be present (and I don't want to cakeify my whole code base).
Moving the code out into a source file and then loading that with :load also always gives a successful exit code.
I found out about scala -e but that does strange things on my machine (see the error log below).
This is Scala 2.12.
$ scala -e '1'
cat: /release: No such file or directory
Exception in thread "main" java.net.UnknownHostException: <my-host-name-here>: <my-host-name-here>: Name or service not known
You could generate a file and run it like any other test file:
(sourceGenerators in Test) += Def.task {
val contents = """object TestRepl {
{{}}
}""".replace("{{}}", (initialCommands in console).value)
val file = (sourceManaged in Test).value / "repltest.scala"
IO.write(file, contents)
Seq(file)
}.taskValue
I have a Spark Streaming application built with Maven (as jar) and deployed with the spark-submit script. The application project layout follows the standard directory layout:
myApp
src
main
scala
com.mycompany.package
MyApp.scala
DoSomething.scala
...
resources
aPerlScript.pl
...
test
scala
com.mycompany.package
MyAppTest.scala
...
target
...
pom.xml
In the DoSomething.scala object I have a method (let's call it doSomething()) that tries to execute a Perl script -- aPerlScript.pl (from the resources folder) -- using scala.sys.process.Process and passing two arguments to the script (the first one is the absolute path to a binary file used as input, the second one is the path/name of the produced output file). I call then DoSomething.doSomething().
The issue is that I was not able to access the script, not with absolute paths, relative paths, getClass.getClassLoader.getResource, getClass.getResource, I have specified the resources folder in my pom.xml. None of my attempts succeeded. I don't know how to find the stuff I put in src/main/resources.
I will appreciate any help.
SIDE NOTES:
I use an external Process instead of a Spark pipe because, at this step of my workflow, I must handle binary files as input and output.
I'm using Spark-streaming 1.1.0, Scala 2.10.4 and Java 7. I build the jar with "Maven install" from within Eclipse (Kepler)
When I use the getClass.getClassLoader.getResource "standard" method to access resources I find that the actual classpath is the spark-submit script's one.
There are a few solutions. The simplest is to use Scala's process infrastructure:
import scala.sys.process._
object RunScript {
val arg = "some argument"
val stream = RunScript.getClass.getClassLoader.getResourceAsStream("aPerlScript.pl")
val ret: Int = (s"/usr/bin/perl - $arg" #< stream).!
}
In this case, ret is the return code for the process and any output from the process is directed to stdout.
A second (longer) solution is to copy the file aPerlScript.pl from the jar file to some temporary location and execute it from there. This code snippet should have most of what you need.
object RunScript {
// Set up copy destination from the Java temporary directory. This is /tmp on Linux
val destDir = System.getProperty("java.io.tmpdir") + "/"
// Get a stream to the script in the resources dir
val source = Channels.newChannel(RunScript.getClass.getClassLoader.getResourceAsStream("aPerlScript.pl"))
val fileOut = new File(destDir, "aPerlScript.pl")
val dest = new FileOutputStream(fileOut)
// Copy file to temporary directory
dest.getChannel.transferFrom(source, 0, Long.MaxValue)
source.close()
dest.close()
}
// Schedule the file for deletion for when the JVM quits
sys.addShutdownHook {
new File(destDir, "aPerlScript.pl").delete
}
// Now you can execute the script.
This approach allows you to bundle native libraries in JAR files. Copying them out allows the libraries to be loaded at runtime for whatever JNI mischief you have planned.
I have a subproject named oppenheimer in my project. It's very simple to run this project from the sbt console.
[myproject] $ oppenheimer/run
I can also pass in a command line argument as such:
[myproject] $ oppenheimer/run migrate
[myproject] $ oppenheimer/run clean
How can I do this from build.sbt? Is it possible to define a task that does this? It would suffice to have something like this:
val customMigrate = ...
val customClean = ...
And this is so that I could use it elsewhere in the project, like such:
(test in Test) <<= (test in Test).dependsOn(customMigrate)
The answer is given in the sbt FAQ section "How can I create a custom run task, in addition to run?". Basically:
lazy val customMigrate = taskKey[Unit]("custom run task")
fullRunTask(customMigrate, Test, "foo.Main", "migrate")