I'm trying to use Slick 2.1 with MS SQL Server in my Play 2.3 application. I'm stuck with trying to make Slick's code generator to work in Build.scala. This is because MS SQL Server JDBC driver is not on maven and I'm just manually putting the driver jar file in the /lib folder. At the moment I'm getting this error:
[error] (run-main-0) java.lang.ClassNotFoundException: com.microsoft.jdbc.sqlserver.SQLServerDriver
This is how the part where code generator is looks like:
lazy val slickCodeGenTask = (sourceManaged, dependencyClasspath in Compile, runner in Compile, streams) map { (dir, cp, r, s) =>
val outputDir = (dir / "slick").getPath
val url = "jdbc:sqlserver://localhost:1433;databaseName=mydb"
val jdbcDriver = "com.microsoft.jdbc.sqlserver.SQLServerDriver"
val slickDriver = "com.typesafe.slick.driver.ms.SQLServerDriver"
val pkg = "db"
toError(r.run("scala.slick.codegen.SourceCodeGenerator", cp.files, Array(slickDriver, jdbcDriver, url, outputDir, pkg, user, pswd), s.log))
val fname = outputDir + "/db/Tables.scala"
Seq(file(fname))
}
How do I make MS SQL Server JDBC driver play along with Slick code generator?
SBT in Play 2.3 doesn't seem to automatically add contents of lib folder to classpath, so I've stumbled on a solution after running idea command for my Intellij IDEA which among other things adds jars from lib to classpath.
Also, after lots of digging around the final solution to this particular problem was to use open source jtds driver for slick code generator and ms jdbc driver for the rest of the application. This is what generator code ended up looking like:
lazy val slickCodeGenTask = (sourceManaged, dependencyClasspath in Compile, runner in Compile, streams) map { (dir, cp, r, s) =>
val outputDir = "app"
val url = "jdbc:jtds:sqlserver://localhost:1433;databaseName=mydb"
val jdbcDriver = "net.sourceforge.jtds.jdbc.Driver"
val slickDriver = "com.typesafe.slick.driver.ms.SQLServerDriver"
val pkg = "db"
toError(r.run("scala.slick.codegen.SourceCodeGenerator", cp.files, Array(slickDriver, jdbcDriver, url, outputDir, pkg, user, pswd), s.log))
val fname = "Tables.scala"
Seq(file(fname))
}
On a tangential note, for sbt 0.13.x, it is recommended to change scala.slick.codegen.SourceCodeGenerator -> slick.codegen.SourceCodeGenerator
Related
I have a map reduce .scala file like this:
import org.apache.spark._
object WordCount {
def main(args: Array[String]){
val inputDir = args(0)
//val inputDir = "/Users/eksi/Desktop/sherlock.txt"
val outputDir = args(1)
//val outputDir = "/Users/eksi/Desktop/out.txt"
val cnf = new SparkConf().setAppName("Example MapReduce Spark Job")
val sc = new SparkContext(cnf)
val textFile = sc.textFile(inputDir)
val counts = textFile.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile(outputDir)
sc.stop()
}
}
When I run my code, with setMaster("local[1]") parameters it works fine.
I want to put this code in a .jar and throw it to S3 to work with AWS EMR. Therefore, I use the following build.sbt to do so.
name := "word-count"
version := "0.0.1"
scalaVersion := "2.11.7"
// additional libraries
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % "1.0.2"
)
It generates a jar file, however none of my scala code is in there. What I see is just a manifest file when I extract the .jar
When I run sbt package this is what I get:
[myMacBook-Pro] > sbt package
[info] Loading project definition from /Users/lele/bigdata/wordcount/project
[info] Set current project to word-count (in build file:/Users/lele/bigdata/wordcount/)
[info] Packaging /Users/lele/bigdata/wordcount/target/scala-2.11/word-count_2.11-0.0.1.jar ...
[info] Done packaging.
[success] Total time: 0 s, completed Jul 27, 2016 10:33:26 PM
What should I do to create a proper jar file that works like
WordCount.jar WordCount
Ref: It generates a jar file, however none of my scala code is in there. What I see is just a manifest file when I extract the .jar
Make sure your WordCount.scala is in the root or in src/main/scala
From http://www.scala-sbt.org/1.0/docs/Directories.html
Source code can be placed in the project’s base directory as with hello/hw.scala. However, most people don’t do this for real projects; too much clutter.
sbt uses the same directory structure as Maven for source files by default (all paths are relative to the base directory):
I have added this to build.sbt
libraryDependencies += "com.typesafe.slick" %% "slick-codegen" % "2.1.0"
lazy val slickGenerate = TaskKey[Seq[File]]("slick code generation")
slickGenerate <<= slickGenerateTask
lazy val slickGenerateTask = {
(sourceManaged in Compile, dependencyClasspath in Compile, runner in Compile, streams) map { (dir, cp, r, s) =>
val dbName = "dbname"
val userName = "user"
val password = "password"
val url = s"jdbc:mysql://server:port/$dbName"
val jdbcDriver = "com.mysql.jdbc.Driver"
val slickDriver = "scala.slick.driver.MySQLDriver"
val targetPackageName = "models"
val outputDir = (dir / dbName).getPath // place generated files in sbt's managed sources folder
val fname = outputDir + s"/$targetPackageName/Tables.scala"
println(s"\nauto-generating slick source for database schema at $url...")
println(s"output source file file: file://$fname\n")
r.run("scala.slick.codegen.SourceCodeGenerator", cp.files, Array(slickDriver, jdbcDriver, url, outputDir, targetPackageName, userName, password), s.log)
Seq(file(fname))
}
}
The task's code itself isn't very exciting. It just needs to create an auto-generated scala source file. Problem is, sbt starts fine, yet this new task is evidently not recognized by sbt and cannot be run in the sbt prompt. I have also had very little luck with the := syntax for task definition. Existing documentation has been just confounding.
How can this new task be made available in the sbt prompt?
This works
libraryDependencies += "com.typesafe.slick" %% "slick-codegen" % "2.1.0"
lazy val slickGenerate = taskKey[Seq[File]]("slick code generation")
slickGenerate := {
val dbName = "dbname"
val userName = "user"
val password = "password"
val url = s"jdbc:mysql://server:port/$dbName"
val jdbcDriver = "com.mysql.jdbc.Driver"
val slickDriver = "scala.slick.driver.MySQLDriver"
val targetPackageName = "models"
val outputDir = ((sourceManaged in Compile).value / dbName).getPath // place generated files in sbt's managed sources folder
val fname = outputDir + s"/$targetPackageName/Tables.scala"
println(s"\nauto-generating slick source for database schema at $url...")
println(s"output source file file: file://$fname\n")
(runner in Compile).value.run("scala.slick.codegen.SourceCodeGenerator", (dependencyClasspath in Compile).value.files, Array(slickDriver, jdbcDriver, url, outputDir, targetPackageName, userName, password), streams.value.log)
Seq(file(fname))
}
In sbt 0.13.x you don't need all those blabla map sameblabla boilerplates. Just access value as is (runner in Compile).value - macro will do everything else for you.
> slickGenerate
[info] Updating {file:/Users/user/slick/}slick...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
auto-generating slick source for database schema at jdbc:mysql://server:port/dbname...
output source file file: file:///Users/user/slick/target/scala-2.10/src_managed/main/dbname/models/Tables.scala
> help slickGenerate
slick code generation
Talking about <<= - your TaskKey is incorrect, see the definition:
def apply[T](label : scala.Predef.String, description : scala.Predef.String // not description
So, the old definition <<= uses "generate slick code" as label, while the new := uses the code-given name for command (new style), so it uses your "generate slick code" as a doc. Which looks strange and inconsistent, but that's a fact and it's partially reasoned by backward-compatibility.
So, correct old-style version is:
import sbt.Keys._
lazy val slickGenerate = TaskKey[Seq[File]]("slick-generate", "generate slick code")
slickGenerate <<= slickGenerateTask
def slickGenerateTask =
(sourceManaged in Compile, dependencyClasspath in Compile, runner in Compile, streams) map { (dir, cp, r, s) =>
...
}
It works in the same way as previous. Note, that you have to use "slickGenerate", not "slick-generate", the last one doesn't work for "help" command.
By the way, you're using Bare build definition now - you may want to switch to Multi-project .sbt definition as it's recommended by sbt docs, see also.
An overview of my environments:
Mac OS Yosemite, Play framework 2.3.7, sbt 0.13.7, Intellij Idea 14, java 1.8.0_25
I tried to run a simple Spark program in Play framework, so I just create a Play 2 project in Intellij, and change some files as follows:
app/Controllers/Application.scala:
package controllers
import play.api._
import play.api.libs.iteratee.Enumerator
import play.api.mvc._
object Application extends Controller {
def index = Action {
Ok(views.html.index("Your new application is ready."))
}
def trySpark = Action {
Ok.chunked(Enumerator(utils.TrySpark.runSpark))
}
}
app/utils/TrySpark.scala:
package utils
import org.apache.spark.{SparkContext, SparkConf}
object TrySpark {
def runSpark: String = {
val conf = new SparkConf().setAppName("trySpark").setMaster("local[4]")
val sc = new SparkContext(conf)
val data = sc.textFile("public/data/array.txt")
val array = data.map ( line => line.split(' ').map(_.toDouble) )
val sum = array.first().reduce( (a, b) => a + b )
return sum.toString
}
}
public/data/array.txt:
1 2 3 4 5 6 7
conf/routes:
GET / controllers.Application.index
GET /spark controllers.Application.trySpark
GET /assets/*file controllers.Assets.at(path="/public", file)
build.sbt:
name := "trySpark"
version := "1.0"
lazy val `tryspark` = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.10.4"
libraryDependencies ++= Seq( jdbc , anorm , cache , ws,
"org.apache.spark" % "spark-core_2.10" % "1.2.0")
unmanagedResourceDirectories in Test <+= baseDirectory ( _ /"target/web/public/test" )
I type activator run to run this app in development mode then type localhost:9000/spark in the browser, it shows result 28 as expected. However, when I want type activator start to run this app in production mode it shows the following error message:
[info] play - Application started (Prod)
[info] play - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
[error] application -
! #6kik15fee - Internal server error, for (GET) [/spark] ->
play.api.Application$$anon$1: Execution exception[[InvalidInputException: Input path does not exist: file:/Path/to/my/project/target/universal/stage/public/data/array.txt]]
at play.api.Application$class.handleError(Application.scala:296) ~[com.typesafe.play.play_2.10-2.3.7.jar:2.3.7]
at play.api.DefaultApplication.handleError(Application.scala:402) [com.typesafe.play.play_2.10-2.3.7.jar:2.3.7]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$14$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:205) [com.typesafe.play.play_2.10-2.3.7.jar:2.3.7]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$14$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:202) [com.typesafe.play.play_2.10-2.3.7.jar:2.3.7]
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) [org.scala-lang.scala-library-2.10.4.jar:na]
Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/Path/to/my/project/target/universal/stage/public/data/array.txt
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.2.0.jar:na]
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270) ~[org.apache.hadoop.hadoop-mapreduce-client-core-2.2.0.jar:na]
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201) ~[org.apache.spark.spark-core_2.10-1.2.0.jar:1.2.0]
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) ~[org.apache.spark.spark-core_2.10-1.2.0.jar:1.2.0]
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) ~[org.apache.spark.spark-core_2.10-1.2.0.jar:1.2.0]
It seems that my array.txt file is not loaded in the production mode. How can solve this problem?
The problem here is that the public directory will not be available in your root project dir when you run in production. It is packaged as a jar (usually in STAGE_DIR/lib/PROJ_NAME-VERSION-assets.jar) so you will not be able to access them this way.
I can see two solutions here:
1) Place the file in the conf directory. This will work, but seems very dirty especially if you intend to use more data files;
2) Place those files in some directory and tell sbt to package it as well. You can keep using the public directory although it seems better to use a different dir especially if you would want to have many more files.
Supposing array.txt is placed in a dir named datafiles in your project root, you can add this to build.sbt:
mappings in Universal ++=
(baseDirectory.value / "datafiles" * "*" get) map
(x => x -> ("datafiles/" + x.getName))
Don't forget to change the paths in your app code:
// (...)
val data = sc.textFile("datafiles/array.txt")
Then just do a clean and when you run either start, stage or dist those files will be available.
Does anyone knows how to configure a SBT project to run an annotation processor (APT)? I'm doing some lab on a web project, using some Java tools like QueryDSL, and i need to generate querydsl classes for my JPA model classes, in a similar way QueryDSL Maven plugin does.
Thanks in advance.
You could manually run the annotation processor (see command below) or implement an SBT task similar to the following:
lazy val processAnnotations = taskKey[Unit]("Process annotations")
processAnnotations := {
val log = streams.value.log
log.info("Processing annotations ...")
val classpath = ((products in Compile).value ++ ((dependencyClasspath in Compile).value.files)) mkString ":"
val destinationDirectory = (classDirectory in Compile).value
val processor = "com.package.PluginProcessor"
val classesToProcess = Seq("com.package.Class1", "com.package.Class2") mkString " "
val command = s"javac -cp $classpath -proc:only -processor $processor -XprintRounds -d $destinationDirectory $classesToProcess"
failIfNonZeroExitStatus(command, "Failed to process annotations.", log)
log.info("Done processing annotations.")
}
def failIfNonZeroExitStatus(command: String, message: => String, log: Logger) {
val result = command !
if (result != 0) {
log.error(message)
sys.error("Failed running command: " + command)
}
}
packageBin in Compile <<= (packageBin in Compile) dependsOn (processAnnotations in Compile)
Update destinationDirectory, processor, and classesToProcess as necessary.
You might also change the "-d" flag to "-s" depending on the type of annotation processor you have (see options for javac).
I have an existing database, and i would like to connect to it with scala/slick.
I'd rather not have to manually write all of the slick classes, to wrap around my tables.
is there a quick way to just read the definitions from the database, from slick? or, possibly, is there another component in the scala standard library or standard toolset, which will do this work for me?
Use the Slick schema generator, you simply need to add this to your Build.scala:
lazy val slick = TaskKey[Seq[File]]("gen-tables")
lazy val slickCodeGenTask = (sourceManaged, dependencyClasspath in Compile, runner in Compile, streams) map {
(dir, cp, r, s) => {
val outputDir = (dir / "slick").getPath
val url = "your db url"
val jdbcDriver = "dbms drivers"
val slickDriver = "slick drivers"
val pkg = "schema"
toError(r.run("scala.slick.model.codegen.SourceCodeGenerator", cp.files, Array(slickDriver, jdbcDriver, url, outputDir, pkg), s.log))
val fname = outputDir + "/path/to/Tables.scala"
Seq(file(fname))
}
}
Add the task to the settings:
val main = play.Project(appName, appVersion, Seq()).settings(
Keys.fork in (Test) := false,
libraryDependencies := Seq(
...
),
slick <<= slickCodeGenTask // register manual sbt command
)
And then call genTables form SBT, this will create a scala file called Tables.scala to the specified path with the whole schema from the database.
This was the Github example I looked up the first time.