Using sbt, I'd like to copy some dependency jars to a lib output folder. If possible, I'd like to use the %provided% keyword, like I can with sbt-assembly.
So given a build.sbt somewhat similar to the following, how do create a task that copies the ark-tweet-nlp but NOt the spark-core dependencies to target/scala-%ver%/lib ?
retrieveManaged := true simply copies everything, which is not what I want.
...
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.0.0" % "provided"
libraryDependencies += "edu.cmu.cs" % "ark-tweet-nlp" % "0.3.2"
retrieveManaged := true
...
You could write a task like this.
build.sbt
val retrieveNotProvided = taskKey[Unit]("Copies non provided and non internal dependencies")
def isInternalOrProvided(conf: String) = conf.contains("-internal") || conf == "provided"
retrieveNotProvided := {
val toCopy = new collection.mutable.HashSet[(File, File)]
val pattern = retrievePattern.value
val output = managedDirectory.value
update.value.retrieve { (conf, mid, art, cached) =>
import org.apache.ivy.core.IvyPatternHelper
val fileName = IvyPatternHelper.substitute(
pattern, mid.organization, mid.name, mid.revision, art.name, art.`type`, art.extension, conf
)
if (!isInternalOrProvided(conf)) toCopy += (cached -> output / fileName)
cached
}
IO.copy(toCopy)
}
You'll have to remove retrieveManaged := true from your build.sbt, because otherwise sbt will trigger the original retrieve function.
Related
We have a build.sbt file like this, which is working fine:
name := "Foo"
version := "0.1"
scalaVersion := "2.12.8"
def aws(module: String): ModuleID = "com.amazonaws" % module % "1.11.250"
lazy val Core = project
.settings(
libraryDependencies ++= Seq(
aws("aws-java-sdk-s3"),
aws("aws-java-sdk-dynamodb"),
)
)
Basically, the project has a few AWS SDK library dependencies and we want to avoid typing the groupID (e.g. "com.amazonaws") and the revision (e.g. "1.11.250") multiple times and that's why we have this line:
def aws(module: String): ModuleID = "com.amazonaws" % module % "1.11.250"
However, since we have many repos like this and we want to move this definition to a custom sbt-plugin. To begin with, we try this:
name := "Foo"
version := "0.1"
scalaVersion := "2.12.8"
val awsVersion = settingKey[String]("The version of aws SDK used for building.") // line 5
def aws(module: String): ModuleID = "com.amazonaws" % module % awsVersion.value // line 6
awsVersion := "1.11.250"
lazy val Core = project
.settings(
libraryDependencies ++= Seq(
aws("aws-java-sdk-s3"),
aws("aws-java-sdk-dynamodb"),
)
)
However, line 6 is producing an error:
error: value can only be used within a task or setting macro, such as :=, +=, ++=, Def.task, or Def.setting.
The idea is that we'll move line 5 and 6 above to our plugin eventually so that we can use it like this:
name := "Foo"
version := "0.1"
scalaVersion := "2.12.8"
awsVersion := "1.11.250"
lazy val Core = project
.settings(
libraryDependencies ++= Seq(
aws("aws-java-sdk-s3"),
aws("aws-java-sdk-dynamodb"),
)
)
Any solution or work around for the error above?
We've also tried this:
def aws(module: String, version: String): ModuleID = "com.amazonaws" % module % version
... which is then used like this:
awsVersion := "1.11.250"
lazy val Core = project
.settings(
libraryDependencies ++= Seq(
aws("aws-java-sdk-s3", awsVersion.value),
aws("aws-java-sdk-dynamodb", awsVersion.value),
)
)
That works fine though a bit annoying to use and it defeats the purpose of using a settingKey to begin with.
You cannot use setting or task values in locally defined methods like aws. The values can be used only within other setting or task definitions, ie the error message such as :=, +=, ++=, Def.task, or Def.setting.
This is what you could do.
Create AutoPlugin in project folder.
import sbt.{AutoPlugin, Def, ModuleID, settingKey}
import sbt.PluginTrigger.AllRequirements
import sbt._
object AwsPlugin extends AutoPlugin {
override def trigger = AllRequirements
type GetAWS = String => ModuleID
object autoImport {
val awsVersion =
settingKey[String]("The version of aws SDK used for building.")
val awsLibrary = settingKey[GetAWS]("Builds given AWS library")
}
import autoImport._
override def projectSettings: Seq[Def.Setting[_]] = Seq(
awsLibrary := { id =>
"com.amazonaws" % id % awsVersion.value
}
)
}
Use it in this way in build.sbt
awsVersion in ThisBuild := "1.11.250"
lazy val Core = project
.settings(
libraryDependencies ++= Seq(
awsLibrary.value("aws-java-sdk-s3"),
awsLibrary.value("aws-java-sdk-dynamodb"),
)
)
I am reading this sample which shows me how to generate source code using Slick-CodeGen
https://github.com/slick/slick-codegen-example/blob/master/build.sbt
And while this sample, is good, I want to modify it so that it reads the database config from application.conf using typesafe config.
Otherwise I will have to replicate the database connection configuration here and also in application.conf file.
Does anyone know, how can this sample be modified so that we can use the typesafe config to read the config values from application.conf?
Edit: Based on the suggestion below, I tried the following
I created a file called build.sbt in the project folder
libraryDependencies += "com.typesafe" % "config" % "1.3.1"
modified my main build.sbt file (in project root) as
val slickVersion = "3.1.1"
lazy val mainProject = Project(
id = "FooBar",
base=file("."),
settings = Defaults.coreDefaultSettings ++ Seq(
scalaVersion := "2.11.8",
libraryDependencies ++= Seq(
"com.typesafe.slick" %% "slick" % slickVersion,
"com.typesafe.slick" %% "slick-codegen" % slickVersion,
"mysql" % "mysql-connector-java" % "5.1.35",
"com.typesafe" % "config" % "1.3.1"
),
myConf := {
ConfigFactory.parseFile(new File("src/main/resources/application.conf"))
},
slick <<= slickCodeGenTask,
sourceGenerators in Compile <+= slickCodeGenTask
)
)
lazy val slick = TaskKey[Seq[File]]("gen-tables")
lazy val myConf = settingKey[Config]("The application properties")
lazy val slickCodeGenTask = (sourceManaged, dependencyClasspath in Compile, runner in Compile, streams) map {(dir, cp, r, s) =>
val outputDir = (dir / "slick").getPath
val username = myConf.value.getString("mysql.username")
val password = myConf.value.getString("mysql.password")
val port = myConf.value.getInt("mysql.port")
val db = myConf.value.getString("mysql.db")
val server = myConf.value.getString("mysql.server")
val url = s"jdbc:mysql://$server:$port/$db?username=$username&password=$password"
val jdbcDriver = myConf.value.getString("mysql.jdbcDriver")
val slickDriver = myConf.value.getString("mysql.slickDriver")
val pkg = "sql"
val fname = outputDir + "/db/Tables.scala"
toError(r.run("slick.codegen.SourceCodeGenerator", cp.files, Array(slickDriver, jdbcDriver, url, outputDir, pkg), s.log))
Seq(file(fname))
}
But it cannot resolve the Config and the ConfigFactory classes.
Declare a dependency on Typesafe Config in project/build.sbt:
libraryDependencies += "com.typesafe" % "config" % "1.3.1"
And define a setting holding your config file in build.sbt:
lazy val myConf = settingKey[Config]("The application properties")
myConf := {
ConfigFactory.parseFile(new File("src/main/resources/application.conf"))
}
Now you can use myConf.value.getString("xyz") to get hold of your configuration values in other tasks or settings.
I am having a really strange problem:
I have a multi-module sbt project containing libraries that I publish locally.
Every once in a while the published jar is really a POM file!
Meaning I can open it with a text editor and see the POM for the library in question instead of the class files.
cleaning and rebuilding/republishing sometimes fixes it temporarily.
Also note that the jar in 'target/scala-2.11' is also a POM file.
Here is the Build.scala and sbt file for the libraries. (names are changed to protect the guilty :) due to company policies)
//////////////////Build.scala
import java.io._
import java.nio.file.{Paths, Files}
import sbt._
import xerial.sbt.Pack._
object Library extends Build {
lazy val rootProj = Project(id = "library", base = file(".")) aggregate(
projA,
projB,
projC,
projD
)
lazy val projA = Project(id = "projectA", base = file("ProjectA"))
lazy val projB = Project(id = "projectB", base = file("ProjectB"))
lazy val projC = Project(id = "projectC", base = file("ProjectC"))
.dependsOn(spatialMathProj)
lazy val projD = Project(id = "projectD", base = file("ProjectD"))
}
//////////////////build.sbt for projectD
name := "ProjectD"
version := "2.0-SNAPSHOT"
scalaVersion := "2.11.8"
publishArtifact in (Compile, packageDoc) := false
publishArtifact in (Compile, packageSrc) := false
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.4.2"
libraryDependencies += "com.typesafe.akka" %% "akka-remote" % "2.4.2"
libraryDependencies += "net.sf.jung" % "jung2" % "2.0.1"
libraryDependencies += "net.sf.jung" % "jung-graph-impl" % "2.0.1"
libraryDependencies += "net.sf.jung" % "jung-visualization" % "2.0.1"
libraryDependencies += "com.plexsys" % "api_2.11" % "2.0-SNAPSHOT"
Has anyone else seen this?
Why is this happening?
I am using sbt version 0.13.7.
Thanks for any help you can offer.
I am trying to make a simple NetLogo extension that is based on akka.
However, whenever I try to load the extension in NetLogo, I get the error:
Caused by: com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'akka.version'
Which obviously means that some configuration is missing. I then proceded to add reference.conf to my resources folder but with no luck.
The last thing I tried was to use the sbt-assemblty plugin, but I keep getting the same error. So this is my build.sbt:
name := "TestAkka"
version := "1.0"
scalaVersion := "2.11.7"
scalaSource in Compile <<= baseDirectory(_ / "src")
scalacOptions ++= Seq("-deprecation", "-unchecked", "-Xfatal-warnings",
"-encoding", "us-ascii")
libraryDependencies ++= Seq(
"org.nlogo" % "NetLogo" % "5.3.0" from
"http://ccl.northwestern.edu/devel/NetLogo-5.3-17964bb.jar",
"asm" % "asm-all" % "3.3.1",
"org.picocontainer" % "picocontainer" % "2.13.6",
"com.typesafe" % "config" % "1.3.0",
"com.typesafe.akka" %% "akka-actor" % "2.4.1",
"com.typesafe.akka" %% "akka-remote" % "2.4.1"
)
artifactName := { (_, _, _) => "sample-scala.jar" }
packageOptions := Seq(
Package.ManifestAttributes(
("Extension-Name", "sample-scala"),
("Class-Manager", "main.scala.akkatest.TestClassManager"),
("NetLogo-Extension-API-Version", "5.3")))
packageBin in Compile <<= (packageBin in Compile, baseDirectory, streams) map {
(jar, base, s) =>
IO.copyFile(jar, base / "sample-scala.jar")
Process("pack200 --modification-time=latest --effort=9 --strip-debug " +
"--no-keep-file-order --unknown-attribute=strip " +
"sample-scala.jar.pack.gz sample-scala.jar").!!
if(Process("git diff --quiet --exit-code HEAD").! == 0) {
Process("git archive -o sample-scala.zip --prefix=sample-scala/ HEAD").!!
IO.createDirectory(base / "sample-scala")
IO.copyFile(base / "sample-scala.jar", base / "sample-scala" / "sample-scala.jar")
IO.copyFile(base / "sample-scala.jar.pack.gz", base / "sample-scala" / "sample-scala.jar.pack.gz")
Process("zip sample-scala.zip sample-scala/sample-scala.jar sample-scala/sample-scala.jar.pack.gz").!!
IO.delete(base / "sample-scala")
}
else {
s.log.warn("working tree not clean; no zip archive made")
IO.delete(base / "sample-scala.zip")
}
jar
}
cleanFiles <++= baseDirectory { base =>
Seq(base / "sample-scala.jar",
base / "sample-scala.jar.pack.gz",
base / "sample-scala.zip") }
I have an project/assembly.sbt with the contents:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.1")
I have a assembly.sbt in root with the contents:
import sbtassembly.AssemblyKeys._
baseAssemblySettings
In my scala code I have:
val configString = ConfigFactory.parseString(
"""
akka {
loglevel = "INFO"
actor {
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
enabled-transports = ["akka.remote.netty.tcp"]
netty.tcp {
hostname = "127.0.0.1"
port = "9500"
}
log-sent-messages = on
log-received-messages = on
}
}
""".stripMargin)
val config = ConfigFactory.load(configString)
The resources folder contains an application.conf which I don't use at the moment. Greping the output of jar tf command with the expression "reference", clearly shows that reference.conf is present:
How do I package this akka example for a netlogo extension?
Note: I have included akka-actor and akka-remote as library dependencies. I am using Intellij and SBT 0.13.8 on a OS X platform.
EDIT:
After taking the advice from Ayush, I get the following output from the command sbt assembly, however the same exception is still present:
I think the problem is that while using sbt:assembly the default merge strategy excludes all the reference.conf files. This is what i found in documentation.
If multiple files share the same relative path (e.g. a resource named
application.conf in multiple dependency JARs), the default strategy is
to verify that all candidates have the same contents and error out
otherwise.
Can you try adding a MergeStrategy as follows
assemblyMergeStrategy in assembly := {
case PathList("reference.conf") => MergeStrategy.concat
}
there's an extra trick to solving this with newer akka libraries as akka does not include all their default configurations in the resource.conf
https://stackoverflow.com/a/72325132/1286583
I'm trying to write a concise multi project Build.sbt, so I tried to put all library dependencies in root project and then make others depends on it. My Build.sbt looks like the following:
object KataBuild extends Build {
lazy val fizzBuzz = Project(
id = "fizzBuzz",
base = file("fizzBuzz"),
settings = Project.defaultSettings ++ Seq(
name := "fizzBuzz",
version := "1.0",
scalaVersion := "2.10.3"
)
)
lazy val kata = Project(
id = "scala-kata",
base = file("."),
settings = Project.defaultSettings ++ Seq(
name := "scala-kata",
version := "1.0",
scalaVersion := "2.10.3",
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "2.1.0" % "test"
)
)
) aggregate(fizzBuzz)
fizzBuzz dependsOn(kata)
}
But running test from the main project (scala-kata) fails to build test for fizzBuzz. What am I missing?
Your question is similar to this one. In short, fizzBuzz.dependsOn(kata) means that its compile configuration depends on the kata's compile configuration, but you want to link the test configurations.
The 'Per-configuration classpath dependencies' section of the sbt docs show you how you can make a test->test dependency instead.
However, if you are not going to use kata's test sources but are just looking for a way to include Scala-Test in fizzBuzz, just add it explicitly to fizzBuzz's library dependencies, too. You can define a helper value
lazy val scalaTest = "org.scalatest" %% "scalatest" % "2.1.0" % "test"
Then you can add it to be sub project's library dependencies (libraryDependencies += scalaTest).