I am using Sass as my CSS preprocesser, and I'm trying to have it run via the asset pipeline. I've tried implementing this sassTask as a source file task and as a web asset task, but I'm running into problems both ways.
If I run Sass as a source task (see below), it gets triggered during activator run when a page is requested and updated files are found upon page reloads. The problem I'm running into is that the resulting CSS files are all getting dumped directly into target/web/public/main/lib, instead of into the subdirectories reflecting the ones they are getting built into under the resources-managed directory. I can't figure out how to make this happen.
Instead, I tried implementing Sass compilation as a web asset task (see below). Working this way, as far as I can tell, resources-managed does not come into play, and so I compile my files directly into target/web/public/main/lib. I'm sure I'm not doing this dynamically enough, but I don't know how to do it any better. But the biggest problem here is that the pipeline apparently does not run when working through activator run. I can get it to run using activator stage, but I really need this to work in the regular development workflow so that I can change style files as the dev server is running, same as with Scala files.
I have tried combing through these forums, through the sbt-web docs, and through some of the existing plugins, but I am finding this process to be highly frustrating, due to the complexity of SBT and the opaqueness of what is actually happening in the build process.
Sass compilation as a source file task:
lazy val sassTask = TaskKey[Seq[java.io.File]]("sassTask", "Compiles Sass files")
sassTask := {
import sys.process._
val x = (WebKeys.nodeModules in Assets).value
val sourceDir = (sourceDirectory in Assets).value
val targetDir = (resourceManaged in Assets).value
Seq("sass", "-I", "target/web/web-modules/main/webjars/lib/susy/sass", "--update", s"$sourceDir:$targetDir").!
val sources = sourceDir ** "*.scss"
val mappings = sources pair relativeTo(sourceDir)
val renamed = mappings map { case (file, path) => file -> path.replaceAll("scss", "css") }
val copies = renamed map { case (file, path) => file -> targetDir / path }
copies map (_._2)
}
sourceGenerators in Assets <+= sassTask
Sass compilation as web asset task:
lazy val sassTask = taskKey[Pipeline.Stage]("Compiles Sass files")
sassTask := {
(mappings: Seq[PathMapping]) =>
import sys.process._
val sourceDir = (sourceDirectory in Assets).value
val targetDir = target.value / "web" / "public" / "main"
val libDir = (target.value / "web" / "web-modules" / "main" / "webjars" / "lib" / "susy" / "sass").toString
Seq("sass", "-I", libDir, "--update", s"$sourceDir:$targetDir").!
val sources = sourceDir ** "*.scss"
val mappings = sources pair relativeTo(sourceDir)
val renamed = mappings map { case (file, path) => file -> path.replaceAll("scss", "css") }
renamed
}
pipelineStages := Seq(sassTask)
I think that according to the documentation related to the Asset Pipeline, a Source File task is a way to go:
Examples of source file tasks as plugins are CoffeeScript, LESS and
JSHint. Some of these take a source file and produce a target web
asset e.g. CoffeeScript produces JS files. Plugins in this category
are mutually exclusive to each other in terms of their function i.e.
only one CoffeeScript plugin will take CoffeeScript sources and
produce target JS files. In summary, source file plugins produce web
assets.
I think what you try to achieve falls into this category.
TL;DR; - build.sbt
val sassTask = taskKey[Seq[File]]("Compiles Sass files")
val sassOutputDir = settingKey[File]("Output directory for Sass generated files")
sassOutputDir := target.value / "web" / "sass" / "main"
resourceDirectories in Assets += sassOutputDir.value
sassTask := {
val sourceDir = (sourceDirectory in Assets).value
val outputDir = sassOutputDir.value
val sourceFiles = (sourceDir ** "*.scss").get
Seq("sass", "--update", s"$sourceDir:$outputDir").!
(outputDir ** "*.css").get
}
sourceGenerators in Assets += sassTask.taskValue
Explanation
Assuming you have sass file in a app/assets/<whatever> directory, and that you want to create css files in web/public/main/<whatever> directory, this is what you could do.
Create a task, which will read in files in the app/assets/<whatever> directory and subdirectories, and output them to our defined sassOutputDir.
val sassTask = taskKey[Seq[File]]("Compiles Sass files")
val sassOutputDir = settingKey[File]("Output directory for Sass generated files")
sassOutputDir := target.value / "web" / "sass" / "main"
resourceDirectories in Assets += sassOutputDir.value
sassTask := {
val sourceDir = (sourceDirectory in Assets).value
val outputDir = sassOutputDir.value
val sourceFiles = (sourceDir ** "*.scss").get
Seq("sass", "--update", s"$sourceDir:$outputDir").!
(outputDir ** "*.css").get
}
This is not enough though. If you want to keep the directory structure you have to add your sassOutputDir to the resourceDirectories in Assets. This is because mappings in sbt-web are declared like this:
mappings := {
val files = (sources.value ++ resources.value ++ webModules.value) ---
(sourceDirectories.value ++ resourceDirectories.value ++ webModuleDirectories.value)
files pair relativeTo(sourceDirectories.value ++ resourceDirectories.value ++ webModuleDirectories.value) | flat
}
which means that all unmapped files are mapped using an alternative flat strategy. However the fix for it is simple, just add this to your build.sbt
resourceDirectories in Assets += sassOutputDir.value
This will make sure the directory structure is preserved.
Related
I have written a plugin to process some SQL files and generate new ones as managed resources. When I run 'sbt compile' the files are generated in to the target/resource_managed/main/sql folder. When I run 'sbt run' or 'sbt test' they are not copied into the target/classes directory like I expect, so the code that is looking for them on the classpath cannot find them.
Here is the code for the plugin:
object SqlProcessorPlugin extends AutoPlugin {
import autoImport._
override def requires = plugins.JvmPlugin
override def trigger = noTrigger
object autoImport {
lazy val processorSettings = taskKey[File]("Settings for sql processing")
lazy val processSqlTask = taskKey[Seq[File]]("Process Sql")
def configProcessor(cfg: Configuration) = {
inConfig(cfg) {
Seq(
target in processorSettings := resourceManaged.value / "sql",
sourceDirectory in processorSettings := sourceDirectory.value / "sql",
processSqlTask / fileInputs += (sourceDirectory in processorSettings).value.toGlob / ** / "*.sql",
processSqlTask := {
SqlProcessor.process(
processSqlTask.inputFileChanges,
(target in processorSettings).value
)
},
resourceGenerators += processSqlTask.taskValue,
)
}
}
override val projectSettings = configProcessor(Compile)
}
}
I've try lots of variations on this based upon examples from other questions and from other plugins, but nothing has resulted in the generated files being copied to the class path.
What an I missing/doing wrong here?
I figured ou the issue. I was using resourceManage.value instead of (Compile/resourceManaged).value as the target directory. Also, I think as a result it was messing up the relative path on the output files resulting in them being copied to the wrong place.
My initial setup had two separate projects (sbt 1.2.6);
a web app (huge codebase, lot of dependencies, slow compile)
a command-line app (basically one file with 3-4 separate dependencies)
The feature request came in; we should show the "valid" values in the command line app. The valid values are in an enum in the web app. So I fired up the sbt documentation and came up with an idea which looked like this;
//main webapp
lazy val core = project
.in(file("."))
.withId("core") //I tried this just in case, not helped...
//... here comes all the plugins and deps
//my hack to get a single-file compile
lazy val `feature-signer-helper` = project
.in(file("."))
.withId("feature-signer-helper")
.settings(
target := { baseDirectory.value / "target" / "features" },
sources in Compile := {
((scalaSource in Compile).value ** "Features.scala").get
}
)
//the command line app
lazy val `feature-signer` = project
.in(file("feature-signer"))
.dependsOn(`feature-signer-helper`)
.settings(
libraryDependencies ++= signerDeps
)
The problem is that it seems like, that whatever the last lazy val xxx = project.in(file(y)) that will be the only project for the y dir.
Also, I don't want to move that one file to a separate directory structure... And logically the command line app and the web app are not "depends on" each other, they have different dependencies (and really different build times).
My questions are;
is there any quick-win in this situation? (I will copy the file worst-case...)
why we have this rich project and source settings if I can't bind them to the same dir?
EDIT:
The below code can copy the needed file (if you have the same dir structure). I'm not super happy with it, but it works. Still interested in other methods.
import sbt._
import Keys._
object FeaturesCopyTask {
val featuresCopyTask = {
sourceGenerators in Compile += Def.task {
val outFile = (sourceManaged in Compile).value / "Features.scala"
val rootDirSrc = (Compile / baseDirectory).value / ".." / "src"
val inFile: File = (rootDirSrc ** "Features.scala").get().head
IO.copyFile(inFile, outFile, preserveLastModified = true)
Seq(outFile)
}.taskValue
}
}
lazy val `feature-signer` = project
.in(file("feature-signer"))
.settings(
libraryDependencies ++= signerDeps,
FeaturesCopyTask.featuresCopyTask
)
I would have the tree be something more like
+- core/
+- webapp/
+- cli/
core is the small amount (mostly model type things) that webapp and cli have in common
webapp depends on core (among many other things)
cli depends on core (and not much else)
So the build.sbt would be something like
lazy val core = (project in file("core"))
// yadda yadda yadda
lazy val webapp = (project in file("webapp"))
.dependsOn(core)
// yadda yadda yadda
lazy val cli = (project in file("cli"))
.dependsOn(core)
// yadda yadda yadda
lazy val root = (project in file("."))
.aggregate(
core,
webapp,
cli
)
Building my project on Scala with sbt, I want to have a task that will run prior to actual Scala compilation and will generate a Version.scala file with project version information. Here's a task I've came up with:
lazy val generateVersionTask = Def.task {
// Generate contents of Version.scala
val contents = s"""package io.kaitai.struct
|
|object Version {
| val name = "${name.value}"
| val version = "${version.value}"
|}
|""".stripMargin
// Update Version.scala file, if needed
val file = (sourceManaged in Compile).value / "version" / "Version.scala"
println(s"Version file generated: $file")
IO.write(file, contents)
Seq(file)
}
This task seems to work, but the problem is how to plug it in, given that it's a cross project, targeting Scala/JVM, Scala/JS, etc.
This is how build.sbt looked before I started touching it:
lazy val root = project.in(file(".")).
aggregate(fooJS, fooJVM).
settings(
publish := {},
publishLocal := {}
)
lazy val foo = crossProject.in(file(".")).
settings(
name := "foo",
version := sys.env.getOrElse("CI_VERSION", "0.1"),
// ...
).
jvmSettings(/* JVM-specific settings */).
jsSettings(/* JS-specific settings */)
lazy val fooJVM = foo.jvm
lazy val fooJS = foo.js
and, on the filesystem, I have:
shared/ — cross-platform code shared between JS/JVM builds
jvm/ — JVM-specific code
js/ — JS-specific code
The best I've came up so far with is adding this task to foo crossProject:
lazy val foo = crossProject.in(file(".")).
settings(
name := "foo",
version := sys.env.getOrElse("CI_VERSION", "0.1"),
sourceGenerators in Compile += generateVersionTask.taskValue, // <== !
// ...
).
jvmSettings(/* JVM-specific settings */).
jsSettings(/* JS-specific settings */)
This works, but in a very awkward way, not really compatible with "shared" codebase. It generates 2 distinct Version.scala files for JS and JVM:
sbt:root> compile
Version file generated: /foo/js/target/scala-2.12/src_managed/main/version/Version.scala
Version file generated: /foo/jvm/target/scala-2.12/src_managed/main/version/Version.scala
Naturally, it's impossible to access contents of these files from shared, and this is where I want to access it.
So far, I've came with a very sloppy workaround:
There is a var declared in singleton object in shared
in both JVM and JS main entry points, the very first thing I do is that I assign that variable to match constants defined in Version.scala
Also, I've tried the same trick with sbt-buildinfo plugin — the result is exactly the same, it generated per-platform BuildInfo.scala, which I can't use directly from shared sources.
Are there any better solutions available?
Consider pointing sourceManaged to shared/src/main/scala/src_managed directory and scoping generateVersionTask to the root project like so
val sharedSourceManaged = Def.setting(
baseDirectory.value / "shared" / "src" / "main" / "scala" / "src_managed"
)
lazy val root = project.in(file(".")).
aggregate(fooJS, fooJVM).
settings(
publish := {},
publishLocal := {},
sourceManaged := sharedSourceManaged.value,
sourceGenerators in Compile += generateVersionTask.taskValue,
cleanFiles += sharedSourceManaged.value
)
Now sbt compile should output something like
Version file generated: /Users/mario/IdeaProjects/scalajs-cross-compile-example/shared/src/main/scala/src_managed/version/Version.scala
...
[info] Compiling 3 Scala sources to /Users/mario/IdeaProjects/scalajs-cross-compile-example/js/target/scala-2.12/classes ...
[info] Compiling 1 Scala source to /Users/mario/IdeaProjects/scalajs-cross-compile-example/target/scala-2.12/classes ...
[info] Compiling 3 Scala sources to /Users/mario/IdeaProjects/scalajs-cross-compile-example/jvm/target/scala-2.12/classes ...
I would like to generate sources from files which are part of the project (I have currently placed them in a resource directory, but this is not a requirement).
This is my attempt on it:
sourceGenerators in Test += (sourceManaged in Test map { src =>
(unmanagedResourceDirectories in Test).value map { dir =>
val file = dir / "demo" / src.name
IO.write(file, "Prefix---" + IO.read(src) + "---Postfix")
file
}
}).taskValue
This gives me an error:
error: Illegal dynamic dependency
(unmanagedResourceDirectories in Test).value map { src =>
What is a correct way to do this?
What has worked eventually is this (inspired by this code, referenced in a comment to a question SBT sourceGenerators task - execute only if a file changes):
sourceGenerators in Test += Def.task {
val sources = (unmanagedResources in Test).value filter ( _.isFile )
val dir = (sourceManaged in Test).value
sources map { src =>
IO.write(dir / src.name, "Prefix---" + IO.read(src) + "---Postfix")
f
}
}.taskValue
The important part was reading the settings inside of the task.
I think Dynamic tasks are the correct way to do it
http://www.scala-sbt.org/0.13/docs/Tasks.html#Dynamic+Computations+with
I have a map reduce .scala file like this:
import org.apache.spark._
object WordCount {
def main(args: Array[String]){
val inputDir = args(0)
//val inputDir = "/Users/eksi/Desktop/sherlock.txt"
val outputDir = args(1)
//val outputDir = "/Users/eksi/Desktop/out.txt"
val cnf = new SparkConf().setAppName("Example MapReduce Spark Job")
val sc = new SparkContext(cnf)
val textFile = sc.textFile(inputDir)
val counts = textFile.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile(outputDir)
sc.stop()
}
}
When I run my code, with setMaster("local[1]") parameters it works fine.
I want to put this code in a .jar and throw it to S3 to work with AWS EMR. Therefore, I use the following build.sbt to do so.
name := "word-count"
version := "0.0.1"
scalaVersion := "2.11.7"
// additional libraries
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % "1.0.2"
)
It generates a jar file, however none of my scala code is in there. What I see is just a manifest file when I extract the .jar
When I run sbt package this is what I get:
[myMacBook-Pro] > sbt package
[info] Loading project definition from /Users/lele/bigdata/wordcount/project
[info] Set current project to word-count (in build file:/Users/lele/bigdata/wordcount/)
[info] Packaging /Users/lele/bigdata/wordcount/target/scala-2.11/word-count_2.11-0.0.1.jar ...
[info] Done packaging.
[success] Total time: 0 s, completed Jul 27, 2016 10:33:26 PM
What should I do to create a proper jar file that works like
WordCount.jar WordCount
Ref: It generates a jar file, however none of my scala code is in there. What I see is just a manifest file when I extract the .jar
Make sure your WordCount.scala is in the root or in src/main/scala
From http://www.scala-sbt.org/1.0/docs/Directories.html
Source code can be placed in the project’s base directory as with hello/hw.scala. However, most people don’t do this for real projects; too much clutter.
sbt uses the same directory structure as Maven for source files by default (all paths are relative to the base directory):