Running Wget in scala build.sbt - scala

I have a requirement where I need to download a bunch of jars from a url and then place them inside a lib directory and then add them to unmanaged dependancy.
I am kind of stuck on how to do this in build.sbt. Went over sbt documentation and I found processbuilder. With that in mind, I came up with the below code.
for(i <- jarUrls) {
val wget = s"wget -P $libDir $anonUser#$hgRootURL/$i"
wget !
}
This runs wget on a bunch of jars and then places the file in the mentioned folder. Pretty simple code, but I am unable to run this. The error that I get is "Expression of type Unit must confirm to DslEntry in SBT file".
How to accomplish this?

build.sbt isn't just scala file, sbt does special preprocessing on it (that's why you don't have to def project = etc).
Your problem happens because every line of code (except imports and definitions) in build.sbt must return expression of type DslEntry as sbt sees every line of code as setting. When do you want your wget to get executed? usual way is to define Task:
lazy val wget = taskKey[Unit]("Wget")
wget := {
for(i <- List(1,2,3)) {
val wget = s"wget -P $libDir $anonUser#$hgRootURL/$i"
wget !
}
()
}
and run it like sbt wget.
You can also make wget task dependent on some other task (or you can think of them as events) in sbt.
See http://www.scala-sbt.org/0.13/docs/Tasks.html
Of course, there are tricky unsafe ways, like:
val init: Unit = {
//any code you want here
}
But I wouldn't recommend it since you probably want those files during let's say compile stage or something:
wget := {
your code here
} dependsOn compile
you can also use regular scala build instead of build.sbt: http://www.scala-sbt.org/0.13/docs/Full-Def.html

Related

How to execute shell command before compile task?

I'd like to execute a shell command - rm -r a directory - whenever my sbt project builds. This would be before compile.
Reasoning: There's a cache file that never gets updated. If I delete it before each compile, it forces the update.
Please advise.
You can create a task that deletes the file:
val removeCacheTask = TaskKey[Unit]("removeCacheFile", "Deletes a cache file")
val removeCacheSettings = removeCacheTask := {
import sys.process._
Seq("rm", "/path/to/file") !
}
Then require that the task be run before compilation by adding these settings to your project:
Project(...).settings(
removeCacheSettings,
compile in Compile <<= (compile in Compile).dependsOn(removeCacheTask)
)
Source: https://groups.google.com/forum/#!topic/play-framework/4DMWSTNM4kQ
In build.sbt it would look like this:
lazy val removeCacheTask = TaskKey[Unit]("removeCacheFile", "Deletes a cache file")
removeCacheTask := {
import sys.process._
Seq("rm", "/path/to/file")!
}
compile in Compile <<= (compile in Compile).dependsOn(removeCacheTask)
#LimbSoup answer is fine, but there're some improvements to consider.
The command to execute rm might not be available on other non-rm OSes, like Windows, so it's much safer to use sbt.IO.delete method.
Since it's about deleting files, there's the clean task that relies on def doClean(clean: Seq[File], preserve: Seq[File]): Unit method (from sbt.Defaults) and with proper changes it could also be of some help. I think it would require a new config, though.
See How to exclude files in a custom clean task? for some guidance regarding a custom clean task.

Watch for project files also

I use sbt in the following fashion: I run ~ test:compile in sbt and then work in IDE, watching occasionaly if the project still compiles, because the IDE's presentation compiler tends to be buggy. When I git pull some code, there might be changes in the project/ files, so I want to have reload. Is there a way, how to watch both source files and project files, so when there is change in project files, I actually get the update?
As jsuereth explained this isn't a task SBT can handle in 1 instance. What's required is a reboot of SBT to abort the watching process and reload it's own configuration.
The following Scala script does exactly this, it uses Java NIO WatchService and Scala Process to monitor a path and restart a process. The code should be fairly simple to understand:
#!/usr/bin/env scala
import java.nio.file._
import scala.collection.JavaConversions._
import scala.sys.process._
val file = Paths.get(args(0))
val cmd = args(1)
val watcher = FileSystems.getDefault.newWatchService
file.register(
watcher,
StandardWatchEventKinds.ENTRY_CREATE,
StandardWatchEventKinds.ENTRY_MODIFY,
StandardWatchEventKinds.ENTRY_DELETE
)
def exec = cmd run true
#scala.annotation.tailrec
def watch(proc: Process): Unit = {
val key = watcher.take
val events = key.pollEvents
val newProc =
if (!events.isEmpty) {
proc.destroy()
exec
} else proc
if (key.reset) watch(newProc)
else println("aborted")
}
watch(exec)
Usage in your sbt dir would be:
watchr.scala project/ "sbt ~ test:compile"
If anything is unclear don't hesitate to ask, of course any scripting language could be used to implement this behavior.
You actually can't use ~ <task> and have it rebuild the project itself right now, because ~ <task> needs to read the build definition itself to determine:
What source files to watch
How to run the task.
What you're doing is altering the config whe project/ changes. This requires a full reload or reboot of sbt to pull in the new configuration.
So, as of sbt 0.13, this isn't possible. You can have it so it will rebuild your source code when project/ changes, but without rebuilding the build definition, not much help.
You could create a new sbt prompt, or task, that when run could check to see if source files in project/ are updated and issue a warning/error so you know to reboot. That's probably the best option right now.

How do I invoke a shell command during a Play 2.0 build (sbt)?

I would like to add a symlink from the .git/hooks directory to a file in my working tree during a regular Play! framework 2.0 build. According to the Play documentation, all sbt functionality is available as normal in a Play build. Based on google searches, I'm trying to add this code to the ApplicationBuild object in my project/Build.scala file:
val symlinkGitPrepushHookTask = compile in Compile <<= compile in Compile map {comp =>
val output = "ln -sf ../../.hooks/pre-push.py .git/hooks/pre-push".!!
print(output)
comp
}
From my reading of the sbt docs, this should be adding a dependency to the compile task in the Compile scope. The dependency is on its existing value, but with my additional function mapped to it. Now when the compile task runs, my anonymous function should be run too. This does not successfully create the symlink, and does not even seem to run.
Immediately after posting this, I thought I would try adding the example I had found to the project/plugins.sbt file. This works, although it seems an abuse of a file to specify plugins.
... existing plugins.sbt content ...
compile in Compile <<= compile in Compile map { comp =>
"ln -sf ../../.hooks/pre-push.py .git/hooks/pre-push".!!
comp
}
The blank line is critical, as this is the delimiter in the .sbt format.

How to configure build.sbt so that xsbt-web-plugin a creates war file without compression?

I am using Scala 2.10.1 with sbt to package my webapp as a war file.
For the purpose of efficient rsync deltas, I'd like to have the war packaged as a .war file, but without zip compression. I just need to know how to configure my build for this.
UPDATE:
All these plugin docs assume all this knowledge of how the syntax works and how to combine tasks into a new task, etc. I can't even tell how to create a new task that does package then command. None of the answers so far have said specifically, "here's what you do.."
Just to be clear, this is all I'm asking for:
I need a Task "packnozip" that does this:
1) run "package"
2) run shell commands:$ mkdir ./Whatever
$ pushd ./Whatever
$ jar xvf ../Whatever.war
$ popd
$ mv ./Whatever.war ./Whatever.war.orig
$ jar cvM0f ./Whatever.war -C ./Whatever .
So what i'm saying is i want to type "packnozip" into the sbt console and have it do #1 then #2.
For now i'm just manually doing #2 which seems silly if it can be automated.
Also watching a 30MB file get completely resent by rsync b/c it is not diffable seems quite silly when a 34MB uncompressed file is only 13% more data, and takes a fraction of second to send b/c of efficient diffs, not to mention "-z" will compress the transfer anyways.
If you have your war file unzipped in a directory you can:
zip -r -0 project.war project/
That should be zero compression. In case you don't see those options, this is my setup:
[node#hip1 dev]$ zip -v
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Which, you could execute as a run task I believe, after the war is packaged.
UPDATE 1
I believe this is the best way to achieve your needs:
http://www.scala-sbt.org/release/docs/Detailed-Topics/Process
val exitcode = "zip -r -0 project.war project/"!
However, if you need to work from a specific directory (Please see Update 2 below):
Modified this to execute within directory but place .war above directory. The path (2nd) argument should include the directory, so that the zip is performed inside of it:
Process("zip" :: "-r" :: "-0" :: "../project.war" :: "." :: Nil, "/path/to/project/") !
Here's another SO question on the ProcessBuilder that may help as well:
How does the “scala.sys.process” from Scala 2.9 work?
(Note: you don't need to import scala.sys.process._)
UPDATE 2
For readers in the future, please note that zipping the project directory itself will not work, one needs to perform the zip of the war inside the directory by using pushd, putting the resulting war outside of the directory as mentioned by the OP in the comments below this answer. As Orange80 mentioned:
pushd ./project && zip -r -0 ../project.war ./ && popd
UPDATE 3
Check out this, it may do exactly what you need, with a 0 for options to specify no compression:
https://github.com/sbt/sbt-onejar
a plugin that lets you create a single executable jar, which, with options (for example "0" as in a command like "jar 0f blah.jar blah/") can be made I think as you mentioned in the comments below to create the jar file without compression.
For usage I found this on SO:
SBT one-jar plugin
And also, if it needs to be modified, it's a pretty reasonable example of a plugin as well, which if you drop it in your home ~/.sbt/plugins it will be global and can be used in your build in the fashion noted in the SO answer above. I hope that helps at least a little bit/
There is no way to do this directly via sbt configuration, since sbt assumes that any files within zip and jar artifacts should be compressed.
One workaround is to unzip and re-zip (without compression) the war file. You can do this by adding the following setting to your project (e.g. in build.sbt):
packageWar in Compile <<= packageWar in Compile map { file =>
println("(Re)packaging with zero compression...")
import java.io.{FileInputStream,FileOutputStream,ByteArrayOutputStream}
import java.util.zip.{CRC32,ZipEntry,ZipInputStream,ZipOutputStream}
val zis = new ZipInputStream(new FileInputStream(file))
val tmp = new File(file.getAbsolutePath + "_decompressed")
val zos = new ZipOutputStream(new FileOutputStream(tmp))
zos.setMethod(ZipOutputStream.STORED)
Iterator.continually(zis.getNextEntry).
takeWhile(ze => ze != null).
foreach { ze =>
val baos = new ByteArrayOutputStream
Iterator.continually(zis.read()).
takeWhile(-1 !=).
foreach(baos.write)
val bytes = baos.toByteArray
ze.setMethod(ZipEntry.STORED)
ze.setSize(baos.size)
ze.setCompressedSize(baos.size)
val crc = new CRC32
crc.update(bytes)
ze.setCrc(crc.getValue)
zos.putNextEntry(ze)
zos.write(bytes)
zos.closeEntry
zis.closeEntry
}
zos.close
zis.close
tmp.renameTo(file)
file
}
Now when you run package in sbt, the final war file will be uncompressed, which you can verify with unzip -vl path/to/package.war.

How to create a compiler Action for SBT

I want to create an Action to automate GCJ compilation. Since I couldn't make it work with Ant, I decided to try SBT. The docs say how to create an Action and how to run an external process. What I don't yet see is how to reuse the directory tree traversal which exists for java and scala compiler Actions. In this case my input files would be all the .class files under a certain root folder. I would also need to specify a specific classpath for GCJ. Any pointers for this would be appreciated too.
I haven't used GCJ much at all and I'm still pretty new at SBT, but this is how I believe you could write a quick task to do exactly what you are looking for with SBT 0.7.1. You can use a PathFinder to grab all of the class files like so:
val allClasses = (outputPath ##) ** "*.class"
Using that PathFinder and the "compileClasspath" top level method, you can construct a task like this which will run gcj using the current project's classpath and compose all of the .class files into one gcjFile:
val gcj = "/usr/local/bin/gcj"
val gcjFile = "target/my_executable.o"
val allClasses = (outputPath ##) ** "*.class"
lazy val gcjCompile = execTask {
<x>{gcj} --classpath={compileClasspath.get.map(_.absolutePath).mkString(":")} -c {allClasses.get.map(_.absolutePath).mkString("-c ")} -o {gcjFile}</x>
} dependsOn(compile) describedAs("Create a GCJ executable object")