Datanucleus enhancement with Bazel - datanucleus

I'm trying to migrate Maven project to Bazel and having troubles with Datanucleus enhancement.
After jar-file is build, Datanucleus looks inside it and does some byte-code manipulation to enhance persistable classes. The way to perform this in Bazel is by defining a rule that takes the *.jar output of java_library rule and creates a new enhanced version of the library.
The problem that I have is that for my rule I need datanucleus-core package from external libraries. When I try to access it from a genrule by $(location //third_party:datanucleus_core) it point to a jar which has no classes:
(genrule) cmd = "echo $(location //third_party:datanucleus_core)"
bazel-out/local-fastbuild/bin/third_party/liborg_datanucleus_datanucleus_core.jar
(genrule) cmd = "jar tf $(location //third_party:datanucleus_core)"
META-INF/
META-INF/MANIFEST.MF
The jar-file resolved by Bazel in genrule from $(location //third_party:datanucleus_core) contains only META-INF/MANIFEST.MF with the following content:
Manifest-Version: 1.0
Created-By: blaze
I tried to use java_binary rule that adds a correct datanucleus_core.jar into classpath, but Datanucleus enhances my libary in-place and fails to write its changes on disk (rewrite the rule's input file). Also java_binary rule is not supposed to be used for building.
So the question is what is the best way to enhance jar library in Bazel running Datanucleus utility, which is provided as a third-party dependency in Maven repository?
Bazel build label: 0.3.2-homebrew, OS: OS X El Capitan (10.11.6), java: 1.8.0_92
Update
Datanucleus dependency declaration:
# WORKSPACE
maven_jar(
name = "org_datanucleus_datanucleus_core",
artifact = "org.datanucleus:datanucleus-core:5.0.3",
)
# third_party/BUILD
java_library(
name = "org_datanucleus_datanucleus_core",
visibility = ["//visibility:public"],
exports = ["#org_datanucleus_datanucleus_core//jar"],
)
(in my question I shortened org_datanucleus_datanucleus_core to datanucleus_core)

As Neil Stockton mentioned, you cannot enhance classes in a jar. So, the basic strategy will be:
Create the jar.
Unjar the class files.
Run the enhancements.
Jar it back up.
Steps 2 & 3 have to be rolled into 4, as Bazel insists that you declare all inputs & outputs to a build rule (and you cannot know what .class files a .java file will generate, so Bazel always jars them up).
Create a datanucleus.bzl file to declare your enhancement rule in. It should look something like:
# Run datastore enhancements on the java_library named "jarname".
def enhance(jarname):
# src is the name of the jar file your java_library rule generates.
src = "lib" + jarname + ".jar"
native.genrule(
name = jarname + "-enhancement",
srcs = [
src,
"//third_party:datanucleus_core"
],
outs = [jarname + "-enhanced.jar"],
cmd = """
# Un-jar the .class files.
jar tf $(location {0})
# Run the enhance.
classes=""
for $$class in $$(find . -name *.class); do
java -cp {0}:$(location //third_party:datanucleus_core) $$class
classes="$$classes $$class"
done
# jar them back up.
jar cf $# $$classes""".format(src),
)
(I'm not too familiar with datastore so the cmd might need some modification, but it should be that general idea.)
Then, in your BUILD file, you'd do:
java_library(
name = "my-lib",
srcs = glob(["*.java"]),
deps = ["..."],
)
# import the rule you wrote.
load('//:datanucleus.bzl', 'enhance')
enhance("my-lib")
Now you can do:
bazel build //:my-lib-enhanced.jar
and use my-lib-enhanced.jar as a dependency in other java_ rules.
More info on .bzl files: https://bazel.build/versions/master/docs/skylark/concepts.html.
Edited to add more info on depending on a jar:
There are a couple of options to get a jar that contains the content of datanucleus. First, you don't need the layer of indirection: you can just say:
srcs = [
src,
"#datanucleus_core//jar"
],
This will give you the actual jar.
If, for some reason, you need the jar to be in third_party, you can modify third_party/BUILD to create a deploy jar, which is a java binary that bundles up all of its dependencies for deployment (since you're not actually going to using it as a binary, you can use whatever you want for the main class name):
java_binary(
name = "datanucleus-core",
main_class = "whatever",
runtime_deps = ["#org_datanucleus_datanucleus_core//jar"],
)
genrule(
name = "your-lib",
srcs = [":datanucleus-core_deploy.jar", ...],
)
The :datanucleus-core_deploy.jar is called an implicit target: it's only built if requested, but it can be generated from your java_binary declaration.

Related

How do I copy a zip dependency to the target directory in an SBT build?

I am working on an SBT project that generates an RPM via the sat-native-packager. One of the items that I want to pull into the RPM is a ZIP file that was published from a separate project using the sat-pack plugin. This ZIP file contains a number of JAR files, along with multiple scripts for invoking them.
I have the following in my RPM project's build.sbt:
libraryDependencies += ("com.mycompany" %% "aputils" % "1.0.0-SNAPSHOT").artifacts(Artifact("aputils", "zip", "zip"))
// Task to download and unpack the utils bundle
lazy val unpackUtilsTask = taskKey[Unit]("Download the utils bundle to the target directory")
unpackUtilsTask := {
val log = streams.value.log
val report: UpdateReport = (update in Rpm).value
val filter = artifactFilter(extension = "zip")
val matches: Seq[File] = report.matching(filter)
matches.foreach{ f =>
log.info(s"Filter match: ${f}")
IO.copyFile(f, target.value)
}
}
When I run this task, it does not match any entries in the UpdateReport. Nothing is printed, and no files are copied to target/. If I modify the task to instead print all of the files in the UpdateReport:
report.allFiles.foreach(f => log.info(s"All files: $f))
I see a number of JAR files, but not my ZIP file. The JAR files turn out to be all of the JAR files that are contained in the ZIP file. I am not sure why the ZIP is being unpacked and its contents are being listed as dependencies like this. If I mark the dependency as notTransitive, then the contained JARs are not listed in the report, but the ZIP still isn't included either.
This project is using SBT 0.13.15. I would prefer not to update it to 1.x at this time, but will do so if I must.
I will need to unzip the ZIP file under target/ eventually so I can define one or more packageMapping entries to pull the files into the RPM, but that seems easy enough to do via sbt.IO, if I can first just get a reference to the original ZIP file that is pulled down from our Artifactory server.
This didn't get any responses after a couple days, but I'll post the answer that I was able to come up with after more trial and error.
I was on the right track by examining the UpdateReport, but I wasn't looking at the right data within it. I needed to drill down to find a ModuleReport, which would show me where the .zip file was being downloaded to on the build machine. Once I have that path, it is trivial to unpack it to target/ using IO.unzip(). Here is how my task ended up looking:
libraryDependencies += ("com.mycompany" %% "aputils" % "1.0.0-SNAPSHOT").artifacts(Artifact("aputils", "zip", "zip"))
// Task to unzip the utils ZIP file to the target directory so we can define a package mapping
lazy val unpackUtilsTask = taskKey[Unit]("Download the utils bundle to the target directory")
unpackUtilsTask := {
val log = streams.value.log
val cReport: ConfigurationReport = (update in Compile).value.configuration("compile").get
cReport.modules.foreach{ mReport =>
if (mReport.module.name.startsWith("aputils")) {
mReport.artifacts.foreach{ case (art, f) =>
log.info(s"Unpacking aputils bundle: ${f.getAbsolutePath}")
IO.unzip(f, target.value)
}
}
}
}
packageBin in Rpm := ((packageBin in Rpm).dependsOn(unpackUtilsTask)).value
The last line attaches the task to the task that builds the RPM, so it will be unzipped before the RPM is built, and we can define packageMappings to put the contents of the .zip file into the generated RPM.

Pull GitHub repository using Bazel

I need to download an entire GitHub repository using Bazel. Since I'm quite new to this tool I'm not really sure how to achieve that.
My main idea is this:
write a custom repository rule in downloadgithubrepo.bzl (which is located in the project root just like the WORKSPACE file) such as:
def _impl(repository_ctx):
repository_ctx.download("url_to_zipped_github_repo", output='relative_path_to_output_file')
github = repository_rule(
implementation = _impl
and in the WORKSPACE file to write something like this:
load("//:downloadgithubrepo.bzl", "github")
and to invoke a build a BUILD file is needed (also located at the project root)
its contents are the following:
cc_library(
name = "testrun",
srcs = "main.c",
)
I had to add the main.c file otherwise the build is failing - that is one issue and the real issue is that this does not work, as in the build is passing but the GitHub repository is not downloaded.
Am I on the right path at all?? Has anyone done something like this before?
What you're looking might already be implemented in the new_git_repository repository rule, or the git_repository rule if the GitHub project already has Bazel BUILD files wired in.
If the GitHub project does not have BUILD files, a BUILD file is required when using new_git_repository. For example, if you want to depend on a file target (e.g. /foo/bar.txt) or rule target (e.g. a cc_library) in https://github.com/example/repository, and the repository does not have BUILD files, write these lines in your project's WORKSPACE file:
new_git_repository(
name = "example_repository",
remote = "https://github.com/example/repository.git",
build_file_content = """
exports_files(["foo/bar.txt"])
# you can also create targets
cc_library(
name = "remote_cc_library",
srcs = ["..."],
hdrs = ["..."],
)
""",
)
In your BUILD file, reference the external repository's targets using the # prefix:
cc_library(
name = "testrun",
srcs = ["main.c"],
data = ["#example_repository//:foo/bar.txt"],
deps = ["#example_repository//:remote_cc_library"],
)
When you run bazel build //:testrun, Bazel will..
Analyze the dependencies of //:testrun, which include the file main.c and targets from the external repository #example_repository.
Look up the WORKSPACE file for an external repository named example_repository, and finds the new_git_repository declaration.
Perform a git clone on the remote attribute specified in the example_repository declaration.
Write a BUILD file containing the build_file_content string at the project root of the cloned repository.
Analyze the targets #example_repository//:foo/bar.txt and #example_repository//:remote_cc_library
Build the dependencies, and hands them to your //:testrun cc_library.
Build //:testrun.
If the GitHub project does have BUILD files, you do not need to provide an BUILD file. You can refer to the targets directly after specifying the WORKSPACE dependency with git_repository:
git_repository(
name = "example_repository",
remote = "https://github.com/example/repository.git",
)
For more information, check out Bazel's documentation on External Repositories.

Scala Standalone JAR with a conf Folder

I'm using the sbt assembly jar plugin to create a standalone jar file. My project folder structure would look like this:
MyProject
-src
- main
- scala
- mypackages and source files
- conf // contains application.conf, application.test.conf and so on
- test
-project // contains all the build related files
- README.md
I now want to be able to run the fat jar that I produce against a version of the application.conf that I specify as a System property!
So here is what I do in my unit test!
System.setProperty("environment", "test")
And this is how I load the config in one of the files in my src folder:
val someEnv = Option(System.getProperty("environment", "")).filter(_.nonEmpty) // gives me some(test)
val name = s"application.${someEnv.get}.conf"
I can see that the environment variable is set and I get the environment passed it. But later on I load the application.test.conf as below:
ConfigFactory.load(name).resolve()
It however loads just the edfault application.conf and not the one that I specify!
What is wrong in my case? Where should I put the conf folder? I'm trying to run it against my unit test which is inside the test folder!
I believe you need to specify the full name of the configuration file. The .conf is optional. Try
ConfigFactory.load(s"application.${someEnv.get}").resolve()
The docs for ConfigFactory.load(String) indicate you need to supply
name (optionally without extension) of a resource on classpath
Ok! Here is what I had to do! Change the name of the folder where the config file is located. I originally had it as conf and I had to rename it to resources and bang it worked!

Scala.js compilation destination

I'm working on a Scala.js cross project where the jvm folder represents my server application and jsrepresents my scala.js code.
Whenever i compile my scala.js code via sbt crossJS/fastOptJS the compiled JS ends up in ./js/target/scala-2.11/web-fastopt.js.
I need to have this compiled JS file accessible in the resources of the server project in the jvm folder, so i can server it through my web application. I think i have to do something with artifactPath but i can't seem to get any results from my experiments thus far.
You can simply set the artifactPath of the fastOptJS task (or the fullOptJS task) to the (managed) resources directory of your JVM project:
// In the JS project's settings
artifactPath in fastOptJS in Compile :=
(resourceManaged in jvm in Compile).value /
((moduleName in fastOptJS).value + "-fastopt.js"))
This will put it in the directory, if the you run the fastOptJS task. However, it will not be included in sbt's resources task and it will not automatically be triggered, if you launch your server. Therefore:
// In the JVM project's settings
resources in Compile += (fastOptJS in js).value.data
A couple of notes:
The first step is only necessary, if your web-server does only serve specific directories. Otherwise the second one is enough, as this adds the file to the resources already; where it lies is secondary.
Setting the crossTarget, as in #ochrons' answer will also output all the .class and .sjsir files in the resource directory.
Have a look at Vincent Munier's sbt-play-scalajs for out-of-the-box sbt-web / Scala.js integration (it follows a slightly different approach: It copies the file from the js project, rather than directly placing it in the JVM project. Useful if you have multiple JVM projects).
You can configure the Scala.js SBT plugin to output the JavaScript file in folder of your choosing. For example like this:
// configure a specific directory for scalajs output
val scalajsOutputDir = Def.settingKey[File]("directory for javascript files output by scalajs")
// make all JS builds use the output dir defined later
lazy val js2jvmSettings = Seq(fastOptJS, fullOptJS, packageJSDependencies) map { packageJSKey =>
crossTarget in(js, Compile, packageJSKey) := scalajsOutputDir.value
}
// instantiate the JVM project for SBT with some additional settings
lazy val jvm: Project = sharedProject.jvm.settings(js2jvmSettings: _*).settings(
// scala.js output is directed under "web/js" dir in the jvm project
scalajsOutputDir := (classDirectory in Compile).value / "web" / "js",
This will also store -jsdeps.js and .js.map files in the same folder, in case you want to use those in your web app.
For a more complete example, check out this tutorial which addresses many other issues of creating a more complex Scala.js application.

How to configure build.sbt so that xsbt-web-plugin a creates war file without compression?

I am using Scala 2.10.1 with sbt to package my webapp as a war file.
For the purpose of efficient rsync deltas, I'd like to have the war packaged as a .war file, but without zip compression. I just need to know how to configure my build for this.
UPDATE:
All these plugin docs assume all this knowledge of how the syntax works and how to combine tasks into a new task, etc. I can't even tell how to create a new task that does package then command. None of the answers so far have said specifically, "here's what you do.."
Just to be clear, this is all I'm asking for:
I need a Task "packnozip" that does this:
1) run "package"
2) run shell commands:$ mkdir ./Whatever
$ pushd ./Whatever
$ jar xvf ../Whatever.war
$ popd
$ mv ./Whatever.war ./Whatever.war.orig
$ jar cvM0f ./Whatever.war -C ./Whatever .
So what i'm saying is i want to type "packnozip" into the sbt console and have it do #1 then #2.
For now i'm just manually doing #2 which seems silly if it can be automated.
Also watching a 30MB file get completely resent by rsync b/c it is not diffable seems quite silly when a 34MB uncompressed file is only 13% more data, and takes a fraction of second to send b/c of efficient diffs, not to mention "-z" will compress the transfer anyways.
If you have your war file unzipped in a directory you can:
zip -r -0 project.war project/
That should be zero compression. In case you don't see those options, this is my setup:
[node#hip1 dev]$ zip -v
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Which, you could execute as a run task I believe, after the war is packaged.
UPDATE 1
I believe this is the best way to achieve your needs:
http://www.scala-sbt.org/release/docs/Detailed-Topics/Process
val exitcode = "zip -r -0 project.war project/"!
However, if you need to work from a specific directory (Please see Update 2 below):
Modified this to execute within directory but place .war above directory. The path (2nd) argument should include the directory, so that the zip is performed inside of it:
Process("zip" :: "-r" :: "-0" :: "../project.war" :: "." :: Nil, "/path/to/project/") !
Here's another SO question on the ProcessBuilder that may help as well:
How does the “scala.sys.process” from Scala 2.9 work?
(Note: you don't need to import scala.sys.process._)
UPDATE 2
For readers in the future, please note that zipping the project directory itself will not work, one needs to perform the zip of the war inside the directory by using pushd, putting the resulting war outside of the directory as mentioned by the OP in the comments below this answer. As Orange80 mentioned:
pushd ./project && zip -r -0 ../project.war ./ && popd
UPDATE 3
Check out this, it may do exactly what you need, with a 0 for options to specify no compression:
https://github.com/sbt/sbt-onejar
a plugin that lets you create a single executable jar, which, with options (for example "0" as in a command like "jar 0f blah.jar blah/") can be made I think as you mentioned in the comments below to create the jar file without compression.
For usage I found this on SO:
SBT one-jar plugin
And also, if it needs to be modified, it's a pretty reasonable example of a plugin as well, which if you drop it in your home ~/.sbt/plugins it will be global and can be used in your build in the fashion noted in the SO answer above. I hope that helps at least a little bit/
There is no way to do this directly via sbt configuration, since sbt assumes that any files within zip and jar artifacts should be compressed.
One workaround is to unzip and re-zip (without compression) the war file. You can do this by adding the following setting to your project (e.g. in build.sbt):
packageWar in Compile <<= packageWar in Compile map { file =>
println("(Re)packaging with zero compression...")
import java.io.{FileInputStream,FileOutputStream,ByteArrayOutputStream}
import java.util.zip.{CRC32,ZipEntry,ZipInputStream,ZipOutputStream}
val zis = new ZipInputStream(new FileInputStream(file))
val tmp = new File(file.getAbsolutePath + "_decompressed")
val zos = new ZipOutputStream(new FileOutputStream(tmp))
zos.setMethod(ZipOutputStream.STORED)
Iterator.continually(zis.getNextEntry).
takeWhile(ze => ze != null).
foreach { ze =>
val baos = new ByteArrayOutputStream
Iterator.continually(zis.read()).
takeWhile(-1 !=).
foreach(baos.write)
val bytes = baos.toByteArray
ze.setMethod(ZipEntry.STORED)
ze.setSize(baos.size)
ze.setCompressedSize(baos.size)
val crc = new CRC32
crc.update(bytes)
ze.setCrc(crc.getValue)
zos.putNextEntry(ze)
zos.write(bytes)
zos.closeEntry
zis.closeEntry
}
zos.close
zis.close
tmp.renameTo(file)
file
}
Now when you run package in sbt, the final war file will be uncompressed, which you can verify with unzip -vl path/to/package.war.