When deploying a Scala application, we use SBT on Jenkins. Currently our build action is specified as clean assembly (using Assembly plugin to produce fat JARs). Our build currently takes between 2-3 minutes, which is sensible, but as the project will become larger and deployments for frequent, it might become a bottleneck.
I remember when doing C++ deployment with Visual Studio, clean (Rebuild All) was necessary, otherwise builds were sometimes (say 0.1%) broken (most likely because build missed some changed dependency in headers).
Is this a concern with SBT? Is clean considered a necessary practice to get reliable builds?
My experience is that sometimes SBT gets mixed-up, the most common thing I've seen is that it can't find classes that are part of the project (and not compiled this time around). I haven't had the inclination to really debug it since doing a clean fixes it every time, but for a CI server I would go for clean every time.
Related
I'd like to use SBT for a build structured around a single Git repository with tens of projects. I'd like to have the following possibilities from the build:
One-click build/test/release of all projects in the build.
Define and apply common settings for all the projects in the build.
Define project-specific settings in the project sub-directories, keeping the root of the build clean.
Project should be able depend on each other, but...
Most of the projects will not depend on each other in the classpath sense.
Some of the projects will be the SBT plugins that should be included into other projects (but not to all of them).
Now with these requirements in mind, what should be the structure of the build? Because of requirement 3, I can't just go with a single build.sbt in a root of a build, because I don't want to put all the projects settings there, since it will be a lot of text, and every change in a single project will be reflected on the top level.
I've also heard the usage of *.sbt files both for root project and sub-projects is error-prone and not generally recommended (Producing no artifact for root project with package under multi-project build in SBT, or How can I use an sbt plugin as a dependency in a multi-project build?, SBT: plugins.sbt in subproject is ignored? etc.). I've tried only simple multi-projects builds with *.sbt files on different levels, and it just worked. Which pitfalls do I need to keep in mind if I'll go for a multi *.sbt files approach, given the requirements above?
Ok, since no one have posted anything so far, I've figured I post what I've learned from doing mono repository with SBT so far.
Some facts first: our SBT build consists of 40+ projects, with 10+ common projects (in the sense that other projects depend on them), and 5 groups of projects related to a single product (5-7 projects in each group). In each group, there's typically one group-common project.
Build organization
We have the following build structure:
One main build.sbt for the whole build.
One build.sbt per project.
Several local SBT plugins in project directory.
Let's talk about each of these items.
1. Main build.sbt
In this file, the general build structure is defined. Namely, we keep cross-project dependencies in there. We don't use the standard commonSettings approach, as in:
val commonSettings = Seq(scalaVersion := "2.12.3", ...)
...
val proj1 = (project in file("p1")).settings(commonSettings)
val proj2 = (project in file("p2")).settings(commonSettings)
...
This would be too wordy and easy to get wrong for a new project. Instead we use a local SBT project that automatically applies to every project in the build (more on this later).
2. Per-project build.sbt
In those files, we generally define all the project settings (non-auto plugins, library dependencies, etc.). We don't define cross-project dependencies in these files, because this doesn't really work as expected. SBT loads all the *.sbt files in certain order, and project definition in every build overrides the previously found ones. In other words, if you avoid (re-)defining projects in per-project *.sbt files, things will work well. All the other settings can be kept there, to avoid too much clutter in main build.sbt.
3. Local SBT plugins
We use a trick to define SBT auto-plugin in <root_dir>/project/ directory, and make them load automatically for all the projects in the build. We use those plugins to automatically define the settings and tasks for all the projects (for things like Scalastyle, scalafmt, Sonar, deployment, etc.). We also keep common settings there (scalaVersion, etc.). Another thing we keep in <root_dir>/project/ is common dependencies versions (not in a plugin, just pure *.scala file).
Summary
Using SBT for a mono repository seems to work, and has certain advantages and disadvantages.
Advantages: It's super easy to re-use the code between products. Also common SBT stuff like Scalastyle, scalafmt, etc. is defined once, and all new projects get it for free. Upgrading a dependency version is done in one place for all the projects, so when someone upgrades the version, he or she does this for all the projects at once, so that different teams benefit from that. This requires certain discipline between teams, but it worked for us so far.
Another advantage is use of common tooling. We have a Gerrit+Jenkins continuous integration, and we have a single job for e.g. pre-submit verification. New projects get a lot of this machinery pretty much for free, again.
Disadvantages: For one, the build load time. On top 13" MacBook Pro it can easily last 30+ seconds (this is time from starting SBT to getting to SBT's command prompt). This is not that bad if you can keep the SBT constantly running though. It's much worse for Intellij refreshing the build information, where it can take around 15 minutes. I don't know why it takes so much longer than in SBT, but here's that. Can be mitigated by avoiding refreshing the Intellij unless absolutely necessary, but it's a real pain.
Yet another problem is that you can't load an individual project or group of projects into Intellij IDEA. You are forced to load the build of whole mono repository. If that would be possible, then, I guess, Intellij's situation could have been better.
Another disadvantage is the fact that one can't use different versions of same SBT plugin for different projects. When one project can't be upgraded to a new plugin version for some reason, the whole repository has to wait. Sometimes this is useful, that is, it expedites maintenance work, and forces us to keep projects in maintenance mode up to date. But sometimes for legacy projects it can be challenging.
Conclusion
All in all, we have worked for a around a year in this mode, and we intend to keep doing so in the foreseeable future. What concerns us is the long Intellij IDEA refresh time, and it only gets worse as we add more projects into this build. We might evaluate alternative build systems later that would avoid us loading projects in isolation to help with Intellij performance, but SBT seems to be up to a task.
Here is an update on this discussion 5 years later.
For those who understand french, there is this presentation from ScalaIO 2019:
François Sarradin: SBT monorepo et livraison, https://www.youtube.com/watch?v=nT8YhC5iRco
There is also this video that is not specific to Sbt but give good tips that was recommanded in that gitter discussion: https://gitter.im/sbt/sbt?at=63584746f00b697fec5b7184
Gil Tayar: FOUR PILLARS AND A BASE: THE NUTS AND BOLTS OF A MONOREPO jsday 2020, https://www.youtube.com/watch?v=cIGFyv1KuGI
Currently all our project builds with Maven on Windows. We were not successful with making incrementally compiled code to work in run time (50% of the cases it was failing with some kind of error), so to benefit from warm compiler and (maybe?) properly working incremental compilation we think about moving to SBT. However currently I have only one sprint to work on it, and I'm afraid to put all the eggs in a single basket and try to migrate whole project in a sprint. I need to find a way to make this change gradual, so I could advance one module at a time. So here are the main questions:
How can I include SBT modules in Maven build (or maybe vise versa, having my "parent" in SBT, yet part of the modules still building with Maven)?
How can we still benefit from IDE support (currently IntelliJ 13), like updated indices on changes in pom / Build.scala, task & goals invocation and so on?
Any advises on subject are highly appreciated.
Eventually we did the switch and don't regret it. Writing SBT tasks is easier, because it's plain Scala. Incremental compilation works now (used to fail in Maven with java.lang.InternalError: Enclosing method not found when deployed to JBoss) and build time is significantly faster. Unfortunately we did not find a way to make a gradual switch, so we had to take the risk. Incremental compiled jars still didn't work, yet Typesafe are about to fix this issue in 2.11.6
i am looking to make our deployments here not suck and i need some help, if you can help me with these few things i owe you beer
right now whenever i make a change thats not to the jsps i need to clean-including-tomcat otherwise my change doesnt take. this is really annoying.
any clues as to what i can change to make it work?
my current build is really simple, just the regular old, javac, war, deploy
one thing that isnt done is that there is no build dir, the project itself contains a web-inf and the javac is done in place, then the war excludes all the .java resources and wars the project.
edit:
I am looking to fix this problem with least amount of effort - so while switching to maven and learning how to use it might solve this problem, but it will create another problem ;)
You've already identified some of the weaknesses, in your current build.
The easiest way that I can suggest to clean it up would be to start with the directory structure.
I highly recommend using the maven directory structure, I would go further to suggest using maven as a build tool instead of ant, however for some folk that remains open for debate.
The maven directory structure has been well thought out, I really like working on projects that use the maven directory structure, because they follow a convention that allows me to save a lot of time, by knowing from previous experience where to find the application components
java source
unit test source
resources etc.
Also by following the convention, the maven plugins work with less configuration required.
Another useful advantage that I get from working on maven based projects is good code metrics, to measure the health of the application. There are various report available as maven plugins, which will give you new insight into your codebase, including:
checkstyle
pmd
findbugs
and more.
Created a build directory where everything got copied before build
Added some flags to not copy over things that rarely change, like images (also to not remove them on clean)
Started using ant-reload task after deploying code
Now i don't need to restart tomcat on every build, and build takes much less time.
My goal is to reduce Scala compilation times. I'm using Intellij IDEA 10 with Scala 2.8.1-RC1.
I've read that using SBT will reduce compilation times because it is clever about re-building only the files/classes that have changed since the last build.
I understand that JRebel has a very similar purpose. Is there any point in using JRebel instead of, or in addition to, SBT?
They do different things. SBT has a pretty speedy continuous incremental build system. JRebel dynamically reloads classes into a running program as they are rebuilt. It's something like a Java debugger's ability to modify a running program but with far fewer annoying limitations. SBT and JRebel are complementary.
You might also say that SBT obviates Hudson or other CI tools. Or that it obviates the red squiggly lines your IDE generates when you enter code that won't compile. But those tools are still useful even with SBT, and SBT offers lots of great features beyond CI and hot deployment.
It does obviate Maven.
SBT overlaps with JRebel for me a bit - like when I use SBT to continuously compile and redeploy a web application to jetty (~prepare-webapp). That feels quite a lot like using JRebel to continuously push changes out to a Java application container.
It offers "poor mans continuous integration" with quick code change detection and testing: http://devblog.point2.com/2009/07/27/scala-continuous-testing-with-sbt/
In general, SBT is the make/ant/maven replacement you should use for Scala. I'm constantly impressed with how it streamlines development, and I miss it when I go back to Java/Maven (even with JRebel). You should use it regardless of what other tools and frameworks you find useful.
Hope that helps :)
What is the difference between a Full Build, an Incremental Build and Deploy within Powerbuilder? The only difference I can see is that Deploy allows you to update the PBDs for, well, deployment.
Full Build regenerates every object. Incremental Build regenerates changed objects and objects that reference changed objects. (Reality is that incremental is a little more complex than that, but that is the intended idea.) Deploy launches whichever project is selected in the properties for the target, so it can do an incremental or full build, depending on what is defined in the project, plus generate whatever executables are defined by the project.
Good luck,
Terry.
I set my projects to Full build. It seems to be more reliable than incremental. Full build is very fast even on LARGE projects anyways. There is not a lot of time savings using incremental build.
Deploy is used when you want to build the project AND run it immediately after the build.