I'd like to use SBT for a build structured around a single Git repository with tens of projects. I'd like to have the following possibilities from the build:
One-click build/test/release of all projects in the build.
Define and apply common settings for all the projects in the build.
Define project-specific settings in the project sub-directories, keeping the root of the build clean.
Project should be able depend on each other, but...
Most of the projects will not depend on each other in the classpath sense.
Some of the projects will be the SBT plugins that should be included into other projects (but not to all of them).
Now with these requirements in mind, what should be the structure of the build? Because of requirement 3, I can't just go with a single build.sbt in a root of a build, because I don't want to put all the projects settings there, since it will be a lot of text, and every change in a single project will be reflected on the top level.
I've also heard the usage of *.sbt files both for root project and sub-projects is error-prone and not generally recommended (Producing no artifact for root project with package under multi-project build in SBT, or How can I use an sbt plugin as a dependency in a multi-project build?, SBT: plugins.sbt in subproject is ignored? etc.). I've tried only simple multi-projects builds with *.sbt files on different levels, and it just worked. Which pitfalls do I need to keep in mind if I'll go for a multi *.sbt files approach, given the requirements above?
Ok, since no one have posted anything so far, I've figured I post what I've learned from doing mono repository with SBT so far.
Some facts first: our SBT build consists of 40+ projects, with 10+ common projects (in the sense that other projects depend on them), and 5 groups of projects related to a single product (5-7 projects in each group). In each group, there's typically one group-common project.
Build organization
We have the following build structure:
One main build.sbt for the whole build.
One build.sbt per project.
Several local SBT plugins in project directory.
Let's talk about each of these items.
1. Main build.sbt
In this file, the general build structure is defined. Namely, we keep cross-project dependencies in there. We don't use the standard commonSettings approach, as in:
val commonSettings = Seq(scalaVersion := "2.12.3", ...)
...
val proj1 = (project in file("p1")).settings(commonSettings)
val proj2 = (project in file("p2")).settings(commonSettings)
...
This would be too wordy and easy to get wrong for a new project. Instead we use a local SBT project that automatically applies to every project in the build (more on this later).
2. Per-project build.sbt
In those files, we generally define all the project settings (non-auto plugins, library dependencies, etc.). We don't define cross-project dependencies in these files, because this doesn't really work as expected. SBT loads all the *.sbt files in certain order, and project definition in every build overrides the previously found ones. In other words, if you avoid (re-)defining projects in per-project *.sbt files, things will work well. All the other settings can be kept there, to avoid too much clutter in main build.sbt.
3. Local SBT plugins
We use a trick to define SBT auto-plugin in <root_dir>/project/ directory, and make them load automatically for all the projects in the build. We use those plugins to automatically define the settings and tasks for all the projects (for things like Scalastyle, scalafmt, Sonar, deployment, etc.). We also keep common settings there (scalaVersion, etc.). Another thing we keep in <root_dir>/project/ is common dependencies versions (not in a plugin, just pure *.scala file).
Summary
Using SBT for a mono repository seems to work, and has certain advantages and disadvantages.
Advantages: It's super easy to re-use the code between products. Also common SBT stuff like Scalastyle, scalafmt, etc. is defined once, and all new projects get it for free. Upgrading a dependency version is done in one place for all the projects, so when someone upgrades the version, he or she does this for all the projects at once, so that different teams benefit from that. This requires certain discipline between teams, but it worked for us so far.
Another advantage is use of common tooling. We have a Gerrit+Jenkins continuous integration, and we have a single job for e.g. pre-submit verification. New projects get a lot of this machinery pretty much for free, again.
Disadvantages: For one, the build load time. On top 13" MacBook Pro it can easily last 30+ seconds (this is time from starting SBT to getting to SBT's command prompt). This is not that bad if you can keep the SBT constantly running though. It's much worse for Intellij refreshing the build information, where it can take around 15 minutes. I don't know why it takes so much longer than in SBT, but here's that. Can be mitigated by avoiding refreshing the Intellij unless absolutely necessary, but it's a real pain.
Yet another problem is that you can't load an individual project or group of projects into Intellij IDEA. You are forced to load the build of whole mono repository. If that would be possible, then, I guess, Intellij's situation could have been better.
Another disadvantage is the fact that one can't use different versions of same SBT plugin for different projects. When one project can't be upgraded to a new plugin version for some reason, the whole repository has to wait. Sometimes this is useful, that is, it expedites maintenance work, and forces us to keep projects in maintenance mode up to date. But sometimes for legacy projects it can be challenging.
Conclusion
All in all, we have worked for a around a year in this mode, and we intend to keep doing so in the foreseeable future. What concerns us is the long Intellij IDEA refresh time, and it only gets worse as we add more projects into this build. We might evaluate alternative build systems later that would avoid us loading projects in isolation to help with Intellij performance, but SBT seems to be up to a task.
Here is an update on this discussion 5 years later.
For those who understand french, there is this presentation from ScalaIO 2019:
François Sarradin: SBT monorepo et livraison, https://www.youtube.com/watch?v=nT8YhC5iRco
There is also this video that is not specific to Sbt but give good tips that was recommanded in that gitter discussion: https://gitter.im/sbt/sbt?at=63584746f00b697fec5b7184
Gil Tayar: FOUR PILLARS AND A BASE: THE NUTS AND BOLTS OF A MONOREPO jsday 2020, https://www.youtube.com/watch?v=cIGFyv1KuGI
Related
I used to package my various Eclipse RCP products with PDE, for years.
With my latest upgrade attempt to Eclipse Oxygen, I got some new strange resolution errors which I could not solve, and I decided it really was time to give Tycho a try. I followed the excellent article about Tycho by Lars Vogel, and after a bit of tweaking, it worked well (and I was not stucked by the same resolution errors as in PDE! Yay!).
But indeed it was a simple test: I created a folder for my bundles, another for my features, created my poms, and so on. Now I look at the degree of automating in my PDE and find quite a huge gap.
In PDE, there is a build.properties where you give your master feature file and a map file, and the process will, seemingly:
parse the master feature
parse the features in it (recursively)
parse the plugins in them
find in the map file the plugins to be packaged (the other dependencies are supposed to be in the target platform)
download the relevant git repos
move the relevant plugins/features to the working directories
launch compile, p2 and so on
(note : the git part needs you provide the egit fetchfactory)
Now in Tycho, I have to create poms, but it is not the problem. I have to create some master poms, and for the individual plugin poms, I have either the pomless option, or the pom generator. The pom generator also seems to have the advantage of creating the parent pom which contains all the plugins as modules. So far so good.
But I have to fill the features and plugins folders, and I'm stuck here.
I do not have PSFs for my products, because I never needed it: in PDE, map + product definition does the trick.
Does it mean I will have to maintain PSFs from now on or is there another tycho solution I did not find? (Tycho doc is quite scarce, in my opinion). Maintaining PSFs seems redundant to me because I have product and map, and also because I have many products, many plugins, and many of which are common to several products.
(Indeed, a basic solution would be to take the git repositories mentioned in the map file, dump them all and launch tycho. Tycho would compile all the plugins, and then the p2 part would package only the product-relevant ones. The problem is that I have plenty of different products that rely on plenty of different repositories. And even in a given git repo, I have plugins that may or may not be relevant for a given product. Thus, I would compile hundreds of useless plugins in the process.)
My need is to copy in the tycho folders only the plugins and features which are referenced in my product and which are not already in my target platform. Generating a PSF from my product and my map would be shifting the problem.
Indeed, I can code this, and I will if needed.
But given that all this is already automated in PDE, is there at least some parts of the process that could be automated with some tycho plugins I did not uncover?
After some time of digging, here is the solution I finally chose.
In order to fetch the relevant features and plugins, I used... PDE ! I digged in PDE and found the various steps in its process. The first one is to fetch (it is an ant task named eclipse.fetch). I externalized this part, and my script launches it, then generates the master poms by scanning the fetched features names and fecthed plugins names, then adds the other tycho confugurations and then launches tycho.
In the end, granted, it is not a full tycho solution, it is a hybrid one PDE + Tycho. But it works like a charm, and the build/package process is Tycho, only the initial fetch is delegated to PDE. (anyway, PDE build/package process does not work in my case, as stated initially)
I have a setup with 13 different eclipse projects (mostly scala and java). All projects have dependencies on each other in different ways. Now the project is starting to get big so we want to transition to a build tool and I wanted to try SBT.
First question: Is there any way to export the build files from eclipse? I mean, I have everything working in eclipse so It feels like an "export build.sbt" would be possible.
Second question: I have not found any easy way to add the project dependencies in a sbt file. Some sites say that I should publish all projects to a local maven repo and then using dependencies to be able to build it, but that requirement seems a little extreme.
I found my answers by a friendly person on the #sbt irc-channel.
For the first question: No, there seems to be none at the moment.
For the second qestion: I should create a multi-project build and define dependencies between projects that way (following the guide at: http://www.scala-sbt.org/release/docs/Getting-Started/Multi-Project.html)
How to initialize a new Scala project in sbt, Eclipse and github, so that it all plays together...
A new Scala project typically requires being set up for sbt, eclipse (if you so choose) and github such that it all works together. After investing some time on this setup, it may help to have this list for aligning these 3 tools/services, for as long as simpler ways are not available. The series of steps that works for me follows. It assumes you have the Scala IDE plugin installed in eclipse.
Create a new repo in Github.
Decide a directory location for the new project
In eclipse, use the Git Repositories View to import the Github repo into that location. Alternatively you can use command line git for that.
Locate to that same location you've chosen for the project and run sbt eclipse. This makes sure eclipse will be able to handle the sbt project structure, so that your project can be built by sbt while also being intelligible for eclipse. If sbt eclipse doesn't work, the sbt eclipse plugin is probably not installed in sbt - install it.
In eclipse, use File --> Import --> General --> Existing Projects into Workspace, selecting that same location, so that eclipse builds its project structure for the file structure having just been prepared by sbt.
Make git ignore all but the core of your new project by updating the .gitignore file to ignore eclipse and sbt files. The following seems to be currently fine.
*.class
*.log
# sbt specific
dist/*
target/
lib_managed/
src_managed/
project/boot/
project/plugins/project/
# Scala-IDE specific
.scala_dependencies
# Eclipse specific
.project
.classpath
.cache
You should now be able to run the project in eclipse, and in sbt, and commit and push code changes through git. To see the empty project run, which may very well make sense at this stage, you can add a scala class in eclipse to it, containing merely the following code. Note that scala sources should typically sit under src/main/scala. If this path doesn't exist yet, create it through e.g. mkdir -p src/main/scala on Unix.
object hello {
def main(args: Array[String]) {
println("Main starting")
}
}
Or alternatively only this code:
object app extends App {
println("Application starting")
}
It should work now.
Need to disclaim that future versions of eclipse, sbt, etc may render this outdated.
If this is dead wrong in your environment, you can add a better answer.
Previous answer/s is/are of high importance as well as the question since these tools are not self explanatory. Based on personal experience with this challenge which seemed impossible but when one acknowledges few key insurance policy reference points things will become reality:
Implementation of development environment may require radical
re-installation as new tool is adopted (if you adopt GIT after you
already had SBT in place you may need to empty SBT's working
directory as GIT require the cloning process to have empty house on
local working directory)
Keep really well updated backups especially when considering
re-setting any of this "domino" environment layers as some of the GUI
or command initiated cleansing operations truly take care of that
sometimes with surprising effectiveness and without any regard to
other tools / layers requirements
Keep really good documentation (including previous pragmatic answer) on key stages to
make things a reality (again)
I work for a small company of software consultants, that does development primarily (currently: solely) for the JVM using Eclipse as our IDE of choice.
Until now, we have not been using any Project Management/Build Tools. We have used a company SVN repository for version control of our projects along with any binary dependencies (well JAR dependencies) of same. This has gotten a bit messy, and we have a lot of extraneous *.jar files in lib folders across our projects.
With our recent shift to Scala as our PL of choice, we have also looked into using SBT as build tool. We have set up a Sonatype Nexus repository, and have begun declaring SBT files for our projects
The thing is, that we have a set of utility projects that are used across a lot of projects for our customers. Until now, we have managed inter-project dependencies by having an Eclipse workspace for each customer project, with each of these having a local version of the utility projects on which it has dependencies. Keeping local versions up to date has therefore been handled solely through SVN commits and updates.
I guess that there are a couple of different ways that we could go about structuring our projects and workspaces. Using SBT to define inter-project dependencies, it would make sense to have a workspace for the utility projects alone, and then importing snapshots/releases as managed dependencies in our customer projects. However, since we often make enhancements to the utility projects during development of the customer projects, it would be nice to still be able to do this in the same workspace and as much on the fly as possible.
From: http://www.devdaily.com/scala/using-github-projects-scala-library-dependencies-sbt-sbteclipse it seems that a plugin exists for Git that grabs current versions from a Git repository, builds the source and includes them in the Eclipse classpath. This sort of setup would allow us to continue to use the revision control system as the main means of distributing project updates, though I have not been able to find something similar for SVN...
Is anyone using SBT with SVN and eclipse, and/or knows of a smart way to setup interproject dependencies without firing a lot of publish/publish-locals alongside the SVN commands?
It would be nice to somehow declare interproject dependencies using SBT, since this would allow us to dump the responsibility for transient managed dependencies on SBT...
Here is the situation. A development team has a large number (hundreds) of Eclipse projects. The code is very much in churn - new projects are being created; projects are being renamed and project dependencies are constantly changing. The external build system is ant. It is proving extremely challenging to keep the dependencies defined in the ant build files in sync with the state of the world in Eclipse. The external ant build needs constant changes to keep up. For various reasons, using ant as the default builder in Eclipse is not an option. The developers want to continue using Eclipse as the build and edit environment for local use.
Question: Is there a tool which will allow a single set of dependencies to be maintained which can be used by Eclipse as well as an external build system like ant?
I have heard of Gradle but never used it before. Would it make sense in this context? I am pretty sure Maven wouldnt work for what is needed
The typical workflow should be:
1. Developers continue working as they currently do - creating and changing Eclipse project dependencies at will and using the default Eclipse builder to compile and test locally.
2. Some mechanism exists by which these dependencies can be carried into an external build system like ant and an external continuous build triggered on every checkin.
Appreciate your feedback - thanks!
We have been quite successful at using Gradle to tackle a similar problem. Here's the outline of the setup
Each project contains a build.gradle that defines project specific dependencies and tasks (may even be empty).
A special master project contains build.gradle that sets up common dependencies and tasks for child projects, and/or injects settings pertinent to a group of child projects.
Logically master project is the parent project, but it exists as a sibling folder so that Eclipse can be more comfortable with it.
Gradle contains a built-in Eclipse plugin which allows generation of Eclipse settings files for each of the projects from the dependencies information (including inter-project dependencies). It works nicely for simple projects, and as for more complicated ones Gradle allows you to tinker with the settings files, so you can do pretty much everything. From here you have two options:
Not to store Eclipse settings file in the repository and call the generation task every time you do a fresh check-out (I prefer this option).
Tell Gradle to use custom variables to make it generate generic settings files which can be checked-in to the repository. You'll then only need to run the generation task when dependencies or other configuration changes.
(Optional) It's a little tricky, but you can make Gradle parse existing project ivy.xml files and set up dependencies from there. I had some success with this, although I would recommend converting dependencies into Gradle format for more flexibility.
Continuous build system integrate with Gradle very well (same as ant). If you are using Jenkins (Hudson) there is a Gradle plugin.
The advantage of using Gradle is that it scales pretty well, and you can support other IDEs like IntelliJ or Netbeans at the same time without much effort (unless you have lots of crazy custom settings). An advantage and a disadvantage is that Gradle is a powerful build system which requires learning Groovy and Gradle DSL which may take some time to acquire. Also the documentation is awesome.
Gradle has a very active community with the sole purpose of tackling exactly this kind of problem.
Hope this helps, and best of luck!
How about parsing the .classpath files, generate a dependency tree and start building from the root. What you need is a convention on the layout of your projects or an generic (ant-) buildfile that could be changed in each project, if needed (e.g. different project layouts). I´m not sure if Eclipse Tycho could be used for that, since it´s a maven plugin(s) to build eclipse plugins or projects. But it´s able to resolve the bundle and project dependencies against maven repositories and eclipse update sites.