Why does sbt not choose the most recent dependency version within range? - scala

in a build.sbt we can declare a range for a dependency; for example:
libraryDependencies += groupID %% "mylib" % "[2.0,)"
There are (in my case) multiple packages that fall within this range (I have 2.0.2, 2.0.3, 2.0.4 and even some more exotic versions like 2.0.1.dev and 2.0.3-dev of mylib available in my repository).
When I try to build this project, sbt consequently chooses version 2.0.2 for no apparent reason. I would expect him to take the biggest number within the range, but he never does. Here are the requirements I tried and the version he selected (so only when specifying an exact version number does he select a more recent version):
revision range | version selected
-----------------------------------------
[2.0,) | 2.0.2
[2.0.0,) | 2.0.2
]2.0,) | 2.0.2
2.0.+ | 2.0.2
latest.integration | 2.0.2
2.0.3 | 2.0.3
2.0.4 | 2.0.4
I have already tried the following workarounds:
removing the local ~/.sbt and ~/.ivy2 directories
running sbt -no-share -sbt-dir /tmp/sbt -ivy-dir /tmp/ivy to avoid any caching effect
searching my home directory for any file containing mylib in the name; only found ind ~/.ivy2
I have run most of my tests with sbt 0.13.16, but I have tried with 0.13.15 and 0.13.17
Note that all my builds run each time in a clean clone of my git repository; so no target directory remains.
the versions 2.0.2, 2.0.3 and 2.0.4 of mylib contain the same code and dependencies (I republished them for testing purposes)

The problem (for once) does not seem to reside in sbt.
I tried with sbt 1.2.1 and range 2.1.x. Now sbt complains that he cannot find this dependency; and he lists the available versions:
[warn] ==== my-maven-repo: tried
[warn] http://my-server/nexus/repository/my-repo/eu/company/mylib_2.12/[revision]/mylib_2.12-[revision].pom
[warn] [2.0.2, 2.0.2-dev.20180719.16.55.39.develop.dc3a706, 2.0.1-dev.20180719.16.49.57.develop.dc3a706, 2.0.1-dev.20180719.16.42.31.develop.dc3a706, 2.0.1.dev.20180719.16.31.59.develop.dc3a706]
so strangely he does not list all packages available on my nexus-maven repository.
After deleting one package using the nexus interface; it seems like this list was now complete.
So the real problem seems to be that sbt publish to a nexus repository does not recreate the index in that repository.
Workaround
I have created a custom sbt-publish-wrapper script to force nexus to recreate the repository metadata when publishing a new library:
#!/bin/bash
set -e
# just run sbt publish with the passed arguments as-is
sbt $# publish
# now make sure the repository on nexus is re-indexed
# this triggers a task to recreate metadata for all maven repositories. This task was manually configured in nexus.
# the id was obtained running `curl -v -u admin:***** -X GET http://my-company/nexus/service/rest/v1/tasks`
# the recreate-maven-repos needs only the nx-tasks-run privilege
curl -q -u recreate-maven-repos:recreate -X POST http://my-company/nexus/service/rest/v1/tasks/c42ab5f5-4bd6-4ed3-b2f1-d061c24a9b90/run

Related

Why does sbt keep downloading my snapshot dependencies?

I have an SBT project that depends on two snapshot dependencies. Every time I build it, it goes off to the remote repository to fetch the dependencies. This is true even if I set offline := true.
When I look at how it is trying to resolve the local dependencies, the build is saying it is looking in "local", i.e., ~/.ivy2/local/... -- which is a nonexistent directory.
The jars are in ~/.ivy2/cache/... and this is where SBT downloads them when it pulls the dependencies from the remote server.
I have searched my .sbt and .scala build files and the string "local" does not appear in them in connection with a repository or cache.
SBT is at version 0.13.11 building against scala 2.11.8.
Why is SBT doing this, and how can I get it to see the cached jars?
If you want to prevent SBT from trying to download from official repositories you could simply create a file project/offline-repositories:
[repositories]
mirror-central: file:////nexus/central
mirror-maven-central-org: file:////nexus/maven-central-org
...
(/nexus/central and /nexus/maven-central-org should contain a (partial) mirror of what you need offline)
Then call sbt with the sbt.repository.config property configured:
-Dsbt.override.build.repos=true \
-Dsbt.repository.config=./project/offline-repositories
For Reference:
http://www.scala-sbt.org/0.13/docs/Proxy-Repositories.html
How to prevent SBT from trying to download from official repositories?
EDIT
If you want to use your ~/.m2 cache:
[repositories]
mirror-central: file:////home/XXXXX/.m2/repository
mirror-maven-central-org: file:////home/XXXXX/.m2/repository
...
This apparently is because in my Ivy cache I had a file named ~/.ivy2/cache/com.xxx/xxx-utils/ivy-2.3.2-SNAPSHOT.xml.original , which the build was trying and failing to parse. I'm not sure where this file came from; conceivably it was put there manually ages ago.

Prefetch SBT versions, Scala, and Ivy Resources with SBT for creating an image

I wish to create a docker image with several SBT components built in (docker part not relevant). If I just install sbt from deb or rpm and not do anything, the first time it is invoked it still has to download the internet before it's able to start.
Thus, I need to be able to specify the following:
1 SBT Versions - e.g. 0.12, 0.13.1, 0.13.2, ...
2 Scala Versions - 2.9.1, 2.10.4, 2.11.5, ...
3 Commonly used libraries - play framework, etc.
I'm wondering what's the best way to ensure those are already pre-cached.
Here is what I do in my Dockerfile to prefetch multiple scala versions :
RUN echo 'crossScalaVersions := Seq("2.10.6", "2.11.7")' > build.sbt \
&& echo 'object Hi { def main(args: Array[String]) = println("Done") }' > src.scala \
&& sbt "+run" \
&& rm build.sbt src.scala
You can easily customize this further by specifying the sbt version on cli ( sbt -Dsbt.version=0.x.y "+run" ), or add some libraryDependencies to the build.sbt.
HTH
Well, if you create one project which has everything you want listed as dependency -- including SBT and Scala versions, and then run update to retrieve all these dependencies, they'll all be cached on .ivy2\cache. This should serve for pretty much everything, but SBT itself seems to have tons of separate dependencies as seen here, or on your local disk at the ivy2 cache.

"Error during sbt execution: No Scala version specified or detected" with SBT from Homebrew

I have installed Scala 2.10.3 and SBT 0.13.1. When I execute SBT following Hello, World from the SBT documentation I get the following error message:
"Error during sbt execution: No Scala version specified or detected"
I've tried to add a build.sbt file with the scalaVersion but the error keeps showing up.
Please advise.
I've just downgraded sbt to 0.12.3 and it works ... so what's wrong with 0.13.X ?
Its not an issue with the upgrade of 0.13.* version. Its the repository cache thats causing the issue. There are possibilities where you have broken / no ivys (when only the jar is downloaded through maven repo) for the sbt jars in your cache.
Once upgraded remove your ivy repo cache (or rename the folder) and remove the .sbt/boot folder (removing this will enable to start from the scratch)
Edit your ./sbt/repositories file and provide preference to the ivy repositories. If you are using proxy repositories which includes both ivy and maven style artifacts, the add it as below
[repositories]
local
maven-local
maven-central-ivy: http://<repository url>, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
maven-central: http://<repository url>
This will enable especially (org.scala-sbt/sbt/*) to be download with the Ivys xml from where the version / revision and dependencies are picked up for the builds. If only the jar is downloaded from "maven-central", then the error occurs.
It is the problem of IDE you are using specially the scala plugin in it.
You can get ride of this just go to the folder where the build.sbt is created. Delete the "project" folder and open cmd, run "sbt" command on the folder, a new project folder will be created. Now you can open that in IDE.
It may be that you may find some problem very first in ide, if you build the project once , all the problems will go away.
I did exactly what #Gopi proposed but this is my repositories file content:
[repositories]
local
jcenter: https://jcenter.bintray.com/
maven-central
typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
sbt-ivy-snapshots: https://repo.scala-sbt.org/scalasbt/ivy-snapshots/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
My SBT Version: 1.1.1
java version 1.8
Windows 10
Hope that helps,
BR. Paul
I've also installed both Scala and SBT on my machine through homebrew and can just run the given simple hello example, so i guess you did something wrong :-)
Please provide a bit more diagnostics, for example:
What is the output of both commands 'which sbt' and 'which scala'?
You should see the results as below:
$ which scala
/usr/local/bin/scala
$ which sbt
/usr/local/bin/sbt
$ ls -laF /usr/local/bin/sbt
lrwxr-xr-x 1 rjk admin 28 Feb 9 13:32 /usr/local/bin/sbt# -> ../Cellar/sbt/0.13.1/bin/sbt
$ ls -laF /usr/local/bin/scala
lrwxr-xr-x 1 rjk admin 32 Feb 9 13:38 /usr/local/bin/scala# -> ../Cellar/scala/2.10.3/bin/scala
And given the directory where you ran the example code from, is it completely empty besides the hw.scala example? Or did you also add the build.sbt or project/build.properties? If that's the case it would be wise to also list that here to spot any problems with them.

sbt retrieveManaged doesn't pick up jars produced using publish-local

Suppose I have a project in which I've used sbt publish-local to generate local copies of the jar files in ~/.ivy2/local.
Another sbt project on the same machine can successfully locate these jars to satisfy dependencies. However, if in that project I use the option retrieveManaged := true, sbt never copies the jars out of ~/.iv2/local into the lib_managed directory.
How can I make this happen? (Is this behaviour intended?)
(I'm using sbt 0.12.1.)
I am using the same setup. When I update my locally published dependencies, I remove the managed jars first and then run the update:
$ rm -r lib_managed/jars/my-package-prefix
$ sbt test:compile
(test:compile will figure out that the jars are missing and do the update by itself).
In older versions of sbt I believe there was an issue when using -SNAPSHOT versions, but I haven't seen this problem anymore with sbt 0.12.
In rare situations, something messes up with the Ivy cache and somehow you cannot enforce the updates. This is perhaps the case when corrupt Ivy meta data files have been used. Then the only solution is to wipe all occurrences of your dependency from ~/.ivy2/local and ~/.ivy2/cache, re-publish, and re-update.

how do I get sbt to use a local maven proxy repository (Nexus)?

I've got an sbt (Scala) project that currently pulls artifacts from the web. We'd like to move towards a corporate-standardized Nexus repository that would cache artifacts. From the Nexus documentation, I understand how to do that for Maven projects. But sbt obviously uses a different approach. (I understand Ivy is involved somehow, but I've never used it and don't understand how it works.)
How do I tell sbt and/or the underlying Ivy to use the corporate Nexus repository system for all dependencies? I'd like the answer to use some sort of project-level configuration file, so that new clones of our source repository will automatically use the proxy. (I.e., mucking about with per-user config files in a dot-directory is not viable.)
Thanks!
Step 1: Follow the instructions at Detailed Topics: Proxy Repositories, which I have summarised and added to below:
(If you are using Artifactory, you can skip this step.) Create an entirely separate Maven proxy repository (or group) on your corporate Maven repository, to proxy ivy-style repositories such as these two important ones:
http://repo.typesafe.com/typesafe/ivy-releases/
http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/
This is needed because some repository managers cannot handle Ivy-style and Maven-style repositories being mixed together.
Create a file repositories, listing both your main corporate repository and any extra one that you created in step 1, in the format shown below:
[repositories]
my-maven-proxy-releases: http://repo.example.com/maven-releases/
my-ivy-proxy-releases: http://repo.example.com/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
Either save that file in the .sbt directory inside your home directory, or specify it on the sbt command line:
sbt -Dsbt.repository.config=<path-to-your-repo-file>
Good news for those using older versions of sbt: Even though, in the sbt 0.12.0 launcher jar at least, the boot properties files for older sbt versions don't contain the required line (the one that mentions repository.config), it will still work for those versions of sbt if you edit those files to add the required line, and repackage them into the sbt 0.12.0 launcher jar! This is because the feature is implemented in the launcher, not in sbt itself. And the sbt 0.12.0 launcher is claimed to be able to launch all versions of sbt, right back to 0.7!
Step 2: To make sure external repositories are not being used, remove the default repositories from your resolvers. This can be done in one of two ways:
Add the command line option -Dsbt.override.build.repos=true mentioned on the Detailed Topics page above. This will cause the repositories you specified in the file to override any repositories specified in any of your sbt files. This might only work in sbt 0.12 and above, though - I haven't tried it yet.
Use fullResolvers := Seq( resolver(s) for your corporate maven repositories ) in your build files, instead of resolvers ++= or resolvers := or whatever you used to use.
OK, with some help from Mark Harrah on the sbt mailing list, I have an answer that works.
My build class now looks like the following (plus some other repos):
import sbt._
//By extending DefaultWebProject, we get Jetty support
class OurApplication(info: ProjectInfo) extends DefaultWebProject(info) {
// This skips adding the default repositories and only uses the ones you added
// explicitly. --Mark Harrah
override def repositories = Set("OurNexus" at "http://our.nexus.server:9001/nexus/content/groups/public/")
override def ivyRepositories = Seq(Resolver.defaultLocal(None)) ++ repositories
/* Squeryl */
val squeryl = "org.squeryl" % "squeryl_2.8.0.RC3" % "0.9.4beta5"
/* DATE4J */
val date4j = "hirondelle.date4j" % "date4j" % "1.0" from "http://www.date4j.net/date4j.jar"
// etc
}
Now, if I delete the Squeryl tree from my machine's .ivy2/cache directory, sbt tries to grab it from the Nexus tree with the appropriate URL. Problem solved!
All you need is to define a property file sbt.boot.properties which will allow you to:
redefine the ivy cache location (I need that because it would be otherwise part of our roaming Windows profile, which is severely limited in disk space in our shop. See Issue 74)
define any other Maven repo you want
C:\HOMEWARE\apps\sbt-0.74\sbt.boot.properties
[scala]
version: 2.7.7
# classifiers: sources, javadoc
[app]
org: org.scala-tools.sbt
name: sbt
version: read(sbt.version)
class: sbt.xMain
components: xsbti
cross-versioned: true
classifiers: sources, javadoc
[repositories]
local
my-nexus: http://my.nexus/nexus/content/repositories/scala-tools/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
maven-local
# sbt-db: http://databinder.net/repo/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
# maven-central
# scala-tools-releases
# scala-tools-snapshots
[boot]
directory: project/boot
properties: project/build.properties
prompt-create: Project does not exist, create new project?
prompt-fill: true
quick-option: true
[log]
level: debug
[app-properties]
project.name: quick=set(test), new=prompt(Name)[p], fill=prompt(Name)
project.organization: new=prompt(Organization)[org.vonc]
project.version: quick=set(1.0), new=prompt(Version)[1.0], fill=prompt(Version)[1.0]
build.scala.versions: quick=set(2.8.0.RC2), new=prompt(Scala version)[2.8.0.RC2], fill=prompt(Scala version)[2.8.0.RC2]
sbt.version: quick=set(0.7.4), new=prompt(sbt version)[0.7.4], fill=prompt(sbt version)[0.7.4]
project.scratch: quick=set(true)
project.initialize: quick=set(true), new=set(true)
[ivy]
cache-directory: C:\HOMEWARE\projects\.ivy2\cache
Note: this sbt.boot.properties file is inspired from:
the one mentioned in the "Generalized Launcher" page of the sbt project.
the one found within sbt-0.74 itself!
I have commented any external Maven repository definition, and added a reference to my own Nexus Maven repo.
The launcher may be configured in one of the following ways in increasing order of precedence:
Replace the /sbt/sbt.boot.properties file in the jar.
Put a configuration file named sbt.boot.properties on the classpath. Put it in the classpath root without the /sbt prefix.
Specify the location of an alternate configuration on the command line. This can be done by:
either specifying the location as the system property sbt.boot.properties
or as the first argument to the launcher prefixed by '#'.
The system property has lower precedence.
Resolution of a relative path is:
first attempted against the current working directory,
then against the user's home directory,
and then against the directory containing the launcher jar.
An error is generated if none of these attempts succeed.
Define a sbt.bat wrapper (in order to be sure to specify your sbt.boot.properties) like:
C:\HOMEWARE>more C:\HOMEWARE\bin\sbt.BAT
#echo off
set t=%~dp0
set adp0=%t:C:\="%"
set SBT_DIR=%adp0%..\apps\sbt-0.74
dir C:\%SBT_DIR%\sbt-launch-0.7.4.jar
# if needed, add your proxy settings
set PROXY_OPTIONS=-Dhttp.proxyHost=my.proxy -Dhttp.proxyPort=80xx -Dhttp.proxyUser=auser -Dhttp.proxyPassword=yyyy
set JAVA_OPTIONS=-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -Xmx512M -cp C:\HOMEWARE\apps\sbt-0.74\sbt-launch-0.7.4
set SBT_BOOT_PROPERTIES=-Dsbt.boot.properties="sbt.boot.properties"
cmd /C C:\HOMEWARE\apps\jdk4eclipse\bin\java.exe %PROXY_OPTIONS% %JAVA_OPTIONS% %SBT_BOOT_PROPERTIES% -jar C:\HOMEWARE\apps\sbt-0.74\sbt-launch-0.7.4.jar %*
And your sbt will download artifacts only from:
your Nexus
your local Maven repo.
Just tested at home with an old Nexus opensource 1.6 I had running, java 1.6, sbt07.4
C:\Prog\Java\jdk1.6.0_18\jre\bin\java -Xmx512M -Dsbt.boot.properties=sbt.boot.properties - jar "c:\Prog\Scala\sbt\sbt-launch-0.7.4.jar"
That gives:
[success] Build completed successfully.
C:\Prog\Scala\tests\pp>sbt
Getting Scala 2.8.0 ...
downloading http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.8.0/scala-compiler-2.
8.0.jar ...
[SUCCESSFUL ] org.scala-lang#scala-compiler;2.8.0!scala-compiler.jar (311ms)
downloading http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-library/2.8.0/scala-library-2.8.
0.jar ...
[SUCCESSFUL ] org.scala-lang#scala-library;2.8.0!scala-library.jar (185ms)
:: retrieving :: org.scala-tools.sbt#boot-scala
confs: [default]
2 artifacts copied, 0 already retrieved (14484kB/167ms)
[info] Building project test 0.1 against Scala 2.8.0
[info] using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
If I try a funny value in the sbt.boot.properties file:
C:\Prog\Scala\tests\pp>sbt
Getting Scala 2.9.7 ...
:: problems summary ::
:::: WARNINGS
module not found: org.scala-lang#scala-compiler;2.9.7
==== nexus: tried
http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.9.7/scala-compiler-2.9.7.pom
-- artifact org.scala-lang#scala-compiler;2.9.7!scala-compiler.jar:
http://localhost:8081/nexus/content/repositories/scala/org/scala-lang/scala-compiler/2.9.7/scala-compiler-2.9.7.jar
So it does limit itself to the two repo I defined:
[repositories]
nexus: http://localhost:8081/nexus/content/repositories/scala
nexus2: http://localhost:8081/nexus/content/repositories/scala, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
(I commented everything else: local, maven-local, ...)
If I comment all repositories and put a funny value (2.7.9) for the scala version in the sbt.boot.properties, I do get (like the OP did)
C:\Prog\Scala\tests\pp>sbt
Error during sbt execution: No repositories defined.
If I put 2.7.7 (while still having all repo commented), yes, it won't generate an error:
C:\Prog\Scala\tests\pp>sbt
[info] Building project test 0.1 against Scala 2.8.0
[info] using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
But that's only because it already had downloaded scala2.8.0 during my previous tries.
If I remove that library from my project/boot directory, then it will throw an Exception:
[info] using sbt.DefaultProject with sbt 0.7.4 and Scala 2.7.7
> C:\Prog\Scala\tests\pp>sbt
Error during sbt execution: No repositories defined.
at xsbt.boot.Pre$.error(Pre.scala:18)
at xsbt.boot.Update.addResolvers(Update.scala:197)
...
at xsbt.boot.Boot$.main(Boot.scala:15)
at xsbt.boot.Boot.main(Boot.scala)
Error loading project: Error during sbt execution: No repositories defined.
edit the config file in sbt_home/conf "sbtconfig.txt"
add two line
-Dsbt.override.build.repos=true
-Dsbt.repository.config="C:/Program Files (x86)/sbt/conf/repo.properties"
the repo.properties content is
[repositories]
local
public: http://222.vvfox.com/public <-fix this ,write your local nexus group url
Well this has bugged me for a while so I found a guy that has written an SBT plugin for maven out on github called maven-sbt so all you have to do is include it in your plugins project and make your project mixin with maven.MavenDependencies and all your operations like update and publish-local work with your local maven. The nice thing about that is if you are like me, your org is all maven. So, all you libs are in you local maven repo but if for some reason you build with sbt first, then you start getting a bunch or jars in ivy too. What a waste of space, and time since you will still need to get them for your maven builds.
That said, I wish this were built into sbt so I would not need to add it to every project. Maybe as a processor at least. He mentioned in one thing I read that he would like to add it to 0.9 but I have not been able to find it.
I got this error because I had a blank file in ~/.sbt/repositories. Both adding repositories to the file and removing the file solved the problem.