Background
I'm working on a project where we host a Java software and an R companion package on Github. The R package can be easily installed directly from source using devtools::install_github() but the Jar file has to be built first.
As the R package depends on functions in the Jar file, this creates a problem since Jar and R package are often out of sync and users can't utilize newer functions until we make a new release. Since the changes in the Jar are often tiny it doesn't always make sense to create a new release though.
Ideal solution (?)
There are two approaches which I could think of but I'm not quite sure which makes more sense and would have to spend some more time figuring each one out:
Write a function in R which downloads the source from GitHub and builds the Jar via something like system("jar cfe myJar.jar myClass myClass.class". Ideally, there is a package for this but I couldn't find one. I would also first have to figure out how to download the source files though.
Implement an automatic deploy that uploads the Jar to the same release after every commit. I saw that Travis CI can build the Jar file but it creates a new release for every commit or only from tagged commits, which is not ideal. It would be better if we could have one release which is overwritten with the most current Jar file on every commit.
Does anybody have experience with this/can point me to a repository where something like this is implemented/can show me a better solution?
Related
We are building all the solutions to a shared bin directory. Having different projects reference different versions of the same dependency is not healthy for our build.
So, we consolidated the dependencies - great. But now the versions start to drift again. We do not want to consolidate them manually every now and then - we want to prevent the drift completely.
Why we do not want to use Paket? The main reason is that it seems we would lose the ability to migrate the NuGet package dependencies to the new PackageReference items in the projects. So, currently we have package.config files, but we plan to replace them with the respective PackageReferences. That means we will use internal NuGet support by msbuild, which seems to leave no place for Paket.
Now, I assume we are not unique in this world and others have the same problem as we are. How do you solve it?
EDIT 1
We have our internal NuGet repo, but we use it for dependencies which do not have organic representation in Nuget.org and for sharing our own internal packages.
One approach is to consume only from the internal NuGet repository. This has challenges, like:
Who uploads the dependencies there? Developers? But then how to make sure they do not upload different versions of the same dependency? Dedicated people? Then they become a bottleneck.
Small thing, but we need to block commits to the central NuGet.config
Uploading a dependency to the internal NuGet repo is not immediate. You cannot just download it from NuGet.org and upload to the internal one, because that would miss any transitive dependencies. So, a process should be built around it.
It is all possible, but I am reluctant to go down that route ... Must be a better way.
EDIT 2
While we do plan to migrate to PackageReference, it will take time. And unfortunately as long as we have Silverlight (another year, at least) a whole bunch of projects in the dedicated Silverlight solution (80+) will not be migrated to PackageReference, because by doing so it becomes impossible to debug the code with VS 2015.
Next, suppose we do migrate ALL the projects and then externalize all the PackageReference items to a single targets file imported by all the projects. This is feasible when using a shared bin directory as we plan to do. But when inspected in VS 2017 this setup communicates a wrongful picture that every single project depends on the entire set of NuGet dependencies.
I would rather avoid this.
Once you move to PackageReference, you can take advantage of MSBuild. For example, you can have a MSBuild file that contains all your dependency versions. It could be a file that you need to <Import ... /> in all your csproj files, or you could use Directory.Build.props. Finally, in each of your projects, change the version number in any <PackageReference to a MSBuild variable that uses the property you previously defined. Most of Microsoft's open source repositories use this technique, with minor variations about file names and whether it's imported automatically with Directory.Build.props, or an explicit <Import ... />.
While you can still use the Package Manager UI in Visual Studio to check for updates, you won't be able to update the package versions with it (at least, it won't preserve how and where the versions are defined). However, just make sure your MSBuild file that defines the versions is in your solution, so you can trivially open the file in Solution Explorer and then type the new version number in. Adding new package references is slightly more effort, but it's generally not done often, and it's still very easy with SDK-style projects, since Visual Studio lets you edit the csproj while the project is still loaded.
Since you didn't accept the other solution, maybe you could take a look at paket
It's a package manager for dotnet than (among other feature) holds solution wide dependency lock file. It is very customizable, and while it solves LOTS of problems, as any tool, creates some new ones. In my experience, the new ones are far less infuriating :)
I want to be able to compile my project once and pass it through multiple build steps on a CI server. But SBT puts files in a staging area like the one below.
/home/vagrant/.sbt/0.13/staging/
This means the project is not stand-alone and for every CI step it is going to compile it again.
How can I tell SBT to keep things simple and stand-alone and to make sure everything it needs is inside the project directory?
FYI, the staging area is used for the target files when the source folder is not read/write. Making the source folder read/write should fix this.
If you pass -Dsbt.global.staging=./.staging to sbt when starting it up, the staging directory will be .staging in the project's directory.
I figured that out by looking at the sbt source and patching that together with how Paul P's sbt runner passes the value for the sbt boot path.
If that doesn't accomplish what you want, then you might be able to make something go with a custom resolver. The sbt Build Loaders page talks about creating a custom resolver that lets you specify more detail about where dependencies are written. If my solution doesn't get you what you want, you'd probably need to do something like that.
From Gradle's documentation:
The scripts generated by this task are intended to be committed to
your version control system. This task also generates a small
gradle-wrapper.jar bootstrap JAR file and properties file which should
also be committed to your VCS. The scripts delegates to this JAR.
From: What should NOT be under source control?
I think generated files should not be in the VCS.
When are gradlew and gradle/gradle-wrapper.jar needed?
Why not store a gradle version in the build.gradle file?
Because the whole point of the gradle wrapper is to be able, without having ever installed gradle, and without even knowing how it works, where to download it from, which version, to clone the project from the VCS, to execute the gradlew script it contains, and to build the project without any additional step.
If all you had was a gradle version number in a build.gradle file, you would need a README explaining everyone that gradle version X must be downloaded from URL Y and installed, and you would have to do it every time the version is incremented.
Because the whole point of the Gradle wrapper is to be able, without having ever installed Gradle
Same argument goes for the JDK, do you want to commit that also? Do you also commit all your dependency libraries?
The dependencies should be upgraded continuously as new versions are released to get security and other bug fixes, and because if you get to far behind it can be a very time consuming task to get up to date again.
If the Gradle wrapper is incremented for every new release, and it is committed, the repo will grow very large. The problem is obvious when working with distributed VCS where a clone will download all versions of everything.
and without even knowing how it works
Create a build script that downloads the wrapper and uses it to build. Everyone does not need to know how the script works, they need to agree that the project is build by executing it.
where to download it from, which version
task wrapper(type: Wrapper) {
gradleVersion = 'X.X'
}
for Gradle version >= 5:
wrapper {
gradleVersion = 'X.X'
}
and then
gradle wrapper
to download the correct version.
to clone the project from the VCS, to execute the gradlew script it contains, and to build the project without any additional step.
Solved by the steps above. Downloading the Gradle wrapper is not different from downloading any other dependency. The script could be smart enough to check for any current Gradle wrapper and only download it if there is a new version.
If the developer has never used Gradle before and maybe doesn't know the project is built with Gradle, then it is more obvious to run a build.sh compared to running gradlew build.
If all you had was a gradle version number in a build.gradle file, you would need a README explaining everyone that gradle version X must be downloaded from URL Y an installed,
No, you would not need a README. You could have one, but we are developers and we should automate as much as possible. Creating a script is better.
and you would have to do it every time the version is incremented.
If the developers agree that the correct process is to:
Clone repo
Run build script
Then there upgrading to latest Gradle wrapper is no problem. If the version is incremented since last run, the script could download the new version.
I would like to recommend a simple approach.
In your project's README, document that an installation step is required, namely:
gradle wrapper --gradle-version 3.3
This works with Gradle 2.4 or higher. This creates a wrapper without requiring a dedicated task to be added to "build.gradle".
With this option, ignore (do not check in) these files/folders for version control:
./gradle
gradlew
gradlew.bat
The key benefit is that you don't have to check-in a downloaded file to source control. It costs one extra step on installation. I think it is worth it.
According to Gradle docs, adding gradle-wrapper.jar to VCS is expected as making Gradle Wrapper available to developers is part of the Gradle approach:
To make the Wrapper files available to other developers and execution environments you’ll need to check them into version control. All Wrapper files including the JAR file are very small in size. Adding the JAR file to version control is expected. Some organizations do not allow projects to submit binary files to version control. At the moment there are no alternative options to the approach.
What is the "project"?
Maybe there is a technical definition of this idiom that excludes build scripts. But if we accept this definition, then we must say your "project" is not all the things that you need to versioned!
But if we say "your project" is everything you have done. Then we can say you must include it and only it into VCS.
This is very theoretical and maybe not practical in case of our development works. So we change it to "your project is every file (or folder) you need to editing them directly".
"directly" means "not indirectly" and "indirectly" means by editing another file and then an effect will be reflected into this file.
So we reach the same that OP said (and is said here):
I think Generated files should not be in the VCS.
Yes. Because you haven't created them. So they are not part of "your project" according to the second definition.
What is the result about these files:
build.gradle: Yes. We need to edit it. Our works should be versioned.
Note: There is no difference where you edit it. Whether in your text editor environment or in Project Structure GUI environment. Anyway you doing it directly!
gradle-wrapper.properties: Yes. We need to at least determine Gradle version in this file.
gradle-wrapper.jar and gradlew[.bat]: I haven't created or edited them in any of my development works, till this moment! So the answer is "No". If you have done so, the answer is "Yes" about you at that work (and about the same file you edited).
The important note about the latest case is the user who clones your repo, needs to execute this command on repo's <root-directory> to auto-generate wrapper files:
> gradle wrapper --gradle-version=$v --distribution-type=$distType
$v and $distType are determined from gradle-wrapper.properties:
distributionUrl=https\://services.gradle.org/distributions/gradle-{$v}-{$distType}.zip
See https://gradle.org/install/ for more information.
gradle executable is bin/gradle[.bat] in local distribution. It's not required that local distribution be same as that determined in the repo. After wrapper files created then gradlew[.bat] can download determined Gradle distribution automatically (if not exists locally). Then he/she probably must regenerate wrapper files using new gradle executable (in downloaded distribution) using above instructions.
Note: In the above instructions, supposed the user has at least one Gradle distribution locally (e.g. ~/.gradle/wrapper/dists/gradle-4.10-bin/bg6py687nqv2mbe6e1hdtk57h/gradle-4.10). It covers almost all real cases. But what happens if the user hasn't any distribution already?
He/She can download it manually using the URL in .properties file. But if he/she doesn't locate it in the path that the wrapper expected, the wrapper will download it again! The expected path is completely predictable but is out of the subject (see here for the most complex part).
There are also some easier (but dirty) ways. For example, he/she can copy wrapper files (except .properties file) from any other local/remote repository to his/her repository and then run gradlew on his/her repository. It will automatically download the suitable distribution.
Old question, fresh answer. If you don't upgrade gradle often (most of us don't), it's better to commit it to VCS. And the main reason for me is to increase the build speed on the CI server. Nowadays, most of the projects are getting built and installed by CI servers, different server instance every time.
If you don't commit it, CI server will download a jar for every build and it significantly increases a build time. There are other ways to handle this problem, but I find this one the easiest to maintain.
I had a package structure which I changed, and this lead to the corresponding folders in /bin/ being removed, so now I get a ClassNotFoundException.
More specifically, I had the ususal /src/ and /bin/, and inside /src/ I had /main and /test, and this was presumably mirrored in /bin/. I shuffled the files around a bit and ended up with /main/model/ and /test/model/.
I have tried to manually create the corresponding folders (mirroring the packages in /src/), but this does not seem to help. I managed to compile the classes in one of the packages, but not in the test package, since it uses ScalaTest* and I didn't manage to compile the classes with the scalatest jar file (or something was wrong with the classpath).
So my question is:
How do I fix this within this peoject? (I don't want to simply create a new project and copy the source files over)
I'm guessing that I should learn more about build systems to be more resilient to such annoyances in the future? If so, what should I read up on, specifically to become better at troubleshooting and having more fine grained control over the build in the context of IDEs in the future (making builds independent of IDEs is not a priority for me at this point)?
An answer to one of these questions would be sufficient.
*All the source files are .scala files, if that might matter.
Udate
I did a clean of the project (project -> clean). This seemed to fix the problem: I was able to run the test classes from within Eclipse. All the binary files were in there, too. I made a new package, main.controller, with one class, and when I tried to run it, it said that it couldn't find the class. I tried to run the tests again, but those gave me a ClassNotFoundException, too. When I looked in /bin/ it turned out that all the folders and files were gone. I've tried to clean the project again but to no avail. I don't understand how I was able to clean the project, but now it can't fix it?
Update 2
To test if this was reproducible with a Java project, I made a Java project with two packages; main and test. I had the main class in main, which used a class from test (so there were dependencies across the packages). It ran succesfully. Then I added packages so that I had main.model and test.model and moved the corresponding files there. It also ran. Then I tried to delete all the files and folders in /bin/, and then the main class would not run. But if I did a clean of the project, then it cleaned it succesfully and the Main class was able to run. Then I made a Git repository for it, placing it outside of /workspace/ (in my git folder) and tried to do the same there. Eclipse was able to clean the project succesfully everytime.
So I don't understand why it can't manage to clean my Scala project.
I had errors in the "Problems" tab (Window -> Show View -> Problems). Now that I've made them go away, my /bin/ is correct and I can run both my Main class and my tests. This Question helped find out what the problem was:
Scala project won't compile in Eclipse; "Could not find the main class."
I have a bit of code for a dll that is needed by two or more projects in eclipse. Currently each project has a copy of the code and builds the DLL separately. I want to separate the dll code into a separate eclipse project so there is a common location. But I want to avoid the situation where we have to build the dll in the one project, then copy the dll back to the other projects and check the dll to each respective project. This will create a dll for each project that isn't traceable to the exact code that it was built with.
Is there a way to somehow symbolically link the dlls to another eclipse project that is using CVS as the version control system so that it is possible to tell which version of the code was used to create the dll? Am I making this too complicated or missing something obvious?
I thought about working sets in the package manager for eclipse, but I have to investigate more on how to use them with CVS to avoid making it a nightmare for the next person who checks it out and can't figure out why their project won't compile.
Thanks.
What about creating a new folder in a separate project. In the advanced section of creating a new folder there is an option to link to another location on the file system.
Or you could also create a container project that makes use of a projectset.psf file. Have the projectset file link to the different projects in your repository. When you want to check out that project, check out the container instead and right click on the projectset file and select Import Project Set...
If you are working with one workspace, you end up with three projects, each mirrored in CVS: One is the dll, the others are the projects using the dll (configured as a project dependency of these projects upon the dll project).
With three projects I wouldn't aim for working sets - they are good for managing a lot of projects within one workspace, for three projects, I'd consider them overkill. I usually tend to aim for several workspaces instead of working sets.
Regarding the next person working with these projects: You need to keep some kind of documentation about how to setup your projects. You might say that your eclipse project files do just that (as they define a project dependency upon another project) but this is for the machine - humans tend to like other communication means.
If you are worried about changes to the dll being incompatible to one project (because the person applying these changes doesn't care about the other project), aim for a build server. This will build all projects and dependent projects whenever something under version control changes, run all tests, provide a build number and package it all ready for use. This way you can be sure that - whatever is in your deliverable - can be reproduced, because the buildserver is not able to make local (uncommitted) changes to the code. Also a buildserver will signal failure (either broken API or broken tests) at the moment of the last commit (well - a few minutes later) and place the burden of repairing the damage on the one causing the damage.