Project management of both mixed and binary files - version-control

we develop software which also includes a lot of art assets (binary files). We would like to version control our source code, but do not keep track of binary files changes (artists works separately and upload new art assets). However, one checkout should produce full tree of both source and art. This is needed because we want to run continuous integration system afterwards (nightly builds, testing). What you could recommend? Maybe any of those web hosted apps allows one to do that.

I would recommend storing binaries in an artifact repository like Nexus: it offers a simple referential based on the filesystem, with simple directories based on a naming convention.
This is very easy to maintain.
Such a model is linked to the source control through a declarative process: you declare in a file (pom.xml) what version of which artifact you need for your specific version of your code.
That pom.xml file is managed like the rest of your sources.

Related

How to prevent NuGet version drift in a big monolithic (sigh :-() application (multiple solutions, many projects in each)?

We are building all the solutions to a shared bin directory. Having different projects reference different versions of the same dependency is not healthy for our build.
So, we consolidated the dependencies - great. But now the versions start to drift again. We do not want to consolidate them manually every now and then - we want to prevent the drift completely.
Why we do not want to use Paket? The main reason is that it seems we would lose the ability to migrate the NuGet package dependencies to the new PackageReference items in the projects. So, currently we have package.config files, but we plan to replace them with the respective PackageReferences. That means we will use internal NuGet support by msbuild, which seems to leave no place for Paket.
Now, I assume we are not unique in this world and others have the same problem as we are. How do you solve it?
EDIT 1
We have our internal NuGet repo, but we use it for dependencies which do not have organic representation in Nuget.org and for sharing our own internal packages.
One approach is to consume only from the internal NuGet repository. This has challenges, like:
Who uploads the dependencies there? Developers? But then how to make sure they do not upload different versions of the same dependency? Dedicated people? Then they become a bottleneck.
Small thing, but we need to block commits to the central NuGet.config
Uploading a dependency to the internal NuGet repo is not immediate. You cannot just download it from NuGet.org and upload to the internal one, because that would miss any transitive dependencies. So, a process should be built around it.
It is all possible, but I am reluctant to go down that route ... Must be a better way.
EDIT 2
While we do plan to migrate to PackageReference, it will take time. And unfortunately as long as we have Silverlight (another year, at least) a whole bunch of projects in the dedicated Silverlight solution (80+) will not be migrated to PackageReference, because by doing so it becomes impossible to debug the code with VS 2015.
Next, suppose we do migrate ALL the projects and then externalize all the PackageReference items to a single targets file imported by all the projects. This is feasible when using a shared bin directory as we plan to do. But when inspected in VS 2017 this setup communicates a wrongful picture that every single project depends on the entire set of NuGet dependencies.
I would rather avoid this.
Once you move to PackageReference, you can take advantage of MSBuild. For example, you can have a MSBuild file that contains all your dependency versions. It could be a file that you need to <Import ... /> in all your csproj files, or you could use Directory.Build.props. Finally, in each of your projects, change the version number in any <PackageReference to a MSBuild variable that uses the property you previously defined. Most of Microsoft's open source repositories use this technique, with minor variations about file names and whether it's imported automatically with Directory.Build.props, or an explicit <Import ... />.
While you can still use the Package Manager UI in Visual Studio to check for updates, you won't be able to update the package versions with it (at least, it won't preserve how and where the versions are defined). However, just make sure your MSBuild file that defines the versions is in your solution, so you can trivially open the file in Solution Explorer and then type the new version number in. Adding new package references is slightly more effort, but it's generally not done often, and it's still very easy with SDK-style projects, since Visual Studio lets you edit the csproj while the project is still loaded.
Since you didn't accept the other solution, maybe you could take a look at paket
It's a package manager for dotnet than (among other feature) holds solution wide dependency lock file. It is very customizable, and while it solves LOTS of problems, as any tool, creates some new ones. In my experience, the new ones are far less infuriating :)

Best way to share a Target Platform for Eclipse RCP project

What is the best way to share a Target Platform?
E.g. together with the source code of some RCP project.
I can define a .target file and fill it with remote p2 sites and share this file. The problem here is first, those sites tend to be very slow and unreliable. From experience, resolving such a Target Platform fails from time to time.
A more reliable and faster (in terms of loading this Target Platform) approach is to define a local directory, that contains all plugins and features.
This directory can either be part of the source repository itself or I can provide a (fast and reliable) remote site from which I can download this target platform anytime.
The difficulty here is, how can I translate a list of p2 sites into a directory, that contains those plugins and features which are provided by the update site?
When I set a specific target platform from within Eclipse, where are those artifacts actually saved? I assume I could just copy this directory.
Despite the sometimes slow Eclipse p2 repositories and despite the crappy target platform editor, I still recommend using .target files. They are easy to share because they can be stored in the source code repository.
While resolving a target platform, PDE caches bundles in the .metadata/.plugins/org.eclipse.pde.core/.bundle_pool directory of the workspace.
Using .target files also allows you to use Tycho as a build tool. Note, however, that Tycho cannot read from local (i.e. file://) repositories.
It is the most common and most accepted way of providing dependencies for RCP/plug-in development.
To mitigate the unreliable performance of Eclipse p2 repositories you may want to mirror these and specify the mirrored sites in your target platform definition.
There is also Target Platform Definition DSL and Generator that you might try as a drop-in replacement for if you are dissatisfied with the reliability of the the PDE target platform editor.

Perforce, Projects shared among multiple Solutions

UPDATE:
So it turns out that we may have found a bug in Visual Studio 2003 (I know...no surprise). We found out that if the solutions were added to the repository using Visual Studio (using Add Solution to Source Control) everything went fine...go figure.
So we're converting our VSS repository (if it can be called that) to Perforce and we're running into issues with projects included in multiple solutions.
Our repository might look like this...
//Depot
DevMain
Solution1
Project1 (Builds to a DLL)
Solution2 (Has Project1 as a project reference)
Project2
Solution3 (Has Project1 as a project reference)
Project3
When using the integrated Source Control in Visual Studio for Solution2 it complains that the projects are not under the current solutions folder and we may want to move it. Because multiple solutions reference Project 1 we can never organize it where one solution won't complain...
Is the best practice to just build Project1 to DLL and store it in a Lib folder? Or is there a better way?
Thanks.
We had a similar problem and were approaching it very much like #Toby Allen mentions in his answer, via client specs. However, in time this becomes very painful (setting up a new team member becomes more and more difficult as client specs become more and more convoluted; also automation is much more complicated because things are... "in flux" :-) ).
Eventually, we evolved our strategy to use a directory structure and branching instead. The directory structure is as follows:
//depot
/products
/(product_name)
/doc
/lib
/(3rd_party_libname)
(DLLs)
/(another_3rd_party_libname)
(DLLs)
/src
/Project1
(files, csproj, vbproj, etc)
/Project2
(files, csproj, vbproj, etc)
/Project3
(files, csproj, vbproj, etc)
Solution1.sln (includes src/Project1)
Solution2.sln (includes src/Project1, src/Project2)
Solution3.sln (includes src/Project1, src/Project3)
/(another_product_name)
/doc
/lib
/src
(solutions)
/shared
/(shared_lib_name)
/doc
/lib
/src
(solutions)
/(another_shared_lib_name)
/doc
/lib
/src
(solutions)
Note that the same structure is repeated throughout the structure (doc/lib/src/solutions). Lib contains "external" libraries - 3rd party libraries that are included in project references. Src contains a flat list of all projects that are part of a particular product. Solutions are then used to "combine" projects in any number of ways. I think of src directory as a container with "what is available", solutions are then picking from this container and combining projects (as needed).
Libraries that are shared among multiple products go into shared directory. Once in shared directory, they are treated as independent from products - they have their own release cycle and are never joined to products as source. Shared libraries are pulled into products by branching the shared library release assembly/assemblies into product's lib directory -> from product's perspective there is no difference between a 3rd party library and a shared library. This allows us to control what product is using what version of a shared library (when a product wants new features, it has to explicitely branch in newer release of a shared library, just like it would include a new release of a 3rd party library, with all pros and cons that go with it).
In summary, our structure has concept of two "types" of shared libraries:
projects local to a product, used by multiple solutions (included in a flat list of projects in src directory, multiple solutions can reference them)
projects used by multiple products (added to shared directory, treated as 3rd party libraries with releases independant from products)
The solution should be to rebind Solution1 to the source control server each time it's added to a new project. (It's under File->Source Control->Change Source Control.)
This should only need to be done once per desktop per solution.
I don't know much about using Visual Studio with Perforce, but you might consider creating workspace views that map Project1 into Solution2, Solution3, etc., where needed.
If I understand correctly you want how your files are stored on disk to differ from how it is stored in Perforce. If this is the case, and you cant simply redefine the reference within VS, then a cleverly designed client can do the trick.
Client: Client_Solution2
Description:
Created by Me.
Root: C:/Development/Solutions
View:
//depot/Devmain/Solution1/... //Client_Solution2/Solution2/Solution1/...
//depot/Devmain/Solution2/... //Client_Solution2/Solution2/...
This will give you a structure where Solution1 is a sub directory of solution 2.

How do you store third party libraries in your source control?

How do you store third party libraries that you use in your project in your source control?
When would you store binaries in your source control?
When would you store the code in your source control?
Would you ever store both? In what situations would you do this?
(Btw, I'm using .NET but it doesn't really matter for this question)
How: a vendor branch is generally a good approach
When (thirdparties): for minimizing the number of referential involved: you could add those libraries into a separate external referential (like Maven), but that mean you need to access that extra referential for each of your environment (development - integration - homologation - preproduction - production)
When (code): for managing the complexity of the changes, when you know updates and fixes for current versions running into production will be needed while new development are in progress.
Why (store both): for deployment reason: you can manage a complete configuration (list of elements you need) in one referential and query it wherever and whenever you need for:
development (you query what you need to develop and execute your code, including thirdparties needed for compilation/execution)
tests (integration, homologation): you query the exacts tags you want to update your testing workspace with
production: you identify exactly what goes into production from one source: your SCM.
For test and production environments, that also mean your own product (the packaged result of what you are building) should also go into your SCM (only the official releases, not the intermediate ones used internally).
If other projects depend on your product, they will build their own project against your packaged version stored in the SCM, not against your source code they somehow recompiled.
Why this is important ?
Because in the end, what will run in production is that packaged version of your product, not your "source code re-compiled". Hence the importance to make all your test with the target final form of your product, clearly stored and tagged in your SCM.
Martin Lazar raises a legitimate point in his answer
Source control is called "source" control, because it is supposed to control sources.
While that may have been historically true, every current RCS have evolved toward SCM (Source Code Management), which does not just control sources, but also manages changes to documents, programs, and other information stored as computer files.
Binaries can then been stored (even stored with binary-delta)
Plus that allows some of those SCM to propose S"C"M feature (as in Source Configuration Management).
That SCM (Configuration) not only stores any kind of "set of files", but also their relationships (aka dependencies) between those sets, in order for you to query one set of file, and to "pull" every other deliveries on which that set depends on (to build or to deploy or to run)
How do you store third party libraries that you use in your project in your source control?
As binary or source or both. Depends on the library.
When would you store binaries in your source control?
A third party library for which we don't have the source or an internal library which we haven't made any changes to or the library is too huge to be built around.
When would you store the code in your source control?
Say, we use an internal library A but have made some bug fixed specific to product X. Then product X's depot will keep the source and build it.
Would you ever store both? In what situations would you do this?
Yes, all the latest binaries are stored in source control.
So the tree looks like this:
product
|-- src
|-- build
|-- lib
|- 3rdparty
|- internal
...
Assuming you are using .Net:
I create a "Libraries" folder in my project and source control that contains any third party assemblies.
My solution then references those assemblies and my build process pulls that folder down to our build server.
Any one pulling your code from source control should be able to compile it without having to hunt down references and assemblies.
Source control is called "source" control, because it is supposed to control sources..
In Java it's common pattern to use some version control system to store sources and other resources like configuration XML files or pictures, and than to use some dependency management tool like Apache Maven, which will store, download and manage your project's dependencies on 3rd party libraries. Then when you reinstall your OS, Maven can automatically download your dependecies from central repository (or your own private repositories as well) and store them in a local cache on your disk. You don't even have to know where the cache is :)
Maven can be also used with other languages and as far as I know plugins for .net and C/C++ are available, but I haven't used it with anything else than Java.
I don't put 3rd party source or binaries in SC. My rationale was that I would not have to update SC just to update the libraries. I am starting to regret this though. When I have had to recreate the project I find myself running around looking for the lib sources instead of just syncing to SC.
On a recent Java project I switched to using Maven - this was quite nice since it meant I didn't need to store any third party jars in a lib/ directory. At compile time maven would pull in the dependencies. One nice side affect is the jars have the version number in their filename.
My experience has been to create a "lib" folder and keep all 3rd party binaries in there. I will create a totally separate tree for the Source Code to these third parties if it is available.
Some places where this might be different is if you are using an open source vs. a retail 3rd party, with open source solutions I tend to just include the code in my projects and not check-in the binaries.
You don't need to store third party libraries in your source control repository. Those libraries (think of SDL, libcurl, etc.) should always be available on the web.
Just two raccomandations:
make sure to state clearly in your code which version of the library you should compile against
be sure that that specific version is always available on the web
Generally speaking, I would do roughly what has been prescribed by other users.
In the case of Subversion, and I admit I can't speak to the inclusion of the feature in the case of other systems, one can use an External to link in a repository maintained elsewhere. In a checkout/export, you'll get a full copy of the source including the updated code from that external repository all in one go.
This was particularly nice for me with PHP/JS development as there is no issue regarding storing binaries. I'd keep an external of the Zend Framework, Doctrine ORM, and jQuery all in a /lib/ folder of the project. Every update would give me a complete, updated copy of -all- the necessary files without any more work than adding that repository URL to an svn property.
If you are using git (which I recommend), and you have the source of the third party library, then storing the library in its own git repository and including it as a submodule is an excellent setup.
http://book.git-scm.com/5_submodules.html
http://git-scm.com/docs/git-submodule
If the source library is also using git then you can clone their library and push it to your own server so you'll never lose it.
Git submodules allow you to specify which revision of a library is required for a project, which is great for maintaining compatibility.
Conceptually you need to store at least the binaries (and the headers if you do C/C++)
That's usually the only way with third party libraries where you don't have the source.
If you have the source you can opt to store the source and build the third party libraries in your build process.
You should be able to install a fresh OS, get your sources from source control, built and run. So yes, you should put them in source control.
It depends on how big they are. When binaries or installers are too big, it can cause havoc for remote users. The upside of storing binaries and installers is that everything a developer needs to get up and running is in source control and the versions are correct. If you have a separate installation location, versions can get messed up. So, in general I like to store small or moderate binaries in source control, but larger ones I leave out.
Edit: Oh, and I call mine "BinRef" :)
When would you store binaries in your source control?
I store binaries in source control when I want they ability to quickly revert back to the old version of an application. There are number of reason I would do this, those include the following:
Application testing in an environment that exactly matches production is not always possible.
The previous programmer may not have used source control or may have used it incorrectly. So it can be difficult to be sure the source code you are changing is the version that matches what the user is working with.
On source code and compiled libraries (most likely standalone):
If I don't use the third party components (compiled libraries) or provided source to build inhouse software components, I simply take them off the shelf and install them as prescribed (which might include a compile, pl/sql code for example). I would not install them in a dependency management repo or any source control system for the simple reason that I don't want them incorporated accidently or otherwise in components I'm building and I don't want to track them in any software cycle. This is an asset (software asset) that should be tracked with other tools. If I'm not using them for software development, I don't and shouldn't see them in tools I use for software development.
If I depend on them for building my own software, I would store them all day long.

Managing dependencies with Eclipse and CVS

I have a bit of code for a dll that is needed by two or more projects in eclipse. Currently each project has a copy of the code and builds the DLL separately. I want to separate the dll code into a separate eclipse project so there is a common location. But I want to avoid the situation where we have to build the dll in the one project, then copy the dll back to the other projects and check the dll to each respective project. This will create a dll for each project that isn't traceable to the exact code that it was built with.
Is there a way to somehow symbolically link the dlls to another eclipse project that is using CVS as the version control system so that it is possible to tell which version of the code was used to create the dll? Am I making this too complicated or missing something obvious?
I thought about working sets in the package manager for eclipse, but I have to investigate more on how to use them with CVS to avoid making it a nightmare for the next person who checks it out and can't figure out why their project won't compile.
Thanks.
What about creating a new folder in a separate project. In the advanced section of creating a new folder there is an option to link to another location on the file system.
Or you could also create a container project that makes use of a projectset.psf file. Have the projectset file link to the different projects in your repository. When you want to check out that project, check out the container instead and right click on the projectset file and select Import Project Set...
If you are working with one workspace, you end up with three projects, each mirrored in CVS: One is the dll, the others are the projects using the dll (configured as a project dependency of these projects upon the dll project).
With three projects I wouldn't aim for working sets - they are good for managing a lot of projects within one workspace, for three projects, I'd consider them overkill. I usually tend to aim for several workspaces instead of working sets.
Regarding the next person working with these projects: You need to keep some kind of documentation about how to setup your projects. You might say that your eclipse project files do just that (as they define a project dependency upon another project) but this is for the machine - humans tend to like other communication means.
If you are worried about changes to the dll being incompatible to one project (because the person applying these changes doesn't care about the other project), aim for a build server. This will build all projects and dependent projects whenever something under version control changes, run all tests, provide a build number and package it all ready for use. This way you can be sure that - whatever is in your deliverable - can be reproduced, because the buildserver is not able to make local (uncommitted) changes to the code. Also a buildserver will signal failure (either broken API or broken tests) at the moment of the last commit (well - a few minutes later) and place the burden of repairing the damage on the one causing the damage.