I've got a project in a git repository that uses some custom (and so far unversioned) setup scripts for the build environment etc. I'd like to put these under version control (hopefully git) but keep them versioned separate from the project itself, while still living in the base directory of the project - I've considered options like local branches but these seem to have the problem that switch back to master (or any other "real" branch) will throw away the working copies of the setup scripts.
I'm on Windows using msysgit so I've got a few tools to play with; does anyone have a recommendation or solution?
If you really need them separate from your main git repo while still living directly within it, you could try:
creating a new repo with those script within it
and:
adding that new repo as a submodule to your repo. Except:
a/ those scripts won't live directly in the base directory, but in a subfolder representing the submodule
b/ you need of course to not publish (push) that new repo, in order for other cloning your main repo to not get those setup files
or:
merging that new repo into your main repo (with the subtree project), but:
you need to split back your project to get rid of those files
for a project with a large history, and with frequent push, that step (the split) can be long and cumbersome.
I would consider a simpler solution, involving some evolution to your current setup files:
a private repo (as in "not pushed") with those setup files
environment variables with the path of your main git repo in order for your setup files (which would not be directly within the base directory of said main repo) to do their job in the right directory (like beginning for instance with a 'cd right_main_git_repo_dir').
I want to share an additional solution and some samples from which to start.
I've has a similar problem in attempting to build Mozilla Firefox with Buildbot -- I need to have some files in the root folder (namely the .mozconfig file and some helper scripts) and I wanted to version them separately.
My solution is as follow:
checkout the Firefox code from the Mercurial repository;
checkout an additional repository with the additional file I need;
before starting the build, I copy these file to the folder with the Firefox code.
This approach is implemented in the following repositories:
buildconfig-mozilla-central: it contains the Buildbot configuration, which
pulls both repositories
copies the files from the scripts repository
and start the build;
buildscripts-mozilla-central: the repository with the build configuration and helper scripts.
Please note that the code might not be well factored (for example the paths) but it should be a good starting point.
This procedure is tailored for Firefox, but it can be applied to any repository.
Related
To set expectations, I'm new to build tooling. We're currently using a hosted agent but we're open to other options.
We've got a local application that kicks off a build using the VSTS API. The hosted build tasks involve the Get sources step from a GitHub repo to the local file system in VSO. The next step we need to copy over a large number of files (upwards of about 10000 files), building the solution, and running the tests.
The problem is that the cloned GitHub repo is in the file system in Visual Studio Online, and my 10000 input files are on a local machine. That seems like a bit much, especially since we plan on doing CI and may have many builds being kicked off per day.
What is the best way to move the input files into the cloned repo so that we can build it? Should we be using a hosted agent for this? Or is it best to do this on our local system? I've looked in the VSO docs but haven't found an answer there. I'm not sure if I asking the right questions here.
There are some ways to handle the situation, you can follow the way which is closest to your situations.
Option 1. Add the large files to the github repo
If the local files are only related to the code of the github repo, you should add the files into the same repo so that all the required files will be cloned in Get Sources step, then you can build directly without copy files step.
Option 2. Manage the large files in another git repo, and then add the git repo as submodule for the github repo
If the local large files are also used for other code, you can manage the large files in a separate repo, and treat it as submodule for github repo by git submodule add <URL for the separate repo>. And in your VSTS build definition, select Checkout submodules in Get sources step. Then the large files can be used directly when you build the github code.
Option 3. Use private agent on your local machine
If you don’t want add the large files in the github repo or a separate git repo for some reasons, you can use a private agent instead. But the build run time may not improve obviously, because the changed run time is only the different between copying local files to server and copying local files to the same local machine.
I have a remote Git repository with approx. the following repository structure:
service1/
service1/image1/
service1/image1/docker/
service1/image1/docker/Dockerfile
service2/
service2/image2/
service2/iamge2/docker/
service2/image2/docker/Dockerfile
I can clone it from https://<my-git-server>/my-project.git to obtain a directory my-project/ and a corresponding file structure underneath it.
My service2 is a Tomcat server that hosts a webapp whose source code I would like to push into the same remote repository next.
Eclipse's file structure inside the project folder is approx. as follows (as is normal for Dynamic Web Projects in Eclipse):
my-project/WebContent/
my-project/build/
my-project/src/
If possible, I would like to arrive at a new repository structure for the remote Git repository as follows:
service1/
service1/image1/docker/
service1/image1/docker/Dockerfile
service2/
service2/image2/
service2/image2/docker/
service2/image2/docker/Dockerfile
service2/image2/eclipse/
service2/image2/eclipse/my-project/
service2/image2/eclipse/my-project/WebContent/
service2/image2/eclipse/my-project/build/
service2/image2/eclipse/my-project/src/
My question is: how I can specify the additional path component eclipse/myproject? I understand that pushing from Eclipse to https://<my-git-server>/my-project.git would result in WebContent, build and src becoming siblings of service1 and service2, and that pushing from Eclipse to https://<my-git-server>/my-project.git/service2/image2/eclipse/my-project would not work either.
So how does one accomplish such a thing?
Upon further study and reflection I have decided on managing source code for my project in two Git repositories. I have no outside dependencies that would prevent me from doing so and have come to understand that monorepos and Git don't fit so well.
For reference, here are the new repository structures:
service1/
service1/image1/docker/
service1/image1/docker/Dockerfile
service2/
service2/image2/
service2/image2/docker/
service2/image2/docker/Dockerfile
WebContent/
build/
src/
If I am not mistaken, Git subtrees or submodules could now serve for cloning from these repositories into a single "mono" (as in monorepo) local view, if ever required.
I am using CVS and I have this hierarchy:
/ROOT
/JAVA
/JavaProject1
/JavaProject2
.project
/PHP
/PHPProject1
/PHPProject2
.project
In Eclipse > CVS Repository Exploring, I can see this hierarchy and I can Check Out only the project that I want.
Also I can check out (import) JAVA and PHP folders (I created them as Eclipse General project for import) to Eclipse Package Explorer and can synchronize and commit all together.
When I want to use Git, it only supports one project.
I don't want a flat hierarchy (near all JAVA and PHP project together), I want to use tree hierarchy and I want to check out only the project that I want as with CVS.
Is my CVS hierarchy possible in Git or what technique should I use?
I think you're mixing what you want to do locally with how you want
to arrange things remotely. All git commands access only the local
repository. The 'push' and 'fetch' commands appear to access a remote
repository, but in fact they effectively start each other on the remote
machine running against the local repository on that machine. So the
tasks you can do remotely are very limited. Specifically, copying "branch"
and "tag" references and the commit histories those references point at.
This means for the simple case there is ONLY the local repository,
it exists in the .git directory in the working directory.
You can arrange working directories, with their .git directories however
you wish on your local machine. Likewise, you can arrange the the remote
repositories in any way allowed by the remote hosting service. The
layouts do not have to match. If the remote is your own Linux server you
can make the layout just like your local. If the remote is (for example)
Github you're more limited.
You'll need to backup the .git directory to backup
the repository; the rest of the working directory is probably not significant. You can use git push to do this backup, as long as you never use '--force'.
Git isn't really very keen on you having multiple working directories
for one repository. It is possible, however, in the simple case they
will each have their own copy of the repository and you will need to
push/pull the updates individually either to a "central" repository
or more "randomly". None of these repositories have to be physically
"remote".
Git much prefers you to switch between branches in one working directory
and use make install style processes to send builds out.
It is also possible to have unrelated branches in one repository, but most people find this too confusing as you still only have one working directory.
I'd like to create Git repositories for some of my existing STM32 embedded C projects that I'm developing with Eclipse.
Currently I have two separate projects in their own folders, and a common folder containing ST's official STM32 Cube peripheral drivers (.c and .h) files that both projects use.
The driver folder shouldn't change (except when new versions are released) but the project files will be edited as I develop them.
How should I place them on Git? Should I:
Have three repos (one for each project and another for the drivers)?
Have two repos and change my project structure to place the drivers with both projects?
Something else?
The first option seems OK, but if I don't clone both the project and driver repos, and do them to specific locations, I can't see how I would configure Eclipse properly.
The second seems more hassle free but redundant (it won't be as easy to update all projects to use new drivers as they are released, and it will use more server space).
EDIT: Just to clarify, I'm not using the STM32CubeMX software. I'm simply using the STM32CubeF4 peripheral drivers, which are basically just embedded C .c/.h source/header files for the chip's internal peripherals (and possibly a few .lib library files for DSP calculations). See the "download" button at the bottom of this page for the .zip archive (current version is 1.4.0): http://www.st.com/web/en/catalog/tools/PF259243
If you use a common driver in several other repos, it looks like a job for git submodules.
You have three separate repos, for example on your server, and both projects include the driver code as a submodule. What is stored in each project repo is:
an URL to the driver repo (on your server)
which commit of the driver repo is to be used
the path where to put the driver clone, relative to the project repo
After cloning a project repo, you run git submodule update --init, and the repo at URL is cloned into the relative path, and its commit is checked out.
git submodule add --name <driver> /server/src/driver.git driver
(/server/src/driver.git is URL, driver is path. <driver> is just an optional reference to ease future work with submodules. Don't use it if it is the same as path)
git submodules feel a bit cumbersome at first. For example, cloning or checking out commits in the parent (project) repo does not make the submodules working files up to date. You have to manually git submodule update for that.
You will end up with two clones of your driver, one in each project's working files structure. But this is not really redundant, as they are clones of a common driver repo on your server.
For every update of your driver, you push the driver changes to your server. To update to the latest driver version in another project, you go to that project's submodule, do the usual git push (you might be in detached state, in which case you need to checkout master first). The update will show up in the project's (not the submodule's) git status, which will tell you "New commits" in the driver submodule. What has changed is which SHA1 is checked out in your submodule; commit the changes in the project will only update which submodule commit is to be checked out.
I used computer A via the Terminal to create a) create a git repository, b) add an index.html file to the repo, c) add a remote origin, d) push to the remote origin. All OK.
Then, i used computer B to clone that repository via Terminal. Then, I opened Eclipse (equipped with Egit), and created a new project in the folder that was created by the cloning process. Then I used Eclipse to push any changes to the remote origin.
Returning to computer A, I used Eclipse to create a project in the original repo folder, and then I attempted to pull from the remote origin, in order to get the changes that were pushed when using computer B.
Eclipse will not do it. It complains the I have items such as .settings, .project and similar and since they are not under version control it won't overwrite them by fetching files from the server. I had to manually delete those files (via Terminal) and then Eclipse worked as expected.
Please provide information on how to avoid this.
Should I create the local repo from within Eclipse and then push it to the remote origin, so that items such as (.settings) are under version control and (if so) how would that cause trouble to people cloning the repo and use different versions of Eclipse?
Should I gitignore those items?
Should I ask Eclipse to save its own affiliated files to another folder (not that i am aware how to do that, i only know that NetBeans does it)?
Looks like you didn't gitignored eclipse files.
Probably, when you commit/push via egit, you also commit and push those files you already had unversioned in your machine A, so git complains, because you are asking to override existing unversioned files.
I strongly recommend you to gitignore those eclipse files. You can see examples of .gitignore files in the github gitignore repo.
Hope it helps.
It complains because if you pull the changes from your remote it will overwrite your local files. That is the problem. The other answerer has right. You should better add all the eclipse project files and and target .settings and classpath to gitignore. You can use a global gitignore for your computers as well, before creating projects. You could use maven for example, then you can import your projects only from the pom.xml-s given in the git repository.
I use them the same. Egit and other guis are a bit too complex to work with. Git repositories can get easy in an inconsistent state where you should use the oldfashioned terminal to solve things. Like, rebasing, merging on conflicts. Gits learning curve is solid.
Now you can solve your problem if on the first computer save a backup of your original and clones your project later, after fixed it on the second. On the second git remove all this files, but use the --cached option to avoid deleting them. Before you do it so, check the help of git remove! after you have done this, put them into the .gitignore as filenames with wildcards. You can also use a global gitignore file in your user folder. Creating a .gitconfig file where you can specifiy the global ignore with the following :
[core]
excludesfile = ~/.gitignore_global
Than just create the .gitignore_global like this :
/nbproject
/bin
/build.xml
.idea
chess.iml
target/
bin
( This file is for idea and netbeans. you can add eclipse project files here )
You can have .gitignore files per project too. You can commit them to the repository, so on the next machine you do not have to do this again. The nicest way I think is having a dotfiles git repository, which is a git repo of your home directory and the dotfiles in it. I also use it for different windows and linux distros.
That's all. You should keep all of your configuration in a safe place. And source code management can do it. But do not commit private stuff to public a place! ;)
Oh I wanted to mention that, you can also have a .gitignore entry in your .gitignore file. That can be very useful when you do not want to touch a repository but need to add a gitignore to hide some stuff especially from the given repo.