To set expectations, I'm new to build tooling. We're currently using a hosted agent but we're open to other options.
We've got a local application that kicks off a build using the VSTS API. The hosted build tasks involve the Get sources step from a GitHub repo to the local file system in VSO. The next step we need to copy over a large number of files (upwards of about 10000 files), building the solution, and running the tests.
The problem is that the cloned GitHub repo is in the file system in Visual Studio Online, and my 10000 input files are on a local machine. That seems like a bit much, especially since we plan on doing CI and may have many builds being kicked off per day.
What is the best way to move the input files into the cloned repo so that we can build it? Should we be using a hosted agent for this? Or is it best to do this on our local system? I've looked in the VSO docs but haven't found an answer there. I'm not sure if I asking the right questions here.
There are some ways to handle the situation, you can follow the way which is closest to your situations.
Option 1. Add the large files to the github repo
If the local files are only related to the code of the github repo, you should add the files into the same repo so that all the required files will be cloned in Get Sources step, then you can build directly without copy files step.
Option 2. Manage the large files in another git repo, and then add the git repo as submodule for the github repo
If the local large files are also used for other code, you can manage the large files in a separate repo, and treat it as submodule for github repo by git submodule add <URL for the separate repo>. And in your VSTS build definition, select Checkout submodules in Get sources step. Then the large files can be used directly when you build the github code.
Option 3. Use private agent on your local machine
If you don’t want add the large files in the github repo or a separate git repo for some reasons, you can use a private agent instead. But the build run time may not improve obviously, because the changed run time is only the different between copying local files to server and copying local files to the same local machine.
Related
How to config the _data folder to another git repository? The whole folder is clone from github and want to continue due to easy update. But the _data folder need to sync with another data repository. How can I config it?
Actually just one yaml file : _data/authorlist.yaml
What to put into package.js if want a script to sync the yaml file with github?
Thank you.
You can use Git submodules to keep a shared repository of data files that you can use in multiple projects. This will allow you to keep a reference to the data repository in your projects, and pull updates to this repository with a single command. The great thing is that submodules are a built-in feature of Git, so it's independent of any NPM scripts, environments (like a bash script would be) or frameworks. See the link above for documentation on how to set up and work with submodules.
We are trying to set up Eclipse so that two users can share the same project directory on our server. Is this possible? Every time we try, it creates a new folder and project.
Thanks!
Chris
No, this isn't possible. Eclipse only supports a single user accessing a workspace (not just a project) at a time.
Use a source control system such as Git or SVN to share code. Eclipse supports many such systems and has extensive sharing support in the 'Team' menus.
The best way to do this would be to use source control.
Sharing the actual workspace or the files with different eclipse instance is a recipe for trouble.
An easy way to do this would be to install git on your machine and also on his machine. Eclipse actually already has git in it ready to go so you probably dont need to install anything.
The one with the files locally will create a repo locally on his computer and commit the files to it.
Next you want to init a new empty repository on a shared folder and push your local chances to this as you would to github for example.
Your partner can then git clone from this repository to his machine and work locally.
Each of you will develop on your own copy and commit your changes locally. You will share your changes by pushing your commits in that central repo and pulling from it to get changes from your partner.
You could also just open an account on GitHub, GitLab or BitBucket (there are many others too) and use that instead of a shared folder. big advantage with these services is that they will be available from anywhere.
We are implementing an environment in Visual Studio Team Services (VSTS).
We have a Git Repo tied to VSTS
The problem is how to keep the Config files separate so they retain their unique values in their local environments? But without uploading Configs VSTS fails to Build within it's environment.
We don't want the same config settings that are in VSTS to always Sync to Local Environments nor do we want to Push our local configs to the Master. Obviously, we can Exclude on Push but the question is how to configure VSTS in a manner that allows it to Build successfully without requiring config files to be uploaded to the Repo?
Reviewing this post, I'm not sure whether or not Repo based configs are required: How to handle multiple configurations in VSTS Release management?
And yes we will eventually have multiple configs to allow Staging and Production releases.
The direct answer is No. Usually git only track source code for projects, and VSTS usually can build successful without config files. I’m not sure what’s your project is, so we can deal with the situation for that you need to push the config file to VSTS but also do not effect local settings when git pull (assume the name of the config file is project.config):
Option 1:
If it’s ok for you to rename project.config when you build in VSTS, you can use project.config for your local environments and projectRemote.config for the remote repo: gitnore project.config in .gitignore, and create projectRemote.config file to push to remote.
Option 2:
Keep local project.config version when pull changes from remote. You can keep local versions by:
touch .gitattributes
echo 'project.config merge=ours' >> .gitattributes
git config --global merge.ours.driver true
Note: the merge strategies seems only works when the pull is not fast forward.
I'm currently helping to maintain a project for a client remotely. I'm the only developer ergo some of my unorthodox approaches/thinking.
the problem
The client is using Visual Studio 2010 + Team Foundation Server for their source control. I am working on a Mac over VPN and have tried several approaches to make committing to their TFS workable. I've tried TFS plugin for Eclipse with no luck (VPN really hoses the connection to TFS). Currently I am having to do a full "checkout for edit" through a virtual machine to the TFS, then transferring the project over the VPN to overwrite those files. Not a sustainable solution to say the least.
the solution?
I'm wondering if there is a way to:
get a list of changed files from GIT (I think this is the solution
(How to list all the files in a commit?)
then use that list as a means to go in and fetch those file, maintaining their folder structure
from there I can do my dump over
VPN into the VM that has the project mapped in TFS.
Or if there is something I've overlooked or hadn't thought of, please do recommend them, I'm all ears.
First, I'm assuming you are running the VM on or near the TFS server, not on your Mac. If not, you can just share a directory using VMware/VirtualBox and edit away on your Mac...
It sounds like you could achieve what you want with plain old Git. If you:
Create a bare repository on the VM (git init --bare)
Add a post-receive hook to copy the files from the master branch (for example) into the TFS directory, overwriting merrily (http://git-scm.com/book/en/Customizing-Git-Git-Hooks)
Initialise your local copy of the source as a Git repository (git init)
Add the remote repository. Assuming it's a Windows box you can use an SMB shared folder over the VPN so your remote is "local" as far as Git is concerned. (git remote add tfsserver file:///Volumes/tfsmount/code
Your first push will be expensive (but you could prepopulate the remote repo to get around that), but subsequent pushes would be just the changesets. The post-receive hook would then take care of updating the files, and you're laughing.
Of course, you then get to impress them with how amazing Git is, get them to migrate, and your problem goes away forever :).
Update: Here's a link which describes these steps in more detail, under the guise of updating a remote website: http://toroid.org/ams/git-website-howto.
I've got a project in a git repository that uses some custom (and so far unversioned) setup scripts for the build environment etc. I'd like to put these under version control (hopefully git) but keep them versioned separate from the project itself, while still living in the base directory of the project - I've considered options like local branches but these seem to have the problem that switch back to master (or any other "real" branch) will throw away the working copies of the setup scripts.
I'm on Windows using msysgit so I've got a few tools to play with; does anyone have a recommendation or solution?
If you really need them separate from your main git repo while still living directly within it, you could try:
creating a new repo with those script within it
and:
adding that new repo as a submodule to your repo. Except:
a/ those scripts won't live directly in the base directory, but in a subfolder representing the submodule
b/ you need of course to not publish (push) that new repo, in order for other cloning your main repo to not get those setup files
or:
merging that new repo into your main repo (with the subtree project), but:
you need to split back your project to get rid of those files
for a project with a large history, and with frequent push, that step (the split) can be long and cumbersome.
I would consider a simpler solution, involving some evolution to your current setup files:
a private repo (as in "not pushed") with those setup files
environment variables with the path of your main git repo in order for your setup files (which would not be directly within the base directory of said main repo) to do their job in the right directory (like beginning for instance with a 'cd right_main_git_repo_dir').
I want to share an additional solution and some samples from which to start.
I've has a similar problem in attempting to build Mozilla Firefox with Buildbot -- I need to have some files in the root folder (namely the .mozconfig file and some helper scripts) and I wanted to version them separately.
My solution is as follow:
checkout the Firefox code from the Mercurial repository;
checkout an additional repository with the additional file I need;
before starting the build, I copy these file to the folder with the Firefox code.
This approach is implemented in the following repositories:
buildconfig-mozilla-central: it contains the Buildbot configuration, which
pulls both repositories
copies the files from the scripts repository
and start the build;
buildscripts-mozilla-central: the repository with the build configuration and helper scripts.
Please note that the code might not be well factored (for example the paths) but it should be a good starting point.
This procedure is tailored for Firefox, but it can be applied to any repository.