Is anyone aware of a hybrid version control and synchronising system?
I'm currently a happy mercurial user, but my projects usually contain a mixture of files.
Most of these (code, documentation, ...) I want to be version-controlled. This is why I use mercurial.
However, on the rare occasion I have files that I would like to synchronise between my working copies, but not version control.
For example, I version control the code I write to do image processing. This code can produce a whole bunch of output images which I'd like to have synchronised so I don't have to remember to shuffle them around my various computers, but there's no point having these version controlled.
To clarify - I am aware of extension to mercurial such as bfiles and bigfiles, which are handy for my image example, but I was just wondering if anyone out there knows of alternative ways to handle this. I just want the one system that I can tell "version control all files except those ones, which should be synced but have no history".
cheers!
EDIT: I could do something like adding a hg marksync <filename> that added <filename> to a list of files to be synced, and then adding a hook to hg push/hg pull that would (say) run rsync (or whichever sync tool) in the background, but I wondered if there was a less hacky solution (I think bfiles/bigfiles do something along these lines anyway).
Version Control System (any) doesn't care about synchronization of
not versioned data
besides default pathes
If you want sync any files - use specially designed for this task tools: f.e. rsync
This code can produce a whole bunch of output images which I'd like to have synchronised
Is this DATA or part of your CODE?
If data: Keep out of your versioning system, just don't go there. If it is part of your code (like layout images) check it in. Those are the only ways which are the generally accepted.
A nice solution for the data would be syncing OR generating them. So you might add a step after deployment to a server: GenerateImages().
edit: In addition to the comment made by the thread starter:
If the images are data and you need to process them on a different system don't think about the version control for your code. It is unrelated. The steps which would make sense to me, in order of processing:
Start with updating your image code, check it in versioning. Then deploy (yes this is deployment) the updated code to the cruncher computer. Now code is done.
Then you have tasks which the number cruncher should handle. Like processing the images. So start that processing from either the cruncher itself (probably some queue happens there) or from a central dispatcher.
Then you have the results locally at the cruncher. Now something has to happen with that data, so that's also part of your software. Decide whether you want the cruncher to send them to some central storage, your workstation or another location. Let the software handle that. This is the most hard part as I read through your question. Many solutions are possible from just FTP/network transfers to specific storage solutions. Willing to help but need more info about the real issues, amounts, sizes etc. on these parts.
If the new updated version of the image processor makes the old generated images obsolete implement that also in your code, by for example attaching an attribute to the files generated, a seperate folder or another indication. That way you could request the cruncher after update to re-generate any obsolete files.
Related
Short version: see topic
Detailed version:
I want to use a specific 3rd party theme for Windows. I'm already using an open source solution which I've compiled myself to disable Window's restriction on Themes.
In the past, when using 3rd party theme related mods that come with DLLs (for example authui.dll for the login ui, or imageres.dll for modding system icons), I avoid using unknown DLLs by simply copying the unknown DLL's theme related resources (such as Bitmaps, Icon groups or UI scripts) unto it's virgin MS Dll counterpart. I call this resource grafting, where resource are changed but the executable elements of DLLs or exes are left alone.
Going back to the theme I want to install, I used sha256 hashing to determine that only aero.msstyles which is also modifiable by resource hacker. So I did the same thing I usually do and transferred resources from the third party theme to Window's own aero.msstyles. Problem is that I ran into a type of resource that I am unable to read or know the contents of. It's called VARIANT. From some experiments done in a VM, it seems to be some kind of binary UI script that resource hacker is unable to decompile. I usually like to be able to read any UI scripts that I transfer but I am unable to do so with this one.
Would this constitute any real security risks? Can UI scripts be side-loaded with some kind of exploit? Seems unlikely to me since the function of a theme file (msstyle) is to coordinate the appearance of the system UI but I don't know enough about the inner working of the whole theming system to be sure. I thought I'd get some other point of views before I take the theme out of the Virtual Machine.
I used vBinDiff to compare the hex code the altered VARIANT/NORMAL binary to that of the original theme. You can also copy the binhexes and save them to two text files which you would compare with WinMerge.
vBinDiff and WinMerge will highlight what modifications and what additions/substractions were made to the binaries, displaying them side by side. I read through the differences, 90% of them were no larger than 4 octals (4bytes), typically what you would expect to see when modding colors using a hex editor. The biggest divergence was an added 32bytes of code.
There are two possible explanations for the such an addition: (1) the author added extra image resources and added the entries necessary to reference them, (2) there is some kind of unwanted code that has been slipped in.
To address the possibility of 2, I did a search to see how small trojan code can get. How likely is it that a trojan has been stuffed into 32bytes if compiled UI scripts? I found a few mentions of an old 17byte virus from the DOS era called trivial which I disegarded right away because it would become apparent very quickly given it's known behavior. As far as full fledged trojans with backdoor and downloading abilities, the smallest I found was 20kb (trojan tinba), discovered in 2012. There is also Catchy32 which is still considered a Trojan but with simpler and very specific functionalities and that one's about 580 bytes (reference). Based on this info, I established that it is highly unlikely (if not impossible) to slip any code in 32bytes of code and established that the resource in question is clean.
Mind you, this doesn't answer the question I asked (can binary UI script resources in a theme carry mal-code) but it does solve my dilemma. Thought I'd share it.
I want to manage different sets of file versions locally on a machine without using complex version control tools like TFS/Git/SVN...etc. here is my use case:
I have a Windows virtual machine that contains many xml, xslt, xsl, txt...etc. files, the virtual machine gets updated with every release of my product.
Often I need to analyze errors in this virtual machine, so I change many files and run the product and start analyzing, let us call these file changes FileChangeSet1.
based on the results above I need to change other files and maybe some of the files in FileChangeSet1 and do another test.
again based on the results, I need to change more files, eventually I end up with FileChangeSet1, FileChangeSet2...FileChangeSet(n)
I want to:
be able to switch between these file change sets easily and quickly, e.g. have a GUI that shows my my tree of FileChangeSets then click one of them and all files of that change are used.
create file change sets from other file change sets e.g. copy FileChangeSet1 in FileChangeSet2 and change only one file in set 2
I don't want to configure and install a complex version/source control system like TFS/Git/SVN where I have to create a database of all my files first.
Making snapshots of the virtual machine is not an option because it is extremely slow.
I think you would not have much advantage with version control tools even because they are made to version text files. For binary files, I think you would end up like managing several diffent copies of the binary files anyway (at least for older tools such as CVS and SVN).
If you are running in linux, you may want to use cmp/diff tools. Take a look on incremental diff and diff tools such as patchutils.
Consider also to create a checksum of huge files to avoid comparing them for nothing.
ps. also take a look on this - http://jojodiff.sourceforge.net/ - haven't tried but it seems simple to use and promising.
Mercurial is the right tool for me. With it I can solve my business case easily as follows:
Install mercurial on Windows, it integrates in the Windows file explorer.
Create a local version control mercurial database by right clicking my root folder.
Now I can open all my files under my root folder in different text editors e.g. notepad++ and modify these files.
When I want to save/remember a specific status I simply commit the files to mercurial by right clicking the root folder, I can provide a commit note.
Later I can change my files in a different way and test how my system reacts to them, again I can commit these files locally.
Over time I have a history of change sets in Mercurial, I can go back to any change set, branch it, merge it...etc.
I have a huge and complex system that contains thousands of files, my root folder is actually the C:\ drive, I can easily and quickly make out of c: a version control database using mercurial.
All with a simple and intuitive GUI, no command line learning needed.
We have a file share on a debian server which people will be putting mostly small plain text, word, excel, pdf, misc, files into. We want to keep file level versioning of everything placed in that file share. We'd like all of the versioning to happen automatically in the background every time a change is made to a file.
What's the easiest open source way to do this? What's the best versioning system to do this with? Is there an automated versioning system like this out there somewhere?
I'm not asking for complete instructions. I'm just looking for recommendations or keywords to search with (other than "automatic versioning files" which didn't find me much).
We're already backing up the volume many times a day. We don't want to do volume snapshots etc. We're looking for some kind of automated file level versioning.
Update:
Using incron to autocommit changes in a folder:
http://andrew.mcmillan.net.nz/blog/using_incron_to_autoversion_a_directory
Thanks to using the new keyword "autoversioning" I found the above which I'm interested in. It's a script which is triggered from incron every time a file changes. One problem is incron isn't recursive.
I would try a versioned filesystem, such as ext3cow or nilfs.
I have no experience with them, but they have always looked like a very interesting idea to me.
The SVNAutoVersioning directive to mod_dav_svn is one solution.
I have a bad fortune of using ClearCase with UCM at work. We try to find a way to get user experience smoother while using it. One of the main problems is that all files are read-only until checked out. This interferes with using IDEs and their refactoring capabilities. Unreserved checkout changes little as it still requires perform check out explicitly. Can Edit-Merge approach be somehow emulated? For example, can ClearCase somehow make unreserved checkout on file modification automatically and not impose read only mode on them?
The way we approach the interaction between:
refactoring operations
ClearCase
is to use an IDE with a good ClearCase plugin:
The only way to preserve the history for files is doing the refactoring through an IDE like Eclipse without checkouting the files first: The ClearCase plugin will 'cleartool move' the files from their old name to their new names and checkout/checkin them, preserving their history.
The other manual way is less ideal:
You isolate that operation in a snapshot view in which, just for this refactoring, we clear up the read-only bit from all the files (so they are all in "hijacked" state).
Once the refactoring is done, we use the result as a source to import in a dynamic view.
clearfimport -mirror.
You can use a dynamic view as the destination, but the end result is new files with no link to their former filename.
The main idea being: you don't refactor your code every 5 minutes or even every day, so , with the right tool, you could have an acceptable way of mitigating the file-centric nature of ClearCase.
We (occasionally!) have to issue hot fixes for our product and do this by reissuing the affected files directly rather than with a new installer. The product has a large number of pieces, some managed code, some unmanaged.
Currently development flags which build artifacts (exes, dlls) need to be shipped in a hot fix. We'd like to be able to identify these automatically by comparing them to the previous build. A simple binary diff doesn't work since the version numbers on all the files have changed as stamping the files with a new number if part of the build.
Are there any tools that will do a more intelligent comparison and decide which files should be included? We'd still have a developer check the list, this is more to catch files the developer didn't think of than the other way around.
(Note: changing the hot fix/build process is not an immediate option, whether or not we should be shipping individual files is a different discussion!)
These are the options I see:
On your build machine get a report of the files that were changed and use the directory structure of the file path to determine which dlls were really updated. Not sure if this breaks your "no build process changes" rule or not.
If you want to wait until after the build I would recommend using a binary file diff tool like http://www.romeotango.com/Downloads/FileCompReadMe.txt. Using that you can get back a set of diffs so you just need to get your script that uses the tool to ignore the diff that occurs as a result of the version number. You can figure out the pattern to how the version number appears by using a controlled scenario where you know the two binary files are the same except for the version number and note where the differences are. Do that for a few of your dlls and hopefully a pattern emerges enough so that you can script it.