All change comments to perforce branch between 2 labels? (including merges) - version-control

Our perforce admin limits "max-row" scans so that my first idea of running the following will not work:
All changes including integrates into a branch at particular label time 1
All changes including integrates into a branch at particular earlier label time 2
Subtract time 2 changes from time 1 to get the new changes with comments.
Is there an alternative way of getting the same result without having such a massive query(when perforce contains 7yrs of history and -i triggers a scan back to the dawn of history)
Based on Gregs comments added this comment:
Basically the point is to see what bugs got fixed in a particular release branch between 2 labels(or more commonly, some old label and today).
I wish to simplify(speedup) way too complex script that we currently have which looks at changes that went into a release branch, it follows files that went into them at least 2 branches up in order to printout all the changeset comments from the original change(the interim merge comments tend to just say something like merge123 etc instead of description of the actual change comments, so we need to walk up the tree to the original comment as well), script finally outputs something like below(we put quality center IDs into changeset comments):
qualityCenterId123 - fixed some bug
in gui qcId124 - fixed some other
bug qcId125 - fixed some other bug
merge123
UPDATE based on comments:
Problem with Toby's approach is that most of the changes into the code branch came via integrations, -i would include those change, but as stated that explodes the query to such a degree that due the load on perforce server our admin won't allow it to run. So this is why I am looking for an alternative approach to get the same result.

I can't see an easy answer to this, but do have a couple more suggestions that perhaps may help point in the right direction.
Persuade your admin to raise the maxscan rows limit. If he is nervous that this will lead to problems with the whole user base, just get him to add you to a new user group (e.g. "Scripting"), and set the limits for just that group. That will have the effect that only members of that group can use the upper limits, and you can then negotiate for suitable times to run the script. You could even do it overnight.
Have a look at the P4 admin guide and see if any of the hints on scripting will help - e.g. maybe a tighter view on the data will limit the query enough to not break the maxscanrows limits.
How's your SQL? You may be able to construct an efficient query using the P4Report tool.
Try asking the question on the Perforce mailing list. It's a very active list that has a lot of very experienced people who are very helpful. See this link for the sign-up page. There's a good chance that they will suggest some good approaches.
Probably too late for yoru existing labels, but consider using the job system to track work. Perforce has inbuilt query tools to track what jobs have made it into different branches. It does require a working-practice change for your team, however.
Sorry I can't provide a more specific answer.

Are your labels more than simply the most recent changelist when they were created? Eg did you really need to record specific files in client workspaces? If not you can very easily compare the two changelists closest to labels.
Say the closest change to your first label date is 23000 and your closes change to your second label date is 25000 then
p4 changes //depot/PATHTOMYCODE/...#23000,#25000
will give you all changes to your code path between these two changelists.

Won't a normal Label-diff do what you want?
From P4V, Tools->Diff. Select the two labels
From P4Win, right click label, select diff files in 2 labels
From command line, p4 diff2 //codeline/...#label1 //codeline/...#label2
Or am I missing exactly what you are after?
Further suggestion after Ville's comment on the above
If you are only after the info per changelist, rather than per file, then try "p4 interchanges" from the command line. This will give you just the summary of what changes in one branch have not happened in another, and you can supply a revision range to limit it to the labels you need.
Do "p4 help interchanges" from command line for details.
Unfortunately the interchanges command is not yet exposed in P4V or P4Win.

Toby Allen's Answer is the best approach if your labels are simple changelists.
If the labels are more complicated, then I think you need to look at all the files in each label and where their versions differ, find the changelist where the version changed.
You can get the file and version lists with:
p4 fstat -Of //...#MyLabel
EDIT:
Consider two complex labels:
VERSION_A:
//depot/file_A.cpp#4
//depot/file_B.cpp#7
//depot/file_C.cpp#1
VERSION_B:
//depot/file_A.cpp#6
//depot/file_B.cpp#5
//depot/file_C.cpp#4
In this example, the labels do not describe a particular changelist, the head change for each file may be different.
If you can have labels like this, then you can run the p4 fstat command on each label and then find the differences. In this example, file_A.cpp has changed twice and file_C.cpp has changed 3 times. file_B.cpp is older in the second label, so it can be ignored.
So now you need to look at the changes that involved these versions:
file_A.cpp#5
file_A.cpp#6
file_C.cpp#2
file_C.cpp#3
file_C.cpp#4
Those changes can be retrieved with p4 filelog, so you want to run something like this:
p4 filelog file_A.cpp#6
p4 filelog file_C.cpp#4
Then you need to remove any duplicates and any history for earlier versions.
Like I said, you only need this if you have messy lables. If there's any way to make your labels represent changelists, you should use Toby Allen's answer.

Related

VSTS: finding project-wide changes between two change lists or labels

I have a large project that is hosted in VSTS. I'd like see the differences in all files, project-wide, between two change lists. (It would also be helpful to do the same between two labels, since I have labels at the two points of interest.)
We have a problem that manifests between one build of our product and another; one way to go searching for the issue is to comprehensively review the code that changed between those two builds.
How can I show the differences in all files, project-wide, between one change list and another?
I've tried this:
tf diff . /recursive /version:100~150
and it seems to work, but it takes a very long time; and the output is a plain text diff. Can I browse the changes graphically, like in BeyondCompare or the IDE? Anything I can do to speed up the comparison?

ClearCase merge-conflict-on-rebase mystery -- why does manual merges are sometimes required when doing a rebase for a file that has NO local changes?

Here's a rather advanced question for true ClearCase experts:
I frequently perform a rebase on a ClearCase snapshot view that has just a very limited number of small changes in few files (e.g. file1.c, file2.c, file3.c).
I do it with the following on the UNIX command line:
cleartool rebase -recommended -complete
Sometimes, while this command runs, out of the blue, and for no exlained reasons (yet), I get prompted for manual input to solve some "merge" conflicts. But they make no sense to me, as they happen in file(s) that I NEVER EVER TOUCHED -- and which ONLY ONE OTHER DEVELOPER EVER TOUCHES.
The "merge" prompts I see when this scenario happens during a rebase look usually like:
"Do you want INSERTION from file x? [yes/no]"
or
"Do you want DELETION from file y? [yes/no]"
or
"Do you want CHANGES from file z? [yes/no]".
Etc.
I have no clue why these "conflicts" are happening. Additionally, it's really hard (see impossible) to make good decisions because the details are shown with a very narrow number of columns, and there's hardly any way to guess right. Using graphical merging is not an option here because this is meant for an automation script that should ideally never ask for user input.
What I do know about this scenario is:
We have a team of 6 developers. 5 of us usually work the same limited number of files... say file1.c, file2.c, file.3.c
I work on a child development stream on these three files. And when I'm done, I normally deliver up to the default parent stream.
On the occasions where the "merge conflicts" on rebase happened, it's always on a totally DIFFERENT FILE -- one that is ONLY EVER TOUCHED by JUST ONE other developer in the team (it's a module that HE owns, NO ONE EVER TOUCHES THAT FILE BUT HIM). Let's call him developer #6.
When this strange "merge conflict" on rebase happens, I've usually been working for an extended time in my own development child stream (always with a snapshot view), and I've done a couple rebases (at least 3) to bring other changes ALL made by other developers (in file1.c, file2.c and file3.c) and which I needed to complete my work.
But, the other developer (#6), the ONLY ONE working on banana.c, had made changes to that file, in at least two of the three rebases activities that were created in the snapshot view of my child development stream.
Again, I repeat, I NEVER touched banana.c, and none of the 5 other developers ever did, except for the guy (#6) who owns the banana.c module.
And there, it happened - ClearCase asked me for manual input to solve a "merge" conflict with banana.c when doing "cleartool rebase -recommended -complete".
How can this be possible???
How can a file require "merging" when doing rebase if there is ONLY EVER one single developer making changes to it?
It's as if ClearCase got confused between different versions of banana.c in at least two of the three rebase activities automatically created in my stream (which both modified banana.c) and prompted me for "merge conflict" resolution (even though I never ever touched banana.c and only one developer (#6) ever did modify that file).
.
.
.
UPDATE: AUGUST 31ST 2015
Here's a log from an occurrence of the problem on August 28th 2015
Needs Merge "/view/MYDYNAMICVIEW/vobs/DIRLEVEL1/DIRLEVEL2/SOMEFILE.cpp" to /main/MAIN_INT_STREAM/SUB_STREAM/CHECKEDOUT from /main/MAIN_INT_STREAM/SUB_STREAM/MY_DEV_STREAM/37 base /main/
MAIN_INT_STREAM/SUB_STREAM/150
********************************
<<< file 1: /view/MYDYNAMICVIEW/vobs/DIRLEVEL1/DIRLEVEL2/SOMEFILE.cpp##/main/MYDYNAMICVIEW/SUB_STREAM/150
>>> file 2: /view/MYDYNAMICVIEW/vobs/DIRLEVEL1/DIRLEVEL2/SOMEFILE.cpp##/main/MYDYNAMICVIEW/SUB_STREAM/MY_DEV_STREAM/37
>>> file 3: /view/MYDYNAMICVIEW/vobs/DIRLEVEL1/DIRLEVEL2/SOMEFILE.cpp
********************************
...CUT FOR BREVITY...
*** Automatic: Applying DELETION from file 3 [deleting base line 123]
...CUT FOR BREVITY...
*** Automatic: Applying INSERT from file 3 [lines 123-124]
...CUT FOR BREVITY...
*** Automatic: Applying CHANGE from file 3 [lines 1329-1335]
...CUT FOR BREVITY...
*** No Automatic Decision Possible
merge: Warning: *** Aborting...
Missing charsets in String to FontSet conversion
Missing charsets in String to FontSet conversion
Missing charsets in String to FontSet conversion
Cannot convert string "-misc-kochi mincho-medium-r-normal--0-*-*-*-*-*-*-*" to type FontSet
...GMERGE POPPED HERE...
Moved contributor "/view/MYDYNAMICVIEW/vobs/DIRLEVEL1/DIRLEVEL2/SOMEFILE.cpp" to "/view/MYDYNAMICVIEW/vobs/DIRLEVEL1/DIRLEVEL2/SOMEFILE.cpp.contrib".
Output of merge is in "/view/MYDYNAMICVIEW/vobs/DIRLEVEL1/DIRLEVEL2/SOMEFILE.cpp".
Recorded merge of "/view/MYDYNAMICVIEW/vobs/DIRLEVEL1/DIRLEVEL2/SOMEFILE.cpp".
I never touched SOMEFILE.cpp - only ONE other developer ever changes it- why does it require a merge?
My net impression at the moment is that ClearCase's automatic merge is doing a bad job.
Could it be a good idea to think of using the "-qall" or "-qntrivial" options to disable ALL/MOST automatic merging -- and do EVERY/MOST merge manually? (or with an external tool?)
1 & 2 How can this be possible???
This "Do you want the CHANGE made in file 2? [yes] no" message only appears when 2 contributors differ from the base contributor.
In that case, a cleartool lsvtree (not with -graph, since you don't have X-Window server) might help seeing the version involved, and trying to make some cleartool diff to see the difference (compared to the base contributor)
3/: that is one example where, if possible, working collaboratively on the same stream/branch (instead of working each developer in one own's stream) would be better.
Regarding the update of August 2015, the key error message is:
Missing charsets in String to FontSet conversion
See technote "Using GUI results in "Missing charsets in String to FontSet conversion" warning"
Possible causes include:
An improper setting of the locale variable. For example it may be set to UTF-8.
The file of interest is the Linux/palette, which defines the actual fonts used in the environment. This file is read to determine the fonts that can be displayed in the ClearCase GUI.
The palette file does not contain the correct fontset.
This issue was identified as a product defect and logged under APAR PK30799.

Search code history for closest matching version based on content

I have a file that was forked from a project at an unknown moment in the past. I want to identify as closely as possible the moment of that fork. The file has been changed since the fork-moment.
Winmerge highlights about about 20% of the lines, with about half of those being just a few characters within the line, a path change or inline function turned into a variable or function call for instance. (20% after ignoring whitespace change and enabling moved-block detection that is, closer to ~40% without that.)
I don't have to worry about branches, the original version control system was CVS. (I don't have access to the CVS file system). I have a git imported version with tags corresponding to the CVS commits, and could generate the same with Mercurial for little effort if need be.
I don't care about matching the specific CSV commit date/time/number/whatever. The goal is to identify when the content of new file started drifting, and step forward through the revision history, cherry picking what to merge to the forked file.
For this project I could brute force it, there only a dozen or so revisions where the fork has mostly likely occurred and the file is less than 500 lines. However it's not hard to imagine a scenario where this is not feasible and I'm curious about what an elegant solution might be.
How would you go about solving this?
"Brute force" sounds as if you were contemplating testing all revisions. Normally one would use a binary search. To decide if it was a good match, I'd normally use just the numbers from diffstat (since you say there are post-fork changes). Accounting for block-moves complicates things, though.

How to join two files in a version control system

I am doing a refactoring of my C++ project containing many source files.
The current refactoring step includes joining two files (say, x.cpp and y.cpp) into a bigger one (say, xy.cpp) with some code being thrown out, and some more code added to it.
I would like to tell my version control system (Perforce, in my case) that the resulting file is based on two previous files, so in future, when i look at the revision history of xy.cpp, i also see all the changes ever done to x.cpp and y.cpp.
Perforce supports renaming files, so if y.cpp didn't exist i would know exactly what to do. Perforce also supports merging, so if i had 2 different versions of xy.cpp it could create one version from it. From this, i figure out that joining two different files is possible (not sure about it); however, i searched through some documentation on Perforce and other source control systems and didn't find anything useful.
Is what i am trying to do possible at all?
Does it have a conventional name (searching the documentation on "merging" or "joining" was unsuccessful)?
You could try integrating with baseless merges (-i on the command line). If I understand the documentation correctly (and I've never used it myself), this will force the integration of two files. You would then need to resolve the integration however you choose, resulting in something close to the file you are envisioning.
After doing this, I assume the Perforce history would show the integration from the unrelated file in it's integration history, allowing you to track back to that file when desired.
I don't think it can be done in a classic VCS.
Those versioning systems come in two flavors (slide 50+ of Getting git by Scott Chacon):
delta-based history: you take one file, and record its delta. In this case, the unit being the file, you cannot associate its history with another file.
DAG-based history: you take one content and record its patches. In this case, the file itself can vary (it can be renamed/moved at will), and it can be the result of two other contents (so it is close of what you want)... but still within the history of one file (the contents coming from different branches of its DAG).
The easy part would be this:
p4 edit x.cpp y.cpp
p4 move x.cpp xy.cpp
p4 move y.cpp xy.cpp
Then the tricky part becomes resolving the move of y.cpp and doing your refactoring. But this will tell Perforce that the files are combined.

Team Foundation Server - Moving Source with History

I was wondering what the best approach might be in moving source code, with history, from one Team Project to another Team Project. I am not concerned with work items, reporting, or SharePoint sites, as the system we are going to be restoring from did not use these functionalities. The reason for wanting to move to a different Team Project also is driven by the fact that the original implementation (being restored from a backup that was maintained by a third party) were using a third-party process template that we do not wish to use going forward. We want to start utilizing work item tracking and reporting after the migration is complete.
The TFS Integration Platform seems to be one likely scenario. It can be used to change the process template, according to the documentation. However, I was curious if the tf.exe move syntax might work? Something like:
tf.exe move $/ProjectA $/ProjectB
It is my understanding that this command operates much like a rename operation, whereas moving with the "Move" context menu item in Source Control Explorer is more like a delete and add operation. Also, would the tf.exe move path actually associate the code under the folders with the appropriate Team Project, assuming that $/ProjectA is the root source control folder for one project and $/ProjectB is the root source control folder for the other? The key is to be able to preserve the history, if possible.
Any advice or tips would be greatly appreciated!
Edit - Could branching to another project handle this scenario - much like Microsoft discusses in the Branching Guidance documentation? I think that this could be the answer, since the history would likely be preserved with the branch. However, I do not have access to a Team Foundation Server 2008 instance at the moment to test it.
Move and Rename are aliases. There is absolutely no difference, in any version of TFS, from the command line or the UI.
Both of them preserve history. At least in 2005/2008, you keep the same physical item in the VersionedItem table no matter how often or how drastically the name and/or parent path changes. There is actually no way to get a "fake" rename (delete + add) without a lot of manual work on your part.
However, while this versioning model is very pure in a theoretical sense, it has some practical gotchas. Because different items can occupy the same name at different points in time, TFS needs the full name + version to uniquely identify any inputs you send it. Normally you don't notice this restriction, but once have you renamed items in the system, if you say tf [doSomething] $/newname -version:oldversion then it will get confused and either throw an error or operate on an item you may not have intended. You have to be careful to pass valid combinations (newname+newversion or oldname+oldversion) to ensure commands behave the way you want.
TFS 2010 changes the story somewhat: it's a branch+delete under the covers, causing the itemID to change. Even so, everyday commands like Get and History are "faked" very well; old clients are about 95% compatible. The advantage is that when you have multiple renames in the system and path-based item lookups start to become ambiguous as alluded to above, the server will simply accept the name you specify and run with it. This improves overall system performance and eliminates several traps that unfamiliar users often fell into, at the cost of not being quite as flexible and not preserving history with 100% precision (eg when there are name collisions during a Merge of two branches).
Returning to the problem at hand...
It's not as simple as saying tf rename $/projectA $/projectB. Top level folders in the source control tree are reserved for the Team Project Creation Wizard; you can't run standard tf commands against them. What you need is a script like:
Get-TfsChildItem $/ProjectA |
select -Skip 1 | # skip the root dir
foreach {
tf rename $_.serveritem $_.serveritem.replace("$/ProjectA", "$/ProjectB")
}
[of course, you can do it by hand if there aren't too many children under $/ProjectA]
As far as the gotchas I mentioned, I'll elaborate on one right now since looking up old history seems very important to you. Once you checkin the rename, tf history $/ProjectA/somefile.cs will NOT work. By default, tf commands assume version = "latest." Any of these alternatives will the full history you want:
tf history $/ProjectA/somefile.cs;1234 where changeset 1234 was before the move
tf history $/ProjectB/somefile.cs;5678 where changeset 5678 was after the move. Or you could just omit the version.
A final alternative for completeness & debugging purposes:
tf history $/ProjectA/somefile.cs -slotmode. You will only see the changes that happened prior to the move; however you'll also see the history of any other items that may have lived in the $/ProjectA/somefile.cs "slot" prior to or subsequent to the item you moved underneath B.
(In TFS 2010, "slot mode" is the default behavior; there's an -ItemMode option to request that your lookup be traced across history like it was 2008 rather than path-based.)
EDIT - no, branching is not a great alternative. While branching does leave enough metadata in the system to trace the full history to & from ProjectB, it's not terribly user friendly in 2008. Plan to spend a lot of time learning the tf merges command (no UI equivalent). 2010 dramatically improves your ability to visualize changes across multiple branches, but it's still not the clean unified experience you'd get from a Rename.
Richard's answer above is well written and explains the situation well. I did have a couple more practical gotcha to add, however.
In TFS2010, the default behavior makes it seem like moving a file causes you to lose all history from before the move. The command my users are likely to use (and the one used, it seems, by the VS2010 GUI) is:
tf history $/ProjectB/somefile.cs
My users intend to get all the history of somefile.cs, both before and after the move. They want "the history of the code that is currently stored in $/ProjectB/somefile.cs", regardless of the filename at any point in time. Maybe other people see it differently.
The first gotcha is that the GUI that appears for me in VS2010 using TFS2010 initally shows only the history since the move. The least-recent item in the list is the rename operation. It can be expanded with a subtle little drop-down arrow. Underneath is the history from the previous location. If you don't know to look for this, it can look like your history is gone.
The second gotcha is that if you later delete ProjectA (because you've finished the migration to ProjectB, say), the history really is gone. Expanding the drop-down in the history for $/ProjectB/somefile.cs doesn't produce the older history.
Another option (and I think easier) is to import to Git then export back to TFS using the Git-TF command line tools.
Install Git-TF from binary, Choclatey or source code.
Clone a TFS folder:
git tf clone https://myAcc.visualstudio.com/mycollection $/TeamProjectA/Main --deep
Disassociate the Git Repo from the TFS server by deleting the .Git/tf folder and the .Git/git-tf file.
Configure the new Git Repo to connect to an empty TFS folder.
git tf configure https://myAcc.visualstudio.com/mycollection $/TeamProjectB/Main --deep
Don't forget the --deep
git tf pull
You should get a message at this point "git-tf: this is a newly configured repository. There is nothing to fetch from tfs."
git commit -a -m "merge commit"
git tf checkin --deep
Regarding the original command above:-
Get-TfsChildItem $/ProjectA |
select -Skip 1 | # skip the root dir
foreach {
tf rename $_.serveritem $_.serveritem.replace("$/ProjectA", "$/ProjectB")
}
Both the source and target names need to be surrounded by quotes in case there are any spaces in the full pathfilename. I found it difficult to do this on $_.ServerItem as surrounding it with escaped " returns that whole child object and not just the .serverItem string. Or if I did manage to get the string, I got unwanted carriage returns such as
"
$proj/folder/file
"
Eventually I got the command to work with the following, but I found that the history still doesn't get transferred, which was the whole point! So I believe this command is the direct equivalent of just using a right mouse click in source explorer and selecting rename (or move).
$tfsServerString = "http://machine:8080/tfs/DefaultCollection"
$tfs = Get-TfsServer $tfsServerString
Get-TfsChildItem -server $tfs "$/Dest Project/MyTestProject" | select -Skip 1 | foreach { $sourceName = $_.serveritem; $targetName = $_.serveritem.replace("$/Dest Project/MyTestProject/", "$/Dest Project/Source/StoreControllers/dSprint/dSprint2/") ; ./tf rename `"$sourceName`" `"$targetName`" /login:myUser }
Also note, it requires use of the backtick ` to escape "