In our project we are following agile practices ( Sprint ). So every day nightly build will be done. We are able to ensure the correctness of build till day before formal build. But unfortunately most of the time people are doing some major check-in at final day.
We wanted to lock some of the highly sensitive elements which would cause more trouble.
We do not want to lock the integration stream itself. We just wanted to lock some files and folders automatically. Is there any way to do it using Cleartool , (or cleartool commands in powershell)
I would not recommend locking the vob or the files:
both options would lock everything (ie any modification in any branch) for all (or most) users.
you need (from the cleartool lock man page) to be the type owner, VOB owner or root to be able to lock the files or a vob: if one of those sensitive files isn't created by you, the lock will fail (and the vob itself has likely been created by an admin)
the maintenance is too cumbersome for files (you need to maintain the list of files you want to lock)
Locking the stream or at least the branch is still your best option.
It is one simple atomic operation target to lock the right environment.
Combined with the -nusers option, you can still authorized some users to do what they need (checkout/checkins)
The OP comments:
Actually I want to prevent all the users from delivering those sensitive files.
If I lock the stream for particular user it will not serve the purpose. It will stop them delivering other files too
The -nuser option lock for all users except a few.
The idea behind the integration stream is that is is not the user who make the deliver, but the stream integration owner who, at his/her own time, makes the deliver. If that stream is locked for everyone but the integrator, he/she can control the deliver
However, that puts the control of those sensitive file on the integrator (again, locking just those file would be a bad idea, and would make sure that any deliver fails, because of those locks)
If you still want them to deliver while being able to control that the build only use a certain version of those files, then I would rather recommend:
not locking the stream
putting a baseline before final day
tweaking your build script in order for it to:
use whatever version found on final day
except for those "sensitive files" where the script would fetch their baselined version (and not the LATEST version found on final day, because said LATEST version might have been changed by some final deliver).
See for instance "Clearcase command to export an element" or
"In ClearCase, how can I view old version of a file in a static view, from the command line?".
Related
This must have been solved somewhere but I can't find a straightforward answer.
The Perforce depot has code, master data, and derived data. But only the code and master data need to be source-controlled. The derived data can be generated during nightly-build on the build machine.
Here is the problem: The users want to sync with the depot as usual and get all three above. So the derived data must be downloaded after user clicks "Get Latest Version" in P4V. They don't want to run extra scripts on their local machines either.
Is there anything I can do on the server side to make this happen?
EDIT
The reason why the derived data had better not be submitted:
The derived data are owned by a particular team as their products. These data are constantly edited by only that team on multiple machines and could have constant conflicts. So for their convenience, the data should not need to be checked out and checked in on every single edits because conflicts should be handled from the master-data end, so the derived data better be excluded from SCM.
The rest of the project teams simply consume these derived data in their work and require no change at all. They should only get one healthy batch of the derived data from the depot, say, once a day.
Submit the derived data from the build machine each night after it's rebuilt.
Have the team that needs to rebuild it themselves exclude it from their client views. This is easy to automate in various ways, e.g. via virtual streams or client spec triggers, but even if it's done "manually," it's only done once per workspace, so there's no maintenance cost.
For everyone else the derived data just syncs down normally, and you can use protections to make it read-only to everyone but the build machine if you want to make sure that nobody is checking it in when they shouldn't be.
Our Main branch was apparently just deleted and there's no record of why. (The branch still appears in Source Control Explorer - When I view the history of the branch it's empty). When I get latest on the branch it deletes everything locally. We have numerous children branches that all appear to be fine, but Main is now empty with no record of how/why. Anybody have any idea how we can figure out what happened and recover it? We have a child branch that should be a duplicate so we should be OK, but we'd really like to figure out what happened!
What may have happened
There are a few things I can think of, the most logical in this case is that someone issued a tf destroy $/project/Branch/* /recursive, that would have the observed effect.
It could also be that someone has renamed the branch, that would not be visible in the history per se, unless you turn on the "Show Deleted Items" option in the options of the Team Foundation Source control options.
Your Application Tier's version control cache may have become corrupted, the chance of this happening is very slim, but it may have caused this. Ensure you have a good backup of your databases even if this may seem the case, if it isn't you're going to need the database backup and the older it is, the more unlikely it is data marked for deletion will still be there.
How can you find out what happened?
Check the tbl_command in the Project Collection Database or access the hidden _oi activity log page on the web access server. You may be able to find the command that caused the deletion.
If that doesn't tell you, analyze the transaction logs of the SQL Server (if your server is configured to keep these).
What to do now?!
Make a backup of your TFS server or secure the ones you have if you haven't done so
If the version control cache is the culprit clearing it (on Application Tier machines) may solve your problem, the cache location shows on the TFS Admin Control panel:
Best way to go about this, is to stop the TFS server temporarily and then delete the contents of this folder.
There seem to be a few ways out:
Forget about it, take the contents of the most up-to-date branch and use that to repopulate the missing data. Just add them to the empty folder, check them in and then re-merge all other branches and resolve all conflicts.
Pro: Fast
Con: you loose history, resolving conflicts will be a horrible task.
Restore the project collection database to a previous point in time (warning! may require restore of all project collections to a previous point in time)
Pro: You get all your history back
Con: You loose changes made since the last known good backup, takes alot of work, will impact all projects in the same collection, possibly all projects on the same server.
Restore the whole server to temporary server and restore the collection with the missing data to the last known good configuration. Use a tool like OpsHub or Team Foundation Migration Toolkit to replay the changes since the disaster.
Pro: You get back to the most up to date point in time
Con: Takes a lot of time and expertise in TFS Migration
Restore the collection database and use the transaction logs to replay as much of the changes to the collection , then skip the transactions that perform the destroy. Be careful though, usually the destroy action marks files as deleted, but a job does the actual deletion in the background.
Pro: You get back to the most up to date point in time
Con: Takes a lot of time and expertise in SQL
Contact Microsoft Support and get a Field expert in the house. They may be able to restore the deletion if it was done without immediately triggering the cleanup job.
Pro: You will get back into the best state possible
Con: it will be costly
Whatever you do, make sure you have a backup of your current situation, that allows you to try different tactics, should your first attempts fail.
Consider splitting the project collection to allow other projects to continue working. You will end up in a situation were this one project ends up in an isolated Project Collection on its own, but it will allow you to move forward quickly.
OK - this is one for the record books, because inexplicably the project reappeared later in the day. All of it's history is back as well. I would have thought that perhaps the DBAs here did a database restore, but that's not possible since all of the checkins that have been happening all day are still there.
So if this happens to you in the future, just cross your fingers and wait a few hours!
p.s. I did look in the SQL logs but couldn't find anything. Bizarre!
It seems rather pointless to have everybody creating the same client for a project in Perforce, so, is there any one one could create a "public" client in Perforce from where everybody could sync from?
Edit: I meant clients like the ones you create in Perforce from a client spec
It's easier to understand the architecture, I believe, if you use the term 'workspace' rather than 'client'. Perforce applications manage files in a designated area of your local disk, called your workspace. As the name implies, your workspace is where you do most of your work. You can have more than one client workspace, even on the same workstation.
Since two different users are generally working independently, on separate workstations or laptops, they each need their own copy of the code, and they each need their own workspace so that they can control when they sync up with the changes in the server.
If you and I try to share a single copy of the code, on a single workstation, we'll find ourselves quickly confused about whose changes are whose; it's much easier for us to work independently, and to merge our changes as separate submissions to the server.
If the issue in your case is that client definitions are complex, with very intricate view definitions, then you may wish to investigate the 'template client' feature: set up a single master client with the view and options that you prefer, and then your other users can use 'client -t' to create workspace definitions that copy the view and options details from the template client.
It's possible to do this, but not advisable. Since Perforce keeps a server-side record of what files are synced to each client, you could run into a situation where:
User Fred syncs using the shared client and gets a fresh set of files.
Before any changes are committed, user Jim syncs using the shared client and gets nothing because the Perforce server thinks that the client already has an up to date set of files.
Jim could get around this using "p4 sync -f" which will force all the latest files to be synced to his workspace, but that's a kludge around the way Perforce is designed to be used.
Perforce clients are very lightweight in terms of the resources they take up on the server, so it's better not to have shared clients.
I tried to find a more complete explanation of why clients should not be shared in the online Perforce documentation, but it's not very helpful. The book "Practical Perforce" has the best overview I've seen if you happen to have a copy around.
Use a template workspace as Bryan mentioned, or consider using streams. In the streams framework you define the stream view (composition) once, and workspaces are generated automatically.
p4 sync -f is too slow. Because firstly it will delete all the files in your local and then reload the files from central depot! there is a tricky way to do. It is to create a havelist and do sync, when wanting do sync -f. details is 1,get the clientspec, 2, save it to local. 3, delete the client 4, create a same client using the saved clientspec. Therefore we save the time for delete local files.
In our office, the software we create is sent to our client's office along with an engineer and a laptop. They modify the code at the customer site, based on the customer requests, and deploy the exe.
When the engineer returns to the office, the changed/latest code is not updated to the server, thereby causing us all sorts of problems in the source code on the development boxes and laptops.
I tried to use a version control system like svn, but sometimes the engineer forgets to update the latest code to the svn server. Is there an automatic way that when the laptop connects to the domain, the version control system should automatically check for changes and prompt the user to update the code on the server, or automatically update the code to the server.
I think that the key to this is to require the on-site engineers to use a VCS at the customer site, and to make it a condition of their continued employment that the code at the customer site is in fact reloaded into the VCS on return to the office. You could say that the engineers sent on-site need to be trained in their duties, and they should be held accountable for not doing the complete job - the job isn't finished until the paperwork is done (where 'paperwork' in this context includes updating the source repositories with the customer's custom adaptations of the software).
It seems to me that it might be better to use a DVCS such as Git or Mercurial rather than SVN in this context. However, you should be able to work with SVN if the laptop dispatched to the server has a suitable working copy created for the customization work.
That said, the question is "can we make this easier and more nearly automatic". In part, that might depend on your infrastructure - it also might depend on Windows capabilities about which I'm clueless. There might be a way to get a particular program to run when the laptop connects to a new domain. An alternative (Unix-ish) approach would be to use some regularly scheduled job that runs, say, every hour and looks to see whether it is on the home domain and whether there are changes that should be submitted to the main repository.
I'm writing an application that monitors a directory for new input files by polling the directory every few seconds. New files may often be several megabytes, and so take some time to fully arrive in the input directory (eg: on copy from a remote share).
Is there a simple way to detect whether a file is currently in the process of being copied? Ideally any method would be platform and filesystem agnostic, but failing that specific strategies might be required for different platforms.
I've already considered taking two directory listings separaetd by a few seconds and comparing file sizes, but this introduces a time/reliability trade-off that my superiors aren't happy with unless there is no alternative.
For background, the application is being written as a set of Matlab M-files, so no JRE/CLR tricks I'm afraid...
Edit: files are arriving in the input directly by straight move/copy operation, either from a network drive or from another location on a local filesystem. This copy operation will probably be initiated by a human user rather than another application.
As a result, it's pretty difficult to place any responsibility on the file provider to add control files or use an intermediate staging area...
Conclusion: it seems like there's no easy way to do this, so I've settled for a belt-and-braces approach - a file is ready for processing if:
its size doesn't change in a certain period of time, and
it's possible to open the file in read-only mode (some copying processes place a lock on the file).
Thanks to everyone for their responses!
The safest method is to have the application(s) that put files in the directory first put them in a different, temporary directory, and then move them to the real one (which should be an atomic operation even when using FTP or file shares). You could also use naming conventions to achieve the same result within one directory.
Edit:
It really depends on the filesystem, on whether its copy functionality even has the concept of a "completed file". I don't know the SMB protocol well, but if it has that concept, you could write an app that exposes an SMB interface (or patch Samba) and an API to get notified for completed file copies. Probably a lot of work though.
This is a middleware problem as old as the hills, and the short answer is: no.
The two 'solutions' put the onus on the file-uploader: (1) upload the file in a staging directory and then move it into the destination directory (2) upload the file, and then create/upload a 'ready' file that indicates the state of the content file.
The 1st one is the better, but both are inelegant. The truth is that better communication media exist than the filesystem. Consider using some IPC that involves only a push or a pull (and not both, as does the filesystem) such as an HTTP POST, a JMS or MSMQ queue, etc. Furthermore, this can also be synchronous, allowing the process receiving the file to acknowledge the content, even check it for worthiness, and hand the client a receipt - this is the righteous road to non-repudiation. Follow this, and you will never suffer arguments over whether a file was or was not delivered to your server for processing.
M.
One simple possibility would be to poll at a fairly large interval (2 to 5 minutes) and only acknowledge the new file the second time you see it.
I don't know of a way in any OS to determine whether a file is still being copied, other than maybe checking if the file is locked.
How are the files getting there? Can you set an attribute on them as they are written and then change the attribute when write is complete? This would need to be done by the thing doing the writing ... which sounds like it isn't an option.
Otherwise, caching the listing and treating a file as new if it has the same file size for two consecutive listings is the best way I can think of.
Alternatively, you could use the modified time on the file - the file has to be new and have a modified time that is at least x in the past. But I think this will be about equivalent to caching the listing.
It you are polling the folder every few seconds, its not much of a time penalty is it? And its platform agnostic.
Also, linux only: http://www.linux.com/feature/144666
Like cron but for files. Not sure how it deals with your specific problem - but may be of use?
What is your OS. In unix you can use the "lsof" utility to determine if a user has the file open for write. Apparently somewhere in the MS Windows Process Explorer there is the same functionality.
Alternativly you could just try an exclusive open on the file and bail out of this fails. But this can be a little unreliable and its easy to tread on your own toes.