Over the last few years, I've started dabbling in scripting using Python, R and Matlab. My background is not in computer science, rather I'm an ecologist by training. I've written and gathered dozens of scripts from each programming environment since I began dabbling as a computer ecologist. To further complicate matters, I have copies of scripts from a work server that I run on personal computers. Things have really become out of control and I hope to restructure my organizational scheme. To date, I've loosely organized scripts by project within folders--which has led to dozens of scripts within dozens of hidden folders. I also have folders within folders of previous versions of scripts. Finally, my question for the experienced programmers out there: What are some best practices for organizing years worth of old scripts and starting fresh? For example, should I focus on a creating a central repository or is a project-based approach more appropriate? How should I incorporate handling different, older versions of scripts? Any advice on how to proceed would be appreciated.
I keep mine by language, for example
/opt/python
/opt/r
/opt/matlab
I also have folders within folders of previous versions of scripts.
You my friend should discover the greatness of git
Related
I often find myself working on two different computers in different places and i was wondering how to share as much as possible between these PCs with the least issues possible.
At the moment i'm using dropbox to share the projects i work most often on (2-3 projects, so no big deal) but my whole workspace is actually bigger than my dropbox space.
What i would like to do is to share all my projects, Eclipse and also the working environments (or at least the databases) and i was thinking to either use an USB hard drive or a repository to do this. There are a lot of issues i can think of about this and i'm pretty sure there are even more issues i haven't thought of:
First of all i know one day i'll forget the hard drive home when going to work and i'll screw up a job day by going back and forth between home and my office so using the repository sounds like a better idea
How bad can be to share database and IDE files? Though my computers run Windows 7 and 8 and both have a 64bit architecture i reckon that sharing such things might lead to bad behavior or things simply not working (for example if somehow one my PCs will assign a different drive letter to it things might not work and i'll have to manually assign it, leading to a waste of time).
Also using a repository for this might not be ideal since it might create a lot of useless versions. I'd love to not have to configure and upgrade each IDE separately and to not update databases manually but i can live without this feature as long as i can share my projects
In case i use an USB hard drive and put even Eclipse and the databases in it, how bad it could be performance-wise? I'm pretty sure using an USB drive will affect performances a lot
A lot of the projects i work on already use a repository, I have projects using Git, SVN and Mercurial. Should i use one of these to share all my projects how will interfere with already existing repository information?
Am i right in thinking that sharing a whole Eclipse workspace (so including will cause issues if the paths between the 2 computers are different? I might have to set thing up differently for each PC (for example if i use external programs and libraries, such as Android SDK, they might be in different places)
And i'm sure there will be more on this. So, my questions are:
1) Is it possible to share IDE and databases (that is, without using periodic import/export) between two PCs without issues or is it better to just share workspaces or even just single projects?
2) Taking into account the issues above, what's the best strategy to share files? I can accept different ways to share files than the two i thought of.
I think you can have an identical IDE installed on computers (All the configuration should be the same, you can configure your Eclipse and zip the Eclipse files copy it on another computer then you have an identical IDE) also using maven as build tool makes it easier for you to share project between IDEs (on paper, you even don't need same IDEs) and for the workspace i guess the best solution would be a source control system,having share folders/usb drives can easily end up in mess.
I'm currently working with a tool that over the years has evolved naturally from a number of perl scripts accessed through an apache web server, to a huge collection of tools using a common database and web site (still running apache, but using catalyst instead of CGI).
The problem we're having is that different departments have done local branches from the common main branch for implementing their own new functionality and adaptations.
We're now charged with the task of deciding how a common platform can be made available where certain base functionality is made one track instead of having all these different branches.
These kind of problems must spring up all the time so I'm hoping someone have a good strategy to offer as to how we should continue from here. Any thoughts would be appreciated.
In general, make sure you have buy in from everybody involved. Trying to do this kind of project without having people on board will just make your life more difficult.
Look for the quick wins. What functionality, if it changed, would have the fastest and clearest beneficial effect across all departments. If it takes you three months to get some good out of it, people won't rate the good results very highly.
Break functionality down as far as you can. One of the biggest problems in forked legacy systems is that a seemingly innocuous change in one place can have huge ramifications elsewhere because of the assumptions made about state. Isolating state in different features should help you out there.
I need to come up with a CM process for PLC code.
Currently, the system is developed using RSLogix 5000. The build product is a monolithic file that can be loaded onto a PLC for execution and edited directly in the development environment. With multiple developers, this has become a problem. They're stepping on each others changes.
As an analogy, it's as if, when doing Java development, the only wway to edit and save the source would be to load up a *.jar file into your IDE, make the change, and then save it back to the jar file. This is less than ideal.
How can I coordinate changes between multiple developers working with PLC's?
If we are talking about one big binary files, then a VCS (centralized or decentralized) is not the best tool for the job.
An external referencial (a shared disk for instance) where a batch will copy and label the current PCL state is better.
See "Tracking Software History"
To avert discontinuities in the historical record of revisions, old versions of programs must be stored.
“We take it a step further, though. Using our MDT AutoSave, we actually go out and interrogate the equipment. Overnight or at whatever frequency is specified, the software reads the programs in the PLCs and then compares that information to the last known program. The version-control software will copy the new program and store it and [then] compare it to the last one.
Launching version control is fairly simple. Required is software installation and then hardware configuration. “You would need a server and a couple of weeks of engineering and you’re good to go,” Perysyn says. However, his company uses a “shrink-wrap approach” that involves installing the software and then customization by users filling in the blanks.
That being said, when you have multiple changes from multiple developers, you need an integration environment where a first delivery can be done and validated, before pushing it to the actual server.
See also this post.
I use Unity Pro, so this may not apply for other brands.
Unity can export an "archive" file which is XML which describes the PLC program and IO setup in its entirety. After commissioning changes, I create an export and check it in to my local Git repo. This gets me an annotated history of changes, but no visual comparison. I can always use UnityDiff for comparison.
Check out http://www.mdtsoft.com/ also
You need specialized versioning system for PLCs like VersionDog.
From the manufacturer:
"Special support with Smart Compares for SIMATIC S5, SIMATIC S7,
SIMATIC PCS 7, WinCC, WinCC flexible, InTouch, CoDeSys, TwinCAT,
Phoenix PC WORX, RSLogix, Schneider Modsoft, Schneider Concept,
Schneider Unity, SINUMERIK 840D, Bosch IndraWorks and more. Also robot
programs from ABB and Kuka and office related data formats like
Microsoft Word, Microsoft Excel and Adobe PDF are perfectly supported
by versiondog.
Update: Here is a screenshot showing ladder version compare. I guess that's what most PLC folks are interested in. We also use it to schedule e-mail report if PLC offline and online application versions are a match, as an alarm that something has been changed in PLC but not put into version control server.
About RSLogix5000 specifically, I have seen developers use an emulated PLC and make their changes online. The final product once developed is then put together with all the comments (as they are not contained in the PLC) and then commissioned. There are issues with changes that cannot be done online, such as AOIs. There are tools in place to stop two people editing the same logic online at once and to take ownership of sections. Backups can be done in the form of uploads, but there isn't any way to track changes.
It is a messy problem, messier still for when you are maintaining a system as you want an .ACD that you can go online with, as unless you are somehow doing a diff with the RSLogix compare tool you just see unreadable machine code like "+|Éû³´¬ÙÆW×晵‚>Ù,"
The most common revision control I have seen (sadly) is just saving the the latest file, then taking a copy and adding the current date to the file name, like the recommended control.com post described.
RSLogix5000 has always prohibited multiple users from opening and editing on the same .ACD simultaneously. However, if multiple users have identical .ACD files, open them, and all make connections to the same target controller, they each can edit on the controller simultaneously, but only if they are working on different routines. Other's edits appear automatically, if they were to look at another programmers routine.
Note that working online like this is usually done with the PLC running, even sometimes with the target system (some kind of machine) operating. This kind of arrangement for the purpose of completing work faster, or in some cases because the system is huge. No one develops like this, as it is really a debug tool and impractical for significant changes.
If one programmer finishes, and another is not done, the unfinished work of the other will be saved to the first programmer's .ACD when they save. Whoever saves last will have everyone's work.
Like others have mentioned in this thread, using file date is fairly reasonable. Some companies use a version control variable that is usually displayed on a connected HMI. Other companies use a separate document that documents who and what changes. Sometimes version notes are placed in a lengthy rung comment in the main routine.
My company uses a separate change log, and dated archive copies are maintained. Multiple programmers are only used in the most extreme cases. Someone is always designated to maintain the offline file integrity, usually the person who will be working the longest, or the project manager.
It is important to note that rung comments are not carried from one user to another before RSLogix5000 v21 because previous versions didn't store comments on the controller.
All this said, you might be trying to manage offline development. I haven't seen any sophisticated methods for this. Usually programmers write the needed routines separately, and a project manager will assemble them into a single project. The cleanest approach I've seen is where a project manager will create an architecture with global functionality, and assign routine work to others, giving them a copy of the .ACD to work with. They return the .ACD with changes, and the project manager copies and pastes their routines into the "master" project.
This is a very good question and it really depends on what you want it to do.
If you are only using Rockwell equipment it might be helpfull to look at their solution, I think it's called FactoryTalk AssetCentre.
Currently I am looking into using Bazaar from Canonical.
One thing that VonC pointed out is that a piece of software that can interogate the PLC is a deffinate plus, not a must in my oppinion but it sure as hell helps.
Am I reading your question properly and you have multiple developers working on the same PLC code at the same time? It's a scary thought but I know it sometimes needs to happen, Siemens PLC's are a bit easier to program with multiple developers but I would assign one person to consolidate and test all the changes before committing to the PLC. Any CVS system will let you create branches for every developer but how you would get them to consolidate their changes is the million dolar question.
Bart.
A simple thing to do would be to do a text diff on the .l5k files so you can easily see whether a developer has been messing with part of the file that is outside of their scope.
I saw this question just now from a link at stack exchange: Are There Realistic/Useful Solutions for Source Control for Ladder Logic Programs. Rather than have a link only answer, I'll dupe my answer here:
There is actually a canned solution - from GE-IP of all places. Check out Proficy Change Management. This product does version control from a PLC control systems point of view, rather than a pure version control of files point of view - it works as a layer sitting on top of a VCS (the scary part is that originally this VCS was Visual SourceSafe) and handles rights management, reporting and checkout/checkin.
While the product is from GE-IP, it is designed to support a variety of PLC and HMI systems out of the box.
Full disclosure, I used for work for a company selling and installing PCM (but that was 7 years ago). So if you ask me what it was like back then I'm likely to tell you where it all went wrong!
In my company we just started a trial with Copia.io
Check it out. Our first tests look very promising!
It brings, branching, merging, ladder diff etc... for multiple PLC platforms (Rockwell, Siemens, Codesys)..
PS. I work for a company that builds machines, we were looking for version-dog alike solutions with a bit more power in collaboration and diffing capabilities. I used tools like Mercurial, Git, Tortoise in past companies (not for PLC though).
I need to create some way to get a local WAR file deployed on a Linux server. What I have been doing until now is the following process:
Upload WAR using WinSCP.
SSH into server using PuTTY.
Move/Rename/Delete certain files folders to prepare for WAR explosion.
Explode WAR.
Send email notifying users of restart.
Stop Tomcat server.
Use tail to make sure server stopped correctly.
Change symlink to point to exploded WAR.
Start Tomcat.
Use tail to make sure server started correctly.
Send email notifying users of completed restart.
This stuff is all relatively straightforward. And I'm sure there are a million and one different ways to do it. Id like to hear about some options. My first thought was a Bash script. I have very little experience with scripting in general but thought this would be a good way to learn. I would also be interested in doing this with Ruby/Python or something current like this as I have little to no experience with these languages. I think as a young developer, I should definitely get some sort of scripting language under my belt. I may also be interested in some sort of software solution that could do this stuff for me, although I think scripting would be a better way to go for the sake of ease and customizability (I might have just made that word up).
Some actual questions for those that made it this far. What language would you recommend to automate the process I've listed above? Would this be a good opportunity for me to learn Bash/Ruby/Python/something else, or should I simply take the 10 minutes to do this by hand 2-3 times a week? I would think the answer to this is obviously no. Can I automate these things from my computer, or will I need to setup the scripts to run within the Linux server? Is the email something I can automate or am I better off doing that part myself?
More questions will almost certainly come up as I do this so thanks to all in advance.
UPDATE
I should mention, I am using Maven to build the WAR. So if I can do all of this with Maven please let me know.
This might be too heavy duty for your needs, but have you looked at build automation tools such as CruiseControl or Hudson? You might also want to look at Integrity, which is more lightweight and written in Ruby (instead of Java like the other two I mentioned). These tools can do everything you said you needed in your question plus way, way more.
Edit
Since you want this to be more of a learning exercise in scripting languages than a practical solution, here's an idea for you. Instead of manually uploading your WAR each time to your server, set up a Mercurial repository on your server and create a hook (see here, here, and especially here) that executes a Ruby (or ant, or maven) script each time a changeset is pushed from a remote computer (i.e. your local workstation). You would write the script so it does all the action items in your list above. That way, you will get to learn three new things: a distributed version control paradigm, how to customize said tool, and how to write Ruby scripts to interact with your operating system (since your actions are very filesystem heavy).
The most common in my experience is ant, it's worth learning, it's all pretty simple, and very usefull.
You should definately automate it, and you should aim to have it happen in 1 step.
What are you using to build the WAR file itself? There's some advantage to using the same tool for build and deployment. On several projects I've used Ant to build a Java project and deploy it to the servers.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
For years, one of my most important tools has been incremental copy utility that compares the contents of two directories and shows me which files are newer / older / added / deleted. Every day I find myself copying folders of source code between my two desktop machines and the server, and such a utility is critical to avoid overwriting newer files with older ones and also to save time by only copying changed files. In addition, the utility allows me to see new files in the source folder that I don't necessarily want to copy (like temp files) that I instead can delete.
Like anyone who subscribes to the NIH way of thinking, I wrote my own utility to compare the contents of two folders and let me mark files to be copied, deleted, diffed or ignored. I've had many versions of this utility going back to DOS, OS/2 and Win32.
I use this utility on a daily basis, and it leaves me wondering: What do others use? Surely there are similar programs out there to do this... My utility doesn't have a diff screen, and it would be occasionally nice to see what the difference is between two changed files.
What do you use for comparing and incrementally copying between folders?
rsync. All the time.
The biggest benefit to rsync is that it trades increased CPU time for decreased transfer bandwidth, as CPUs are super fast nowadays, and even disk-copy is relatively slow, this is a good thing.
I use rsync for some jobs, and unison for others.
For your situation, I would strongly recommend using some version control solution such as Subversion.
As for NIH? While I have written a large number of tools over the years, I always look for an existing tool before writing my own. It saves time, and may have a better solution than I would have used. At the very least, it will give me some "how NOT to do it" examples.
SyncToy is also good at this stuff.
SyncBack (free) or SyncBackSE ($$) is another possible solution. SyncBackSE is one of the few programs I've ever paid for. Health warning: Win only.
IMHO, NIH violates Laziness and Impatience, though it strongly supports Hubris.
I tried Robocopy (included in Vista and available for download for XP) today and it worked fine. To incremental mirror a drive I used: robocopy source destination /MIR
There is also a GUI available: http://technet.microsoft.com/en-us/magazine/cc160891.aspx
robocopy. It's in Vista, and is also part of the Windows Resource Kit. It has a strange command line interface, but it's very powerful & good for this kind of thing.
Still, I find myself wondering whether source control would be a better choice for you.
NIH is not necessarily a bad thing. It can be good when the application has some very personal traits and you want it to be as convenient for you as possible, screw the generality.
I've also rolled my own utility (a Perl script) for this purpose a few years ago, and I'm using it both and home and at work for backups. It's simple to run because it does exactly what I need and only that, and simple to tweak because it's written in a flexible scripting language.
I have been using rsync (Linux to Linux / WinXP to Linux) a lot and like it. Today I tried to get it up and running under Vista and that was quite a challenge. I got it running but are having some issues with network drives and localized chars (i.e. åäö).
SyncToy seems pretty sweet! I noticed that it puts a data file in the synced folders. Anyone knows if it is possible to use it without the data files / have it save the data files to another folder?
I have to try robocopy as well.
Thanks a lot!
Have you considered using version control tool to accomplish this? It will allow you to keep things in sync while also remembering the history of a project.
A few good tools to try here.
Rsync is the ultimate for permissions and deltas on linux but on windows your usually using it with posix so acl permissions aren't perfect.
The people who made delta copy also made a pure windows version from the ground up that is based on rysnc's algorithm:
http://web.synametrics.com/Syncrify.htm
I haven't tried it but hopefully it will bring back good permissions to windows with the amazing incremental offerings of rsync.