Related
From my understanding, Informatica Cloud, Boomi, Talend, JitterBit are all integration tools which have "Connectors" to connect to servers (and I believe these Connectors in turn call APIs to access the required data). I saw many others but none of them are free although some are open source.
Are there any tools that help you visualize the integration process for free? If not, why not?
Tools like Informatica, Boomi provide drag and drop which show the entire flow.
Talend Open Studio is one such tool. not completely visual but almost there.
There are not many free tools as data storage technologies are constantly changing. It would be expensive for developers to keep up with constantly changing technologies without a source of income.
Developing data integration tools are resource intensive. Why would anybody(any enterprise company) spend so much of their effort and give away for free. Also the provider of the tools have to provide support for enterprise level . P1 means 4 hour response which means building the capability of the support team on par with the developer. All of these cost money and time. The only way to recoup is to sell the finished product and provide services.
I may be working with Siebel CRM soon, and I'm looking for advice on using modern development practices and enterprise best practices.
Specifically I'd like advice on the following areas:
How should we set up version control (specifically with Subversion)? What kind of structure should our repository have? How should we handle branches and tags?
How can we do code reviews? How can we peer review configuration changes made through Siebel Tools that don't necessarily have any "code"? We want to review these changes for quality assurance and knowledge transfer, as well as compliance with change management policies.
What sort of change management works well with Siebel? How do we verify that only things listed in our change log are actually changed when we do a new deploy?
How can we automate testing of our application? Is unit testing even possible with Siebel? I saw another question suggesting QTP for web testing, but are there other options that work?
Are there other things we can do to implement Continuous Integration practices with our Siebel development efforts?
What recommendations do you have for naming conventions and other things that would traditionally fall under "coding style" guidelines?
How should we separate development roles from Siebel Administrator roles? What should our build/test/deploy cycle look like?
It's not likely that I'll be able to obtain any new expensive tools for this, but if there's a paid tool that provides really great ROI, feel free to mention it.
If you have other recommendations along these lines, but not specifically addressed by one of my questions, feel free to add that as well.
How should we set up version control (specifically with Subversion)?
use the guidance provided in the documentation for Siebel Tools. But please note that Siebel does not build from the files in SVN so it will only be useful as an archival tool; you cannot manage your code or build from SVN.
What kind of structure should our repository have? How should we handle branches and tags?
Siebel development code is not built or managed in SVN so this is a pretty useless thing to do. Just note the date that you built your SRF and exported your Repo and match with a tag or branch in SVN.
How can we do code reviews? How can we peer review configuration changes made through Siebel Tools that don't necessarily have any "code"? We want to review these changes for quality assurance and knowledge transfer, as well as compliance with change management policies.
Use Siebel Tools to do this. It has a built in 'checking' tool for obvious errors (all devs should be using this before they check in) and a diff tool (you will need to check against an older version of the same object - which you could drag out of SVN if you want). I normally automate the checking tool once a day and review the output logs, and automate build from the Siebel server 5 times a day and look for errors during the compile. Diffs via SVN and a standard diff tool might be possible, but the Siebel objects are stored as XML-like files in SVN and so are hard to read sometimes.
What sort of change management works well with Siebel? How do we verify that only things listed in our change log are actually changed when we do a new deploy?
?
How can we automate testing of our application? Is unit testing even possible with Siebel? I saw another question suggesting QTP for web testing, but are there other options that work?
QTP is the standard way to go - check on the Oracle web site for other vendors that they may recommend. You could also try Sikuli.
Are there other things we can do to implement Continuous Integration practices with our Siebel development efforts?
Not really.
What recommendations do you have for naming conventions and other things that would traditionally fall under "coding style" guidelines?
Checkout the appropriate section of Siebel Bookshelf for current naming guidelines and use these always.
How should we separate development roles from Siebel Administrator roles?
Not sure what you mean.
What should our build/test/deploy cycle look like?
Build a new SRF and export a new Repo from Dev once a night. Once all the dev work has been checked-in and unit tests are done take the next SRF and Repo and push into the test environment. At this point in normal software development you'd branch your SVN and continue to develop on the trunk but Siebel is different because you cannot build from SVN and you cannot easily restore a whole lot of files from SVN into your build environment, so you're best to make hot fixes for test either in dev (and pause mainline dev development until that is done) or in the test environment, and do ugly backports to the development environment (that's what most people do in fact). Build a new SRF and export a new Repo from Test once a night and once that's good, snap a copy for your Production release.
Try to stick to cycles of no more than 4 weeks (1 week for desing/prototyping. 1 week for dev, 1 week for test, 1 week for bug fixes and deployment) - any longer than that and the overhead of planning will become too great.
Hints for an easier life: Avoid eScript except in Business Services (otherwise it becomes unmanageable); use all the Siebel built-in tools instead of rolling-your own; try to avoid any roll-up functionality (it always seems like a good idea but it always destroys performance); keep the number of screens and views to an absolute minimum; do not build views when you should be building reports instead; always make sure that EIM tables match and schema extensions that you make - even if you don't use EIM right now; try to build Integration Objects to match your logical schema - they are always useful (for web services, XML publishing) and a hell of a job to build after the fact; prefer Workflow Policies over run-time Events; don't add new sort or search specifications without indexes - ever ever ever; don't make by-reference links to the LOV table; always patch; if the vendor doesn't say that you can do something, never do it.
We have set up a complete Continuous Integration toolchain for our Siebel systems consisting of Subversion, Hudson, Jira, Siebel ADM and some self-written stuff integrating everything.
This helepd a lot, although Siebel "source code" is not as suitable for standard CI approaches as, say, some Java-based project.
And, YES, it is possible to put your files - including SIF - into your Subversion repository and use this as source for your deployments.
I'm planning to blog about this in http://siebel-ci.blogspot.de/ - stay tuned.
SVN/CVS are not suitable for Siebel, a few reasons being
a) Siebel objects are db objects and SVN/CVS etc store sif equivalent of the changes.
These changes are impossible to query except for some basic queries.
b) The integration between Siebel tools and SVN is a loosely coupled integration.
The ideal integration should be with the Siebel repository and invidual tools.
Take a look at our tool Object Hive it addresses many of the short comings of a files based version control.
http://www.enterprisebeacon.com/siebel_version_control_tool.html
Object Hive has been from the ground up specifically for Siebel version control, some of its features are:
1) Object Based repository similar to Siebel repository that stores all version history.
This makes is very easy to query changes and conduct code reviews based on the changes
2) A browser based GUI that is similar to Siebel tools to query for version history (no combing sif files for changes).
3) Seamless integration - directly integrates with the Siebel repository.
No messy installation for invidual developer.
4) Powerful reporting (realtime and batch) to easily identify changes over any time period.
5) Oracle Exa-ready certified.
Can anyone explain in simple terms what the difference is between configuration management and version control? From the descriptions I've been able to find on various websites, it seems like configuration management is just a fancy term for putting your config files in a source control repository. But others lead me to believe there is a more involved explanation.
Version control is necessary but not sufficient for configuration management. Version control happens in some central or distributed repository, but says nothing about where any particular version is deployed or used.
Configuration management worries about how to take what is in version control and deploy that consistently to the appropriate places, primarily QA and production, but in a large enough development operation developers as well.
For example, you may keep all of your SQL queries in version control, including your table modification scripts, but that doesn't control when those scripts are deployed to the appropriate database server and kept in sync with the deployment of any other code that relies on that database structure.
Configuration management includes, but is not limited to, version control.
Configuration management is everything that you need to manage in terms of a project. This includes software, hardware, tests, documentation, release management, and more. It identifies every end-user component and tracks every proposed and approved change to it from Day 1 of the project to the day the project ends.
Version control is specifically applied to computer files. This includes documents, spreadsheets, emails, source code, and more.
Version control is saving files and keeping different versions of them, so you can see the change over time.
Configuration management is generally referred to as an overall process of which keeps track of what version of the code is on what server, how the servers are setup (and the install scripts to do so at many places). It is how process of what happens after the code goes into source control and how gets to deployed to the servers/desktops etc.
Configuration management is an ambigute term.
In software, it tends to be a superset of version control with emphasis on the entire process to produce a result in a repeatable and predictible manner.
In computing maintenance, it is related to the maintenance of the configuration settings and hardware/firmware/software versions of entire networks and set of attached computing machines (including servers, clients, routers...).
In hardware manufacturing, it represents even a superset of the two above, including the hardware pieces and software modules needed to obtain a product, with the description of the process to manufacture them, and even sometime the entire schemas and configurations of the production lines themselves.
In addition to everything said above I'd like to recommend Bob Aiello's book named "Configuration Management Best Practices" - http://www.amazon.com/dp/0321685865 .
It covers all aspects of Software Configuration Management including version control.
Version control is the control of deliverables whereas configuration management is managing the entire process leading to produce the deliverables. Configuration management involves change management, project management, etc., which generally are not managed by simple version control.
Roughly speaking, version control means you can check out the source for any particular version. Configuration management means you can build and deploy and probably test any particular version.
This can be helpful.
Versions and configurations
Versions:
Ability to maintain several versions of an object.
Commonly found in many software engineering and concurrent engineering environments.
Merging and reconciliation of various versions is left to the application program
Some systems maintain a version graph
Configuration:
A configuration is a collection compatible versions of modules of a software system (a version per module)
Version control is one of the features of a SCM system.
From the subversion user guide:
http://svnbook.red-bean.com/en/1.7/svn-book.html
"Some version control systems are also software configuration management (SCM) systems. These systems are specifically tailored to manage trees of source code and have many features that are specific to software development—such as natively understanding programming languages, or supplying tools for building software. Subversion, however, is not one of these systems. It is a general system that can be used to manage any collection of files. For you, those files might be source code—for others, anything from grocery shopping lists to digital video mixdowns and beyond."
Most of the work I do is with what could be considered enterprise Web applications. These projects have large budgets, longer timelines (from 3-12 months), and heavy customizations. Because as developers we have been touting the idea of the Web as the next desktop OS, customers are coming to expect the software running on this "new OS" to react the same as on the desktop. That includes easy to manage automatic upgrades. In other words, "An update is available. Do you want to upgrade?" Is this even a realistic expectation? Can anyone speak from experience on trying to implement this feature?
At my company we have enterprise installations ranging into the thousands of seats. If we implemented an auto-upgrade, our customers would mutiny!
Large installations have peculiar issues that don't apply to small ones. For example, with 2000 users (not all of whom are, let us say, the most sophisticated of tool users), tool-training is a big deal: training time, internal demos, internal process documents, etc.. They cannot unleash a new feature or UI change without a chance to understand how it fits in their process and therefore what their internal best practices are and how to communicate that to their users.
Also when applications fail, it's the internal IT team who are responsible. Therefore, they want time to install a new version in a test area, beat it up, and deploy on a Saturday only when they're good and ready.
I can see the value in making minor patches more easy to install, particularly when the patch is just for a bug-fix and not for anything that would require retraining, and if the admins still get final say over when it's installed. But even then, I don't believe anyone has ever asked for this! Whether because they don't want it or they are trained to not expect it, it doesn't seem worth it.
Well, it really depends on your business model but for a lot of applications the SaaS model can end up biting you. It's great for a lot of things but for some larger applications the users are not investing as significant amount up front and could possibly move to something else before you've made any money.
See
http://news.zdnet.com/2424-9595_22-218408.html
and here
http://www.25hoursaday.com/weblog/2008/07/21/SoftwareAsAServiceWhenYourBusinessModelBecomesAParadox.aspx
for more information
One of the primary reasons to implement an application as a web application is that you get automatic upgrades for free. Why would users be getting prompted for upgrades on a web app?
For Windows applications, the "update is available, do you want to upgrade?" functionality is provided by Microsoft using ClickOnce, which I have used in an enterprise environment successfully -- there are a few gotchas but for the most part it is a good way to manage automatic deployment and upgrade of Windows apps.
For mobile apps, you can also implement auto-upgrades, although it is a little trickier.
In any case, to answer your question in a broad sense, I don't know if it is expected that all enterprise apps should make upgrading easy, but it certainly is worth the money from an IT support standpoint to architect them to allow for easy upgrading.
If you're providing a hosted solution, I wouldn't bother. Let the upgrade happen silently (perhaps with a notice that you did it). If you're selling an application that's hosted on their servers, let the upgrade decision be made by a single owner, not every user of the app.
Questions #1 through #4 on the Joel Test in my opinion are all about the development tools being used and the support system in place for developers:
Do you use source control?
Can you make a build in one step?
Do you make daily builds?
Do you have a bug database?
I'm just curious what free/cheap (but good) tools exist for the small development shops that don't have large bank accounts to use to achieve a positive answer on these questions.
For source control I know Subversion is a great solution, and if you are a one man shop you could even use SourceGear's Vault.
I use NAnt for my larger projects, but have yet to set up a script to build my installers as well as running the obfusication tools all as a single step. Any other suggestions?
If you can answer yes to the building in a single step, I think creating daily builds would be easy, but what tools would you recommend for automating those daily builds?
For a one or two man team, it's already been discussed on SO that you can use FogBugz On Demand, but what other bug tracking solutions exist for small teams?
source control: Subversion or Mercurial or Git
build automation: NAnt, MSBuild, Rake, Maven
continuous integration: CruiseControl.NET or Continuum or Jenkins
issue tracking: Trac, Bugzilla, Gemini (if it must be .NET and free-ish)
Don't forget automated testing with NUnit, Fit, and WatiN.
1) Subversion
2) Ant / Maven
3) Continuum
4) Bugzilla / Trac
My preferred stack:
1) Subversion. I'm intrigued about distributed source control but haven't had chance to try any in anger yet. For a centralized solution svn is rock solid.
2) Ant. Maven is a joy to use when it's working but as an old ant hacker I find maven to be hard to follow once things go wrong.
3) Hudson. Not been mentioned so far but definitely worth investigating. Incredibly usable and actively maintained tool. PreviousLy we paid for Anthill Pro which seemed flakey and was painful to fix each time it screwed up.
4) We pay for jira. Not cheap but much more usable than the open source options we looked at and very flexible too.
My engineering stack:
Git (I love GitHub, but Git doesn't require a hosted solution)
Rake
CruiseControl.rb
FogBugz
No doubt these choices are influenced by my development stack, which most often includes Ruby, Rails, SQLite, Firefox, and OSX.
You may want to look at an existing question of mine for finding an alternative to Team System. There are plenty of recommendations in there also.
Git
Make
Cron
Trac
I'm a man of few syllables ;-)
Be sure to use some kind of version control where developers can easily create private branches willy-nilly, then take their private branch and squeeze it into a single commit on the main branch. That way, individual developers---as opposed to the organization---can get the benefits of version control without polluting anyone else's code (and slowing down their work) with broken commits.
This feature is what I like about git. I think it's only really present in distributed version control systems; using a DVCS doesn't mean you actually have to do distributed development, though.
Regarding one-step building, make is the default build tool and it works quite well for most tasks. I'd go with that unless you have a good reason not to.
You want daily builds, put the build command in your cron.daily. Set up a procmail hook to handle the mail from cron if need be.
For bug tracking, use $(apt-cache search bug tracking). Basically, as long as it says "bug tracker" on the box and you know other people are using it, it's probably going to work fine. Among the regulars are bugzilla, mantis and trac.
I don't have any tools to suggest, but I do have a suggestion about the daily builds. I always answer yes to that question, even though we don't have daily builds. Instead, we do a build every time someone does a commit. We thereby catch any problems almost immediately. If any of our projects ever has enough LOC that building takes more than trivial time, doing this will also gracefully degrade in the direction of a daily build.
A good issue tracker that was relatively inexpensive was axoSoft OnTime. I used it for years before getting MS TFS.
Nant and CruiseControl are staples of my environment.
I don't think you really need obfuscation on .Net any more (see another response)
I wouldn't consider Vault, SVN is really the market leader at the moment (and free). Git is looking pretty promising but currently is command line only with a steep learning curve.
MSBuild beats NAnt for .Net 2 or 3.5
CC.Net is excellent.
*4) Redmine
I recommend Bitnami for testing out different stacks. It's got Trac, Redmine, and Subversion, as well as several other unrelated ones.
Check out these articles on Continuous Integration using MSBuild, CruiseControl.NET, FxCop, NUnit, NCover and Subversion...
From the software development trenches
I'm currently using SVN but I've generally had a lot or problem with checkouts to a network drive on a dev server. There tend to be locking issues that require a lot of fishing around to fix. It may be that using the WebDav access method, would ease some of these problems, but I haven't experimented yet.
Any of Bugzilla, Trac or Fogbugz will help you with your bug tracking, and each offer an export feature, so you can always change your mind later on. Also, if you can get your team to fully buy in, time management software can also be handy for post-mortems, etc (if everyone is motivated to fully participate.
For build automation and continuous integration take a look at TeamCity from Jetbrains.
It has a lot of features and is really a breeze to set up and use.
If you use Visual Studio 2005/2008 it will build your solution directly without the need for extra scripts (if a build is all you want.)
It will also execute your unit tests and gather stats on build success, unit test execution times, etc, etc.
Best of all: The Pro edition is free for teams with up to 20 users and 3 build agents.
source control: cvs
build gnu make
cron job that calls bash scripts
bugzilla