What to do when you've really screwed up the design of a distributed system? - distributed-computing

Related question: What is the most efficient way to break up a centralised database?
I'm going to try and make this question fairly general so it will benefit others.
About 3 years ago, I implemented an integrated CRM and website. Because I wanted to impress the customer, I implemented the cheapest architecture I could think of, which was to host the central database and website on the web server. I created a desktop application which communicates with the web server via a web service (this application runs from their main office).
In hindsight this was rather foolish, as now that the company has grown, their internet connection becomes slower and slower each month. Now, because of the speed issues, the desktop software times out on a regular basis, the customer is left with 3 options:
Purchase a faster internet connection.
Move the database (and website) to an in-house server.
Re-design the architecture so that the CRM and web databases are separate.
The first option is the "easiest", but certainly not the cheapest long term. Second option; if we move the website to in-house hosting, the client has to combat issues like overloaded/poor/offline internet connection, loss of power, etc. And the final option; the client is loathed to pay a whole whack of cash for me to re-design and re-code the architecture, and I can't afford to do this for free (I need to eat).
Is there any way to recover from when you've screwed up the design of a distributed system so bad, that none of the options work? Or is it a case of cutting your losses and just learning from the mistake? I feel terrible that there's no quick fix for this problem.

You didn't screw up. The customer wanted the cheapest option, you gave it to them, this is the cost that they put off. I hope you haven't assumed blame with your customer. If they're blaming you, it's a classic case of them paying for a Chevy while wanting a Mercedes.
Pursuant to that:
Your customer needs to make a business decision about what to do. Your job is to explain to them the consequences of each of the choices in as honest and professional a way as possible and leave the choice up to them.
Just remember, you didn't screw up! You provided for them a solution that served their needs for years, and they were happy with it until they exceeded the system's design basis. If they don't want to have to maintain the system's scalability again three years from now, they're going to have to be willing to pay for it now. Software isn't magic.

I wouldn't call it a screw up unless:
It was known how much traffic or performance requirements would grow. And
You deliberately designed the system to under-perform. And
You deliberately designed the system to be rigid and non adaptable to change.
A screw up would have been to over-engineer a highly complex system costing more than what the scale at the time demanded.
In fact it is good practice to only invest as much as can currently be leveraged by the business, using growth to fund further investment in scalability, should it be required. It is simple risk management.
Surely as the business has grown over time, presumably with the help of your software, they have also set aside something for the next level up. They should be thanking you for helping grow their business beyond expectations, and throwing money at you so you can help them carry through to the next level of growth.
All of those three options could be good. Which one is the best depends on cost benefits analysis, ROI etc. It is partially a technical decision but mostly a business one.
Congratulations on helping build a growing business up til now, and on to the future.

Are you sure that the cause of the timeouts is the internet connection, and not some performance issues in the web service / CRM system? By timeout I'm going to assume you mean something like ~30 seconds, in which case:
Either the internet connection is to blame and so you would see these sorts of timeouts to other websites (e.g. google), which is clearly unacceptable and so sorting the internet is your only real option.
Or the timeout is caused either by the desktop application, the web serice, or due to exessively large amounts of information being passed backwards and forwards, in which case you should either address the performance issue how you might any other bug, or look into ways of optimising the Desktop application so that less information is passed backwards and forwards.
In sort: the architecture that you currently have seems (fundamentally) fine to me, on the basis that (performance problems aside) access for the company to the CRM system should be comparable to accesss for the public to the system - as long as your customers have reasonable response times, so should the company.

Install a copy of the database on the local network. Then let the client software communicate with the local copy and let the database software do the synchronization between the local database server and the database on the webserver. It depends on which database you use, but some of them have tools to make that work. In MSSQL it is called replication.

First things first how much of the code do you really have to throw away? What language did you use for the Desktop client? Something .NET and you may be able to salvage a good chuck of the logic of the system and only need to redo the UI and some of the connections.
My thoughts are that 1 and 2 are out of the question, while 1 might be a good idea it doesn't solve the real problem. And we as engineers should try and build solutions not dependent on the client when ever possible. And 2 makes them get into something they aren't experts at and it is better to keep the hosting else where.
Also since you mention a web service is all you are really losing the UI? You can alway reuse the webservices for the web server interface.
Lastly you could look at using a framework to help provide a simple web based CRUD to start and then expand from there.

Are you sure the connection is saturated? You could be hitting all sorts of network, I/O and database problems... Unless you've already done so, use wireshark to analyze the traffic; measure the throughput and share the results with us.

Related

General question on Sever Side Google Tag Manager

So I've been looking into page speed and have been reading about Google's Server Side GTM. Wondering if a lot of people have transitioned to this and what the costs could be for a high traffic marketing site. If the site already has a ton of tags, is this something that can be set up by the marketing department or developers, or is it more likely needed to hire someone who specializes in GTM.
If anyone has experienced this transition and has any information to share regarding page speed results that would be greatly appreciated as well.
The page speed change will vary depending on your site and the health of your container. You can measure it. Just go to your site, go to devtools, block GTM, reload the page. Compare that to how fast it loads with it.
General conclusion: Most GTM containers have zero influence on page load speed. Even if the library is not cached. Before it's non-blocking.
Keep in mind that best practice for implementing server-side GTM is using it in conjunction with the regular GTM. Which kinda defeats the purpose since sending GA hits is not something affecting your site speed. Unless you create real hardware load with them. Which is pretty difficult.
Now, to maintain your sGTM, you do need to increase spendings, but the main expense is usually not on maintaining the instance with the container. The main expense has a more long-term nature. The maintenance of the backend instance will require more expensive experts, much tougher debugging, typically involvement of normally more expensive backend devs and so on.
Generally, going sGTM route just because it sounds like fun is a bad idea to do in production. There should be a good business case to justify it.

High traffic site. >10 million user a day. VPS or dedicated server?

We're launching an iPhone app soon, and if everything goes well, we might reach up to tens of millions of user each day.
What server solution would you use for this? I guess a small VPS isn't enough. Is dedicated server a better choice? Is there any good hosting provider that can provide such servers?
I'm a newbie when It comes to servers, and would like some basic info about how to handle this.
Thanks in advance
Unfortunately, you are not really going to know the apps requirements until the app is launched. It all depends on how much the app needs to communicate with the server, and how often users are using the app. Depending on those variables and even more, a VPS might be enough, or you may need a dedicated box, or several. It also depends a lot on the performance of the VPS and dedicated boxes, furthermore it depends on how much access to the system you need.
Ultimately, it seems you may not even know how well the app is going to do, so I suggest you take the cheap/efficient route of using cloud computing. That way you will limit your expenses initially when you app has a small user base. Then your performance can amp up as quickly as your app requires (of course so will the price). That is the benefit of cloud computing, you will not be losing money in the beginning until you have the user base to use your server to its limit. Furthermore, you do not have downtime, etc when/if your server is no longer enough.
Check out Google's Cloud Computing to get a hint of what is possible. I personally like Google's cloud experience, but you have many more options with varying degrees of freedom that you will have to check out. Amazon of course is another possibility.

Old concepts with new names (namely REST and Cloud computing)

It seems that SaaS and Cloud computing are old concepts with new names, and I am curious if I am wrong.
For cloud computing you can look at: Difference between cloud computing and distributed computing?
Basically, it seems that when we have been hosting that that is cloud computing, it is just that now some companies have put in much great resources to ensure better uptime than my local ISP. But, it seems that there is nothing really new here.
For REST, it seems that it is what we have been doing with cgis for 15 years.
Here is a question on REST: What am I not understanding about REST?
It appears that REST is an old concept, and I am curious how it is different than has been done since the early days of the web, and, to a large extent, the early days of using telnet (which http is on top of).
Am I mistaken in my simplification of these? I try to see how what is new is like what I know so I can see what more has to be learned in that topic, but for cloud computing and REST it seems that very little needs to be learned.
You are both right and wrong. You are right in the sense that new ideas are normally similar to old ideas, and indeed cloud computing is based significantly on distributed computing.
What is new in cloud computing is
virtualization
self-service
With virtualization, you can run multiple operating systems on a single hardware. While that, in itself, isn't new, either, it was never considered in distributed systems as a relevant piece of the architecture. Using virtualization allows self-service: users can create their own clusters of nodes without the administrator of the hardware taking any action. This allows a significant acceleration of deployment, and a significant reduction of cost.
For ReST, what you are missing is the client API. It is true that on the server side, a ReST service can be implemented with CGI. What is new here is that it is not an end user which retrieves the URL, but a program.
Saying that HTTP is on top of telnet ignores realities; this is like saying that we made no progress since the introduction of copper wires for communication. Strictly speaking, HTTP is not in top of telnet, but on top of TCP (which telnet is also on top of, these days).
Considering Roy's dissertation coined the term REST back in 2000, you can definitely argue that there is nothing new about REST. Additionally, the REST architectural style was synthesized from successful existing practices, so REST implementations pre-date the definition. Having said that, there is nothing simple about designing REST interfaces. Ever since Netscape first abused cookies to allow servers to maintain session state people have been swimming upstream against the web.
REST's recent resurrection has come mainly from people becoming disillusioned with SOAP based Web Services. SOAP tried to hide HTTP instead of embracing it and I think people are starting to realize how effective HTTP can be as an distributed application protocol that can do more than just deliver HTML to web browsers.
RESTful web applications don't use session state, so one could argue that by that virtue alone it is different than most web applications in existence at the moment.
As for Cloud Computing, I find myself agreeing with Larry Ellison for once in my life.
I'm in agreement on what you've posted. You might consider making this community wiki since it's likely to garner many answers based on opinion. Cloud computing seems to have taken off as a buzzword, and this is largely due to a decrease in cost for mass quantities of hardware. And then there is REST which is really just a formal name and definition for something that has been in place for a long time. Some people like to encapsulate ideas with buzzwords and acronyms. Sometimes it's useful to put a name to an idea though.
Not only this, the concept of things being old concepts with new names is old. It's hard to be original these days :P
You are right about REST -- its mostly old concepts with a lot of added pedantry and not much added substance.
Cloud computing has a small but fundamental difference from distributed computing. In distributed computing you had servers dedicated to particular functions, and usually some sort of directory service to locate the correct server. In cloud computing any server is capable of any task and usually the servers queue up for work which is distributed from a central point.

Convincing a large company to use free software? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I'm currently a developer at my first job right out of college. I work for a large company, and the trend I notice with them is that they tend to go with more expensive, closed source software about 99% of the time, while there are perfectly good open source alternatives that are available, most of which are vastly superior to their closed-source counterparts. For example, we use this absolutely awful source control software that cost a ton of money, while there are quite a few open source and/or free options that in my experience, albiet limited, are much better and offer basically the exact same functionality.
I guess my question is: How would an experienced developer approach management about using more free software?
It appears there is another question very similar to this that did not show up when I made this one: How can I convince IT that F/OSS software isn't evil?
EDIT: Just come clarification. I'm not necessarily trying to change the company's procedure, I'm looking for advice on how to approach management about the subject.
Start using it in small utilities and things which are throwaway and don't need management buyin. This can prove the worth of an open source solution and put a crack in
the door for using it in other
projects.
Present articles from trade magazines showing that other people are using the open source solution.
Go with products which have commercial support options, such as MySQL, which enterprises seem to have an easier time swallowing.
Pick your battles carefully. Wait until they are suffering. If they are happy with what they have, they will not switch, no matter how much cheaper or superior the alternative is. You need to catch them while they're trying to think of ways to save money, or while they're disgusted with the problems of the current system.
Be very careful with what you refer to as free. There is a very large corpus of products that would be perfectly valid for a student to use without paying that an enterprise would have to pay for. Also never forget Total Cost of Ownership (TCO). A lot of relatively expensive software is expensive because you get things like configuration and help support for them whereas that may not be the case for free software.
I think you are not asking the right question. To me, the challenge is to have my Big Corp to buy the BEST softwares for me, be it free softwares or not.
Paying for Windows or paying for Linux is not important (what is 100 $ for a Big Corp ?).
But having things done better is really important.
I think that your request to your boss should not be : "Hey, it's free and it's as good as XYZ, why are we using XYZ ?"
Why you boss would risk something trying the product you told when XYZ seems to be ok ?
It would be much more better to ask : "Hey, here is what I cannot do with XYZ : (your list). With my product, I would be able to do that and much more so fast than I would have a lot of spare time to test our own software !".
Small money is usually not a show stopper. Being able to work faster in order to do much more testing (or any other things that could help your boss have a better image) is definitely an excellent argument !
Best wishes,
Sylvain.
I work in a big company that has recently moved into being more enthusiastic about open source solutions. There have been a few big hurdles:
Customer won't support it - we're defense contractors. We do almost nothing without customer say-so. As the customer's opinions have changed we've been able to change our architectures and tool usage. That said, there are still scenarios were open source is unacceptable and we don't use it.
No tech support = scary - in several cases, it's been possible to make the point that open source may not have a single point of company tech support, but it does have huge communities that will support questios for free, and that there are consultants available as needed for the really hard stuff. Plus many, many releases of new versions for bug fixes. And, several competing expensive products have not been able to service tech support needs. Being able to point to specific internal examples with long. well documented, histories of support problems, has been key.
Fear of security issues - we had to develop a process for scrutinizing and controlling every peice of open source introduced. We've managed to find criteria for what we deem risky, versus what we deem relatively benign based on info-sec policy.
Fear of lawsuit - Being large, and profitable, we fear lawsuits, we're great targets. We now have a process for the legal team to scrutinize every open source license. This has proved to be a win - since the legal team now has briefings on every major version of the typical open source licenses, and they can quickly review most stuff.
Version control - fear that if those wacky developers can just download anything they like the world will self destruct. OK, well, practically speaking, the concept of "how do we know what's in a given product" - being able to show a FOSS version control process that is managed internally has been important.
It was definitely a slow process - small projects proved profitable and customers started encouraging it in proposals. That made it useful for executive management. It's helped that those that support it have been williing to put in some extra time to making the business case in terms of efficiency/cost savings, and have been willing to negotiate repeatedly with various parts of the corporate infrastructure.
Making open source work has taken the effort of IT, the info security folks, the legal team, the procurement team, and technical management. Knowing that before you talk to your manager is probably a key to success.
There's also some political savy - for a first project, don't encroach on any sacred cows - ie, those projects that may not be successful, but are high profile and owned by someone with lots of political power. Instead, choose some wacky new thing that isn't available right now and prove the cost savings in a way that is unlikely to provoke a defensive reaction.
When you try to introduce open source software to a big company (or even a small one, in many cases), the biggest counter-argument you're going to hear is "There's no tech support." Companies tend to be wary of using software that's supported by the community, because there's no guarantee (or in some cases, service agreement) that questions about the software will be answered within a reasonable time frame, or at all. In many cases, you can find a company that will provide support for the open-source package you want to use (for example, Red Hat does this for its Linux distribution, even though the contents of the distribution is mainly open source). Showing management a business entity that can support the software will often go a long way.
The other counter-argument to using open source software that I've heard the most often is "Open source software is buggy." This is a tough one; this opinion is pretty ingrained in some corporate cultures. Two possible responses are "The open-source community fixes bugs quickly" and "Since we have the source code, our engineers can fix bugs"--but that's often not what managers want to hear.
So, in essence, it depends on the company, their attitudes, and how much they trust you to make business-critical recommendations. I've used all of the arguments above with different levels of success in different companies.
Of course, in these economic times, the "free" part may go a long way. :-)
"Free software" doesn't necessarily mean your company is going to get software for free. Many successful open-source projects are also offered with licenses and services that cost real money and are geared to organizations that want or need to be assured of good support. MySQL is an example
The reason for a lot of big companies using closed software is that they can call support and the vendor will issue a hotfix, patch or cumulative update
Changing a large company's habits are often like turning an Oil tanker around... it takes a long time and uses a lot of energy.
If the company were in the process of evaluating the purchase of new software for a specific task, Then I would make sure to write a concise opinion memo about why my choice is better.
If the software is something I would use personally and not a server product that multiple developers are forced to use, then I would just ask my manager to use it.
If the software is in place, does the job (even if I don't like the way it does it), i'd learn as much as I can about it to give it as much chance of work for me, or at least make my life easier. If it still sucks really bad, I probably wouldn't try to change it until it was time for the company to pay for an upgrade.
If the software works but is just annoying... I'd do as above, learning all there is to know about it just to make my life easier and then deal with it.
You're probably right that the system you'd recommend is better than the one currently in place. But like some other posters said, choose your battles, especially when this is your first job out in the real world. You may become expendable quickly.
It's not really so much a matter of what's better, even if your way IS better, it's a matter of the culture and the way things are done and the cost of switching. Even if, hypothetically, their system can be magically transported to your OSS system, with no loss of data, dates, records, or anything, you're still going to have people who say "I liked the old way better."
Remember: Experience is what you get when you don't get what you want. I know it may sound glamorous to be "the new guy who recommended a great new versioning system that everybody loved", but you also could just as easily become "that hotshot who insisted on a new versioning system that everybody hated." It's a much smarter career move to just play by the rules at least for a little while until you have some clout and can make some recommendations. In the meantime you may even learn why the old system is preferred, or learn to like it more the more you use it.
I know what you mean. It took us years to convince our managers that everything would be okay if we moved away from using Interbase (a commercial Relational DB) to it's opensource counterpart Firebird. Mostly it was fear of no support that blocked the move. I think the factors which changed their mind were:
tests showing better performance
that there are companies that provide and charge for supporting the OS alternative
constant pushing of the argument by passionate developers
I think cost savings would have played a part if our company were paying for the site licenses but in fact our customers were.
I look at this question like this. I work with the .NET framework. I could ask my employer to migrate to PHP. This is a disadvantage to me, as well as my company, for many reasons. Let's start with the obvious.
1.) I know PHP, but can do much more, and a lot faster, with .NET.
2.) Paying for a service, usually ensures a better experience. The Visual Studio IDE is second to NONE when developing an application.
3.) I can develop an application much faster in VS than hard-coding PHP.
4.) This is the most important one. If I work with a big company, I want my programmers to develop my app faster, and I expect it to run faster. PHP (an example Open Source language) is fast, and reliable, but if I can spend the money, I'll deploy ASP.NET.
Basically, big business, or even small business, wants to spend their money, as long as it's for a good reason. Your best bet is to say, 'Hey, if you want to deploy ASP.NET (or whatever), send me for some training. Then I'll be able to develop OUR application to my best ability'.
not to sound totally cynical, but:
an experienced developer probably would not approach management about something like this, unless he/she was already an expert with the open source package. Companies like to have a phone number to call and someone to blame when things don't work. Free open source packages do not provide this kind of 'accountability' (yes we know it's a joke, but management doesn't)
it is unlikely that management is going to listen to someone fresh out of college about any major purchasing or technology decision. You have to learn the business and earn everyone's respect first. [sorry!]
Same problem everywhere. Once an organization gets beyond a certain size (e.g., the Dunbar number) it starts to show a certain woodenheaded quality that will confound you. Lots of history, people, agendas that you aren't aware of. And getting everyone to agree on your solution is difficult.
Best to start locally. See if you can persuade your manager or PM to use SVN or CVS or GIT locally for a project and then get it to diffuse.
But that situation is true where I work as well. I use SVN locally for myself, but a commercial product for integrating with others.
Companies will use whatever will ultimately make them the most money. That means whatever software will make their employees more productive. If there is a particular piece of open source software you think they should use then when the time comes to purchase the software to do job X then as long as you can prove it will make the employees more productive and they are able to get reliable support just a phone call away as with commercial software then they will use it.
Big companies need to hire support staff for stuff like that. When they purchase software from a company, they are guaranteed support with the contract. Open source projects can die off a lot easier, whereas a large software vendor can be held responsible for much greater periods of time.
Every company has a culture, and fighting the culture can be something of an uphill battle. But if you're willing to try:
you'll likely have more luck getting BSD and BSD-like projects approved (MIT license, Apache, Boost, etc.); and it doesn't matter if most of the arguments against GPL and LGPL are mainly FUD
you should refer to the projects as "royalty-free"
you should make sure things are approved by somebody that can approve them (your direct manager) because putting the company in a bind -- especially when you're new -- (even if the "bind" is only in their head) is not conducive to long-term employment
you can probably go a long way by simply asking what the procedure is to choose a library or tool
From a configuration management perspective, having developers add free software stuff willy nilly whenever they feel like is a serious PITA to manage.
I've worked at companies where you were allowed to do it whenever you wanted and others where you could never do it.
There's definitely a balance to be found but if you're in a larger company with multiple projects, you do have to keep in mind that each time you add a new 'tool' it complicates the build process.

Is automatic upgrades a realistic feature to expect from enterprise Web applications?

Most of the work I do is with what could be considered enterprise Web applications. These projects have large budgets, longer timelines (from 3-12 months), and heavy customizations. Because as developers we have been touting the idea of the Web as the next desktop OS, customers are coming to expect the software running on this "new OS" to react the same as on the desktop. That includes easy to manage automatic upgrades. In other words, "An update is available. Do you want to upgrade?" Is this even a realistic expectation? Can anyone speak from experience on trying to implement this feature?
At my company we have enterprise installations ranging into the thousands of seats. If we implemented an auto-upgrade, our customers would mutiny!
Large installations have peculiar issues that don't apply to small ones. For example, with 2000 users (not all of whom are, let us say, the most sophisticated of tool users), tool-training is a big deal: training time, internal demos, internal process documents, etc.. They cannot unleash a new feature or UI change without a chance to understand how it fits in their process and therefore what their internal best practices are and how to communicate that to their users.
Also when applications fail, it's the internal IT team who are responsible. Therefore, they want time to install a new version in a test area, beat it up, and deploy on a Saturday only when they're good and ready.
I can see the value in making minor patches more easy to install, particularly when the patch is just for a bug-fix and not for anything that would require retraining, and if the admins still get final say over when it's installed. But even then, I don't believe anyone has ever asked for this! Whether because they don't want it or they are trained to not expect it, it doesn't seem worth it.
Well, it really depends on your business model but for a lot of applications the SaaS model can end up biting you. It's great for a lot of things but for some larger applications the users are not investing as significant amount up front and could possibly move to something else before you've made any money.
See
http://news.zdnet.com/2424-9595_22-218408.html
and here
http://www.25hoursaday.com/weblog/2008/07/21/SoftwareAsAServiceWhenYourBusinessModelBecomesAParadox.aspx
for more information
One of the primary reasons to implement an application as a web application is that you get automatic upgrades for free. Why would users be getting prompted for upgrades on a web app?
For Windows applications, the "update is available, do you want to upgrade?" functionality is provided by Microsoft using ClickOnce, which I have used in an enterprise environment successfully -- there are a few gotchas but for the most part it is a good way to manage automatic deployment and upgrade of Windows apps.
For mobile apps, you can also implement auto-upgrades, although it is a little trickier.
In any case, to answer your question in a broad sense, I don't know if it is expected that all enterprise apps should make upgrading easy, but it certainly is worth the money from an IT support standpoint to architect them to allow for easy upgrading.
If you're providing a hosted solution, I wouldn't bother. Let the upgrade happen silently (perhaps with a notice that you did it). If you're selling an application that's hosted on their servers, let the upgrade decision be made by a single owner, not every user of the app.