I understand this is a loaded question, however I am going to try my luck to see if anyone has info/documentation that I have been unable to locate thus far. Perhaps someone with a better understanding of REST API functionality could point me in the correct direction.
In looking to deploy Remedy 9.1, I am being told REST API will be disabled due to performance concerns of the front end application itself (web interface). I am trying to find out if there is any quality control or prioritization that occurs on the backend that would mitigate this concern.
I understand that there are some obvious savings in not having to dynamically render a webpage or engage the front end virtually at all when making a REST API call, so when pulling data its more lean 1 to 1 to use REST. However if someone where to be reckless with a REST API call, is the ARServer at all equipped to manage this request by assigning it low priority, or would it simply take down the whole system?
In a perfect world I would love if someone could point me to some specific documentation that has something close to a definitive answer either way.
Thanks for any help anyone may be able to send my way.
Deploy a server group, keep one server for the front end and one for the back end. Only make calls to the 'back end' server.
I think the bigger issue that is being avoided is that there are kinds of places for bottlenecks. If they are concerned about that, what guardrails do they have in place to protect from an overloaded database? Or a table lock? The REST API is just another client. There is only so much you can do to protect the system. Since it is another client, it has all the protections at the app level that exist for every other client. Indexing, requiring search criteria, threading, and other performance tactics will help regardless of the client.
RCJ has a good point in that you can dedicate servers in the server group (or even outside of it) to accommodate their needs. But you will always go back to the central database as the ultimate risk.
Related
So I've been looking into page speed and have been reading about Google's Server Side GTM. Wondering if a lot of people have transitioned to this and what the costs could be for a high traffic marketing site. If the site already has a ton of tags, is this something that can be set up by the marketing department or developers, or is it more likely needed to hire someone who specializes in GTM.
If anyone has experienced this transition and has any information to share regarding page speed results that would be greatly appreciated as well.
The page speed change will vary depending on your site and the health of your container. You can measure it. Just go to your site, go to devtools, block GTM, reload the page. Compare that to how fast it loads with it.
General conclusion: Most GTM containers have zero influence on page load speed. Even if the library is not cached. Before it's non-blocking.
Keep in mind that best practice for implementing server-side GTM is using it in conjunction with the regular GTM. Which kinda defeats the purpose since sending GA hits is not something affecting your site speed. Unless you create real hardware load with them. Which is pretty difficult.
Now, to maintain your sGTM, you do need to increase spendings, but the main expense is usually not on maintaining the instance with the container. The main expense has a more long-term nature. The maintenance of the backend instance will require more expensive experts, much tougher debugging, typically involvement of normally more expensive backend devs and so on.
Generally, going sGTM route just because it sounds like fun is a bad idea to do in production. There should be a good business case to justify it.
after reading all the threads on stackoverflow and other platforms, I still wasn't able to find an answer, which satisfies me.
The task:
I want to create a single page application (SPA) which receives data from a REST API. In this SPA, NO authentication should be used. It's a public site.
But the REST API should only be accessible from people who loaded the SPA from my webserver.
I assume this is only solvable with something on server side like sessions, cookies etc. - otherwise I'm open for your suggestions, solutions etc.
Thx in advance!
There's no reasonably easy way to do this. You can easily prevent other domains (in browsers) from accessing a an API on your domain (via CORS), but it's significantly harder to prevent scripts from doing this.
The issue lies in 'how do you detect legit browser traffic from a script'. It turns out that this is not easy. You could try to detect 'unusual behavior' as much as possible (for example a large amount of requests in a short time), but this doesn't stop clients that are slower.
Ultimately if people want your data, they will find some way around whatever restrictions you come up with. You should reevaluate this and use one of the following options:
Don't do an SPA and API. Although one could wonder, if the data exists in HTML it can still be crawled.
Add authentication. But obviously this won't help you in any way if anyone can authenticate.
Re-evaluate why you have this restriction. What are you worried about? If you're worried about people taking your data and using it elsewhere, how does only showing it in a browser from 1 domain help with that? If you're worried about copyright theft, why not use a legal approach to this?
I've seen a lot of these types of questions, but in my opinion I haven't yet seen one that has a legitimate good reason to want this. But, maybe you're the first.
I believe I answered my question myself on a comment 30 minutes ago... I think with captcha I'm able to secure the REST API against unwanted access to my REST API
I'm admittedly unsure whether this post falls within the scope of acceptable SO questions. If not, please advise whether I might be able to adjust it to fit or if perhaps there might be a more appropriate site for it.
I'm a WinForms guy, but I've got a new project where I'm going to be making web service calls for a Point of Sale system. I've read about how CRUD operations are handled in RESTful environments where GET/PUT/POST/etc represent their respective CRUD counterpart. However I've just started working on a project where I need to submit my requirements to a developer who'll be developing a web api for me to use but he tells me that this isn't how the big boys do it.
Instead of making web requests to create a transaction followed by requests to add items to the transaction in the object based approach I'm accustomed to, I will instead use a service based approach to just make a 'prepare' checkout call in order to see the subtotal, tax, total, etc. for the transaction with the items I currently have on it. Then when I'm ready to actually process the transaction I'll make a call to 'complete' checkout.
I quoted a couple words above because I'm curious whether these are common terms that everyone uses or just ones that he happened to choose to explain the process to me. And my question is, where might I go to get up to speed on the way the 'big boys' like Google and Amazon design their APIs? I'm not the one implementing the API, but there seems to be a little bit of an impedance mismatch in regard to how I'm trying to communicate what I need and the way the developer is expecting to hear my requirements.
Not sure wrt the specifics of your application though your general understanding seems ik. There are always corner cases that test the born though.
I would heed that you listen to your dev team on how things should be imolemented and just provide the "what's" (requirements). They should be trusted to know best practice and your company's own interpretation and standards (right or wrong). If they don't give you your requirement (ease-of-use or can't be easily reusable with expanded requirements) then you can review why with an architect or dev mgr.
However, if you are interested and want to debate and perhaps understand, check out Atlassian's best practice here: https://developer.atlassian.com/plugins/servlet/mobile#content/view/4915226.
FYI: Atlassian make really leading dev tools in use in v.large companies. Note also that this best-practices is as a part of refactoring meaning they've been through the mill and know what worked and what hasn't).
FYI2 (edit): Reading between the lines of your question, I think your dev is basically instructing you specifically on how transactions are managed within ReST. That is, you don't typically begin, add, end. Instead, everything that is transactional is rolled within a transaction wrapper and POSTed to the server as a single transaction.
Related question: What is the most efficient way to break up a centralised database?
I'm going to try and make this question fairly general so it will benefit others.
About 3 years ago, I implemented an integrated CRM and website. Because I wanted to impress the customer, I implemented the cheapest architecture I could think of, which was to host the central database and website on the web server. I created a desktop application which communicates with the web server via a web service (this application runs from their main office).
In hindsight this was rather foolish, as now that the company has grown, their internet connection becomes slower and slower each month. Now, because of the speed issues, the desktop software times out on a regular basis, the customer is left with 3 options:
Purchase a faster internet connection.
Move the database (and website) to an in-house server.
Re-design the architecture so that the CRM and web databases are separate.
The first option is the "easiest", but certainly not the cheapest long term. Second option; if we move the website to in-house hosting, the client has to combat issues like overloaded/poor/offline internet connection, loss of power, etc. And the final option; the client is loathed to pay a whole whack of cash for me to re-design and re-code the architecture, and I can't afford to do this for free (I need to eat).
Is there any way to recover from when you've screwed up the design of a distributed system so bad, that none of the options work? Or is it a case of cutting your losses and just learning from the mistake? I feel terrible that there's no quick fix for this problem.
You didn't screw up. The customer wanted the cheapest option, you gave it to them, this is the cost that they put off. I hope you haven't assumed blame with your customer. If they're blaming you, it's a classic case of them paying for a Chevy while wanting a Mercedes.
Pursuant to that:
Your customer needs to make a business decision about what to do. Your job is to explain to them the consequences of each of the choices in as honest and professional a way as possible and leave the choice up to them.
Just remember, you didn't screw up! You provided for them a solution that served their needs for years, and they were happy with it until they exceeded the system's design basis. If they don't want to have to maintain the system's scalability again three years from now, they're going to have to be willing to pay for it now. Software isn't magic.
I wouldn't call it a screw up unless:
It was known how much traffic or performance requirements would grow. And
You deliberately designed the system to under-perform. And
You deliberately designed the system to be rigid and non adaptable to change.
A screw up would have been to over-engineer a highly complex system costing more than what the scale at the time demanded.
In fact it is good practice to only invest as much as can currently be leveraged by the business, using growth to fund further investment in scalability, should it be required. It is simple risk management.
Surely as the business has grown over time, presumably with the help of your software, they have also set aside something for the next level up. They should be thanking you for helping grow their business beyond expectations, and throwing money at you so you can help them carry through to the next level of growth.
All of those three options could be good. Which one is the best depends on cost benefits analysis, ROI etc. It is partially a technical decision but mostly a business one.
Congratulations on helping build a growing business up til now, and on to the future.
Are you sure that the cause of the timeouts is the internet connection, and not some performance issues in the web service / CRM system? By timeout I'm going to assume you mean something like ~30 seconds, in which case:
Either the internet connection is to blame and so you would see these sorts of timeouts to other websites (e.g. google), which is clearly unacceptable and so sorting the internet is your only real option.
Or the timeout is caused either by the desktop application, the web serice, or due to exessively large amounts of information being passed backwards and forwards, in which case you should either address the performance issue how you might any other bug, or look into ways of optimising the Desktop application so that less information is passed backwards and forwards.
In sort: the architecture that you currently have seems (fundamentally) fine to me, on the basis that (performance problems aside) access for the company to the CRM system should be comparable to accesss for the public to the system - as long as your customers have reasonable response times, so should the company.
Install a copy of the database on the local network. Then let the client software communicate with the local copy and let the database software do the synchronization between the local database server and the database on the webserver. It depends on which database you use, but some of them have tools to make that work. In MSSQL it is called replication.
First things first how much of the code do you really have to throw away? What language did you use for the Desktop client? Something .NET and you may be able to salvage a good chuck of the logic of the system and only need to redo the UI and some of the connections.
My thoughts are that 1 and 2 are out of the question, while 1 might be a good idea it doesn't solve the real problem. And we as engineers should try and build solutions not dependent on the client when ever possible. And 2 makes them get into something they aren't experts at and it is better to keep the hosting else where.
Also since you mention a web service is all you are really losing the UI? You can alway reuse the webservices for the web server interface.
Lastly you could look at using a framework to help provide a simple web based CRUD to start and then expand from there.
Are you sure the connection is saturated? You could be hitting all sorts of network, I/O and database problems... Unless you've already done so, use wireshark to analyze the traffic; measure the throughput and share the results with us.
I'm writing a web application for public consumption...How do you get over/ deal with the fear of User Input? As a web developer, you know the tricks and holes that exist that can be exploited particularly on the web which are made all the more easier with add-ons like Firebug etc
Sometimes it's so overwhelming you just want to forget the whole deal (does make you appreciate Intranet Development though!)
Sorry if this isn't a question that can be answered simply, but perhaps ideas or strategies that are helpful...Thanks!
One word: server-side validation (ok, that may have been three words).
There's lots of sound advice in other answers, but I'll add a less "programming" answer:
Have a plan for dealing with it.
Be ready for the contingency that malicious users do manage to sneak something past you. Have plans in place to mitigate damage, restore clean and complete data, and communicate with users (and potentially other interested parties such as the issuers of any credit card details you hold) to tell them what's going on. Know how you will detect the breach and close it. Know that key operational and development personnel are reachable, so that a bad guy striking at 5:01pm on the Friday before a public holiday won't get 72+ clear hours before you can go offline let alone start fixing things.
Having plans in place won't help you stop bad user input, but it should help a bit with overcoming your fears.
If its "security" related concerns you need to just push through it, security and exploits are a fact of life in software, and they need to be addressed head-on as part of the development process.
Here are some suggestions:
Keep it in perspective - Security, Exploits and compromises are going to happen to any application which is popular or useful, be prepared for them and expect them to occur
Test it, then test it again - QA, Acceptance testing and sign off should be first class parts of your design and production process, even if you are a one-man shop. Enlist users to test as a dedicated (and vocal) user will be your most useful tool in finding problems
Know your platform - Make sure you know the technology, and hardware you are deploying on. Ensure that relevant patches and security updates are applied
research - look at applications similar to your own and see what issues they experience, surf their forums, read their bug logs etc.
Be realistic - You are not going to be able to fix every bug and close every hole. Pick the most impactful ones and address those
Lots of eyes - Enlist as many people to review your designs and code as possible. This should be in addition to your QA resources
You don't get over it.
Check everything at server side - validate input again, check permissions, etc.
Sanitize all data.
That's very easy to write in bold letter and a little harder to do in practice.
Something I always did was wrap all user strings in an object, something like StringWrapper which forces you to call an encoding method to get the string. In other words, just provide access to s.htmlEncode() s.urlEncode().htmlEncode() etc. Of course you need to get the raw string so you can have a s.rawString() method, but now you have something you can grep for to review all uses of raw strings.
So when you come to 'echo userString' you will get a type error, and you are then reminded to encode/escape the string through the public methods.
Some other general things:
Prefer white-lists over black lists
Don't go overboard with stripping out bad input. I want to be able to use the < character in posts/comments/etc! Just make sure you encode data correctly
Use parameterized SQL queries. If you are SQL escaping user input yourself, you are doing it wrong.
First, I'll try to comfort you a bit by pointing out that it's good to be paranoid. Just as it's good to be a little scared while driving, it's good to be afraid of user input. Assume the worst as much as you can, and you won't be disappointed.
Second, program defensively. Assume any communication you have with the outside world is entirely compromised. Take in only parameters that the user should be able to control. Expose only that data that the user should be able to see.
Sanitize input. Sanitize sanitize sanitize. If it's input that will be displayed on the site (nicknames for a leaderboard, messages on a forum, anything), sanitize it appropriately. If it's input that might be sent to SQL, sanitize that too. In fact, don't even write SQL directly, use an intermediary of some sort.
There's really only one thing you can't defend from if you're using HTTP. If you use a cookie to identify somebody's identity, there's nothing you can do from preventing somebody else in a coffeehouse from sniffing the cookie of somebody else in that coffee house if they're both using the same wireless connection. As long as they're not using a secure connection, nothing can save you from that. Even Gmail isn't safe from that attack. The only thing you can do is make sure an authorization cookie can't last forever, and consider making them re-login before they do something big like change password or buy something.
But don't sweat it. A lot of the security details have been taken care of by whatever system you're building on top of (you ARE building on top of SOMETHING, aren't you? Spring MVC? Rails? Struts? ). It's really not that tough. If there's big money at stake, you can pay a security auditing company to try and break it. If there's not, just try to think of everything reasonable and fix holes when they're found.
But don't stop being paranoid. They're always out to get you. That's just part of being popular.
P.S. One more hint. If you have javascript like this:
if( document.forms["myForm"]["payment"].value < 0 ) {
alert("You must enter a positive number!");
return false;
}
Then you'd sure as hell have code in the backend that goes:
verify( input.payment >= 0 )
"Quote" everything so that it can not have any meaning in the 'target' language: SQL, HTML, JavaScript, etc.
This will get in the way of course, so you have to be careful to identify when this needs special handling, like through administrative privileges to deal with some if the data.
There are multiple types of injection and cross-site scripting (see this earlier answer), but there are defenses against all of them. You'll clearly want to look at stored procedures, white-listing (e.g. for HTML input), and validation, to start.
Beyond that, it's hard to give general advice. Other people have given some good tips, such as always doing server-side validation and researching past attacks.
Be vigilant, but not afraid.
No validation in web-application layer.
All validations and security checks should be done by the domain layer or business layer.
Throw exceptions with valid error messages and let these execptions be caught and processed at presentation layer or web-application.
You can use validation framework
to automate validations with the help
of custom validation attributes.
http://imar.spaanjaars.com/QuickDocId.aspx?quickdoc=477
There should be some documentation of known exploits for the language/system you're using. I know the Zend PHP Certification covers that issue a bit and you can read the study guide.
Why not hire an expert to audit your applications from time to time? It's a worthwhile investment considering your level of concern.
Our client always say: "Deal with my users as they dont differentiate between the date and text fields!!"
I code in Java, and my code is full of asserts i assume everything is wrong from the client and i check it all at server.
#1 thing for me is to always construct static SQL queries and pass your data as parameters. This limits the quoting issues you have to deal with enormously. See also http://xkcd.com/327/
This also has performance benefits, as you can re-use the prepared queries.
There are actually only 2 things you need to take care with:
Avoid SQL injection. Use parameterized queries to save user-controlled input in database. In Java terms: use PreparedStatement. In PHP terms: use mysql_real_escape_string() or PDO.
Avoid XSS. Escape user-controlled input during display. In Java/JSP terms: use JSTL <c:out>. In PHP terms: use htmlspecialchars().
That's all. You don't need to worry about the format of the data. Just about the way how you handle it.