I'm curious whether if it's possible to set up a server to respond with html fetched from another domain rather than simply redirect the requester to that domain.
For example, I set up a simple node express server that has a GET route /google, which fetches google.com, and then responds with the response from the fetch. However, in this case, it does not respond with the google webpage as I would expect.
It is not only possible but quite common especially in larger server environments. The term you are looking for is reverse-proxy.
Proxying is typically used to distribute the load among several servers, seamlessly show content from different websites, or pass requests for processing to application servers over protocols other than HTTP.
Source: https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/
Most major web servers support it.
More than likely the response you're getting from google (and passing on) is some kind of redirection. Try it with a static web page of your own to rule out any redirection shenanigans.
Related
The situation
There's a REST API which is currently consumed by other backend systems.
Now, that same REST API is going to be used by a single page application soon, and that SPA needs some additional security measures (CSRF token verification among others), but those additional security measures should be enforced only against end users running ordinary browsers, and not against other backends, so that those existing other backends keep working without any changes.
The Question
How do you distinguish between when a browser is consuming a REST API and when another backend is consuming it?
Is there a header that will be sent by any modern browsers and can't be turned off or can't be tampered with?
Maybe the User-Agent? Or do REST libs (in any language) send that too?
Or the Referer? Or Origin? Or some other headers?
Or something else other than a header?
Is there a header that will be sent by any modern browsers and can't be turned off or can't be tampered with?
As far as I know, you aren't going to find what you are looking for.
User-Agent is close
The "User-Agent" header field contains information about the user agent originating the request, which is often used by servers to help identify the scope of reported interoperability problems, to work around or tailor responses to avoid particular user agent limitations, and for analytics regarding browser or operating system use. A user agent SHOULD send a User-Agent field in each request unless specifically configured not to do so.
But it certainly isn't "tamper-proof"; it's just a text header, many user agents will allow you to customize it, etc.
I am writing a REST Api gateway for an Angular SPA and I am confronted with the problem of securing the data exposed by the API for the SPA against "data thiefs". I am aware that I can't do much against HTML scraping, but at least I don't want to offer such data thiefs the user experience and full power of our JSON sent to the SPA.
The difference between most "tutorials" and threads about this topic is that I am exposing this data to a public website (which means no user authentication required) which offers valuable statistics about a video game.
My initial idea on how to protect the Rest API for SPA:
Using JWTs everywhere. When a visitor opens the website the very first time the SPA requests a JWT from my REST Api and saves it in the HTTPS cookies. For all requests the SPA has to use the JWT to get a response.
Problems with that approach
The data thief could simply request the oauth token from our endpoint as well. I have no chance to verify that the token has actually been requested from my SPA or from the data thief?
Even if I solved that the attacker could read the saved JWT from the HTTPS cookies and use it in his own application. Sure I could add time expiration for the JWT
My question:
I am under the impression that this is a common problem and therefore I am wondering if there are any good solutions to protect against others than the SPA having direct access to my REST Api responses?
From the API's point of view, your SPA is in no way different than any other client. You obviously can't include a secret in the SPA as it is sent to anybody and cannot be protected. Also the requests it makes to the API can be easily sniffed and copied by another client.
So in short, as diacussed many times here, you can't authenticate the client application. Anybody can create a different client if they want.
One thing you can actually do is checking the referer/origin of requests. If a client is running in a browser, thr requests it can make are somewhat limited, and one such limitation is the referer and origin headers, which are always controlled by the browser, and not javascript. So you can actually make sure that if (and only if!) the client is running in an unmodified browser, it is downloaded from your domain. This is the default in browsers btw, so if you are not sending CORS headers, you already did this (browsers do, actually). However, this does not keep an attacker from building and running a non-browser client and fake any referer or origin he likes, or just disregard the same origin policy.
Another thing you could do is changing the API regularly just enough to stop rogue clients from working (and changing your client at the same time ofc). Obviously this is not secure at all, but can be annoying enough for an attacker. If downloading all your data once is a concern, this again doesn't help at all.
Some real things you should consider though are:
Does anybody actually want to download your data? How much is it worth? Most of the times nobody wants to create a different client, and nobody is that much interested in the data.
If it is that interesting, you should implement user authentication at the very least, and cover the remaining risk either via points below and/or in your contracts legally.
You could implement throttling to not allow bulk downloading. For example if the typical user accesses 1 record every 5 seconds, and 10 altogether, you can build rules based on the client IP for example to reasonably limit user access. Note though that rate limiting must be based on a parameter the client can't modify arbitrarily, and without authentication, that's pretty much the client IP only, and you will face issues with users behind a NAT (ie. corporate networks for example).
Similarly, you can implement monitoring to discover if somebody is downloading more data than it would be normal or necessary. However, without user authentication, your only option will be to ban the client IP. So again it comes down to knowing who the user is, ie. authentication.
I am currently implementing a Facebook Chat Extension which basically is just a web page displayed in a browser provided by the Facebook Messenger app. This web page communicates with a corporate backend over a REST API (implemented with Python/Flask). Communication is done via HTTPS.
My question: How to secure the communication the Web page and the backend in the sense that the backend cannot be accessed by any clients that we do not control?
I am new to the topic, and would like to avoid making beginners' mistakes or add too complicated protocols to our tech stack.
Short answer: You cant. Everything can be faked by i.e. curl and some scripting.
Slightly longer:
You can make it harder. Non browser clients have to implement everything you do to authenticate your app (like client side certificates and Signet requests) forcing them to reverse engineer every obfuscation you do.
The low hanging fruit is to use CORS and set the Access Allow Origin Header to your domain. Browsers will respect your setting and wont allow requests to your api (they do an options request to determine that.)
But then again a non official client could just use a proxy.
You can't be 100% sure that the given header data from the client is true. It's more about honesty and less about security. ("It's a feature - not a bug.")
Rather think about what could happen if someone uses your API in a malicious way (DDoS or data leak)? And how would he use it? There are probably patterns to recognize an attacker (like an unusual amount of requests).
After you analyzed this situation, you can find more information here about the right approach to secure your API: https://www.incapsula.com/blog/best-practices-for-securing-your-api.html
REST and CORS.. how are they different? is it even correct to compare them? because I have seen a seemingly REST API use custom X- headers to make a pre-flighted request(Docebo LMS API). This means that maybe CORS and REST are used for different purposes.. But on the surface, it seems that both are designed to give access to resources stored on a different server. Also, Simple XMLHTTP requests seem to work like HTTP.(The headers sent and received by the browser are through HTTP).. So, are XMLHTTP objects translated into HTTP by the browser? I am really taking in a ton of information right now and I cant seem to make any real progress in understanding these things... Any help is appreciated.
CORS - Cross Origin Resource Sharing. A concept and set of techniques that enables sharing of resource/data across domains. Example, from your page /yourDomain.net you try to make an ajax call to myDomain.net to post some data. Read this Wikipedia and MDN articles.
REST - REpresentational State Transfer. A set of standards & guidelines that defines a specific way for systems to talk to each other. It follows state-less http like standards where URIs reprsent resource and client can work on them using http verbs. e.g. GET weatherApp.com/weather/rome. Refer this.
HTTP - Hyper Text Transfer Protocol. THE standard protocol to transfer data to/from web servers. Check this W3 specifications and Wikipedia page.
XMLHttp - A type of request generally used to make ajax calls from client (mainly html, javascript) applications to web servers. It works on http standards. Not bound to XML though. Read this and this.
Now, all of REST, XMLHttp, CORS work on HTTP is some way, meaning they all use the http infrastructure.
And any/all of them might be used to create a fully functional modern application. For example, a web application might use XMLHttp request to make REST service call to get some data. It can also utilize CORS to get/post data to another domain. Need not say, the whole system relies on http!
They are totally different things. Rest is a specifical approach to prrforming data calls. Basically is characterized by a systen where the state is not stored on the server but rather passed in calls. You can read more here
Cors is a technique for enabling javascript to perform data ervice calls to domains otheir than the server donain that they came from. Normally web browsers prevent javascript and other web technologies from doing cross origin or cross domain calls. These are calls where a js script came from google.com lets say, and now it wats to call microsoft.com. well the browser would stop that call because google.com and microsoft.com are different domains.
That example is obvious, so lets try a less obvious one. Your script on blogs.yoursite.com tries to call a service at shopping.yoursite.com. now these sires are both yoursite.com but they could still be considered cross domain and usually are. CORS allows you(on the html developer side) to say i trust these domains. And by trusting them, now you can call their webservices even if they would have been a cross domain call.
I have a feedback form on server A. Due to heavy traffic I want to redirect some of these POST requests to another server B based on the category of feedback (a field in the post form).
The problem is the feedback form is distributed across multiple mobile applications and is hard-coded now. We cannot change anything in the Form post now.
What is the best approach? Is it possible to do using DNS redirection? I would ideally like that before the request reaches the Apache server to increase the load, the redirection happens. Please suggest.
You could return a 307 HTTP response, which tells the client to redirect to the other location, while re-posting all data. That could be a disadvantage for large chunks of data (files), but it will be too if you choose to send the data to the other server yourself. Besides that, the first server will receive the data anyhow, so you only gain by this solution if the processing itself is relatively costful.
An even better solution would be to use a load balancer which can determine for you which server to send the data too.
Ideally you need to change the form (not the actual form itself, but supporting code). The idea is to add some JavaScript, that upon submitting a form checks that "category" field and changes the value of form's "action" attribute making it submitted to a desired server.
If that is cannot be done at all (for whatever reason), then this may be implemented on server side. Instead of hitting Server A directly, it should hit Server Zero -- a load balancer, which then can route the request further based on the logic you provide: e.g. 1) choose less busy server; 2) randomly choose server; 3) analyse POST form data and choose right server; 4) other options. #2 can be implemented via DNS (an example is available in the link below).
Any load balancer should be able to implement options 1 & 2. For other options you have to check if specific load balancer (software or hardware) supports it (check feature list).
Links:
http://en.wikipedia.org/wiki/Load_balancer