I'm in a conundrum, and could really use some help...
I'm having difficulty trying to find information regarding how to enable a site -that already has X-FRAME-OPTIONS: SAMEORIGIN encoded- to be loaded into an iframe from a couple of specific domains (i.e. domain.com would be the common TLD). The issue is, that although this would be quite simple to do -using X-FRAME-OPTIONS: ALLOW-FROM http://domain.com-, if that were the only domain which would ever have a need to access the target site via an iframe. In reality however, I actually need to figure out how to set it up for (currently) three sub-domains -with the possibility of allowing from even more in the future- of the original TLD (i.e. example1.domain.com, example2.domain.com, and example3.domain.com), to be able to access the site while loaded inside the intended iframe. The only info I've been able to find regarding this issue so far is quite a bit outdated, and says that there is NO POSSIBLE WAY to allow a wildcard reference (or any other form of multiple domain reference) for a particular domain that would also apply to it's subsequent sub-domains (or anything along those lines) that seems to be effective at both functioning as intended and also preventing 'Clickjackin' by malicious individuals from occurring. I was hoping that someone more knowledgeable (and better versed in X-FRAME-OPTIONS) than myself might actually be able to offer me a feasible resolution.
Thanks in advance.
If you can entertain approaches outside of X-Frame-Options, consider creating a server-to-server API that can be called to access the content in question, and then allow it to be displayed without requiring framing.
That is, instead of ClientSite containing an IFRAME referencing the FramedPage which does the page assembly within the web browser, ClientSite calls an API on the backend to get the content directly from you and inserts the content into the page on the server before delivering the page to the user's web browser.
This gives you substantial flexibility. You could require an API key, apply basic server-to-server IP address whitelisting, or whatever suits, to prevent unwanted callers of your API.
Related
I am writing a REST API using Express Js and I have some questions regarding security.
My first question is what information can hackers get from a request made from the client side. Can they figure out the request link? What about the body and headers? Is body more secure than parameters/vice versa?
My second question is if I implemented a CORS whitelist that only allowed origins that I wanted to access my API, would that prevent anyone else from hitting the API endpoints? Can people find ways around CORS?
When a REST api is called from a browser client, everything should be treated as completely open. Anyone can read urls, headers, bodies, etc. There is no reasonable way around this, and you should design your system with this in mind.
CORS does not prevent someone from writing a script to call your API either. CORS does not add security, it takes it away by making it possible to call your API from browser applications on other domains. Not having CORS technically makes it harder to call your API in some contexts, and removes a potential security concern. The S in CORS stands for 'sharing', not 'security'.
Any security you need should be based on the server. For example, if you have data in your API that can only be read 1 user, then the server needs to make sure that a different user cannot read it. To do this, a user needs to authenticate itself.
Background:
I have a single page application that pulls data from a REST API. The API is designed such that the only URL necessary is the API root, ie https://example.com/api which provides URLs for other resources so that the client doesn't need to have any knowledge of how they are constructed.
API Design
The API has three main classes of data:
Module: Top level container
Category: A sub-container in a specific module
Resource: An item in a category
SPA Design
The app consuming the API has views for listing modules, viewing a particular module's details, and viewing a particular resource. The way the app works is it keeps all loaded data in a store. This store is persistent until the page is closed/refreshed.
The Problem:
My question is, if the user has navigated to a resource's detail view (example.com/resources/1/) and then they refresh the page, how do I load that particular resource without knowing its URL for the API?
Potential Solutions:
Hardcode URLs
Hardcoding the URLs would be fairly straightforward since I control both the API and the client, but I would really prefer to stick to a self describing API where the client doesn't need to know about the URLs.
Recursive Fetch
I could fetch the data recursively. For example, if the user requests a Resource with a particular ID, I could perform the following steps.
Fetch all the modules.
For each module, fetch its categories
Find the category that contains the requested resource and fetch the requested resource's details.
My concern with this is that I would be making a lot of unnecessary requests. If we have 100 modules but the user is only ever going to view 1 of them, we still make 100 requests to get the categories in each module.
Descriptive URLs
If I nested URLs like example.com/modules/123/categories/456/resources/789/, then I could do 3 simple lookups since I could avoid searching through the received data. The issue with this approach is that the URLs quickly become unwieldy, especially if I also wanted to include a slug for each resource. However, since this approach allows me to avoid hardcoding URLs and avoid making unnecessary network requests, it is currently my preferred option.
Notes:
I control both the client application and the API, so I can make changes in either place.
I am open to redesigning the API if necessary
Any ideas for how to address this issue would by greatly appreciated.
Expanding on my comment in an answer.
I think this is a very common problem and one I've struggled with myself. I don't think Nicholas Shanks's answer truly solves this.
This section in particular I take some issues with:
The user reloading example.com/resources/1/ is simply re-affirming the current application state, and the client does not need to do any API traversal to get back here.
Your client application should know the current URL, but that URL is saved on the client machine (in RAM, or disk cache, or a history file, etc.)
The implication I take from this, is that urls on your application are only valid for the life-time of the history file or disk cache, and cannot be shared with other users.
If that is good enough for your use-case, then this is probably the simplest, but I feel that there's a lot of cases where this is not true. The most obvious one indeed being the ability to share urls from the frontend-application.
To solve this, I would sum the issue up as:
You need to be able to statelessly map a url from a frontend to an API
The simplest, but incorrect way might simply be to map a API url such as:
http://api.example.org/resources/1
Directly to url such as:
http://frontend.example.org/resources/1
The issue I have with this, is that there's an implication that /resource/1 is taken from the frontend url and just added on to the api url. This is not something we're supposed to do, because it means we can't really evolve this api. If the server decides to link to a different server for example, the urls break.
Another option is that you generate uris such as:
http://frontend.example.org/http://api.example.org/resources/1
http://frontend.example.org/?uri=http://api.example.org/resources/1
I personally don't think this is too crazy. It does mean that the frontend needs to be able to load that uri and figure out what 'view' to load for the backend uri.
A third possibility is that you add another api that can:
Generate short strings that the frontend can use as unique ids (http://frontend.example.org/[short-string])
This api would return some document to the frontend that informs what view to load and what the (last known) API uri was.
None of these ideas sound super great to me. I want a better solution to this problem, but these are things I came up with as I was contemplating this.
Super curious if there's better ideas out there!
The current URL that the user is viewing, and the steps it took to get to the current place, are both application state (in the HATEOAS sense).
The user reloading example.com/resources/1/ is simply re-affirming the current application state, and the client does not need to do any API traversal to get back here.
Your client application should know the current URL, but that URL is saved on the client machine (in RAM, or disk cache, or a history file, etc.)
The starting point of the API is (well, can be) compiled-in to your client. Commpiled-in URLs are what couple the client to the server, not URLs that the user has visited during use of the client, including the current URL.
Your question, "For example, if the user requests a Resource with a particular ID", indicates that you have not grasped the decoupling that HATEOAS provides.
The user NEVER asks for a resource with such-and-such an ID. The user can click a link to get a query form, and then the server provides a form that generates requests to /collection/{id}. (In HTML, this is only possible for query strings, not path components, but other hypermedia formats don't have this limitation).
When the user submits the form with the ID number in the field, the client can build the request URL from the data supplied by the server+user.
Currently as it stands, if a user reads the source of my web application, they'd be able to determine the direct URIs of all the RESTful services my web application utilizes.
The problem I see is this: My web application knows how to correctly use the API, and I might not have thought of every single validation known to man to prevent bad data from being sent through the API.
And so with that is there a method to prevent "direct" access to the API and limit it only to my web application?
P.S. As an FYI: API calls concerning a user are protected by the presence of a user-specific cookie which is only issued upon login. This means I'm not too afraid of User X being able to directly modify User Y's data through the API.
No.
If the browser is making the request, the user can spoof the request. Period.
My web application knows how to correctly use the API
That's good, but that's leading you down the path of assuming client-side functionality executed as intended. Never make that assumption.
I might not have thought of every single validation known to man to prevent bad data from being sent through the API
This statement leads me to believe that the API itself is more complex than it needs to be. The best thing you can do is simplify. It's difficult to be more specific without seeing specific code, but API requests should be fairly simple and straightforward and the same techniques to prevent malicious code from getting through should be applied universally. The same general rules apply here as in any web application interaction...
Never trust anything that came from the client
Never assume client-side code executed as intended
Never execute input as code, always treat it as a raw value
and so on...
As you mention toward the end, you've already taken care of authentication and authorization for the requests. Given that, if User X is permitted to make a given API call, then what you're essentially asking is, "How do I allow User X to make an API call without allowing User X to make an API call?" The server can't tell the difference. A request is a request.
Sure, there are things you can try, such as always including some custom header in requests made from code. But anybody can inspect that request and spoof that header. The user's browser isn't part of your application and isn't under your control.
I am trying to search and find a content from a site by using Perl Mechanize.It worked fine in the beginning after few execution i am getting 403 Forbidden instead of the search results,
$m = WWW::Mechanize->new();
$url="http://site.com/search?q=$keyword";
$m->get($url);
$c = $m->content;
print "$c";`
how can solve this problem. Please give me some suggestions.
Before beginning to scrape a site, you should make sure that you are authorized to do so. Most sites have a Terms Of Service (TOS), that lay out how you can use the site. Most sites disallow automatic access, and place strong restrictions on the intellectual property.
A site can defend against unwanted access on three levels:
Conventions: The /robots.txt almost every site has should be honored by your programs. Do not assume that a library you are using will take care of that; honoring the robots.txt is your responsibility. Here is a excerpt from the stackoverflow robots.txt:
User-Agent: *
Disallow: /ask/
Disallow: /questions/ask/
Disallow: /search/
So it seems SO doesn't like bots asking questions, or using the site search. Who would have guessed?
It is also expected that a developer will use the API and similar services to access the content. E.g. Stackoverflow has very customizable RSS feeds, has published snapshots of the database, even has an online interface for DB queries, and an API you can use.
Legal: (IANAL!) Before accessing a site for anything other than your personal, immediate consumption, you should read the TOS, or whatever they are called. They state if and how you may access the site and reuse content. Be aware that all content has some copyright. The copyright system is effectively global, so you aren't exempt from the TOS just by being in another country than the site owner.
You implicitly accept the TOS by using a site (by any means).
Some sites license their content to everybody. Good examples are Wikipedia and Stackoverflow, which license user submissions under CC-BY-SA (or rather, the submitting users license their content to the site under this license). They cannot restrict the reuse of content, but can restrict the access to that content. E.g. the Wikipedia TOS contains this a section Refraining from certain activities:
Engaging in Disruptive and Illegal Misuse of Facilities
[…]
Engaging in automated uses of the site that are abusive or disruptive of the services […]
[…] placing an undue burden on a Project website or the networks or servers connected with a Project website;
[…] traffic that suggests no serious intent to use the Project website for its stated purpose;
Knowingly accessing, […] or using any of our non-public areas in our computer systems without authorization […]
Of course, this is just meant to make disallow a DDOS, but while Bots are an important part of Wikipedia, other sites do tend to frown on them.
Technical measures: … like letting connections from an infringing IP time out, or sending a 403 error (which is very polite). Some of these measures may be automated (e.g. triggered by useragent strings, weird referrers, URL hacking, fast requests) or by watchful sysadmins tailing the logs.
If the TOS etc. don't make it clear that you may use a bot on the site, you can always ask the site owner for written permission to do so.
If you think there was a misunderstanding, and you are being blocked despite regular use of a site, you can always contact the owner/admin/webmaster and ask them re-open your access.
our client needs to shortcuts to particular pages
We need to redirect non existent urls like
http://site.com/promotion1
to the actual URL similar to
http://site.com/promotions/promotion1/tabid/799/language/en-AU/Default.aspx
...
I've sent a list of appropriate DNN modules to our client but it may take them forever to get back to me.
In the mean time they still submitting requests to us to create redirects for them.
if there's no cost involved then i wont have to wait for them to get back to me.
so I'm looking for a Quick and free way to enable the clients to set these up on this own.
I've looked at:
MAS.ActionRedirect
Ventrian Friendly URL Provider
DotNetNuke URL Rewriting HTTP Module
But haven't had much luck in the small amount of time i have available.
Has anyone got some suggestions on how to achieve our goal with either the above resources or maybe some additional resource i haven't found yet?
(DNN v4.9)
You should be able to use the built-in friendly URL functionality within DNN, or use a URL rewriter module within IIS.
You can read my answer about using the DNN Friendly URL functionality for more details, or look into the IIS URL Rewrite module.