With AWS services we have the Web application running from the S3 bucket and accessing the data through the REST API from Load Balancer (which is set of Node.js applications running on EC2 instance).
Currently we have specified URL's as following:
API Load Balancer: api.somedomain.com
Static Web App on S3: somedomain.com
But having this setup brought us a set of problems since requests are CORS with this setup. We could workaround CORS with special headers, but that doesn't work with all browsers.
What we want to achieve is running API on the same domain but with different path:
API Load Balancer: somedomain.com/api
Static Web App on S3: somedomain.com
One of the ideas was to attach the API Load Balancer to the CDN and forward all request to Load Balancer if query is coming on the "/api/*" path. But that doesn't work since our API is using not only HEAD and GET requests, but also POST, PUT, DELETE.
Another idea is using second EC2 instance instead of S3 bucket to host website (using some web server like nginx or apache). But that gives too much overhead when everything is in place already (S3 static content hosting). Also if using this scenario we wouldn't get all the benefits of Amazon CloudFront performance.
So, could your recommend how to combine Load Balancer and S3, so they would run on same domain, but with different paths? (API on somedomain.com/api and Web App on somedomain.com)
Thank you!
You can't have an EC2 instance and an S3 bucket with the same host name. Consider what happens when a web browser makes a request to that host name. DNS resolves it to an IP address (or addresses) and the packets of the request are delivered to that address. The address either terminates at the EC2 instance or the S3 bucket, not both.
As I understand your situation, you have static web pages hosted on S3 that include JavaScript code that makes various HTTP requests to the EC2 instance. If the S3 web pages are on a different host than the EC2 instance then the same origin policy will prevent the browser from even attempting some of the requests.
The only solutions I can see are:
Make all requests to the EC2 instance, with it fetching the S3 contents and delivering it to the browser whenever a web page is asked for.
Have your JavaScript use iframes and change the document.domain in the the web pages to a common parent origin. For example, if your web pages are at www.example.com and your EC2 instance is at api.example.com, the JavaScript would change document.domain to just example.com and the browser would permit iframes from from www.example.com to communicate with api.example.com.
Bite the bullet and use CORS. It's really not hard, and it's supported in all remotely recent browsers (IE 8 and 9 do it, but not in a standard way).
The first method is no good, because you almost might as well not use S3 at all in that case.
The second case should be okay for you. It should work in any browser, because it's not really CORS. So no CORS headers are needed. But it's tricky.
The third, CORS, approach should be just fine. Your EC2 instance just has to return the proper headers telling web pages from the S3 bucket that it's safe for them to talk to the EC2 instance.
Just wanted to add an additional bit to the answer that, if we go with CORS approach and preflight requests adds an overhead to the server and network bandwidth, we may even consider adding header "Access-Control-Max-Age" to the CORS response
Access-Control-Max-Age
Related
I have my google bucket connect with a load balancer and CDN enabled in google cloud, but I really don't get how google CDN working for static file, checking in the log viewer i can see the "statusDetails: response_from_cache" and "cacheHit: true" so i can say that the CDN is working properly.
Trying to issue a request for the image in my google CDN bucket from a computer located in Europe, the file return from the frontend IP address of my load balancer. Also the same IP address served my image if i make the request from a computer located in Asia.
So the same IP address was used for serving my static image ignore the location where the request coming from, checking the log viewer again, i can see that both of the request has claimed to go through google CDN, again google log viewer tell me that CDN working properly.
i think that the CDN should serve the file from the nearest server to the end-users, what is the point for using google CDN if the file always served from only 1 single IP address for all user over the world?
I have a free account of cloudflare, once i configure my DNS, the image file go through cloudflare network and if i do the test as above, i will see my static image file returned from multiple IP address which is nearest to my end-users.
Could somebody help me to understand what is the purpose for using google CDN in this case ? did i miss something in the configuration process for google CDN?
Thanks a lot in advance.
Google Cloud CDN uses Google's global edge network to serve content
closer to users and it leverages Google Cloud global external HTTP(S)
load balancers to provide routing, health checking, and Anycast IP
support. The HTTP(S) load balancing configuration specifies the
frontend IP addresses and ports on which Cloud CDN receives requests
and the backends that originate responses to those requests.
Google CDN has a special feature of ‘single anycast IP for the whole
network’ letting all contents served through the load balancer
frontend IP resulting in low latency. So rather than having one load
balancer per region, you can simplify your architecture and have
every instance behind a single global load balancer. Also it has a
feature of HTTP/2 which supports the latest HTTP protocol for faster
performance. For additional information, you can check here.
Cloud CDN reduces latency by serving assets directly at Google's
network edge. To know more about the caching using Cloud CDN, refer
to this caching-overview docs.
I am new to Cloud Foundry.
Is there any way that only specific users can view and update an app deployed in Cloud Foundry?
1.I deployed an app in Cloud Foundry using “cf push”command.
2.After entering “cf push “command I’ve got an message below.
Using manifest file /home/stevemar/node-hello-world/manifest.yml
enter Creating app node-hello-world-example...
name: node-hello-world-example
requested state: started
routes: {route-information}
last uploaded: Mon 14 Sep 13:46:54 UTC 2020
stack: cflinuxfs3
buildpacks: sdk-for-nodejs
type: web
instances: 1/1
memory usage: 256M
3.Using the {route-information} above,I can see the app deployed via browser entering below URL.
https://{route-information}
By this way ,anyone can see app from browser, but I don’t want that to be seen by everyone and limit access to specific user.
I heard that this global IP will be allocated to {route-information} by default.
Is there any way to limit access to only between specific users?
(For example,is there any function like “private registry” at Kubernetes in Cloud Foundry which is not open to public)
Since I am using Cloud Foundry in IBM Cloud it would be better if there is solution using IBM Cloud.
I’ve already granted cloud foundry role to the other user.
Thank you.
The CloudFoundry platform itself does not provide any access controls for applications. If you assign a public route to your application, where the DNS is publicly resolvable and the foundation is on the public Internet, like IBM Bluemix, then anyone can access your app.
There's a number of things you can do to limit access, but they do require some work on your part.
Use a private DNS. You can add any domain you want to Cloud Foundry, even ones that don't resolve. That means you could add my-cool-domain.local which does not resolve anywhere. You could then add a record to /etc/hosts for this domain or perhaps run DNS on your local network to resolve this DNS domain and direct traffic to the CloudFoundry.
With this setup, most people cannot access your application because the DNS domain for the route to your application does not resolve anywhere. It's important to understand that this isn't really security, but obscurity. It would stop most traffic from making it to your app, but if someone knew the domain, they could add their own /etc/hosts header or send fake Host headers to access your application.
This type of setup can work well if you have light security requirements like you just want to hide something while you work on it, or it can work well paired with other options below.
You can set up access controls in your application. Many application servers & frameworks can do things like restrict access by IP address or require user access (Basic auth is easy and it is OK, if you're only allowing HTTPS traffic to your app which you should always do anyway).
You can use OAuth2 to secure apps too. Again, many app servers & frameworks have support for this and make it relatively simple to secure your apps. If you don't have a corporate OAuth2 solution, there are public providers you can use. Exactly how you do OAuth2 in your app is beyond the scope of this question, but there's plenty of material out there on how to do this. Google information for your application language/framework of choice.
You could set up an access Gateway. This would be an application that's job is to proxy traffic to other applications on the foundation. The Gateway could be something like Nginx, Apache HTTPD, or Spring Cloud Gateway. The idea is that the gateway would be publicly accessible, and would almost certainly apply access controls/restrictions (see #2, many of these proxies have access control options that only take a few lines of config). Your actual applications would not be deployed publicly though. When you deploy your actual applications, they would only be on the internal Cloud Foundry domain.
CloudFoundry has local domains, often apps.internal (run cf domains to see if that shows up), which you can use to easily route traffic across the internal container-to-container network. Using this domain and the C2C network, you can have apps deployed to CF that are not accessible to the public Internet, except through your Gateway.
Again, how you configure this exactly is outside the scope of this question, but check out the docs I linked to for info on using the C2C network & internal routes. Then check out your proxy server of choice's documentation.
How to name health check routes among several services behind ALB?
I'm moving my API and database to AWS. Before moving I split up my monolith REST API into four services:
public API (to which apps and websites connects)
admin API (for admin web site)
messaging API (web socket server for realtime communication with apps)
workers (queue based task processors)
I'm now trying to figure out a good organisation of the routes. At first created two subdomains, api.mydomain.com and www.mydomain.com.
I directed the api subdomain to my ALB which routed traffic based on the path only, like this:
"/sockets" -> messaging-api
"/admin" -> admin-api
"/" -> public-api
Now I'm trying to implement the health check routes. I'd like to name them "/health". But the health checks needs to be directed to each target group. Since the ALB only routes based on the path I cannot have /health on more than one server.
Possible solutions:
1. Separate the services via subdomains
I could create a subdomain for each service like:
- api.mydomain.com
- sockets.mydomain.com
- admin.mydomain.com
With this setup I could have a /health in each service without collisions.
2. Separate the health check routes via naming
I could name the health check route differently for each service like:
api.mydomain.com/health-public-api
api.mydomain.com/health-messaging-api
api.mydomain.com/health-admin-api
Suggestions?
Both the above solutions seems viable, but I'd like to know if maybe one of the solutions will bite me later, when for example more services are added, or when I'll add a graphQL API later on.
edit:
I just bumped into one drawback with solution #1. My local
dev-enviromnemt is setup with a docker image for each service and
nginx for routing the requests. On top of this I use ngrok to be able
to reach the dev environment from the Internet.
I think it would be hard to solve the service separation in based on
subdomains, but I really don't need the /health routes in the dev
enviromnent, so I guess I could just pretend they are not there.
Answering my own question as documentation and possibly some input to others.
tl;dr:
I went with a third option, separating all services via the first level in the the path. The main difference from my previous structure is that my main api (aka public-api) has moved from the root to a subpath called /app. I also renamed it to app-api.
api.mydomain.com/app/..
api.mydomain.com/admin/..
api.mydomain.com/sockets/..
api.mydomain.com/auth/..
www.mydomain.com/..
This solution gives me several pros and no cons (I think).
Pros:
Easy to route request both in ALB and in local dev environment via nginx without the extra work needed for SNI
subdomains separates api vs web sites very clearly
/health routes gets unique names by default since they live under separate paths.
the apps (web and smartphones) can use a common api url (api.mydomain.com/) and still reach all services, i.e. they don't need to store several differently initialized Axios connections. No biggie, but still..
I also opted for making /health a little more future proof and standardized on the following structure in each service.
api.mydomain.com/servicename/health
api.mydomain.com/servicename/health/is-up
api.mydomain.com/servicename/health/is-ready
up = responding to requests, ready = all dependencies are connected (i.e. databases, etc)
/health returns status 200 along with a json object describing the readiness.
/health/is-up responds with 200 or nothing (i.e. not reachable at all)
/health/is-ready responds with 200 if all dependencies are ready, otherwise 500.
The target groups in AWS will use /is-ready for health checks, but for now it's the same thing as /is-up since I haven't implemented the readiness tests yet.
I have a gadget[*] that connects to the user's WiFi network and responds to commands over a simple REST interface. The user uses a web app to control this gadget. The web app is currently served over http and the app's javascript does AJAX calls to the gadget's local IP address to control it. This scheme works well and I have no issues with it.
[*] By "gadget" I mean an actual, physical IoT device that the user buys and installs within their home, and configures to connect to their home WiFi network
Now, I want to serve this web app over https. I have no issue setting up https on the hosting side. The problem is, now the browser blocks access to the gadget (since the gadget's REST API is over http and not https).
The obvious solution is to have the gadget serve it's REST API over https. But how? It has a local IP address and no one will issue a certificate for it. (Even if they did, I'd have to buy a boatload of certificates for each possible local IP address.) I could round-trip via the cloud (by adding additional logic on my server side to accept commands from the web app and forward it to the gadget over another connection), but this will increase latencies.
Is there a way around this problem? One possibility that I have in mind is to:
Get a wildcard certificate (say, *.mydomain.com)
Run my own DNS that maps sub-domains to a local IP address following a pattern (For example, 192-168-1-123.mydomain.com would map to 192.168.1.123)
Use the wild-card certificate in all the gadgets
My web app could then make AJAX calls to https://192-168-1-123.mydomain.com instead of http://192.168.1.123 and latencies would remain unaffected aside from the initial DNS lookup
Would this work? It's an expensive experiment to try out (wildcard certificates cost ~$200) and running a DNS server seems like a lot of work. Plus I find myself under-qualified to think through the security implications.
Perhaps there's already a service out there that solves this problem?
While this is a pretty old question, it is still nothing that you find out-of-the-box solutions for today.
Just as #Jaffa-the-cake posted in a comment, you can lean on how Plex did it, which Filippo Valsorda explained in his blog:
https://blog.filippo.io/how-plex-is-doing-https-for-all-its-users/
This is very similar to what you proposed yourself. You don't even need a wildcard certificate, but you can generate certificates on-the-fly using Let's Encrypt. (You can still use wildcard certificates, if you want, which Let's Encrypt supports now, too.)
Just yesterday I did a manual proof-of-concept for that workflow, that can be automated with the following steps:
Write a Web Service that can create DNS entries for individual devices dynamically and generate matching certificates via Let's Encrypt - this is pretty easy using certbot and e.g. Google Cloud DNS. I guess Azure, AWS and others have similar offerings, too. When you use certbot's DNS plugins, you don't even need to have an actual web server running on port 80/443.
On you local device, contact that Web Service to generate a unique DNS entry (e.g. ..yourdns.com) and certificate for that domain
Use that certificate in your local HTTPS server
Browse to that domain instead of your local IP
Now you will have a HTTPS connection to your local server, using a local IP, but a publicly resolved DNS entry.
The downside is that this does not work offline from arbitrary clients. And you need to think of a good security concept to create trust between the client that requests a DNS and certificate, and your web service that will generate those.
BTW, do you mind sharing what kind of gadget it is that you are building?
If all you want is to access the device APIs through the web browser, A Simple solution would be to proxy all the requests to the device through your web server.this was even self signed certs for the devices wont be a problem. Only problem though is that the server would have to be on the same network as your devices.
If you are not on the same network, you can write a simple browser plugin (chrome) to send the api request to IoT device. but then the dependency on the app/plugin will be clumsy.
This question may be a bit subjective but I think will offer some valuable concrete information and solutions to proxying to heroku and debugging latency issues.
I have an app built using Sinatra/Mongo that exposes a REST API at api.example.com. It's on Heroku Cedar. Typically I serve static files through nginx at www and proxy requests to /api through to the api subdomain to avoid cross-domain browser complaints. I have a rackspace cloud instance so I put the front-end there temporarily on nginx and setup the proxy. Now latency is horrible when proxying, every 3 or 4 requests it takes longer than 1 minute, otherwise ~150ms. When going directly to the API (browser to api.example.com) average latency is ~40ms. While I know the setup isn't ideal I didn't expect it to be that bad.
I assume this is in part due to proxying from rackspace - server may well be on the west coast - to heroku on amazon ec2 east. My thought at the moment is that getting an amazon ec2 instance and proxying that to my heroku app would alleviate the problem, but I'd like to verify this somehow rather than guessing blindly (it's also more expensive). Is there any reasonable way to determine where the long latency is coming from? Also, any other suggestions as to how to structure this application? I know I can serve static files on Heroku, but I don't like the idea of my API serving my front-end, would rather these be able to scale independently of one another.
Since you're using Heroku to run your API, what I'd suggest is putting your static files into an Amazon S3 bucket, something named 'myapp-static', and then using Amazon Cloudfront to proxy your static files via a DNS CNAME record (static.myapp.com).
What's good about using S3 over Rackspace is that:
Your files will be faster for you to upload from Heroku, since both your app and storage are on the same network (AWS).
S3 is built for serving static files directly, without the overhead of running your own server proxying requests.
What's good about using Cloudfront is that it will cache your static files as long as you want (reducing multiple HTTP requests), and serve files from an endpoint closest to the user. EG: If a user in California makes an API request and gets a static file from you, it will be served from them from the AWS California servers as opposed to your East Coast Heroku instances.
Lastly, what you'll do on your application end is send the user a LINK to your static asset (eg: http://static.myapp.com/images/background.png) in your REST API, this way the client is responsible for downloading the content directly, and will be able to download the asset as fast as possible.