I've been victim of users using my undocumented API endpoints for quite a while now. But I've been thinking of an idea for the past few weeks, where the endpoint that my webapp uses, changes every 24-48 hours. This would be to discourage these users from reverse-engineering and using my API, subsequently using up my server resources.
How feasible would this be? And do you think there would be a way to automate this?
Related
I need some help with deciding on the architecture of my project (a web app for unlocking discounts). I am first planning on creating the website (React for the front-end & Django for the back-end, PostgreSQL database). In the future, I may create a mobile app too for Android & iOS (unsure what front-end framework yet).
So I have decided I want the front-end and back-end to be completely separated so the back-end is a REST api. This will allow me to not have to create multiple back-ends for mobile apps.
But, after researching, I have found that this could be quite expensive in terms of server costs. This is a new business and I am the only developer so funding isn't high. So I was thinking that I could deploy the front-end & back-end on the same server but as separate apps that talk via nginx?
I have 4 questions about this:
If I do this, would it still be possible to reuse the back-end as a REST api for the mobile apps or is that a no because it's linked to the web front-end?
If it is possible, would I be able to host the mobile front-end in the same server (so have everything hosted on 1 server)?
Is this a stupid idea - would I just be better off deploying everything into separate servers in the long-run (to reduce load)?
Should I just worry about this in the future? And for now just deploy the separated web front-end & back-end to the same server.
I have never really deployed anything into a real life production environment so I'm sorry if my questions seem silly. I haven't started development yet but I want to think about scalability & future extensibility before I start. Thank you.
Nowadays I'd go with a serverless approach. Instead of having servers to maintain you can focus on your app functionalities.
There are a lot of options. You can check, for example, AWS Amplify (https://aws.amazon.com/amplify/) or Netlify (https://www.netlify.com/) for a more "full-stack" approach.
In AWS, you also can keep separated projects, having your backend in lambdas and your frontend served through S3 + CloudFront. You also don't have servers to care about.
There are only examples of how you can solve your problem without servers, but answering your questions:
You can reuse your APIs regardless of the way your app is deployed. It will be more related to how you designed them;
Yes, you can host everything in a single server if you want, but I really don't recommend that;
If you don't want to pay for 24/7 servers, you can go for a serverless approach;
As I told you before, you can do what you want without worrying about servers.
Your main point of focus is to keep the cost lower and to implement a good solution also. My suggestion would be to look for AWS Lightsail. Lightsail offers fixed price VM which you can configure yourself, and it starts from $3.5 / month at the time of writing this answer.
My answers to your questions
If I do this, would it still be possible to reuse the back-end as a REST api for the mobile apps or is that a no because it's linked to the web front-end?
Yes, it's possible. Keep the frontend and backend in different repo, and you can deploy it as docker instances on the same server. You will have 1 frontend docker container and 1 backend docker container, and they can communicate with each other.
If it is possible, would I be able to host the mobile front-end in the same server (so have everything hosted on 1 server)?
For mobile, you will develop a mobile application which you can publish to playstore or deploy to smartphone. Your app can then call the backend service and get the JSON in response. So you have to design your backend in such a way that it can serve data to both requests.
Is this a stupid idea - would I just be better off deploying everything into separate servers in the long-run (to reduce load)?
For long term and design perspective, you need to consider factors like scalability, maintainability, security etc.., so its always better to have multiple server to avoid single point of failure.
Should I just worry about this in the future? And for now just deploy the separated web front-end & back-end to the same server.
My advice to you will be to think carefully now, so you don't get nightmares in the future. Invest your time now and design a stable solution which could help you in long-term. As you mentioned that its a small business, but your solution should be able to easy handle growth.
My suggestion
As suggested by the Paulo, S3 + CloudFront looks good for frontend. You can get 1 year free CDN using Lightsail.
For Backend, you should at least have 2 (I will suggest minimum 3) servers and deploy backend docker containers. You can use docker compose to automate the deployment. If you want to orchestrate then Docker Swarm Mode is best. With this you will avoid single point of failure. You can get very affordable servers from Amazon Lightsail
For database, you need to make it scalable. To ensure scalability and High Avalability we should have replicated DB. Minimum 3 DB instances will be good starting point. MongoDB is a good choice. With simple configuration you can enable DB replication. 1 Master 2 slaves instances.
1 Load-balancer in front of your servers to distribute the load. To save the cost you can configure the Load-balancer yourself but this will add learning curve and you will have to spent time and understanding the details. The better solution is to use a managed load balancer. Lightsail offers Load Balancer for $18 / month at the time writing this answer.
The above mentioned solution is cost-effective and will give you long-term benefit and also you can estimate the cost based on your solution.
Obviously, this can still be improved but I tried to cover the necessary aspects of the question asked.
Background:
I'm building an API service app. The app is just like any other, you send an HTTP request and receive a response. This seems simple up until I start thinking about user registration, payments, authentication, logging and so on.
Application:
tl;dr simple app diagram
Endpoints listening for HTTP requests and doing all the request related work. This is the core of the service, what the service user would use this app for. Directly not accessible to the end user (unless somehow it knows the url). Python flask server, deployed on google cloud RUN.
API gateway acting like a proxy and a single access point forwarding the requests to the endpoints. This is the service access point for the end users. This part will also be responsible for authentication, limitations, logging and tracking the use of the API endpoints. Python flask server, deployed on google cloud RUN.
Website including documentation, demo and show off of API calls through API gateway, registration, payment (thinking of Stripe) etc. VueJS app on NodeJS server on google cloud compute VM.
Database storing credentials of registered users, payment information and auth keys. Not implemented yet.
Problems:
Is this architecture proper? What could be done differently or improved? How could I further simplify all the interactions between separate parts of the app? Am I not missing any essential parts?
Haven't yet implemented the database part and I'm not sure what should I
use? There are plenty of options on google cloud. Also I could go with something simple and just install a DB with http/JSON interface on google cloud compute VM. How do I chose the DB? Given such an app, what would be the best choice?
Please recommend literature/blogs/other sources of info on similar app
architecture for new developers not familiar with it?
This is pretty open ended, but here are some general comments:
Think about how your UI will work. Are you setting up a static app served directly from cloud storage or do you need something rendered on the server? Personally I prefer separating UI from API when I can but you need to be aware of things like search engine optimization. Even if you need to render some content dynamically your site can still be static. Take a look at static site generators like Gatsby. I haven't had to implement a server rendered UI in years and that makes me happy.
API gateway might be fine, but you don't really need it for anything. It might be simpler to start without it and concentrate on what actually matters. If your APIs are being called by an external client you can't trust the calls anyways and any API key you might be using will be exposed. I'd say don't worry about it for a single app. That being said, if you definitely want to use a GW then use one, just be aware that it is mostly a glorified proxy and not some core part of your architecture.
Make sure your API implementations don't store any local state so you can rely on Cloud Run scaling your services up and down. Definitely don't ever store state directly inside your containers. If you need state on the server it needs to be in some external data store.
Use JWTs or an external IDM (that will generate JWTs) for authentication. Keep session data on the client side as much as possible and pass the JWT in every API call to authenticate the caller. If you are implementing login on your own the only APIs you need to expose without tokens are for auth and password recovery, which you can separate into their own service.
Database selection depends on how well you understand your processes, how transactional your services are and your existing skillset. Overall I would use what you are comfortable with, you can probably succeed with a lot of things. Certain NoSQL flavors can seem simple on the surface but if you don't have a clear understanding on the types of queries you need to run they can get tedious to work with. Generally you should stick to relational databases for OLAP style implementations and consider NoSQL for OLTP. Personally I like MongoDB and it is very popular, probably because it sort of sits in the middle of the pack which makes it fit a lot of applications. Using MongoDB also makes you cloud agnostic since it is available on every platform. Using platform specific database flavors can lock you down to a specific vendor.
Whatever you do, don't start installing things on VMs. You can be almost 100% sure you are doing it wrong if this comes up. Remember, the services you consume don't all have to be managed by Google or even run on GCP. You can get MongoDB capacity directly from MongoDB who manage it on your behalf on all of the Big3 cloud vendors.
At least think about the long term, even if you don't necessarily need to have it impact your architecture right now. If you are expecting your app to be up for years try to make it more platform agnostic than less. This might mean sticking away from some really platform specific serverless features that will force you to jump a couple of extra hoops. If you are using Cloud Run you are using containers which already makes your app pretty portable, don't lock it to one platform by using a lot of platform specific features. That being said, don't stay away from them either. You should always go for the low hanging fruit, so don't try to avoid using things like secrets manager etc. If your app has a short lifespan and you need really fast time to market then don't worry about it.
Just my 2c, what you are doing is very generic and can be done in a lot of different ways.
So I am trying to build my first full website, and my idea for this website involves using a publicly available API. The only issue is that most public APIs have a rate limit of a certain amount of requests per hour, and if I am making direct requests from my application to their API, I will probably run out of requests if I have any users whatsoever.
My question is, is there a way to design the website in a way that could not have the outside dependency? What I was thinking was using this public API to build my own API service that my website uses with only the information I need. The only issue I see with this is that the public API is constantly changing, so I will constantly have to run scripts to update my own API with the correct data and would have to redeploy. Is there any clean way of accomplishing this from a design perspective? Thanks
I need to develop a chatbot with these properties:
Platform - Skype for Business On-Premise
Function - Replies to user queries by looking in various knowledgebases (Multiple Platforms - Databases, Web APIs, etc.)
Basic textual conversation to begin with and will gradually evolve to send attachments
No calls/videos, just chat
Will be hosted on an external server with organisation VPN
A simple sip will be created for the chatbot which can be pinged by any user. I should be able to get this through to our IT dept.
Limited time for development
Scalability is an essential requirement but the organisation is fairly new to this, so they might be patient and allow me to make mistakes
My research has led me to these possible approaches:
SfB SDK - I have rejected this approach because it requires the client to be running at all times and doesn't seem to be scalable
UCMA with Application Endpoint - Haven't rejected this approach, but seems like I'll not go ahead with this because creation of Application Platforms seems tedious and requires me to make a lot of SfB server related IT requests
UCMA with User Endpoint - Great affinity towards this. I have experimented Tom Morgan's (thoughtstuff.co.uk) stuff and this seems like something I can start off right away
UCWA with Application Endpoint - Rejected this approach, because UCWA (from my research) appears unsuitable for On-premise and the setup also seems time consuming
UCWA with User Endpoint - Haven't rejected this approach, but I'm not sure if the Web API way is really a good approach for On-premise platform
I'd like to ask how am I doing so far, but that seems too vague
What would you suggest is a good way to achieve this?
Also, can someone be patient enough to reply the drawbacks and advantages of each approach for my use case. I'd like to make an informed decision and not reject any approach, just because of a misunderstood overhead
I have been asking around in my organisation and other circles.
And since I am not receiving any quick responses, I'll keep adding what I have learnt.
This way a person in the dev community will have a log of how I went with this.
UCWA is better suited for S4B online (compared to on premise) and is generally used by people who are comfortable with RESTful and have low familiarity with .NET development
UCMA is apparently THE WAY to go and for any on-premise bot requirements, preferably with an application endpoint.
So for our development, we are starting with UCMA user endpoint so that we can deliver a basic start as a version-one
And meanwhile we shall also get in touch with the IT department and Lync administrators for creation of Application Endpoints
Once we have this the same functionality that we had with the user endpoint will be copied over to the Application Endpoint version
Keep watching this space for further updates
I have been looking for a way to protect my RESTful APIs. This appeared simple, but it seems to not be so simple. First off, I am writing an iOS app connecting to a Play Framework server. None of this has anything to do with Google, Facebook, Twitter, or LinkedIn (shocking I know). Oh, and my current plans do not require custom apps to use my APIs, its just my apps for the time being.
Basic Authentication
What appeared to be simple was a basic user/pass on a /auth method managing a cookie session. That may draw some groans as being too simple or weak but mostly it moved identity to a session key quickly verified. My initial setup was to expire the sessions every day, but that lead to the iOS app forcing a login daily proving to be an annoyance.
OAuth
I posted a question on an iOS board and received a blunt direction towards OAuth. My research of OAuth began but holy sh*t is that complicated and there does not seem to be any server side examples... just plenty of people complaining about how frustrating it is. All the client examples show connecting to Google, Facebook, Twitter, and LinkedIn. Oh Joy!
After watching Eran Hammer's rant about OAuth1 and OAuth2, it seemed fruitless to continue and his OZ idea (which looks really clean) is only at the early stages in node.js.
Question
So, my question to the broad StackOverflow community is... what do you do for securing your REST APIs?
I'd suggest to consider approach used by biggest players i.e. Amazon Web Services or Windows Azure - HMAC. Although it isn't comfortable in implementation, as you can see it's trusted technique.
The general idea is to sign the request's parts (i.e. headers) in the iOS with secret key and try to recalculate it on the Play app to verify that request is authentic and not manipulated. If it won't fail, you can be (almost) sure, that was sent from somebody, who uses valid secret key.
Take a look into Windows' document to get the concept (I think that for common task, you can use the less number of elements used for signing).
There is also other interesting post (based on AWS authentication) which describes whole process even better.
Edit
Of course you should realize that authentication in iOS and securing API requests are different things, even if you'll expire your session every 15 minutes, you can't be sure that somebody won't overhear it and then will be able to send a fake request from the outside. Signing every request should minimize the risk.
On the other hand, if you'll prepare clear rules for signing the requests and will write short doc (which I recommend even for yourself), you can deliver it to the other developer and he'll be able to implement it in (almost) any platform supporting SHA256, so you will have API ready for using from 3-rd party apps - if you'll decide to publish it in the future.
Since Play Framework is in Java, you could use Apache Shiro
I haven't used it yet.. (I am planning to though) So I don't know if it's the best option.
Just do something simple, send the authorization code / password in a custom header over HTTPS .
So the only problem with the Basic Authentication approach was that the user has to login every day? Why not offer the user an option to save his username/password on the device? That way he can choose between security and convenience.