I'm about to launch a SPA app that I built in VueJS but I'm worried about how exposed my API is. Anyone with access to the app could easily just download all of the data I have running through my RESTful API. The API is protected with oAuth, but with signups being free, essentially anyone can literally have all of the data my app provides.
I recently started looking into an SSR framework like Nuxt thinking this would be the solution to my problem. If I make all my RESTful calls server side, I wont have to expose my API to the world and can return HTML partials while still having the niceness of a SPA app. I know even HTML can be scraped, but I'd rather not just hand my whole database to someone in a nice JSON payload.
After playing around with Nuxt, I'm not sure this is what I'm looking for since it only does the initial REST call on the server, and all subsequent ones happen from the client. What are my options when it comes to not exposing my REST API while still having the benefits of a SPA app?
EDIT:
Some backstory if it will help: I have about 30,000 records in a database that are available through a public-facing ASP.NET MVC Core API. I limited the amount of results that can come back with a page request to 60. My VueJS/Nuxt app uses this API to display the data to the user. Since the calls happen from the client-side, my API is open to anyone who wants to hit it. That means some young, enterprising kid could write a small app that loops 500 times and will have the entirety of my database in less than a couple minutes. I know it's impossible to stop scrapers, but I'd like to make this a little harder to accomplish.
Related
I've done almost all of my web projects using Java(Spring MVC + Thymeleaf) and it's MVC technologies. Recently I've heared about REST and started learning some stuff about it. I realized that it is one of the coolest things I've ever seen in my whole life!(I'm just joking around, it's definitely not)
All we need is just to parse data to json type and then return it to the frontend. And in frontend we no longer need to use Model and it's objects. Frontend can get all required data in nice-to-work-with json type using one single GET request!
We don't need to use some weird Thymeleaf constructions to handle errors or to iterate through the list in our template! We can handle all events and process all data using javascript and it's frameworks. It is much more powerful.
Does there exist something I missed? When to use Model? When to use json-type data?
These are two (somewhat overlapping) approaches to frontend development: generating pages on the backend and on the frontend.
Using a backend model and Thymeleaf templates (or any other HTML templates), you generate your web page on the server side. This means the following benefits
you can write frontend and backend logic in one language (Java);
you can enforce data and security constraints in one place - on the backend, using Java code or configuration;
in many cases, it's faster for the user to get their first page, since the rendering happens on the server, and the user gets an already rendered page;
But this approach has the following drawbacks:
the server has to render the pages, which means more load on the server;
most of the modern web sites and applications use JavaScript anyway, so you'll have to write at least some JavaScript;
server-generated pages don't even come close to what's possible to render in the browser.
Providing a REST API for a JavaScript frontend, you have the following benefits:
you unload your server, since most of the UI related work happens on the user's device;
you can achieve much more with modern frontend frameworks;
navigating on an already rendered UI is faster, since you don't need for the server to render every page, you only need to get a relatively small JSON response and then update the page dynamically.
But this approach has the following tradeoffs:
the user generally waits longer to see their first page, as the browser needs to download a lot of scripts and then spend time on dynamic rendering of the page;
in a large application, you need to either have a full-stack developer team, or two teams working on frontend and backend separately;
you have to write your UI logic in JavaScript. Make of it what you will :)
you have to sometimes duplicate the constraints both on the frontend and on the backend: e.g., if a user can't edit a field, you have to both show it as read-only on the frontend, and add validation in your REST service, since the user may try to access your REST API directly and bypass the frontend validation;
you have to be more aware of securing your REST API in general.
I've got a public Java web app created with Spring Boot. I'm ready to release it to production and wanted to get some tips on making sure that it is as secure as possible.
It is basically a search engine that uses Apache Lucene in the backed and a lot of javascript in the front end. I am planning on deploying it to Amazon Web Services, while using my local machine as a backup/test enviroment.
Users can search and browse data. The frontend calls REST endpoints using Javascripts XMLHttpRequest to query the backend for content and then displays it to the user.
The app is completely public and there is no user authentication as of yet.
The app also persists user requests to a database for tracking purposes.
Here's what I've done so far to secure it:
Make sure that the REST endpoints fully verify that the parameters given to them in the requests are valid.
What I plan on doing:
Using HTTPS.
Verifying that any put request or url requests have a reasonable size.
What I am considering adding:
Limiting the number of requests a user can make in a given time period. (not sure if Spring-Boot already has a facility to do this, or I should implement this myself)
Use some kind of API key scheme to make sure that my endpoints are only accessed by my front end. (not sure if this is effective and don't yet know how to do this).
Is there anything else that I should consider doing? Do the things I listed above make sense?
Would greatly appreciate any tips on this.
Let's suppose I was going to design a platform like Airbnb. They have a website as well as native apps on various mobile platforms.
I've been researching app design, and from what I've gathered, the most effective way to do this is to build an API for the back-end, like a REST API using something like node.js, and SQL or mongoDB. The font-end would then be developed natively on each platform which makes calls to the API endpoints to display and update data. This design sounds like it works great for mobile development, but what would be the best way to construct a website that uses the same API?
There are three approaches I can think of:
Use something completely client-side like AangularJS to create a single-page application front end which ties directly into the REST API back-end. This seems OK, but I don't really like the idea of a single-page application and would prefer a more traditional approach
Create a normal web application (in PHP, python, node.js, etc), but rather than tying the data to a typical back end like mySQL, it would basically act as an interface to the REST API. For example when you visit www.example.com/video/3 the server would then call the corresponding REST endpoint (ie api.example.com/video/3/show) and render the HTML for the user. This seems like kind of a messy approach, especially since most web frameworks are designed to work with a SQL backend.
Tie the web interface in directly in with the REST api. For example, The endpoint example.com/video/3/show can return both html or json depending on the HTTP headers. The advantage is that you can share most of your code, however the code would become more complex and you can't decouple your web interface from the API.
What is the best approach for this situation? Do you choose to completely decouple the web application from the REST API? If so, how do you elegantly interface between the two? Or do you choose to merge the REST API and web interface into one code base?
It's a usually a prefered way but one should have a good command of SPA.
Adds a redundant layer from performance perspective. You will basically make twice more requests all the time.
This might work with super simple UI, when it's just a matter of serializing your REST API result into different formats but I believe you want rich UI and going this way will be a nightmare from both implementation and maintainance perspective.
SUGGESTED SOLUTION:
Extract your core logic. Put it into a separate project/assembly and reuse it both in your REST API and UI. This way you will be able to reuse the business logic which is the same both for UI and REST API and keep the representation stuff separately which is different for UI and REST API.
Hope it helps!
Both the first and the second option seem reasonable to me, in the sense that there are certain advantages in decoupling the backend API from the clients (including your web site). For example, you could have dedicated teams per each project, if there's a bug on the web/api you'd only have to release that project, and not both.
Say you're going public with your API. If you're releasing a version that breaks backwards compatibility, with a decoupled web app you'd be able to detect that earlier (say staging environment, given you're developing both in-house). However, if they were tightly coupled they'd probably work just fine, and you'll find out you've broken the other clients only once you release in production.
I would say the first option is preferable one as a generic approach. SPA first load delay problem can be resolved with server side rendering technique.
For second option you will have to face scalability, cpu performance, user session(not on rest api of course because should be stateless), caching issues both on your rest api services and normal website node instances (maybe caching not in all the cases). In most of the cases this intermediate backend layer is just unnecessary, there is not any technical limitation for doing all the stuff in the recent versions of browsers.
The third option violates the separation of concerns, in your case presentational from data models/bussines logic.
I am in the process of scoping the development of an iPhone app for a client. Among other things, the app will allow users to browse through and place orders on specific (tangible) products.
The client has a website that currently does a similar thing and due to their limited budget and the fact that the website runs on a third-party proprietary platform which they have no control over, we are investigating possible alternatives to building a web service.
On the website, user registration and authentication, as well as order placing is done through POST requests via secure HTTP. The response is always a formatted HTML page which will contain strings indicating whether the request was successful or not, and if there was an error, what the error is etc.
So provided I can replicate the POST requests on the phone, and parse the HTML responses to read the results of each request, do you think this is an acceptable alternative to building a web service to handle this?
Apart from the possibility of pages changing (which we can manage) and the fact that I will probably have to download and parse a relatively large HTML response, are there any other drawbacks to this solution and is there anything else that I might be missing?
Many thanks in advance for your thoughts.
Cheers,
Rog
You could create an intermediary server that will communicate with the client server, and on it expose some REST web services with json (small overhead and easy to handle) responses that will be consumed by the iPhone app.
So, you're going to parse HTML and formulate POSTs off a third-party server, and pray that they don't even so much as rename a form field.
Your question is in two parts:
Do I think that a miracle is an acceptable solution? I don't.
Do I think that aside from the fact a miracle is required, are there any other drawbacks? None that I can think of.
You didn't ask, but this is a terrible course of action. Two suggestions.
I spy an assumption that the providers of the third-party platform aren't interested in enabling third-party applications by providing an API. They have a very good business reason for this, which is that it promotes platform lock-in. Reach out to their support department and have a talk with them.
You have to sell the client on building an intermediary web service. To at least try to mitigate the damage that changes on this third-party platform can do to your app, I recommend that you build and operate a proxy that receives requests from your applications, and proxies them over to the third-party platform. You should build into this client-server protocol a means for returning "we are in maintenance mode, go away" messages to apps, for that inevitable day when the third-party server changes something that breaks your app (they swapped the billing and shipping address pages, for instance) and you have to rush through an update through Apple to deal with it.
The proxy could be written in something more flexible and easy to bash stuff out in, such as PHP, Python, Perl, or Ruby. It could be hosted at Amazon in a micro instance.
p.s. This question is inappropriately tagged objective C.
HTML is the worst because of parsing (1-2secs per page), memory, and changes, but you already know that. Check in advance that ALL the data you need is exposed on the HTML.
If you use an intermediary server you are moving work elsewhere and you have another server to maintain. I would only do that if memory is an issue. Check How To Choose The Best XML Parser for Your iPhone Project for memory/performance/xpath support. libxml2 is a good option, but it depends on your needs. And maybe you'll want to check ASIHTTPRequest features before using the SDK.
I think utilising the web language of JSON would contribute to the diminishing of the parsing time. By building a REST service that, when sent a GET request, returns the correct information for easy sorting, you could then display the output a lot faster than that of parsing straight HTML.
I prefer JSON over XML, but everyone has their personal preference. You should look at a few very good libraries that are built specifically for parsing purposes of both XML and JSON.
For XML I recommend using the inbuilt libxml parser. Albeit, this can sometimes deem very difficult to use. A simple Google search will bring up a heap of results that relate specifically to what parser should be used depending on what task is to be completed.
As for a JSON parser, I recommend SBJSON. I am currently using it one of the biggest projects I have undertaken and it is definitely working perfectly for my use.
If you need a good way to connect to a RESTful web service, you should try LRResty.
Don't go for a parsing solution on the iPhone for 4 reasons:
Server can change their design and break your application (AppStore submition is long) + They can also detect that the request are sent from an application based on user agent which you have to update the application to change it.
Some of the requests might be made thru Javascript so you not only have to parse (X)HTML but also Javascript request (which can be in the form of XMLHttpRequest, but don't have to)
Long term evolution of the mobile market : maybe your client want (or will want) an application for android, Blackberry, Bada OS (Samsung), Symbian (Nokia/ OVIStore), Java Mobile or Windows Phone 7?
Of course network traffic, Memory and CPU needed to parse HTML (look the time it takes to the browser to do it?)
Regarding the traffic, if the application will not have a huge traffic you can home-host your proxy. Or you can find some provider to host it for you. I guess you won't need more than a couple of Megabytes of storage but maybe traffic. For less than 100€/year you can find some with unlimited traffic (like OVH Pro plan or Infomaniak). But if you want to go Java have a look at Google App Engine : you pay only if your traffic is important and if your application generate many CPU Cycles. If not : you don't have to pay. And it's hosted on Google server : reliable.
If the client is open, you could consider the paypal API.
I am working on a experimental website (which is accessible through web browser) that will act as a front-end to a restful interface (a sub-system). The website will serve as an interface between a user and the restful interface, as it will make http requests to the restful interface for almost all database operations. Authentication will probably be done using openid and authorization for the database operations will be done via oAuth.
Just out of curiousity, is this a feasible solution or I should develop two systems that accesses the database in parallel (i.e. the website has its own data access logic, and the restful interface has another data access logic)? And what are the pros/cons if I insist on doing it this way (it is just an experiment project for me to learn things like how OpenID and oAuth work in real life anyway) besides there will be more database queries and http requests generated for each transaction?
Your concept sounds quite feasible. I'd say that you'll get some fairly good wins out of this approach. For starters you'll get a large degree of code reuse since you'll be able to put other front ends on top of the RESTful service. Additionally, you'll be able to unit test this architecture with relative ease. Finally, you'll be able to give 3rd party developers access to the same API that you use (subject possibly to some restrictions) which will be a huge win when it comes to attracting customers and developers to your platform.
On the down side, depending on how you structure your back end you could run into the standard problem of granularity. Too much granularity and you'll end up making lots of connections for very little amounts of data. Too little and you'll get more data than you need in some cases. As for security, you should be able to lock down the back end so that requests can only be made under certain conditions: requests contain an authorization token, api key, etc.
Sounds good, but I'd recommend that you do this only if you plan to open up the restful API for other UI's to use, or simply to learn something cool. Support HTML XML and JSON for the interface.
Otherwise, use a great MVC framework instead (asp.net MVC, rails, cakephp). You'll end up with the same basic result but you'll be "strongerly" typed to the database.
with a modern javascript library your approach is quite straightforward.
ExtJS now has always had Ajax support, but it is now able to do this via a REST interface.
So, your ExtJS user interface components populate receive a URL. They populate themselves via a GET to the URL, and store update via POST to the URL.
This has worked really well on a project I'm currently working on. By applying RESTful principles there's an almost clinical separation between the front & backends - meaning it would be trivial undertaking to replace other. Plus, the API barely needs documenting, since it's an implementation of an existing mature standard.
Good luck,
Ian
woow! A question from 2009! And it's funny to read the answers. Many people seem to disagree with the web services approach and JS front end - which has nowadays become kind of standard, known as Single Page Applications..
I think the general approach you outline is quite feasible -- the main pro is flexibility, the main con is that it won't protect clueless users against their own ((expletive deleted)) abuses. As most users are likely to be clueless, this isn't feasible for mass consumption... but, it's fine for really leet users!-)
So to clarify, you want to have your web UI call into your web service, which in turn calls into the database?
This is exactly the path I took for a recent project and I think it was a mistake because you end up creating a lot of extra work. Here's why:
When you are coding your web service, you will create a library to wrap database calls, which is typical. No problem there.
But then when you code your web UI, you will end up creating another library to wrap calls into the REST interface... because otherwise it will get cumbersome making all the raw HTTP calls.
So you essentially created 2 data access libraries, one to wrap DB and the other to wrap the Web service calls. This basically doubles the amount of work you do, because for every operation on a resource, you will end up implementing in both libraries. This gets tiring real fast.
The simpler alternative is to create a single library that wraps access to the database, as before, then use that library from BOTH the web UI and web service.
This is assuming that your web UI and web service reside on the same network and both have direct access to the backend database server (which was the case for me). In this setup having both go directly to the database is also a lot more efficient then having the UI go through the web service.