We have a python web app that clients interact with and that web app directly interacts with a database. We now have the need to develop an API that merchants will use to get and post data from/in our database in JSON. Should we build the API as part of the web app, meaning that each request will pass through our python web app and then interact with the database or should it be separated?
Further considerations include scalability and the fact that in the future we’ll probably want to develop a mobile app or other services that will also need to communicate with the database. As such, we considered the possibility to build an API as the only point of interaction with the database.
However, we are deeply in the development of the flask web app and change it would mean an huge delay in our schedule and we just wanted to weight in the advantages and disadvantages of both solutions.
These two schemes summarize the options we are considering:
Option 1:
Option2:
As you said both options have advantages and disadvantages.
Option 1 gives you Separation of Concerns. The logic for interacting with your database is abstracted behind a single service. Changes to the type of database you use or the schema you use only requires code changes to a single service. For example, say your platform has expanded and you now wish to cache calls to your database. If you have an API, Web App, and Mobile App all communicating directly with the database they must all be updated to take advantage of the cache. These changes would likely also have to be orchestrated to be deployed at the same time. In reality this is going to involve downtime: most often you see this referred to as 'scheduled maintenance'.
However, Option 1 breaks the Single Responsibility Principle. A service should do a single thing and do it well. In Option 1 the service is responsible for both being an interface to the database and rendering HTML for the web app. Changes to the Web App require you to redeploy the service for the API even though the two are not connected.
The advantages and disadvantages for Option 2 are mostly just the opposites of the advantages and disadvantages for Option 1. Multiple services sharing a database can lead to inconsistency in the data, tight coupling (especially in deployment), and debugging being more difficult.
A common design pattern (which I'd recommend) is a slight modification of Option 1. An API sits in front of the database. This is the only service that interacts with the database. This setup should improve your scalability. It's easy to deploy duplicate APIs and then load-balance requests between them. Furthermore, caching database lookups or changing database technology entirely is a (relatively) simple task. Your Web App, or any other services you develop in the future, interact with the API instead of the database. Here you can reap the benefits of Single Responsibility. It is worth noting that with this design every request for your Web App will have to go through two services. However, the benefits of the design arguably outweigh a few extra milliseconds of latency.
One last thing: kudos for thinking about scalability this early on. You may take a hit now if your schedule is delayed but I think you'll be better off in the long term.
Related
I have 2 years in the IT industry,i love to read a lot ,but when i go deep in some subjects i see a lot of contradiction in somes articles,forum or terms that are used interchangeable.
I understand the difference between Soap and Rest.
When we want to communicate between backends, we can use either of these 2 approaches, each with its advantages and disadvantages.
Situation :
If i have an application, which can be monolithic or not, where I have a backend and I will only have a front end that consumes it. Usually we create a Rest Api so that our front end can consume it. But we will never think about exposing our backend with Soap.(Lot of reasons)
Questions:
1 -Is it okay if I say that Rest , in addition to allowing us to exchange information between application and application (backend to backend ), is it also useful when exposing services for our front end? And SOAP is only useful for Server - Server communication?
2 -And finally, if I expose a backend only for a front end, it is ok to say that we expose a web service or conceptually we say that it is a backend for frontend ?
Question 1: No the First question is wrong Assumption. We can say that in SOAP, XML is the only means of communication, while in Rest, the accepted means is JSON, while there are other formats like XML, JSON, PDF, HTML etc. and Ofcourse, XML can be converted back from server into UI Language and XML Request deciphered at Server for a Response. So, its not Ok to say that SOAP is only for Server - Server Communication.
2. No, when you have typically exposed backend only for consumption by a Front end, you can typically say that it is a backend for numerous front end client requests. But IMO, Backend for a front end is a monolithic webapplication, both bundled in WAR. so in that sense, any UI Request can request response from the Back end web service. Hope i am able to clear your understanding about web services.
I see that in your question there are actually 4 embedded independent topics. And probably because they are always used in conjunction it is sometimes tough to understand.
I will give a short answer first:
REST and SOAP both can be used for Client-Server and Server-Server integrations. But the choice will be dependent on the questions like where you want server-side UI technology/client-side UI technology or is it a single page application/portal technology, etc.
If you expose a single-backend for a single-frontend it's technically a BFF although the term BFF is used only in the case where you have separate-backends for each type of frontend application. e.g. one for mobile, one for web, one for IoT devices, etc.
The long answer is to clarify the 4 principles. Let me give a try at this by separating the topics into the below four headings:
1. Backend(Business Layer) vs Backend for frontend(BFF)
In classical 3-tier architecture (UI-Business Layer-Database) world, the middle-layer that consists of the business and integration logic is mostly referred to as backend/business layer.
This layer can be separated from the UI/Frontend using multiple different options like APIs(REST/SOAP), RPC, Servlet Technology, etc. The limitation with this 3-tier architecture is that, it is still tightly coupled to the type of users and use-cases are limited to web/browser based. It is not a good choice when you want to reuse the business-layer for both web and mobile as the mobile applications are required to be light-weight by principle.
That's where we lean on to multi-tier architecture with Backend For Frontend(BFF) as a savior. It's just a methodology to segregate the business-layers based on consumers.
2. Monolith vs SOA vs Microservices(Optimized SOA)
In a monolith world all the code components mostly UI and Business Layer sits in a tightly coupled fashion. The simplest example would be a Java Servlet Pages(JSP) application with Java as Business Layer. These are typically server-side UI technologies.
In Service Oriented Architecture(SOA), the usecases revolve around leveraging reusable business layer functions aka services. Here one would have to deal with UI-Server, Inter-Service and Server-Server integration scenarios. It's heavily service dependent, meaning it's like a spider-web of dependent applications.
The Microservices is an extension of SOA, but the approach is to keep a resource in focus instead of services to reduce the spider-web dependencies. Hence, self-sufficient and standalone service-clusters are the base of micro-services architecture.
3. SOAP vs REST Webservices
SOAP stands for Simple Object Access Protocol, typically used by the business-layer to provide user-defined methods/services to manipulate an object. For example look at the names of the services for accessing a book collection
To get a book getABook()
Get the whole list of books listAllBooks()
Find a book by name searchABook(String name)
Update a book's details updateABookDetails()
On the other hand, REST is representational state transfer which transfers the state of a web-resource to the client using underlying existing HTTP methods. So the above services for accessing a book collection would look like
To get a book /book(HTTP GET)
Get the whole list of books /books(HTTP GET)
Find a book by name /book?name={search}(HTTP GET)
Update a book's details /book/{bookId}(HTTP PATCH/PUT)
4. How to make a correct choice of architecture?
Spot the diversity of the application user groups and usages: This will help to understand the platform(web/mobile/IoT/etc), nature of the application and session-management.
Determine the estimated/required throughput: This will help you to understand the scalability requirements.
How frequently and who will be maintaining the application: This will help to gauge the application and technology complexity, deployment cycles, deployment strategy, appetite for downtime, etc.
In conclusion, always follow the divine rule of KISS: Keep it simple, stupid.
1.)
A webapplication is for H2M communication a webservice is for M2M communication through the web. The interface of the service is more standardized, more structured, so machines can easily use it and parse the messages.
I don't think it matters where your service consumer is, it can run in a browser or it can run on the server. As long as it can communicate with the service on a relative safe channel it is ok.
You design a service usually to decouple it from multiple different consumers, so you don't have to deal with the consumer implementations. This makes sense usually when you have potentially unknown consumers programmed usually by 3rd party programmers you don't even know or want to know about. You version the service or at least the messages to stay compatible with old consumers.
If you have only a single consumer developed by you, then it might be too much extra effort to maintain a service with a quasi-standard interface. You can easily change the code of the consumer when you change the interface of the service, so thinking about interface design, standardization, backward compatibility, etc. does not make much sense. Though you can still use REST or SOAP ad hoc without much design. In this case having a RESTish CRUD API without hypermedia is a better choice I think.
2.)
I think both are good, I would say backend in your scenario.
We have different client applications (each is built with a different UI and is targeted to a different sales channels) that are used to capture orders that ultimately need to be processed by our factory.
At first we decided to offer a single "order" microservice that would be used by all these client applications for business rules execution and data storage. This microservice will also trigger our backoffice processes such as client profile update, order analysis, documents storage to our electronic vault, invoicing, communications, etc.
The challenge we are facing is that these client applications are developed by teams that are external to ours (we are a backoffice team only). Each team responsible to develop a client application will be able to offer a different UX to their users (some will allow to save orders in an incomplete state, some wil allow to capture data using a specific worflow, some will use text fields instead of listboxes for some values, etc.).
This diversity of behaviors from client applications is an issue because our microservice logic will become very complex to be able to support all those UI requirements. Moreover, everytime a change will be made to one of the client applications, we will have to modify our microservice which is a case of strong coupling.
My questions are: What would be your best advice to manage this issue? Should we let each application capture the data the way it wants (and persist it if needed in its own database) and let them call our microservice only when an order is complete and compliant to our API contract?
Should we keep our idea of having a single "order" microservice for everyone and force each client application to capture the data the same way?
Any other option?
We want to reduce the duplication of data and business rules in our ecosystem but in the same time we don't want our 'order' microservice to become a mess.
Many thanks for your help.
Moreover, everytime a change will be made to one of the client applications, we will have to modify our microservice which is a case of strong coupling.
This rings alarm bells for me. A change to a UI shouldn't require a change to a backend service. (The exception would be if a new feature were being added to a system and the backend service needed to play a part in supporting that feature, but I wouldn't just call that a change to a client.) As you have said, it's strong coupling, and that's something to be avoided in a microservices environment.
Ideally, your service should provide a generic, programmatic API that is flexible enough to support multiple UIs (or other non-UI applications) without having any knowledge of how the UIs work.
It sounds like you have some decisions to make about what responsibilities your service will and won't take on:
Does it make more sense for your generic orders service to facilitate the storage/retrieval/completion of incomplete orders, or to force its clients to manage this somewhere else?
Does it make more sense for your generic service to provide facilities to assist in the tracking of workflows, or to force the UIs that need that functionality to find it elsewhere?
For clients that want to show list boxes, does it make sense for your generic orders service to provide APIs that aid in populating those boxes?
Should we let each application capture the data the way it wants (and persist it if needed in its own database) and let them call our microservice only when an order is complete and compliant to our API contract?
It really depends on whether you think that's the most sensible way for your service to behave. Something that will play into that will be how similar or dissimilar the needs of each UI is. If 4 out of 5 UIs have the same needs, it could well make sense to support that generically in your service. If every single UI behaves differently to the others, putting that functionality in your generic orders service would amount to storing frontend code somewhere that it doesn't belong.
It seems like there might also be some organisational considerations to these decisions. If the teams using your service are only frontend teams (i.e. without capacity/skills to build backend services), then someone will still have to build the backend functionality they require.
Should we keep our idea of having a single "order" microservice for everyone and force each client application to capture the data the same way?
Yes to the idea of having a single order service with a generic interface for everyone. With regards to forcing client applications to capture data a certain way, your API will only dictate what they need to do to create an order. You can't (and shouldn't) force anything on them about the way they capture the data before calling your service. They can do it however they like. The questions are really around whether your service supports various models of capture or pushes that responsibility back to the frontend.
What would be your best advice to manage this issue?
Collaborate with the teams that will use the service. Gather as much information as you can about the use cases in which they intend to use it. Discover what is common for the majority and choose what of that you will support. Create a semi-formal spec (e.g. well-documented Open API), share it with the client teams, ask for feedback, and iterate. For the parts of the UIs that aren't common across clients, strongly consider telling those teams they'll need to support those elements of their design themselves, especially if they represent significant work on your end.
Let's suppose I was going to design a platform like Airbnb. They have a website as well as native apps on various mobile platforms.
I've been researching app design, and from what I've gathered, the most effective way to do this is to build an API for the back-end, like a REST API using something like node.js, and SQL or mongoDB. The font-end would then be developed natively on each platform which makes calls to the API endpoints to display and update data. This design sounds like it works great for mobile development, but what would be the best way to construct a website that uses the same API?
There are three approaches I can think of:
Use something completely client-side like AangularJS to create a single-page application front end which ties directly into the REST API back-end. This seems OK, but I don't really like the idea of a single-page application and would prefer a more traditional approach
Create a normal web application (in PHP, python, node.js, etc), but rather than tying the data to a typical back end like mySQL, it would basically act as an interface to the REST API. For example when you visit www.example.com/video/3 the server would then call the corresponding REST endpoint (ie api.example.com/video/3/show) and render the HTML for the user. This seems like kind of a messy approach, especially since most web frameworks are designed to work with a SQL backend.
Tie the web interface in directly in with the REST api. For example, The endpoint example.com/video/3/show can return both html or json depending on the HTTP headers. The advantage is that you can share most of your code, however the code would become more complex and you can't decouple your web interface from the API.
What is the best approach for this situation? Do you choose to completely decouple the web application from the REST API? If so, how do you elegantly interface between the two? Or do you choose to merge the REST API and web interface into one code base?
It's a usually a prefered way but one should have a good command of SPA.
Adds a redundant layer from performance perspective. You will basically make twice more requests all the time.
This might work with super simple UI, when it's just a matter of serializing your REST API result into different formats but I believe you want rich UI and going this way will be a nightmare from both implementation and maintainance perspective.
SUGGESTED SOLUTION:
Extract your core logic. Put it into a separate project/assembly and reuse it both in your REST API and UI. This way you will be able to reuse the business logic which is the same both for UI and REST API and keep the representation stuff separately which is different for UI and REST API.
Hope it helps!
Both the first and the second option seem reasonable to me, in the sense that there are certain advantages in decoupling the backend API from the clients (including your web site). For example, you could have dedicated teams per each project, if there's a bug on the web/api you'd only have to release that project, and not both.
Say you're going public with your API. If you're releasing a version that breaks backwards compatibility, with a decoupled web app you'd be able to detect that earlier (say staging environment, given you're developing both in-house). However, if they were tightly coupled they'd probably work just fine, and you'll find out you've broken the other clients only once you release in production.
I would say the first option is preferable one as a generic approach. SPA first load delay problem can be resolved with server side rendering technique.
For second option you will have to face scalability, cpu performance, user session(not on rest api of course because should be stateless), caching issues both on your rest api services and normal website node instances (maybe caching not in all the cases). In most of the cases this intermediate backend layer is just unnecessary, there is not any technical limitation for doing all the stuff in the recent versions of browsers.
The third option violates the separation of concerns, in your case presentational from data models/bussines logic.
I'm looking to create an ecommerce website, however I have two possible ideas for the structure of the application.
The first idea would utilize MVC 4 and have an EntityFramework layer to communicate to and from the database. This is fairly straight forward but I don't like the idea of having ALL of the processing happening on the front end of the website.
My second idea was to have a web service that sits in between the MVC application and EntityFramework. This would be responsible for most of the website processing, and would expose only necessary information to the front end of the website to minimize the amount of processing on the front end of the website.
Would this be a better solution, or is there another solution which would be better?
You should make a difference between logical and physical layering.
To avoid premature optimization, you can have all the logical layers you want inside the web application itself (the traditional presentation / business / data approach or another pattern, for example onion architecture).
Later, if needed, you can always optimize and move parts into their own physical layer, and in that case you would have a web service or web API in between. But it all depends on your requirements (multiple consumers of the services?) and environment. You could just as easy scale the web site itself, for example.
So in your case, think about what added value this additional abstraction would have, and if it's worth the extra complexity.
Running all traffic an EF access through a web service layer internall has the following benefits:
It runs up your budget because you do add a lot of slow processing overead (string parsing) into a high throughput low level interface.
It absolutely makes sure you have a low performance because of the added latency.
It totally makes sure you spend a lot of more working hours to get a solution because you need to implement and test a web servie for any small database access.
THere is no reason to run anything internally through a web service interface. It is architectural cancer - out of control complexity growth. Web services are a user interface layer and a trust boudary and should be the end of an application. If you make a web commerce website (and most likely that is quite trivial and small compared to something like amazon) there is no need to get involved into elements that make things a lot more compelx without any gain.
DEPENDS!
Building an eCommerce web site from scratch I would use a client mvc web site with a responsive pattern in mind, so it will fit to all devices , and use EF WebApi as the data layer to generate XML/JSON.
I am working on a experimental website (which is accessible through web browser) that will act as a front-end to a restful interface (a sub-system). The website will serve as an interface between a user and the restful interface, as it will make http requests to the restful interface for almost all database operations. Authentication will probably be done using openid and authorization for the database operations will be done via oAuth.
Just out of curiousity, is this a feasible solution or I should develop two systems that accesses the database in parallel (i.e. the website has its own data access logic, and the restful interface has another data access logic)? And what are the pros/cons if I insist on doing it this way (it is just an experiment project for me to learn things like how OpenID and oAuth work in real life anyway) besides there will be more database queries and http requests generated for each transaction?
Your concept sounds quite feasible. I'd say that you'll get some fairly good wins out of this approach. For starters you'll get a large degree of code reuse since you'll be able to put other front ends on top of the RESTful service. Additionally, you'll be able to unit test this architecture with relative ease. Finally, you'll be able to give 3rd party developers access to the same API that you use (subject possibly to some restrictions) which will be a huge win when it comes to attracting customers and developers to your platform.
On the down side, depending on how you structure your back end you could run into the standard problem of granularity. Too much granularity and you'll end up making lots of connections for very little amounts of data. Too little and you'll get more data than you need in some cases. As for security, you should be able to lock down the back end so that requests can only be made under certain conditions: requests contain an authorization token, api key, etc.
Sounds good, but I'd recommend that you do this only if you plan to open up the restful API for other UI's to use, or simply to learn something cool. Support HTML XML and JSON for the interface.
Otherwise, use a great MVC framework instead (asp.net MVC, rails, cakephp). You'll end up with the same basic result but you'll be "strongerly" typed to the database.
with a modern javascript library your approach is quite straightforward.
ExtJS now has always had Ajax support, but it is now able to do this via a REST interface.
So, your ExtJS user interface components populate receive a URL. They populate themselves via a GET to the URL, and store update via POST to the URL.
This has worked really well on a project I'm currently working on. By applying RESTful principles there's an almost clinical separation between the front & backends - meaning it would be trivial undertaking to replace other. Plus, the API barely needs documenting, since it's an implementation of an existing mature standard.
Good luck,
Ian
woow! A question from 2009! And it's funny to read the answers. Many people seem to disagree with the web services approach and JS front end - which has nowadays become kind of standard, known as Single Page Applications..
I think the general approach you outline is quite feasible -- the main pro is flexibility, the main con is that it won't protect clueless users against their own ((expletive deleted)) abuses. As most users are likely to be clueless, this isn't feasible for mass consumption... but, it's fine for really leet users!-)
So to clarify, you want to have your web UI call into your web service, which in turn calls into the database?
This is exactly the path I took for a recent project and I think it was a mistake because you end up creating a lot of extra work. Here's why:
When you are coding your web service, you will create a library to wrap database calls, which is typical. No problem there.
But then when you code your web UI, you will end up creating another library to wrap calls into the REST interface... because otherwise it will get cumbersome making all the raw HTTP calls.
So you essentially created 2 data access libraries, one to wrap DB and the other to wrap the Web service calls. This basically doubles the amount of work you do, because for every operation on a resource, you will end up implementing in both libraries. This gets tiring real fast.
The simpler alternative is to create a single library that wraps access to the database, as before, then use that library from BOTH the web UI and web service.
This is assuming that your web UI and web service reside on the same network and both have direct access to the backend database server (which was the case for me). In this setup having both go directly to the database is also a lot more efficient then having the UI go through the web service.