I work on an existing Seam 2.2.0 + JPA (Hibernate 3.3.1) application that needs to be converted to a 'single database per client' environment where each database schema is the same. The application runs on Glassfish, uses IceFaces, and has several pages that utilize Conversations. It also uses a single EJB for authentication. Unfortunately, the decision to split clients off into their own databases is outside of my control.
As a proof of concept, I have already made the application aware of multiple databases by moving the management of EntityManagerFactory(ies) and DataSource(s) into the application using Spring JPA abstractions, resource local transactions, and ThreadLocal for context information. For example, each time a user logs in a new EntityManagerFactory is initialized using a new DataSource that communicates with their database if it has not already been initialized. This is working well in a test environment with a handful of databases.
My question is, will this approach scale to hundreds of databases? I expect to add application servers to a load balancer to handle additional load, but will the overhead of the Hibernate/JPA first-level cache and/or Seam context management (a.k.a., memory consumption) require significantly more servers to scale compared to a typical load balanced application? If so, can this be mitigated by allocating servers with lots of RAM and/or a large distributed cache?
Any insight would be greatly appreciated.
I worked on an application with this approach and I what I can point out is:
Datasource and EntityManagerFactory management is the hard part. However it seems you did this already in the test environment. Double check if you did things right in respect of the Seam Managed Entity Manager.
Your application won't scale well on hundreds of databases because you have a linear increase of memory consumption for each database. In fact, for each database you are going to have different instances of EntityManagerFactory (Hibernate SessionFactory) each requiring a considerable amount of Ram.
Beware of possible issues if you configure Hibernate second level cache. Since all SessionFactories are created from the same data model the cache region names may collide. I used hibernate.cache.region_prefix configuration parameter to make these names uniques among the various instances using the database id as the cache prefix.
Related
we have a big monolithic project that use EF to manipulates a database.
Now I want to start to create different services or micro-services that use a part of the big-model.
As first step read-only data may be good, but I have to reference all big-model or there is some different way ?
Any help are wellcome.
You can start with all the microservices sharing the same large DbContext, and maintaining the single shared database. This allows you to build new services without complicated refactoring of the existing application or database schema.
We have a DB2 database which is used by legacy applications that we are in the process of decommissioning and we have an Oracle database that we are developing new applications for. In order to maintain compatibility with legacy applications until they are completely decommissioned and keep data in sync, we are using Atomikos for two phase commits. This however is resulting in a lot of duplicated entities and repositories and thus technical debt, because the same entities and repositories cannot be used by the same entity managers so we have to duplicate them and put them under different packages for the entity scanning.
In most cases we want to read data from the legacy database and persist to both, but ideally this would be configurable.
Any help on this would be greatly appreciated.
I am using eclipseLink JPA and i am using same db for two different EJB applications deployed in two different servers. I am getting concurrency issues due to JPA cache if other application is doing some modification on the same object. Is there any way i can manage this concurrency issues. Using native query for db operations is not an option. I am using oracle db and glassfish server. is there any solution available for this.
Let me explain why you have issues:
Imagine you have cached data for object. So the first application will work with that cached data and there are no need to this application to go to database (as it already have the value in memory). It can upgrade that cached data if you will perform another modifying query on the same entityManager. It in no way can know that the other application changed state of your data.
So isolation level won't help as there will be no read query from first application as it already have cached value. And as far as entityManager know is only the operations from that same manager. It have no way to know about another entityManager on different application.
The solution is to not to cache. Or you can configure external cache as your second level cache which you can evict from both applications (like hazelcast cache for example).
Imagine we have 2 services: Product and Order. Based on my understanding of SOA, I know that each service can have its own data store (a separate database, or a group of tables in the same database). But no Service is allowed to touch the data store of another Service directly.
Now, imagine we have stored the product and order data independently inside Product and Order Services. In the Order Service, we can identify products by their ID.
My question is: With this architecture, how can I display the list of orders and product details on the "same" page?
My understanding is that I should get the list of OrderItems from OrderService. Each OrderItem has a ProductID. Now, if I make a separate call to ProductService to retrieve the details about each Product, that would be very inefficient.
How would you approach this problem?
Cheers,
Mosh
I did some research and found 2 different solutions for this.
1- Services can cache data of other Services locally. But this requires a pub/sub mechanism, so any changes in the source of the data should be published so the subscribing Services can update their local cache. This is costly to implement, but is the fastest solution because the Service has the required data locally. It also increases the availability of a Service by preventing it from being dependent to the data of other Services. In other words, if the other Service is not available, it can still do its job by its cache data.
2- Alternatively, a Service can query a "list" of objects from another Service by supplying a list of identifiers. This prevents a separate call to be made to the target service to get the details of a given object. This is easier to implement but performance-wise, is not as fast as solution 1. Also, in case the target Service is not available, the source Service cannot do its job.
Hope this helps others who have come across this issue.
Mosh
DB integration (which is really what you are talking about when two services share a table in a DB) is wrong at so many levels!
It completely breaks some of the major principles of software engineering
loose coupling,
encapsulation
separation of concerns
A service should be (to earn that name) completely independent, namely:
it must not rely on others to ensure the consistency and coherence of its data
it must not rely on others to guaranty the security of its data
it must not depend on external implementations (only interfaces)
Two services that share data at the DB level are unable to guarantee any of the former.
The fact that you "control" both services is completely irrelevant. Today you control... tomorrow you might want to outsource or replace one of the services. That should be as simple as ensuring the proper interfaces are in place.
Imagine both services that share a table with some field (varchar) in it. Now one service needs to change that field to numeric... bang the other service stops functioning - loose coupling goes down the drain.
Most of the time the trick lies in properly defining the service scope and clearly stipulating what a service do and what it doesn't do. You should also avoid turning everything into a service. Set your service granularity to high and services will start popping everywhere and integration headaches will escalate.
That being said, there are some situations where data integration between services poses some challenges. The main premiss do, should always be - data can belong only to one service. Data is intrinsically tied to business logic that affects data consistency and coherence and as such there should never be more than one service controlling any given data.
Another approach would be to have some sort of data source that lives outside of the SOA services. This data source could be considered your cache of the data, your operational data source or even a data warehouse. Extraction packages can export the data from the services (and/or some sort of real time mechanism). You can query this data source how you want.
The advantage of this approach is that the SOA black box is maintained and you can swap out a service knowing how you have coupled it.
Disadvantage is the added complexity and maintenance overhead.
SOA is just a buzz-phrase for deploying components behind web services. How many data stores you have is entirely up to you. In some cases it makes sense to have partitioned data behind individual components, and in other cases all the data lives behind one service, and in other cases many components that expose service interfaces connect to the same database via the database's connection protocol. Approach the problem by approaching the problem, not be imposing artificial constraints.
I don't think there is any principle in SOA that services should have separate data store. In general it is actually impractical. Yes you can have product and order service and the client can do the join using web service call as you said and this may be acceptable in some scenario. But that doesn't mean that you cannot have a specific service for a client if you already know the client's behaviour and performance requirements.
What I mean is that you should have a search service that returns orders and products with the join done in database. This is practical and would solve your business problem.
It is unfortunate to see this whole discussion being deteriorating in a "can I use a shared database or not in SOA" statement quest, which is totally irrelevant and does not help answering the original question at all.
More then often in a real world situation the data is already stored in different systems to start with. Customer data for example is coming from CRM, product data from SAP, contract data from yet another different source.
It is not a quest for getting this data technically together, rather than an understanding there is only one source of the data. To differently put it, there is only one owner of the data within your enterprise, who is solely responsible for maintaining it and ensuring the correct data quality.
Storing data locally for performance reasons means replicating data, which is more than often a dangerous venture, unless you have a solid caching strategy in place. I think Mosh has given some sensible answers when faced with an existing application landscape.
I have developed a number of departmental client-server applications, and am now ready to begin working on moving one of these applications to a SaaS model. I have done some basic web development, but I'm a newbie when it comes to SaaS architectures.
One of the first questions that comes to mind as I try to design the architecture is the question of single vs. multi tenancy. The pros and cons of each vary significantly depending on the type of application and scale required, so I'd like to describe my application and scale needs below, and hope others can comment on how I should get started with the architecture.
The client-server application currently consists of a Firebird database and a Windows application. The database contains about 20 tables containing a few thousand records in 4 primary tables, and a few hundred records in various lookup and related tables. Although the number of records is small, the size can get large, as the database can contain large BLOBS. Each customer sets up their own database and has a handful of users within the organization connected to it. When I update the db schema, a new windows application is released, and it checks the db schema and then applies the updates as needed.
For the SaaS application, I am designing for 100's (not 1000's or millions) of new customers per year. My first thought was to go with a multi tenancy model to make updates easy (shut down apply the updates to one database, and then start up). On the other hand, a single tenancy model would provide a means to roll updates out to a group of customers at a time, and spread the risk of data corruption - i.e. if something goes wrong with a database, it will impact one customer instead of all customers. With this idea, I was thinking of having a single web front-end which would connect to a single customer database upon login. Thus, when a new customer creates an account, a new database would be created (each customer would have their own db with multiple users as needed for the customer).
In this model, a db update would require either a process to go through each db to apply schema changes, or a trigger upon logging in to initiate a schema update similar to the client-server model currently in use.
Can anyone point me to information for similar applications which have been ported from client-server to SaaS? Or provide any pointers to consider? Basically I'm looking for architecture examples of taking a departmental application and making it available as a self service website for multiple customers. Thanks for any suggestions, resources, etc.
Good questions.
One thing that comes to mind is that if you have multiple databases which you roll out in a staged manner to reduce the likelihood of breaking all of your customers, you will have to address the issue of what to do if the db structure changes. You will either have to be very rigorous with respect to maintaining backward compatibility, or else deploy separate versions of your code base and somehow manage which tenants are associated with which databases.
We are providing our application using a SaaS model as well.
It was, initially a Windows app which worked similar to your multiple database proposal. Upon login, the win app would authenticate against a single "licensee" database which would then respond with connection information for a database specific to that licensee. The nice thing about this was that it provided 1) physical separation of licensee data, which our customers liked and 2) enabled us to physically locate the database on a server geographically closer to the users which both improves performance and avoids some potentially tricky legal and regulatory issues with respect to providing data across country boundaries.
Of course, since the app was a thick client app, we could get away with making database changes and pushing them out to one licensee at a time. When we were ready to upgrade, we could push out an updated thick client in conjunction with the new database - thereby ensuring that the codebase was a match with the database. As long as the common "licensee" authentication database stayed consistent, this worked fairly well.
On the other hand, though, this solution brought with it all of the problems of maintaining and managing a thick client approach which finally lead us down the thing client, browser-based approach.
In our new model, everything is in a single database. When we have updates, we push both the code and the db out at the same time. This solves the problem of keeping the code base consistent with the database structure. However, we are now confronted with the issues mentioned in #s 1 and 2, above, which we have yet to resolve.
I hope this provides some food for thought for you.
I, too, am interested in this question.
Thanks for the post.
-S