I have an rest api which deletes from multiple database table. Each table is accessed by different repositories, so in code it deletes by calling each repo in single method which is annotated by #Transactional.
repositoryA.deleteById(id);
repositoryB.deleteById(id);
repositoryC.deleteById(id);
Is it good approach and is there any better way to do it ?
Related
I´m getting PK Violation Exception when using EF Core 2.1 DbContext in an Azure QueueTrigger function. Guess is due to the nature of DbContext not being thread-safe, and the Azure Function running different instances in parallel. I have read quite a few, but I can´t find a good approach to solve this.
Here is my scenario (producer-consumer pattern):
I have a Scheduled Azure Function that is calling an API to get Projects from different external systems. To get all the required info for a project, I need to run different Queries to other external services, so I´m decoupling this to another Azure function, so the Scheduled function just queues a message per Project, as “Sync Project ID 101”.
Another QueueTrigger Function fires every time a message is queued, so, it means different instances running in parallel. This function must gather all the data of a specific Project, and that means more calls to other external services / APIs, to (some kind of) aggregate all the info about a Project. IMHO it´s good to do it that way, as I can process multiple Projects in parallel, and I can scale the Function if I need it.
Once I have all this Project info, I want to persist it in a SQL DB using EF Core (and here comes the issue)
Project data includes Users in the Project, and each user have a specific GUID as PK (coming from the external system). That means I can have repeated Users IDs in different Function instances, and here is the problem, as when I try to persist User info in a SQL Table, I can get PK Duplication exception, as multiple Function instances can try to Insert the same User at the same time (when the instance A check if user exists, it gets False, but another instance B is actually adding this User, so when instance A tries the Insert, it fails).
Guess I can lock DbContext somehow, but not sure if is good, as I also have a website doing Queries to the SQL DB (read-only queries for now, but could be updates in future too).
Another idea could be to send the entire Project info to another Queue / Blob file, and have another function in Singleton mode that Insert the data into SQL.
I´ve created this project simplifying my scenario, but enough to reproduce the issue and understand the problem.
https://github.com/luismanez/queuetrigger-efcore-multithreading
Any other ideas or recommended approaches? (open to change the architecture if find something better)
Many thanks!
A "more easy" way could be to do some kind of upsert in the database. There is a sample of how to do that with EF Core: https://www.flexlabs.org/2018/02/adding-upsert-support-for-entity-framework-core
I am working on a POC (to move part of functionality from relational DB to cloud datastore). I have few questions:
I would need to refresh few "kind" every night as the data comes up
from a different data source (via flat files). I read about it, and
understood that there is not TRUNCATE kind of functionality in
datastore. I believe, only option is to retrieve the keys from the
"kind" in a loop and delete entity by entity. And use import functionality to load the new set of data. Is there any better
option?
Assume I have a kind called department, and a kind called
store. Now, I need a kind called dept-store. So for this parent
nodes are department and store. Is there a way to enforce this kind
of relationship? From the documentation I see that there can only be
one parent.
If i have a child entity in kind1 whose parent is
present in kind2, and they are linked together, is there a way to
query all the properties present in kind1 and kind2 together? From
relational DB perspective, it is like equi-join with "SELECT *". I
am looking for an equivalent functionality in datastore.
In order to answer your questions:
There is two ways to delete multiple entities. First, you can use Cloud Dataflow to delete entities in Bulk [1]. Second, once keys are retrieved you can make a batch delete operation by passing the keys to Datastore delete function, you have the usage example here [2]. In order to retrieve the keys you can run keys-only query [3].
In Datastore an entitiy can have only one parent but can have multiple children. But for your use case you may try to have a third kind, dept-store, and assign its properties as the keys of the entities from the department and the store kinds. This solution might need a good understanding of your neeeds for implementation, as Datastore by nature is Non-relational database.
You can lookup multiple entities providing the keys retrieved from kind1 and kind2 with batch operations [2].
Currently, I'm working on an ASP MVC4 project using EF5 with repository pattern.
I have just joined this project.
In this project, we have implemented many repository class, these repository will responsible for search, update, delete etc using the dbcontext, they also return the DTO classes and in the service layer we use those repositories to get the DTO then convert to the view model.
Every time I want to do some logic with the entities, I will go to the repositories and write code here. So I wonder why we need the service layer and the repositories at the same time, we can write the logic code directly in the service layer or use repositories in the controller directly.
I don't see any advantage here since our source code is too complicated and we need so many classes (DTO, viewmodel...) and I think the performance will be not good compare to using repositories or services directly.
Can you point out the key here? Thank you.
It's very simple:
Repositories are for data access
Services are for business logic
But once you've started to instill business concepts into repositories it becomes very hard to turn the tide.
An example of how easily business concepts mingle with data access concerns is soft deletes. Let's say that there is a table journal_voucher from which rows should never be deleted, only inactivated. So there is a boolean (bit) field IsActive that's set to false if a row should be off the record.
Now it seems obvious to have a Delete method in JournalRepository that sets the IsActive flag in stead of deleting an entity. Likewise, any retrieval methods may automatically filter out inactive records.
Wrong. Being active or inactive is a business concept. For a data access layer the content of any database field is meaningless. It's only supposed to read and write it properly.
Now see what happens: other entities will probably just be hard-deleted. Maybe yet others can't ever be deleted, or, why not, never be created. If one repository has this active/inactive responsibility, the next obvious step is to implement these other CRUD rules in the appropriate repositories as well. Then a business requirement emerges that only records of the current year are interesting... Oh, and we have to check whether a journal_voucher can even be inactivated... And so on and so forth.
You end up with a host of very different repository classes and scattered business logic.
I believe that if you decide to use your own repositories on top of Entity Framework's repositories (DbSets) they should be generic repositories. That is: for each entity class they do exactly the same thing. It's even arguable whether they should return DTOs instead of EF entity objects (I'd vote for the latter).
Everything else is done in services. So there will probably be a JournalService that inactivates journal_vouchers, with proper checking. The service decides that IsActive is set to false and instructs the repository to update the entity. (In fact a unit of work should do that, but that's a different story).
This distinction has many benefits:
The rest of the world only communicates with services.
Therefore, repositories can safely return IQueryable. The services limit the amount of retrieved data.
It's much easier to decide where business logic involving multiple entities belongs (i.e. almost all business logic).
It is much easier with dependency injection.
The repositories can be mocked relatively easily and the services can be readily unit tested without duplicating business rules in mocked repositories.
We have kind of an interesting situation here. We are using the repository pattern with Entity Framework so each database table has its own Repository class that accepts an instance of a DbContext in its constructor. We are also using Ninject for dependency injection. We have defined a single context to be instantiated during a given request so that when a multiple of repositories ask for a DbContext, the same instance is used throughout. This allows us to follow a UnitOfWork pattern so multiple things can occur on multiple repositories and a single commit will commit all changes to the database.
Here is the issue, we are also using SQL Azure Federations which splits up our client data into multiple databases (sharding). We need to be able to jump from one federation member (database) to another within the same request but want to be able to use the same service/repository methods that are dependent upon the injected context. Our first thought was to just execute the USE FEDERATION sql command on the existing context to move to the next database but it seems to work sometimes and not others. After executing the statement we see that the underlying connection on the context really is pointing to the new server but for some reason queries executed on that context end up returning results from the previous connection. I presume using the same instance of a context on multiple database is not something that is really natively supported as you would normally spin up a new context when connecting to a new database. Unfortunately we have a bunch of repositories that are created dynamically using Ninject which then take in the same instance of the context so even if we do spin up a new context for the new database, we have no way to making all the existing repositories suddenly become dependent upon the new context we just spun up instead of the one that was created at the initial request.
Here are a few solutions we can think of but are not sure how to get any of them to work:
Change the database used on an existing context (as explained above but didn't seem to work)
Swap out an injected context for all repositories with a new one so that all existing repositories now become dependent upon a different context
Somehow request from Ninject a new instance of all the repositories passing in the parameters needed for the context once it is requested.
Again, bottom line here is we have a set of repositories and services that are dependent upon a single Context and we want to be able to reuse those services and repositories but swap out or change the context, on which all are dependent, to point to a new server.
Solution was to just completely drop Ninject out of the scenario. Not the best solution but the tools we are using were not really designed to do what we wanted in the environment we are working.
I am writing an application that will move data from one database to another Using JPA EntityManager API. My questions are : 1. Can i use multiple entity managers in a single transaction? 2. Can i read an entity from one database and persist it in the other? what are the issues that am likely to encounter?
Can I use multiple entity managers in a single transaction?
Yes, using JTA. I'm not sure you need a global transaction in your case though. Are you really moving or copying entities from one DB to the other? In the later case, you could use two transactions sequentially.
Can I read an entity from one database and persist it in the other?
Assuming they have the same structure and you don't have any conflicting PK, it should be possible to read an entity using a first entity manager, detach it and then merge it using another entity manager. If you have possible PK conflicts, you'll have to use a DIY approach (vs a simple merge).