Correct pattern to insert or update existed entity with spring data reactive - spring-data

I implement an application with spring data (mongo).
It gets data from external service
Store it to DB
If item does not exist I'd like to insert it to db
If item exists but has same md5 hash with external item - do nothing
If item exists in DB and md5 hashes are not equal I'd like to update it
I have the code
public Mono<Void> processItems() {
return externalService.getItems() //this returns Flux<Item>
.flatMap(this::createOrUpdateItem)
.flatMap(itemRepository::save)
.then()
}
private Mono<Item> createOrUpdateItem(Item item) {
return itemRepository.findById(item.getId) //this returns Mono<Item>
.flatMap(itemFromDb -> updateItem(itemFromDb, item))
.defaultIfEmpty(item)
}
private Mono<Item> updateItem(Item itemFromDb, Item item){
if(itemFromDb.getMd5.equals(item.getMd5)){
return Mono.just(itemFromDb);
}
itemFromDb.setName(item.getName)
.setDescription(item.getDescription);
return Mono.just(itemFromDb);
}
How does it work:
If item does not exist it stored in db
If item exists and has same md5 hash updateItem returns existed in db and version in DB increases
If item exists and md5 hashes are not equal updateItem set feilds and returns existed in db and version in DB increases
The problem is: when md5 hash are equal I don't want to cal DB. Because it increases version of the item in db but there were no changes.
How to properly implement this algorithm?
I can use #EqualsAndHashCode(exclude = "id") but not sure if it is a right way

The only missing ingredient in your example is to continue with an empty stream if nothing needs to be done.
This solution is actually similar to yours, but has a clear separation of concerns.
The first flatMap only loads existing data. The second one contains only the business logic to decide what to do. It's up to you to follow these established principles (Single Responsibility Principle, Integration Operation Segregation Principle etc.).
public Mono<Void> processItems() {
return externalService.getItems() //this returns Flux<Item>
.flatMap(this::loadExistingItem)
.flatMap(this::setupOperation)
.flatMap(this::saveItem)
.then()
}
private Mono<List<Item>> loadExistingItem(Item item) {
return itemRepository.findById(item.getId)
.map(fromDb -> Arrays.asList(item, fromDb))
.defaultIfEmpty(() -> Arrays.asList(item, null));
}
private Mono<Item> setupOperation(List<Item> items) {
Item newItem = items.get(0);
Item existingItem = items.get(1);
if (existingItem == null) {
return Mono.just(newItem);
}
if(newItem.getMd5().equals(existingItem.getMd5())){
return Mono.empty();
}
existingItem.setName(newItem.getName)
existingItem.setDescription(newItem.getDescription);
return Mono.just(existingItem);
}

Related

RxJava - retrieving entry in Observable in second flatMapSingle

We have a vertx verticle which receives an id and uses it see if an entity with the id exist in a database. It contains the following logic:
if (itemFound) {
e.onNext(item_which_was_found)
}
else {
e.onNext(null);
}
Another verticle has an Observable which processes a list of id's. It uses rxSend to pass each id in the list to the first verticle to do the database lookup:
Observable<Object> observable = ...
observable.flatMapSingle(id -> {
return rxSend(VERTICLE_1_ADDRESS, id);
})
.flatMapSingle ( i ->
{
// Logic dependent on if item was found
)
.subscribe();
With the above, it is easy to handle cases where the entity associated with the id was found in the database, because the first vertcle, in onNext(), returns the entity. The question is for the second case, when no entity exists and first verticle returns onNext(null). In this case, how is it possible to retrieve, in the second flatMapSingle, the item in the observable which is currently being processed (that is, the id which has no associated database entity) ? Or is there a better way to structure the code?
Thanks
You can change your observable definition to:
Observable<Object> observable = observable();
observable.flatMapSingle(id -> {
return rxSend(VERTICLE_1_ADDRESS, id).flatMap(i -> {
// Logic dependent on if item was found
// id is visible here
});
}).subscribe();
Then the id will be visible to your second lambda.

ASP.NET Core 2 Controller PUT method failing with Concurrency Core

I have an ASP.NET Core 2.0 Site that has a scaffolded controller built directly from a simple model and simple context. I seeded the data by simply checking for the number of records in the GET method and if 0, then I added 100 records. GET is retrieving records as I would expect repeatedly.
I'm using the inmemory database provider.
services.AddDbContext<MyDbContext>
(opt => opt.UseInMemoryDatabase("CodeCampInMemoryDb"));
When I do a PUT with a record that I know existed in my GET, I get a concurrency error as shown at the bottom of this post. I've not used this method of changing the EntityState of a record I created myself, so I'm not sure how this was suppose to work in the first place, but clearly now it is not working.
Maybe it has something to do with a transaction being processed on the inmemory database? I'm not sure how to avoid that if that is the problem.
// PUT: api/Sessions/5
[HttpPut("{id}")]
public async Task<IActionResult> PutSessionRec([FromRoute] int id, [FromBody] SessionRec sessionRec)
{
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
if (id != sessionRec.Id)
{
return BadRequest();
}
_context.Entry(sessionRec).State = EntityState.Modified;
try
{
await _context.SaveChangesAsync();
}
catch (DbUpdateConcurrencyException xx)
{
if (!SessionRecExists(id))
{
return NotFound();
}
else
{
throw;
}
}
return NoContent();
}
Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: Attempted to update or delete an entity that does not exist in the store.
at Microsoft.EntityFrameworkCore.Storage.Internal.InMemoryTable`1.Update(IUpdateEntry entry)
at Microsoft.EntityFrameworkCore.Storage.Internal.InMemoryStore.ExecuteTransaction
If you marked a property as a Timestamp and don't provide it, you will get this exception every time. You either need to load the entity with the latest Timestamp and update that (not ideal) or you have to send the Timestamp down to the client and have the client send it back up (correct way). However, if you are using an older version of JSON serialization, you have to convert the byte[] to base 64 and then convert it back.

querying in EntityObject

I have partial class TaxReportItem with partial method OnActualVolumeChanging(double value):
public partial class TaxReportItem
{
partial void OnActualVolumeChanging(double value)
{
if (Tax != null)
{
Payment = value*Tax.TaxRate;
}
}
}
In this method i want to get all collection of TaxReportItems that's present in context(something like this):
partial void OnActualVolumeChanging(double value)
{
var sum = 0.0;
if (Tax != null)
{
Payment = value*Tax.TaxRate;
foreach (var taxReportItem in ????)
{
sum += taxReportItem.Sum;
}
}
}
How can i achieve this?
This is actually quite hard because you should never need this. If you need this, design of your entity object is wrong and it is doing something which should be done elsewhere. Single TaxReportItem should never need to know about other tax report items and load them from database unless other items are dependent on this item (they form an aggregate). In such case you should have navigation property to dependent items in the principal one.
To follow your question. If you need to load other items you must have instance of the context to do that. You can either get instance used to load current item and use it to load other entities (bad solution) or you can create a new instance of the context and use it to load other entities (even worse solution).
As you can see from the linked article getting current context instance from the entity itself is not easy and it has some prerequisites which goes back to the first paragraph. It is hard because it is wrong approach.

Entity Framework and 2nd level caching with EF Provider Wrappers

I'm trying to get 2nd level caching to work with entity framework 4. The "EF Provider Wrappers" made by Jarek Kowalski (http://code.msdn.microsoft.com/EFProviderWrappers/Release/ProjectReleases.aspx?ReleaseId=4747) works pretty good, the problem i have is that all cached entries from a table is invalidated as soon as an update is made to the table. Is this intended, or have i made an error in my implementation?
If this is intended, it makes it completely useless on tables which have a lot of updates. Are there any way to remedy this?
This is my implementation of the ICache interface, using ScaleOut StateServer as cache:
public class SossCache : ICache
{
private readonly NamedCache SossCache;
public SossCache(string cacheName)
{
this.SossCache = CacheFactory.GetCache(cacheName);
}
public bool GetItem(string key, out object value)
{
value = this.SossCache.Get(key);
return value != null;
}
public void PutItem(string key, object value, IEnumerable<string> dependentEntitySets, TimeSpan slidingExpiration, DateTime absoluteExpiration)
{
bool isAbsoluteTimeout = slidingExpiration == TimeSpan.Zero;
TimeSpan timeout = isAbsoluteTimeout ? absoluteExpiration.Subtract(DateTime.Now) : slidingExpiration;
CreatePolicy createPolicy = new CreatePolicy(timeout, isAbsoluteTimeout, ObjectPreemptionPriority.Normal, dependentEntitySets.ToArray(), true);
this.SossCache.Insert(key, value, createPolicy, true, false);
}
public void InvalidateItem(string key)
{
this.SossCache.Remove(key);
}
public void InvalidateSets(IEnumerable<string> entitySets)
{
foreach (string key in entitySets)
InvalidateItem(key);
}
}
Yes it is intentional. The author has mentioned it in the same link that you have shared.
"EFCachingProvider is a bit more complex. It uses external caching implementation and caches results of all queries queries that are executed in DbCommand.ExecuteReader(). Whenever update is detected (either UPDATE, INSERT or DELETE) the provider invalidates affected cache entries by evicting all cached queries which were dependent on any of the updated tables."
I am not sure what a clean solution would be for your case. But if your table is very frequently updated you better not cache entries of that table. You can use "CustomCachingPolicy" to exclude that table from being cached.
"CustomCachingPolicy – includes user-configurable list of tables that should and should not be cached, as well as expiration times and result size limits."

EF Code First - Recreate Database If Model Changes

I'm currently working on a project which is using EF Code First with POCOs. I have 5 POCOs that so far depends on the POCO "User".
The POCO "User" should refer to my already existing MemberShip table "aspnet_Users" (which I map it to in the OnModelCreating method of the DbContext).
The problem is that I want to take advantage of the "Recreate Database If Model changes" feature as Scott Gu shows at: http://weblogs.asp.net/scottgu/archive/2010/07/16/code-first-development-with-entity-framework-4.aspx - What the feature basically does is to recreate the database as soon as it sees any changes in my POCOs. What I want it to do is to Recreate the database but to somehow NOT delete the whole Database so that aspnet_Users is still alive. However it seems impossible as it either makes a whole new Database or replaces the current one with..
So my question is: Am I doomed to define my database tables by hand, or can I somehow merge my POCOs into my current database and still take use of the feature without wipeing it all?
As of EF Code First in CTP5, this is not possible. Code First will drop and create your database or it does not touch it at all. I think in your case, you should manually create your full database and then try to come up with an object model that matches the DB.
That said, EF team is actively working on the feature that you are looking for: altering the database instead of recreating it:
Code First Database Evolution (aka Migrations)
I was just able to do this in EF 4.1 with the following considerations:
CodeFirst
DropCreateDatabaseAlways
keeping the same connection string and database name
The database is still deleted and recreated - it has to be to for the schema to reflect your model changes -- but your data remains intact.
Here's how: you read your database into your in-memory POCO objects, and then after the POCO objects have successfully made it into memory, you then let EF drop and recreate the database. Here is an example
public class NorthwindDbContextInitializer : DropCreateDatabaseAlways<NorthindDbContext> {
/// <summary>
/// Connection from which to ead the data from, to insert into the new database.
/// Not the same connection instance as the DbContext, but may have the same connection string.
/// </summary>
DbConnection connection;
Dictionary<Tuple<PropertyInfo,Type>, System.Collections.IEnumerable> map;
public NorthwindDbContextInitializer(DbConnection connection, Dictionary<Tuple<PropertyInfo, Type>, System.Collections.IEnumerable> map = null) {
this.connection = connection;
this.map = map ?? ReadDataIntoMemory();
}
//read data into memory BEFORE database is dropped
Dictionary<Tuple<PropertyInfo, Type>, System.Collections.IEnumerable> ReadDataIntoMemory() {
Dictionary<Tuple<PropertyInfo,Type>, System.Collections.IEnumerable> map = new Dictionary<Tuple<PropertyInfo,Type>,System.Collections.IEnumerable>();
switch (connection.State) {
case System.Data.ConnectionState.Closed:
connection.Open();
break;
}
using (this.connection) {
var metaquery = from p in typeof(NorthindDbContext).GetProperties().Where(p => p.PropertyType.IsGenericType)
let elementType = p.PropertyType.GetGenericArguments()[0]
let dbsetType = typeof(DbSet<>).MakeGenericType(elementType)
where dbsetType.IsAssignableFrom(p.PropertyType)
select new Tuple<PropertyInfo, Type>(p, elementType);
foreach (var tuple in metaquery) {
map.Add(tuple, ExecuteReader(tuple));
}
this.connection.Close();
Database.Delete(this.connection);//call explicitly or else if you let the framework do this implicitly, it will complain the connection is in use.
}
return map;
}
protected override void Seed(NorthindDbContext context) {
foreach (var keyvalue in this.map) {
foreach (var obj in (System.Collections.IEnumerable)keyvalue.Value) {
PropertyInfo p = keyvalue.Key.Item1;
dynamic dbset = p.GetValue(context, null);
dbset.Add(((dynamic)obj));
}
}
context.SaveChanges();
base.Seed(context);
}
System.Collections.IEnumerable ExecuteReader(Tuple<PropertyInfo, Type> tuple) {
DbCommand cmd = this.connection.CreateCommand();
cmd.CommandText = string.Format("select * from [dbo].[{0}]", tuple.Item2.Name);
DbDataReader reader = cmd.ExecuteReader();
using (reader) {
ConstructorInfo ctor = typeof(Test.ObjectReader<>).MakeGenericType(tuple.Item2)
.GetConstructors()[0];
ParameterExpression p = Expression.Parameter(typeof(DbDataReader));
LambdaExpression newlambda = Expression.Lambda(Expression.New(ctor, p), p);
System.Collections.IEnumerable objreader = (System.Collections.IEnumerable)newlambda.Compile().DynamicInvoke(reader);
MethodCallExpression toArray = Expression.Call(typeof(Enumerable),
"ToArray",
new Type[] { tuple.Item2 },
Expression.Constant(objreader));
LambdaExpression lambda = Expression.Lambda(toArray, Expression.Parameter(typeof(IEnumerable<>).MakeGenericType(tuple.Item2)));
var array = (System.Collections.IEnumerable)lambda.Compile().DynamicInvoke(new object[] { objreader });
return array;
}
}
}
This example relies on a ObjectReader class which you can find here if you need it.
I wouldn't bother with the blog articles, read the documentation.
Finally, I would still suggest you always back up your database before running the initialization. (e.g. if the Seed method throws an exception, all your data is in memory, so you risk your data being lost once the program terminates.) A model change isn't exactly an afterthought action anyway, so be sure to back your data up.
One thing you might consider is to use a 'disconnected' foreign key. You can leave the ASPNETDB alone and just reference the user in your DB using the User key (guid). You can access the logged in user as follows:
MembershipUser currentUser = Membership.GetUser(User.Identity.Name, true /* userIsOnline */);
And then use the User's key as a FK in your DB:
Guid UserId = (Guid) currentUser.ProviderUserKey ;
This approach decouples your DB with the ASPNETDB and associated provider architecturally. However, operationally, the data will of course be loosely connected since the IDs will be in each DB. Note also there will be no referential constraints, whcih may or may not be an issue for you.