AspNet Boilerplate Parallel DB Access through Entity Framework from an AppService - entity-framework

We are using ASP.NET Zero and are running into issues with parallel processing from an AppService. We know requests must be transactional, but unfortunately we need to break out to slow running APIs for numerous calls, so we have to do parallel processing.
As expected, we are running into a DbContext contingency issue on the second database call we make:
System.InvalidOperationException: A second operation started on this context
before a previous operation completed. This is usually caused by different
threads using the same instance of DbContext, however instance members are
not guaranteed to be thread safe. This could also be caused by a nested query
being evaluated on the client, if this is the case rewrite the query avoiding
nested invocations.
We read that a new UOW is required, so we tried using both the method attribute and the explicit UowManager, but neither of the two worked.
We also tried creating instances of the referenced AppServices using the IocResolver, but we are still not able to get a unique DbContext per thread (please see below).
public List<InvoiceDto> CreateInvoices(List<InvoiceTemplateLineItemDto> templateLineItems)
{
List<InvoiceDto> invoices = new InvoiceDto[templateLineItems.Count].ToList();
ConcurrentQueue<Exception> exceptions = new ConcurrentQueue<Exception>();
Parallel.ForEach(templateLineItems, async (templateLineItem) =>
{
try
{
XAppService xAppService = _iocResolver.Resolve<XAppService>();
InvoiceDto invoice = await xAppService
.CreateInvoiceInvoiceItem();
invoices.Insert(templateLineItems.IndexOf(templateLineItem), invoice);
}
catch (Exception e)
{
exceptions.Enqueue(e);
}
});
if (exceptions.Count > 0) throw new AggregateException(exceptions);
return invoices;
}
How can we ensure that a new DbContext is availble per thread?

I was able to replicate and resolve the problem with a generic version of ABP. I'm still experiencing the problem in my original solution, which is far more complex. I'll have to do some more digging to determine why it is failing there.
For others that come across this problem, which is exactly the same issue as reference here, you can simply disable the UnitOfWork through an attribute as illustrated in the code below.
public class InvoiceAppService : ApplicationService
{
private readonly InvoiceItemAppService _invoiceItemAppService;
public InvoiceAppService(InvoiceItemAppService invoiceItemAppService)
{
_invoiceItemAppService = invoiceItemAppService;
}
// Just add this attribute
[UnitOfWork(IsDisabled = true)]
public InvoiceDto GetInvoice(List<int> invoiceItemIds)
{
_invoiceItemAppService.Initialize();
ConcurrentQueue<InvoiceItemDto> invoiceItems =
new ConcurrentQueue<InvoiceItemDto>();
ConcurrentQueue<Exception> exceptions = new ConcurrentQueue<Exception>();
Parallel.ForEach(invoiceItemIds, (invoiceItemId) =>
{
try
{
InvoiceItemDto invoiceItemDto =
_invoiceItemAppService.CreateAsync(invoiceItemId).Result;
invoiceItems.Enqueue(invoiceItemDto);
}
catch (Exception e)
{
exceptions.Enqueue(e);
}
});
if (exceptions.Count > 0) {
AggregateException ex = new AggregateException(exceptions);
Logger.Error("Unable to get invoice", ex);
throw ex;
}
return new InvoiceDto {
Date = DateTime.Now,
InvoiceItems = invoiceItems.ToArray()
};
}
}
public class InvoiceItemAppService : ApplicationService
{
private readonly IRepository<InvoiceItem> _invoiceItemRepository;
private readonly IRepository<Token> _tokenRepository;
private readonly IRepository<Credential> _credentialRepository;
private Token _token;
private Credential _credential;
public InvoiceItemAppService(IRepository<InvoiceItem> invoiceItemRepository,
IRepository<Token> tokenRepository,
IRepository<Credential> credentialRepository)
{
_invoiceItemRepository = invoiceItemRepository;
_tokenRepository = tokenRepository;
_credentialRepository = credentialRepository;
}
public void Initialize()
{
_token = _tokenRepository.FirstOrDefault(x => x.Id == 1);
_credential = _credentialRepository.FirstOrDefault(x => x.Id == 1);
}
// Create an invoice item using info from an external API and some db records
public async Task<InvoiceItemDto> CreateAsync(int id)
{
// Get db record
InvoiceItem invoiceItem = await _invoiceItemRepository.GetAsync(id);
// Get price
decimal price = await GetPriceAsync(invoiceItem.Description);
return new InvoiceItemDto {
Id = id,
Description = invoiceItem.Description,
Amount = price
};
}
private async Task<decimal> GetPriceAsync(string description)
{
// Simulate a slow API call to get price using description
// We use the token and credentials here in the real deal
await Task.Delay(5000);
return 100.00M;
}
}

Related

JPA : Update operation without JPA query or entitymanager

I am learning JPA, I found out that we have some functions which is already present in Jparepository like save,saveAll,find, findAll etc. but there is nothing like update,
I come across one scenario where I need to update the table, if the value is already present otherwise I need to insert the record in table.
I created
#Repository
public interface ProductInfoRepository
extends JpaRepository<ProductInfoTable, String>
{
Optional<ProductInfoTable> findByProductName(String productname);
}
public class ProductServiceImpl
implements ProductService
{
#Autowired
private ProductInfoRepository productRepository;
#Override
public ResponseMessage saveProductDetail(ProductInfo productInfo)
{
Optional<ProductInfoTable> productInfoinTable =
productRepository.findByProductName(productInfo.getProductName());
ProductInfoTable productInfoDetail;
Integer quantity = productInfo.getQuantity();
if (productInfoinTable.isPresent())
{
quantity += productInfoinTable.get().getQuantity();
}
productInfoDetail =
new ProductInfoTable(productInfo.getProductName(), quantity + productInfo.getQuantity(),
productInfo.getImage());
productRepository.save(productInfoDetail);
return new ResponseMessage("product saved successfully");
}
}
as you can see, I can save the record if the record is new, but when I am trying to save the record which is already present in table it is giving me error related to primarykeyviolation which is obvious. I checked somewhat, we can do the update by creating the entitymanager object or jpa query but what if I dont want to use both of them. is there any other way we can do so ?
update I also added the instance of EntityManager and trying to merge the code
#Override
public ResponseMessage saveProductDetail(ProductInfo productInfo)
{
Optional<ProductInfoTable> productInfoinTable =
productRepository.findByProductName(productInfo.getProductName());
ProductInfoTable productInfoDetail;
Integer price = productInfo.getPrice();
if (productInfoinTable.isPresent())
{
price = productInfoinTable.get().getPrice();
}
productInfoDetail =
new ProductInfoTable(productInfo.getProductName(), price, productInfo.getImage());
em.merge(productInfoDetail);
return new ResponseMessage("product saved successfully");
but no error, no execution of update statements in log, any possible reasons for that ?
}
I suspect you need code like this to solve the problem
public ResponseMessage saveProductDetail(ProductInfo productInfo)
{
Optional<ProductInfoTable> productInfoinTable =
productRepository.findByProductName(productInfo.getProductName());
final ProductInfoTable productInfoDetail;
if (productInfoinTable.isPresent()) {
// to edit
productInfoDetail = productInfoinTable.get();
Integer quantity = productInfoDetail.getQuantity() + productInfo.getQuantity();
productInfoDetail.setQuantity(quantity);
} else {
// to create new
productInfoDetail = new ProductInfoTable(productInfo.getProductName(),
productInfo.getQuantity(), productInfo.getImage());
}
productRepository.save(productInfoDetail);
return new ResponseMessage("product saved successfully");
}

How to mock a MongoDB collection of type IMongoCollection<T> to return some predefined data?

Below is my code:
Controller/Action:
[HttpPost]
[AllowAnonymous]
[ValidateAntiForgeryToken]
public async Task<IActionResult> Register(CustomerViewModel model, string returnUrl = null)
{
try
{
ViewData["ReturnUrl"] = returnUrl;
// when debugging the test, _dbContext.Customers throws exception
CustomerDoc existingCustomer = await _dbContext.Customers.Find(o => o.email == model.email).FirstOrDefaultAsync();
if (existingCustomer != null)
{
ModelState.AddModelError("Email", "email already used.");
}
// other checkings
if (!ModelState.IsValid)
{
return View(model);
}
// if model state is valid, do something here
}
catch (Exception ex)
{
return View(model);
}
return View(model);
}
And my unit test code is:
[Fact]
public async Task should_return_view_with_errors_when_email_already_exists()
{
IEnumerable<CustomerDoc> customers = new List<CustomerDoc>
{
new CustomerDoc
{
email = "test#test.com"
}
};
_dbContextMock.SetupAllProperties();
// below line is causing the error
_dbContextMock.Setup(c => c.Customers).Returns(() =>(IMongoCollection<CustomerDoc>)customers);
CustomerViewModel model = new CustomerViewModel
{
email = "test#test.com"
};
CreateController();
var result = await _controller.Register(model);
Assert.IsType<ViewResult>(result);
Assert.False(_controller.ModelState.IsValid);
Assert.True(_controller.ModelState.ContainsKey("Email"));
}
As you can see in my unit test code comment, I am trying to mock a IMongoCollection to return some data. But I am not able to do so because _dbContext.Customers is throwing exception.
How can I mock IMongoCollection to return some predefined data?
I am using
asp.net core 2.1.0
mongodb driver 2.7.0
You declare customers as a List:
IEnumerable<CustomerDoc> customers = new List<CustomerDoc>
but then try to cast it to IMongoCollection
() =>(IMongoCollection<CustomerDoc>)customers
There's two immediate directions (but both have further issues to deal with):
1) Just return the list without the cast
() => customers
but I can't see the type of c.Customers so I suspect this will just move the issue. I'll take a guess that it's IMongoCollection<CustomerDoc> which is why you're trying to do the cast in the first place? This is problematic as the .Returns would need to be be associated with a function performing the equivalent of c.Customer.Find(). Even so, it's probably better than the alternative.
2) Changing the customers variable to a type that implements IMongoCollection.
Option 1 feels like the way to go as option 2 forces you to start dealing with lots of logic that really shouldn't be relevant to this piece of code.

Datanucleus. Occasional javax.jdo.JDODetachedFieldAccessException: You have just attempted to access field "linkedObject"

We use Java Datanucleus 5.0.2 with JDO.
We get occasional exception while several threads are retrieving the same information and NO thread is changing this particular "linkedObject" reference.
NOTE: The object is retrieved using the fetch plan and logs show that.
It is hard to write a test case for this to fail as it is a race condition. But nevertheless I want to ask if someone has experienced it?
Caused by: javax.jdo.JDODetachedFieldAccessException: You have just attempted to access field "linkedObject" yet this field was not detached when you detached the object. Either dont access this field, or detach it when detaching the object.
at com.company.BaseClass.dnGetlinkedObject(BaseClass.java)
at com.company.BaseClass.getLinkedObject(BaseClass.java:71)
...
I can see three threads calling this method
private static <T> T getUniqueQueryJDO(final PersistenceManager pm, final JDOQLQuery query) throws PersistenceException {
try {
final javax.jdo.Query jdoQuery = setUpJDOQuery(pm, query);
jdoQuery.setUnique(true);
T result = null;
final T queryResult = (T) jdoQuery.executeWithMap(query.getMapValues());
if (queryResult != null) {
result = pm.detachCopy(queryResult);
}
jdoQuery.closeAll();
return result;
}
and one of them fails randomly
BaseClass.java
#PersistentDomainObject
#PersistenceCapable(table = "BaseClass", detachable = TRUE)
#Inheritance(strategy = InheritanceStrategy.SUPERCLASS_TABLE)
#FetchGroups(
...
#FetchGroup(name = FETCH_LINKED_OBJECT, members = {#Persistent(name = "linkedObject")})})
...
public class BaseClass {
...
public static final String FETCH_LINKED_CLASS = "FETCH_NAME";
...
#Persistent(defaultFetchGroup = FALSE, columns = {#Column(name = "linkedObjectId", allowsNull = FALSE)}, nullValue = NullValue.EXCEPTION)
private LinkedClass linkedObject;
...
public LinkedClass getLinkedObject() {
return linkedObject;
}
}

Test a method that does something as well as throws an exception

I'm getting started with TDD (and MoQ). I have a method that takes in an Order and creates a unique CustomerId for that Order.
public class CustomerService
{
public CustomerService(IOrderRepository orderRepo)
{
_orderRepo = orderRepo;
}
public string Create(Order order)
{
//1. Save the Order that comes in, irrespective of being valid or not.
_orderRepo.Save(order);
//2. If the order is invalid, throw an exception.
if (!isValid(order))
throw new ArgumentException();
//3. Create a Customer, generate a unique CustomerId and return that.
return createCustomerAndGenerateCustomerId(order);
}
}
The following test doesn't seem to be working correctly:
[Test]
[ExpectedException(typeof(ArgumentException))]
public void Create_WhenPassedInvalidOrder_StillPersistsIt()
{
//Arrange
var order = new Order(); //This is invalid, as it has some mandatory fields missing
var mockOrderRepo = new Mock<IOrderRepository>();
var customerService = new CustomerService(mockOrderRepo.Object);
//Act
customerService.Create(order); //This should call Save() on the order repo, but still throw an exception.
//Assert
mockOrderRepo.Verify(o => o.Save(order), Times.Once());
}
This test is always passing, even if I don't call _orderRepo.Save(). What am I doing wrong?
You cannot use ExpectedException in this scenario because Nunit will try/catch the exception on the test level so your mockOrderRepo.Verify never gets called.
So you manually need to try catch your customerService.Create call - and if you want manually assert on the thrown exception - or you the Assert.Throws if you're using Nunit 2.5 or higher:
[Test]
public void Create_WhenPassedInvalidOrder_StillPersistsIt()
{
//Arrange
var order = new Order();
var mockOrderRepo = new Mock<IOrderRepository>();
var customerService = new CustomerService(mockOrderRepo.Object);
//Act
Assert.Throws<ArgumentException>(() => customerService.Create(order));
//Assert
mockOrderRepo.Verify(o => o.Save(order), Times.Once());
}

Entity Framework + ODATA: side-stepping the pagination

The project I'm working on has the Entity Framework on top of an OData layer. The Odata layer has it's server side pagination turned to a value of 75. My reading on the subject leads me to believe that this pagination value is used across the board, rather than a per table basis. The table that I'm currently looking to extract all the data from is, of course, more than 75 rows. Using the entity framework, my code is simply thus:
public IQueryable<ProductColor> GetProductColors()
{
return db.ProductColors;
}
where db is the entity context. This is returning the first 75 records. I read something where I could append a parameter inlinecount set to allpages giving me the following code:
public IQueryable<ProductColor> GetProductColors()
{
return db.ProductColors.AddQueryOption("inlinecount","allpages");
}
However, this too returns 75 rows!
Can anyone shed light on how to truly get all the records regardless of the OData server-side pagination stuff?
important: I cannot remove the pagination or turn it off! It's extremely valuable in other scenarios where performance is a concern.
Update:
Through some more searching I've found an MSDN that describes how to do this task.
I'd love to be able to turn it into a full Generic method but, this was as close as I could get to a generic without using reflection:
public IQueryable<T> TakeAll<T>(QueryOperationResponse<T> qor)
{
var collection = new List<T>();
DataServiceQueryContinuation<T> next = null;
QueryOperationResponse<T> response = qor;
do
{
if (next != null)
{
response = db.Execute<T>(next) as QueryOperationResponse<T>;
}
foreach (var elem in response)
{
collection.Add(elem);
}
} while ((next = response.GetContinuation()) != null);
return collection.AsQueryable();
}
calling it like:
public IQueryable<ProductColor> GetProductColors()
{
QueryOperationResponse<ProductColor> response = db.ProductColors.Execute() as QueryOperationResponse<ProductColor>;
var productColors = this.TakeAll<ProductColor>(response);
return productColors.AsQueryable();
}
If unable turn off paging you'll receive 75 row by call, always. You can get all rows in following ways:
Add another IQueryable<ProductColor> AllProductColors and modify
public static void InitializeService(DataServiceConfiguration config)
{
config.UseVerboseErrors = true;
config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
config.SetEntitySetPageSize("ProductColors", 75); - Note only paged queries are present
config.SetServiceOperationAccessRule("*", ServiceOperationRights.AllRead);
config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
}
You should call ProductColors as many as needed, for example
var cat = new NetflixCatalog(new Uri("http://odata.netflix.com/v1/Catalog/"));
var x = from t in cat.Titles
where t.ReleaseYear == 2009
select t;
var response = (QueryOperationResponse<Title>)((DataServiceQuery<Title>)x).Execute();
while (true)
{
foreach (Title title in response)
{
Console.WriteLine(title.Name);
}
var continuation = response.GetContinuation();
if (continuation == null)
{
break;
}
response = cat.Execute(continuation);
}
I use Rx with following code
public sealed class DataSequence<TEntry> : IObservable<TEntry>
{
private readonly DataServiceContext context;
private readonly Logger logger = LogManager.GetCurrentClassLogger();
private readonly IQueryable<TEntry> query;
public DataSequence(IQueryable<TEntry> query, DataServiceContext context)
{
this.query = query;
this.context = context;
}
public IDisposable Subscribe(IObserver<TEntry> observer)
{
QueryOperationResponse<TEntry> response;
try
{
response = (QueryOperationResponse<TEntry>)((DataServiceQuery<TEntry>)query).Execute();
if (response == null)
{
return Disposable.Empty;
}
}
catch (Exception ex)
{
logger.Error(ex);
return Disposable.Empty;
}
var initialState = new State
{
CanContinue = true,
Response = response
};
IObservable<TEntry> sequence = Observable.Generate(
initialState,
state => state.CanContinue,
MoveToNextState,
GetCurrentValue,
Scheduler.ThreadPool).Merge();
return new CompositeDisposable(initialState, sequence.Subscribe(observer));
}
private static IObservable<TEntry> GetCurrentValue(State state)
{
if (state.Response == null)
{
return Observable.Empty<TEntry>();
}
return state.Response.ToObservable();
}
private State MoveToNextState(State state)
{
DataServiceQueryContinuation<TEntry> continuation = state.Response.GetContinuation();
if (continuation == null)
{
state.CanContinue = false;
return state;
}
QueryOperationResponse<TEntry> response;
try
{
response = context.Execute(continuation);
}
catch (Exception)
{
state.CanContinue = false;
return state;
}
state.Response = response;
return state;
}
private sealed class State : IDisposable
{
public bool CanContinue { get; set; }
public QueryOperationResponse<TEntry> Response { get; set; }
public void Dispose()
{
CanContinue = false;
}
}
}
so for get any data thru OData, create a sequence and Rx does the rest
var sequence = new DataSequence<Product>(context.Products, context);
sequence.OnErrorResumeNext(Observable.Empty<Product>())
.ObserveOnDispatcher().SubscribeOn(Scheduler.NewThread).Subscribe(AddProduct, logger.Error);
The page size is set by the service author and can be set per entity set (but a service may choose to apply the same page size to all entity sets). There's no way to avoid it from the client (which is by design since it's a security feature).
The inlinecount option asks the server to include the total count of the results (just the number), it doesn't disable the paging.
From the client the only way to read all the data is to issue the request which will return the first page and it may contain a next link which you request to read the next page and so on until the last response doesn't have the next link.
If you're using the WCF Data Services client library it has support for continuations (the next link) and a simple sample can be found in this blog post (for example): http://blogs.msdn.com/b/phaniraj/archive/2010/04/25/server-driven-paging-with-wcf-data-services.aspx