storing object in cosmos db returns bad request? - entity-framework-core

I seem to be unable to store a simple object to cosmos db?
this is the database model.
public class HbModel
{
public Guid id { get; set; }
public string FormName { get; set; }
public Dictionary<string, object> Form { get; set; }
}
and this is how I store the data into the database
private static void SeedData(HbModelContext dbContext)
{
var cosmosClient = dbContext.Database.GetCosmosClient();
cosmosClient.ClientOptions.AllowBulkExecution = true;
if (dbContext.Set<HbModel>().FirstOrDefault() == null)
{
// No items could be picked hence try seeding.
var container = cosmosClient.GetContainer("hb", "hb_forms");
HbModel first = new HbModel()
{
Id = Guid.NewGuid(),//Guid.Parse(x["guid"] as string),
FormName = "asda",//x["name"] as string,
Form = new Dictionary<string, object>() //
}
string partitionKey = await GetPartitionKey(container.Database, container.Id);
var response = await container.CreateItemAsync(first, new PartitionKey(partitionKey));
}
else
{
Console.WriteLine("Already have data");
}
}
private static async Task<string> GetPartitionKey(Database database, string containerName)
{
var query = new QueryDefinition("select * from c where c.id = #id")
.WithParameter("#id", containerName);
using var iterator = database.GetContainerQueryIterator<ContainerProperties>(query);
while (iterator.HasMoreResults)
{
foreach (var container in await iterator.ReadNextAsync())
{
return container.PartitionKeyPath;
}
}
return null;
}
but when creating the item I get this error message
A host error has occurred during startup operation '3b06df1f-000c-4223-a374-ca1dc48d59d1'.
[2022-07-11T15:02:12.071Z] Microsoft.Azure.Cosmos.Client: Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 24bac0ba-f1f7-411f-bc57-3f91110c4528; Reason: ();.
Value cannot be null. (Parameter 'provider')
no idea why it fails?
the data should not be formatted incorreclty?
It also fails in case there is data in the dictionary.
What is going wrong?

There are several things wrong with the attached code.
You are enabling Bulk but you are not following the Bulk pattern
cosmosClient.ClientOptions.AllowBulkExecution = true is being set, but you are not parallelizing work. If you are going to use Bulk, make sure you are following the documentation and creating lists of concurrent Tasks. Reference: https://learn.microsoft.com/azure/cosmos-db/sql/tutorial-sql-api-dotnet-bulk-import#step-6-populate-a-list-of-concurrent-tasks. Otherwise don't use Bulk.
You are blocking threads.
The call to container.CreateItemAsync(first, new PartitionKey("/__partitionKey")).Result; is a blocking call, this can lead you to deadlocks. When using async operations (such as CreateItemAsync) please use the async/await pattern. Reference: https://github.com/davidfowl/AspNetCoreDiagnosticScenarios/blob/master/AsyncGuidance.md#avoid-using-taskresult-and-taskwait
The PartitionKey parameter should be the value not the definition.
On the call container.CreateItemAsync(first, new PartitionKey("/__partitionKey")) the Partition Key (second parameter) should be the value. Assuming your container has a Partition Key Definition of /__partitionKey then your documents should have a __partitionKey property and you should pass the Value in this parameter of such property in the current document. Reference: https://learn.microsoft.com/azure/cosmos-db/sql/troubleshoot-bad-request#wrong-partition-key-value
Optionally, if your documents do not contain such a value, just remove the parameter from the call:
container.CreateItemAsync(first)
Be advised though that this solution will not scale, you need to design your database with Partitioning in mind: https://learn.microsoft.com/azure/cosmos-db/partitioning-overview#choose-partitionkey
Missing id
The model has Id but Cosmos DB requires id, make sure the content of the document contains id when serialized.

Related

AspNet Boilerplate Parallel DB Access through Entity Framework from an AppService

We are using ASP.NET Zero and are running into issues with parallel processing from an AppService. We know requests must be transactional, but unfortunately we need to break out to slow running APIs for numerous calls, so we have to do parallel processing.
As expected, we are running into a DbContext contingency issue on the second database call we make:
System.InvalidOperationException: A second operation started on this context
before a previous operation completed. This is usually caused by different
threads using the same instance of DbContext, however instance members are
not guaranteed to be thread safe. This could also be caused by a nested query
being evaluated on the client, if this is the case rewrite the query avoiding
nested invocations.
We read that a new UOW is required, so we tried using both the method attribute and the explicit UowManager, but neither of the two worked.
We also tried creating instances of the referenced AppServices using the IocResolver, but we are still not able to get a unique DbContext per thread (please see below).
public List<InvoiceDto> CreateInvoices(List<InvoiceTemplateLineItemDto> templateLineItems)
{
List<InvoiceDto> invoices = new InvoiceDto[templateLineItems.Count].ToList();
ConcurrentQueue<Exception> exceptions = new ConcurrentQueue<Exception>();
Parallel.ForEach(templateLineItems, async (templateLineItem) =>
{
try
{
XAppService xAppService = _iocResolver.Resolve<XAppService>();
InvoiceDto invoice = await xAppService
.CreateInvoiceInvoiceItem();
invoices.Insert(templateLineItems.IndexOf(templateLineItem), invoice);
}
catch (Exception e)
{
exceptions.Enqueue(e);
}
});
if (exceptions.Count > 0) throw new AggregateException(exceptions);
return invoices;
}
How can we ensure that a new DbContext is availble per thread?
I was able to replicate and resolve the problem with a generic version of ABP. I'm still experiencing the problem in my original solution, which is far more complex. I'll have to do some more digging to determine why it is failing there.
For others that come across this problem, which is exactly the same issue as reference here, you can simply disable the UnitOfWork through an attribute as illustrated in the code below.
public class InvoiceAppService : ApplicationService
{
private readonly InvoiceItemAppService _invoiceItemAppService;
public InvoiceAppService(InvoiceItemAppService invoiceItemAppService)
{
_invoiceItemAppService = invoiceItemAppService;
}
// Just add this attribute
[UnitOfWork(IsDisabled = true)]
public InvoiceDto GetInvoice(List<int> invoiceItemIds)
{
_invoiceItemAppService.Initialize();
ConcurrentQueue<InvoiceItemDto> invoiceItems =
new ConcurrentQueue<InvoiceItemDto>();
ConcurrentQueue<Exception> exceptions = new ConcurrentQueue<Exception>();
Parallel.ForEach(invoiceItemIds, (invoiceItemId) =>
{
try
{
InvoiceItemDto invoiceItemDto =
_invoiceItemAppService.CreateAsync(invoiceItemId).Result;
invoiceItems.Enqueue(invoiceItemDto);
}
catch (Exception e)
{
exceptions.Enqueue(e);
}
});
if (exceptions.Count > 0) {
AggregateException ex = new AggregateException(exceptions);
Logger.Error("Unable to get invoice", ex);
throw ex;
}
return new InvoiceDto {
Date = DateTime.Now,
InvoiceItems = invoiceItems.ToArray()
};
}
}
public class InvoiceItemAppService : ApplicationService
{
private readonly IRepository<InvoiceItem> _invoiceItemRepository;
private readonly IRepository<Token> _tokenRepository;
private readonly IRepository<Credential> _credentialRepository;
private Token _token;
private Credential _credential;
public InvoiceItemAppService(IRepository<InvoiceItem> invoiceItemRepository,
IRepository<Token> tokenRepository,
IRepository<Credential> credentialRepository)
{
_invoiceItemRepository = invoiceItemRepository;
_tokenRepository = tokenRepository;
_credentialRepository = credentialRepository;
}
public void Initialize()
{
_token = _tokenRepository.FirstOrDefault(x => x.Id == 1);
_credential = _credentialRepository.FirstOrDefault(x => x.Id == 1);
}
// Create an invoice item using info from an external API and some db records
public async Task<InvoiceItemDto> CreateAsync(int id)
{
// Get db record
InvoiceItem invoiceItem = await _invoiceItemRepository.GetAsync(id);
// Get price
decimal price = await GetPriceAsync(invoiceItem.Description);
return new InvoiceItemDto {
Id = id,
Description = invoiceItem.Description,
Amount = price
};
}
private async Task<decimal> GetPriceAsync(string description)
{
// Simulate a slow API call to get price using description
// We use the token and credentials here in the real deal
await Task.Delay(5000);
return 100.00M;
}
}

Error code 404 or 405, for GET and PUT methods

Good day I have created a simple Web API for my app.
I was able to successfully make a POST method without conflict, and also GET without parameters(https://www.something.com/api/something/) works too, but when I insert parameter for my GET (https://www.something.com/api/something/1) it gives me a 404 on POSTMAN. When I try PUT Method, and I don't put a parameter, it gives me a 405. And 404 when I put a parameter. Below is my code.
I'm using MongoDB for my database.
database has _id, and category as Partition key.
_id also works as id(Property name JSON)
Controller
// To get a specific record
[HttpGet("{id:length(24)}")]
public ActionResult< SomeModel > Get(string id)
{
var some = _someThing.Get(id);
if (some == null)
{
return NotFound();
}
return some;
}
// For Updating a record
[HttpPut("{id:length(24)}")]
public IActionResult Update(string id, SomeModel pModel)
{
var something = _someModel.Get(id);
if (something == null)
{
return NotFound();
}
_someModel.Update(id, pModel);
return NoContent();
}
Services
// For Finding a specific record
public SomeModel Get(string id) =>
_scores.Find< SomeModel >(scores => scores.id == id).FirstOrDefault();
// For Updating record
public void Update(string id, SomeModel newScore) =>
_scores.ReplaceOne(scores => scores.id == id, newScore);
From this Microsoft page section about route constraints:
Length: Matches a string with the specified length or within a specified range of lengths.
It looks to me like your controller is expecting an ID of 24 characters long.
Try changing it to this:
[HttpGet("{id:length(1,24)}")]
Or using minlength or maxlength instead.

Use EF Core to get a list of Longs

I have a stored procedure that returns me a list of IDs. (I then use this list of IDs as keys for objects.)
I am migrating this from .NET to .NET Core. In normal .NET I could use an extension library to get the numbers out like this:
var getOrderDetailIdsStoredProc = new GetOrderDetailIdsStoredProc()
{
NumberOfOrderDetailIdsNeeded = numberOfOrderDetailIdsNeeded
};
var orderDetailIds = contextProvider.Context.Database
.ExecuteStoredProcedure<long>(getOrderDetailIdsStoredProc);
But that library (EntityFrameworkExtras) is not working with EF Core (I found a version for EF Core, but it doesn't work.)
So I have been looking for other solutions:
DbContext.Database.ExecuteSqlCommand: Cannot return records, only output variables
DbSet.FromSQL: Can only be run on a DbSet<T> (basically it needs an entity type)
Right now, all I can think of is to make an entity called Number:
public class Number
{
public long Value;
}
public DbSet<Number> Numbers;
And then do something like this:
Numbers.FromSql("exec GenerateOrderDetailSequencedIds #numberNeeded", numberNeeded)
Aside from the fact that this is very ugly (making an entity out of a native type), I have no table to hook it up to, so I worry it will not work.
Is there any way in EF Core to run a stored procedure and get back a list of numbers?
NOTE: This worked, but was not compatable with BreezeJs (it could not deal with a DbQuery). See my other answer for what I ended up doing.
OrderDetailIdHolder.cs
public class OrderDetailIdHolder
{
public long NewId { get; set; }
}
MyEntitiesContext
public DbQuery<OrderDetailIdHolder> OrderDetailIdHolders { get; set; }
internal List<long> GetOrderDetailIds(int numberOfIdsNeeded)
{
var result = OrderDetailIdHolders.FromSql($"exec Sales.GenerateOrderDetailIds {numberOfIdsNeeded}").ToList();
return result.Select(x=>x.NewId).ToList();
}
This a bit extra complexity for just a list of longs. But it works.
It is important to note that the property (NewId in this case) must match what is returned from the sproc. Also, the type is not a DbSet. It is a DbQuery.
It is also important to note that this is only for EF Core 2.2. EF Core 3 has a different way to do this (Keyless Entity Types)
This is what I ended up using:
public static List<T> SqlQueryList<T>(this DatabaseFacade database, string query, params SqlParameter[] sqlParameters)
{
// TODO: Add a using statement here so we don't leak the connection's resources.
var conn = database.GetDbConnection();
conn.Open();
var command = conn.CreateCommand();
command.CommandText = query;
command.Parameters.AddRange(sqlParameters);
var reader = command.ExecuteReader();
List<T> result = new List<T>();
while (reader.Read())
{
T typedRow;
var row = reader.GetValue(0);
if (typeof(T).IsValueType)
{
typedRow = (T) row;
}
else
{
typedRow = (T)Convert.ChangeType(result, typeof(T));
}
result.Add(typedRow);
}
return result;
}
Called like this:
var numberOfOrderDetailIdsNeededParam = new SqlParameter
{
ParameterName = "#numberOfOrderDetailIdsNeeded",
SqlDbType = SqlDbType.Int,
Direction = ParameterDirection.Input
};
numberOfOrderDetailIdsNeededParam.Value = numberOfOrderDetailIdsNeeded;
var result = contextProvider.Context.Database.SqlQueryList<long>($"exec Sales.GenerateOrderDetailIds #numberOfOrderDetailIdsNeeded", numberOfOrderDetailIdsNeededParam);
I did it this way because it was compatible with BreezeJs for .NET Core. Note that I only really tested this with Value types.

NET Core MongoDb Update/Replace exclude fields

I'm trying to complete a general repository for all of the entities in my application. I Have a BaseEntity with property Id, CreatorId and LastModifiedUserId. Now I'd like to Update a record in a collection, without having to modify the field CreatorId, so I have (from the client) an Entity valorized with some fields updated that I want to update.
Hi have 2 ways:
UpdateOneAsync
ReplaceOneAsync
The repo is created like this:
public class BaseRepository<T> : IRepository<T> where T : BaseEntity
{
public async Task<T> Replace/Update(T entity){...}
}
So it's very hard to use Update(1), since I should retrieve with reflection all the fields of T and exclude the ones that I don't want to update.
With Replace(2) I cannot find a way to specify which fields i should exclude when replacing an object with another. Projectionproperty in FindOneAndReplaceOptions<T>() just excludes the fields on the document that is returned after the update.
Am I missing a way in the replace method to exclude the fields or should I try to retrieve the fields with reflection and use a Update?
I don't know if this solution is ok for you .. what i do is :
Declare in Base Repo a method like
public virtual bool Update(TEntity entity, string key)
{
var result = _collection.ReplaceOne(x => x.Id.Equals(key), entity, new UpdateOptions
{
IsUpsert = false
});
return result.IsAcknowledged;
}
then in your controller when you want to update your entities is there where you set the prop you want to change .. like:
[HttpPut]
[ProducesResponseType(typeof(OrderDTO), 200)]
[ProducesResponseType(400)]
public async Task<ActionResult<bool>> Put([FromBody] OrderDTO value)
{
try
{
if (!ModelState.IsValid) return BadRequest(ModelState);
var orderOnDb = await _orderService.FindAsync(xx => xx.Id == value.Id);
if (orderOnDb == null) return BadRequest(Constants.Error.NOT_FOUND_ON_MONGO);
// SET PROPERTY TO UPDATE (MANUALLY)
orderOnDb.LastUpdateDate = DateTime.Now;
orderOnDb.PaymentMethod = value.PaymentMethod;
orderOnDb.StateHistory = value.StateHistory;
//Save on db
var res = await _orderRepo.UpdateAsync(orderOnDb, orderOnDb.Id);
return res;
}
catch (Exception ex)
{
_logger.LogCritical(ex, ex.Message);
throw ex;
}
}
Hope it helps you!!!

CacheBuilder using guava cache for query resultant

To reduce the DB hits to read the data from DB using the query, I am planning to keep resultant in the cache. To do this I am using guava caching.
studentController.java
public Map<String, Object> getSomeMethodName(Number departmentId, String departmentType){
ArrayList<Student> studentList = studentManager.getStudentListByDepartmentType(departmentId, departmentType);
----------
----------
}
StudentHibernateDao.java(criteria query )
#Override
public ArrayList<Student> getStudentListByDepartmentType(Number departmentId, String departmentType) {
Criteria criteria =sessionFactory.getCurrentSession().createCriteria(Student.class);
criteria.add(Restrictions.eq("departmentId", departmentId));
criteria.add(Restrictions.eq("departmentType", departmentType));
ArrayList<Student> studentList = (ArrayList)criteria.list();
return studentList;
}
To cache the criteria query resultant i started off with building CacheBuilder, like below.
private static LoadingCache<Number departmentId, String departmentType, ArrayList<Student>> studentListCache = CacheBuilder
.newBuilder().expireAfterAccess(1, TimeUnit.MINUTES)
.maximumSize(1000)
.build(new CacheLoader<Number departmentId, String departmentType, ArrayList<Student>>() {
public ArrayList<Student> load(String key) throws Exception {
return getStudentListByDepartmentType(departmentId, departmentType);
}
});
Here I dont know where to put CacheBuilder function and how to pass multiple key parameters(i.e departmentId and departmentType) to CacheLoader and call it.
Is this the correct way of caching using guava. Am I missing anything?
Guava's cache only accepts two type parameters, a key and a value type. If you want your key to be a compound key then you need to build a new compound type to encapsulate it. Effectively it would need to look like this (I apologize for my syntax, I don't use Java that often):
// Compound Key type
class CompoundDepartmentId {
public CompoundDepartmentId(Long departmentId, String departmentType) {
this.departmentId = departmentId;
this.departmentType = departmentType;
}
}
private static LoadingCache<CompoundDepartmentId, ArrayList<Student>> studentListCache =
CacheBuilder
.newBuilder().expireAfterAccess(1, TimeUnit.MINUTES)
.maximumSize(1000)
.build(new CacheLoader<CompoundDepartmentId, ArrayList<Student>>() {
public ArrayList<Student> load(CompoundDepartmentId key) throws Exception {
return getStudentListByDepartmentType(key.departmentId, key.departmentType);
}
});