Background
I am creating a projection from a parent/child relationship that includes a Name property of the parent and a list of the children's Ids.
Code
private class ParentChildInfo
{
public string Name { get; set; }
public List<int> ChildIds { get; set; }
}
var infos = ctx.Masters.Include(m => m.Children).Select(
m => new ParentChildInfo()
{
Name = m.Name,
ChildIds = m.Children.Where(c => c.SomeProp.StartsWith("SpecialValue"))
.Select(c => c.Id).ToList()
}).ToList();
Unfortunately that produced the error
LINQ to Entities does not recognize the method 'System.Collections.Generic.List`1[System.Int32] ToList[Int32]
That lead me to this post, which suggested (in the comments) making the following changes:
private class ParentChildInfo
{
public string Name { get; set; }
public IEnumerable<int> ChildIds { get; set; } // No longer List<int>
}
var infos = ctx.Masters.Include(m => m.Children).Select(
m => new ParentChildInfo()
{
Name = m.Name,
ChildIds = m.Children.Where(c => c.SomeProp.StartsWith("SpecialValue"))
.Select(c => c.Id) // Dropped the .ToList()
}).ToList();
I originally wanted to get lists rather than enumerables because the code that uses the result runs for several minutes, and I did not want to tie up the DbContext that long.
I use the code like this:
using (MyContext ctx = new MyContext())
{
// code from above that populates infoes
}
foreach (var info in infoes)
{
// use info.ChildIds
}
I planned to move the foreach into the using so that I can enumerate the ChildIds, but hit F5 instead and was surprised to see that the code works.
Question
Given that the DbContext is disposed at that point and ChildIds is an IEnumerable<int> rather than List<int>, why exactly can I enumerate ChildIds?
It is because the ToList() of the infos query actually executes the query. So the collection ctx.Masters is enumerated and the projections are populated. Even without the Include it would notice that Master.Children is addressed and emit the SQL join. The implementing type of IEnumerable<int> ChildIds is probably List<int>.
You did .ToList() on the query so the query was executed and all the results are materialized and the connection to the database should be closed. I assume it would not work if you did not have .ToList() since (at least in EF5) the results are being processed in streaming fashion and entities are materialized when requested (i.e. on each iteration in the loop).
Related
I saw somewhere that with the Go MongoDB driver it is possible to save a document with the order number instead of the field name.
They end up with this in the database:
{
"3": "foo",
"10": 1,
"33": 123456
"107": {
"2": "bar",
"1": "foo"
}
}
I like the idea!
So, I tried to find a way to do the same with the MongoDB C# driver.
I have the code below but I am not sure what I should bring from the protobut-net to get the member order number.
var pack = new ConventionPack();
pack.AddMemberMapConvention("numbered", m => m.SetElementName( WHAT TO PUT HERE ));
ConventionRegistry.Register("numbered", pack, type => true);
The SetElementName takes a string parameter.
How can I grab the order number of a member from protobuf-net?
Something like ...Member.Order.ToString()
I don't know if this whole thing is a great idea but I want to test it.
Thanks
-- UPDATE --
Just to add more information. I am using inheritance for my models to use generics.
[BsonDiscriminator("Base", RootClass = true)]
[DataContract]
public abstract class Base
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
[ProtoMember(1)]
public string Id { get; set; }
[BsonDateTimeOptions]
[ProtoMember(2)]
public DateTime CreatedDate { get; private set; } = DateTime.UtcNow;
[BsonDateTimeOptions]
[ProtoMember(3)]
public DateTime UpdatedDate { get; set; } = DateTime.UtcNow;
}
[ProtoContract]
public class Todo : Base
{
[ProtoMember(10)]
public string Title { get; set; }
[ProtoMember(20)]
public string Content { get; set; }
[ProtoMember(30)]
public string Category { get; set; }
}
And I added this line as shown in the protobuf-net documentation:
RuntimeTypeModel.Default[typeof(Base)].AddSubType(42, typeof(Todo));
So with that and what Marc showed to get the member's number, I end up having a custom Convention Class in MongoDB with <T> so I can use it for other objects:
public class NumberedElementNameConvention<T> : ConventionBase, IMemberMapConvention where T : Base
{
public void Apply(BsonMemberMap memberMap)
{
var members = RuntimeTypeModel.Default[typeof(T)].GetFields();
foreach (var member in members)
{
memberMap.SetElementName(member.FieldNumber.ToString());
}
}
}
And the registration of this Convention is done like so:
var pack = new ConventionPack { new NumberedElementNameConvention<Todo>() };
ConventionRegistry.Register("NumberedName", pack, type => true);
After running this I get this error:
Grpc.AspNetCore.Server.ServerCallHandler[6]
Error when executing service method 'CreateOne'.
MongoDB.Bson.BsonSerializationException: The property 'UpdatedDate' of type 'Nnet.Models.Base' cannot use element name '30' because it is already being used by property 'CreatedDate'...
Also, when I run the code below I am expecting to get all members of the Todo object.
var members = RuntimeTypeModel.Default[typeof(Todo)].GetFields();
foreach (var member in members)
{
Console.WriteLine($"{member.FieldNumber}: {member.Member.Name}");
}
However, I am not getting those inherited from the Base object:
❯ dotnet run
10: Title
20: Content
30: Category
The field metadata for protobuf-net is available from the RuntimeTypeModel API, for example:
var members = RuntimeTypeModel.Default[yourType].GetFields();
foreach (var member in members)
{
Console.WriteLine($"{member.FieldNumber}: {member.Member.Name}");
}
The .FieldNumber gives the protobuf field-number, and .Member gives the MemberInfo of the corresponding field or property. You may want to do some level of caching if the m => m.SetElementName( WHAT TO PUT HERE ) is evaluated lots of times for the same m, so you don't perform unnecessary work - but: before you do, just add some logging to the lambda first, and see how often it gets called: if it isn't too often, maybe don't worry about it.
Note that there is also a lookup on MetaType that allows query by MemberInfo:
var member = RuntimeTypeModel.Default[yourType][memberInfo];
Re the edit; in this region:
var members = RuntimeTypeModel.Default[typeof(T)].GetFields();
foreach (var member in members)
{
memberMap.SetElementName(member.FieldNumber.ToString());
}
I believe you're meant to identify the relevant field from memberMap - i.e. in this context you're only talking about one field at the time; I suspect what is happening is that for each member in turn you're changing the element name multiple times, leaving it at the last protobuf field defined.
Separately, there's a complication of inheritance; protobuf-net doesn't implement inheritance in a flat way - instead, the base type is also expected to be a [ProtoContract] and is meant to define a [ProtoInclude(...)] for each derived type; the field numbers are type-specific, meaning: both the base type and the derived type can legally have a field 1. If you need to describe inheritance, and you are determined to use protobuf-net's model, then you would need to handle this; for example, you could use the [ProtoInclude(...)] number as a prefix on each, so Base.Id is "1", and if we imagine that Todo has field 5 in the [ProtoInclude(...)], then Todo.Title could be "5.10".
Alternatively: if you're not actively using protobuf-net: maybe just use your own attribute for the numbers? or there's usually an inbuilt attribute that the serializer you've chosen would use directly.
Okay now! So after a some investigation I end up with this simple way to do it with Marc's help. In MongoDB instead of using attributes to decorate models and its properties, it is possible to use code within BsonClassMap. Within that class I add the foreach loop that Marc provided and the right parameters, we can now have numbers instead names.
On the Client side and Server side it is this same code:
//Base Model ClassMap
BsonClassMap.RegisterClassMap<Base>(cm =>
{
cm.AutoMap();
foreach (var member in RuntimeTypeModel.Default[typeof(Base)].GetFields())
{
cm.MapMember(typeof(Base).GetMember(member.Member.Name)[0])
.SetElementName(member.FieldNumber.ToString())
.SetOrder(member.FieldNumber);
}
});
//Todo Model ClassMap
BsonClassMap.RegisterClassMap<Todo>(cm =>
{
cm.AutoMap();
foreach (var member in RuntimeTypeModel.Default[typeof(Todo)].GetFields())
{
cm.MapMember(typeof(Todo).GetMember(member.Member.Name)[0])
.SetElementName(member.FieldNumber.ToString())
.SetOrder(member.FieldNumber);
}
});
it's a little ugly but you can rework it.
One thing to note is that MongoDB has the control over the Id. In the database anything that represent the object id become _id. Same thing when you insert a new document in the database a _t field is added if you use Discriminator (I am not sure if it's full related). Basically, every member beginning with a underscore is reserved. See the image below after running de code:
You can refer to the question above in the update section to see if this result represent the models with the given orders (it does).
Here is the code I use for insertion and queries:
// INSERT
var client = channel.CreateGrpcService<IBaseService<Todo>>();
var reply = await client.CreateOneAsync(
new Todo
{
Title = "Some Title"
}
);
// FIND BY ID
var todoId = new UniqueIdentification { Id = "613c110a073055f0d87a0e27"};
var res = await client.GetById(todoId);
// FIND ONE BY QUERY FILTER REQUEST
...
var filter = Builders<Todo>.Filter.Eq("10", "Some Title");
var filterString = filter.Render(documentSerializer, serializerRegistry);
...
The last one above it's a query with the number ("10") of the property Title. But it's possible in the same way to query with the property name, like so:
// FIND ONE BY QUERY FILTER REQUEST
...
var filter = Builders<Todo>.Filter.Eq(e => e.Title, "Some Title");
var filterString = filter.Render(documentSerializer, serializerRegistry);
...
What is great with this approach is that these BsonClassMap are called once on the Client or/and Server when they are initiated.
I just realize that this might not be a good idea because it is going to be painful to prevent collision between numbers. The order numbers in the code below is possible:
[BsonDiscriminator("Base", RootClass = true)]
[DataContract]
public abstract class Base
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
[ProtoMember(1)]
public string Id { get; set; }
[BsonDateTimeOptions]
[ProtoMember(2)]
public DateTime CreatedDate { get; private set; } = DateTime.UtcNow;
[BsonDateTimeOptions]
[ProtoMember(3)]
public DateTime UpdatedDate { get; set; } = DateTime.UtcNow;
}
[ProtoContract]
public class Todo : Base
{
[ProtoMember(1)]
public string Title { get; set; }
[ProtoMember(2)]
public string Content { get; set; }
[ProtoMember(3)]
public string Category { get; set; }
}
but there is going to be three collisions if the foreach loop runs.
Yeah... :/
This is where Marc's second solution comes in, where you put a prefix... I am going to keep the name convention by default.
Cheers!
I have a Category table and it has a Parent Category, I try to iterate over all the categories and get the parents categories with it's Inverse Parent but some of them returns without the inverse parents from unknown reason.
Categories.cs
public partial class Categories
{
public Categories()
{
InverseParent = new HashSet<Categories>();
}
public int Id { get; set; }
public int? ParentId { get; set; }
public DateTime CreateDate { get; set; }
public bool? Status { get; set; }
public virtual Categories Parent { get; set; }
public virtual ICollection<Categories> InverseParent { get; set; }
}
This is how I try to iterate them to create a select list items:
var parentCategories = await _context.Categories.
Include(x => x.Parent).
Where(x => x.Status == true).
Where(x => x.Parent != null).
Select(x => x.Parent).
Distinct().
ToListAsync();
foreach (var parent in parentCategories)
{
SelectListGroup group = new SelectListGroup() { Name = parent.Id.ToString() };
foreach (var category in parent.InverseParent)
{
categories.Add(new SelectListItem { Text = category.Id.ToString(), Value = category.Id.ToString(), Group = group });
}
}
So the problem is that some of my parent categories returns all their children categories and some don't and I don't why.
There are several issues with that code, all having some explaination in the Loading Related Data section of the documentation.
First, you didn't ask EF Core to include InverseParent, so it's more logically to expect it to be always null.
What you get is a result of the following Eager Loading behavior:
Tip
Entity Framework Core will automatically fix-up navigation properties to any other entities that were previously loaded into the context instance. So even if you don't explicitly include the data for a navigation property, the property may still be populated if some or all of the related entities were previously loaded.
Second, since the query is changing it's initial shape (Select, Disctinct), it's falling into Ignored Includes category.
With that being said, you should build the query other way around - starting directly with parent categories and including InverseParent:
var parentCategories = await _context.Categories
.Include(x => x.InverseParent)
.Where(x => x.InverseParent.Any(c => c.Status == true)) // to match your query filter
.ToListAsync();
While you are including Include(x => x.Parent), you don't seem to do the same for InverseParent. This might affect your results exactly the way you describe. Would including it fix it?
parentCategories = await _context.Categories.
Include(x => x.Parent).
Include(x => x.InverseParent).
Where(x => x.Status == true).
Where(x => x.Parent != null).
Select(x => x.Parent).
Distinct().
ToListAsync();
foreach (var parent in parentCategories)
{
SelectListGroup group = new SelectListGroup() { Name = parent.Id.ToString() };
foreach (var category in parent.InverseParent)
{
categories.Add(new SelectListItem { Text = category.Id.ToString(), Value = category.Id.ToString(), Group = group });
}
}
UPD: Since you are selecting x => x.Parent anyway it might be necessary to use ThenInclude() method instead.
So I partially followed from an SO answer on how to store a property with array datatype in Entity Framework. What I didn't follow on that answer is setting the string InternalData to be private instead of public as I find it a code smell if it is set to public (not enough reputation to comment there yet).
I also managed to map the private property in entity framework from this blog.
When I perform CR (create, read) from that entity, all goes well. However, when my LINQ query has a where clause using that property with array datatype, it says that "System.NotSupportedException: 'The specified type member is not supported in LINQ to Entities. Only initializers, entity members, and entity navigation properties are supported.'".
How to work around on this? Here are the relevant code blocks:
public class ReminderSettings
{
[Key]
public string UserID { get; set; }
[Column("RemindForPaymentStatus")]
private string _remindForPaymentStatusCSV { get; set; }
private Status[] _remindForPaymentStatus;
[NotMapped]
public Status[] RemindForPaymentStatus
{
get
{
return Array.ConvertAll(_remindForPaymentStatusCSV.Split(','), e => (Status)Enum.Parse(typeof(Status), e));
}
set
{
_remindForPaymentStatus = value;
_remindForPaymentStatusCSV = String.Join(",", _remindForPaymentStatus.Select(x => x.ToString()).ToArray());
}
}
public static readonly Expression<Func<ReminderSettings, string>> RemindForPaymentStatusExpression = p => p._remindForPaymentStatusCSV;
}
public enum Status
{
NotPaid = 0,
PartiallyPaid = 1,
FullyPaid = 2,
Overpaid = 3
}
protected override void OnModelCreating(DbModelBuuilder modelBuilder)
{
modelBuilder.Entity<ReminderSettings>().Property(ReminderSettings.RemindForPaymentStatusExpression);
}
//This query will cause the error
public IEnumerable<ReminderSettings> GetReminderSettingsByPaymentStatus(Status[] statusArray)
{
var query = ApplicationDbContext.ReminderSettings.Where(x => x.RemindForPaymentStatus.Intersect(statusArray).Any());
return query.ToList(); //System.NotSupportedException: 'The specified type member 'RemindForPaymentStatus' is not supported in LINQ to Entities. Only initializers, entity members, and entity navigation properties are supported.'
}
Entity Framework can not translate LINQ expressions to SQL if they access a property that is annotated as [NotMapped]. (It also can not translate if the property contains custom C# code in its getter/setter).
As a quick (but potentially low performance) workaround, you can execute the part of the query that does not cause problems, then apply additional filtering in-memory.
// execute query on DB server and fetch items into memory
var reminders = dbContext.ReminderSettings.ToList();
// now that we work in-memory, LINQ does not need to translate our custom code to SQL anymore
var filtered = reminders.Where(r => r.RemindForPaymentStatus.Contains(Status.NotPaid));
If this causes performance problems, you have to make the backing field of your NotMapped property public and work directly with it.
var filtered = dbContext.ReminderSettings
.Where(r => r._remindForPaymentStatusCSV.Contains(Status.NotPaid.ToString("D"));
Edit
To handle multiple Status as query parameters, you can attach Where clauses in a loop (which behaves like an AND). This works as long as your Status enum values are distinguishable (i.e. there is no Status "11" if there is also a Status "1").
var query = dbContext.ReminderSettings.Select(r => r);
foreach(var statusParam in queryParams.Status) {
var statusString = statusParam.ToString("D");
query = query.Where(r => r._remindForPaymentStatusCSV.Contains(statusString));
}
var result = query.ToArray();
Supposing I have three EF entity classes:
public class Person {
...
public ICollection Vehicles { get; set; }
}
public class Vehicle {
...
public Person Owner { get; set; }
public CarModel ModelInfo { get; set; }
}
public class CarModel {
...
// properties for make, model, color, etc
}
The Person.Vehicles property is lazy-loaded.
Supposing I have a Person instance already loaded and I want to load its Vehicle collection property such that it also includes the related ModelInfo property.
So I have this:
void LoadVehiclesAndRelated(MyDbContext dbContext, Person person)
{
dbContext.Entry( person )
.Collection( p => p.Vehicles )
.Query()
.Include( v => v.ModelInfo )
.Load();
}
Used like so:
using( MyDbContext dbContext = ... ) {
Person p = GetPerson( 123 );
LoadVehiclesAndRelated( dbContext, p );
}
foreach(Vehicle v in p.Vehicles) {
Console.WriteLine( v.ModelInfo );
}
However when I do this, I get an exception at runtime when it first evaluates the p.Vehicles expression because the property is actually empty (so it wants to load it) but the DbContext is now disposed.
When the .Load() call was made (inside LoadVehiclesAndRelated() I saw the SQL being executed against the server (in SQL Server Profiler) but the collection property remains empty.
How can I then load the property and with the Included sub-properties?
Annoyingly, this scenario is not mentioned in the MSDN guide for explicit-loading: https://msdn.microsoft.com/en-us/data/jj574232.aspx
Looks like calling .Query().Load() is not the same as calling DbCollectionEntry.Load directly, and the important difference is that the former does not set the IsLoaded property, which then is causing triggering lazy load later. Most likely because as explained in the link, the former is intended to be used for filtered (partial) collection load scenarios.
Shortly, to fix the issue, just set IsLoaded to true after loading the collection:
var entry = dbContext.Entry(person);
var vehicleCollection = entry.Collection(p => p.Vehicles);
vehicleCollection.Query()
.Include( v => v.ModelInfo )
.Load();
vehicleCollection.IsLoaded = true;
P.S. For the sake of correctness, this behavior is sort of mentioned at the end of the Applying filters when explicitly loading related entities section:
When using the Query method it is usually best to turn off lazy loading for the navigation property. This is because otherwise the entire collection may get loaded automatically by the lazy loading mechanism either before or after the filtered query has been executed.
I am using Service Stack as my system's API and I'm using Entity Framework to get data from my SQL Server DataBase. Although, I cannot retrieve any data from a list of objects generated by entity framework.
[Route("/getInterventions","GET")]
public class GetInterventions
{
}
public class GetInterventionsResponse
{
public List<Intervention> interventions { get; set; }
public ResponseStatus ResponseStatus { get; set; }
}
public class GetInterventionsService : Service
{
public object Any(GetInterventions request)
{
using (var dbConnection = new operationsContext())
{
List<Intervention> dbItems = dbConnection.Interventions.ToList();
return new GetInterventionsResponse{
interventions = dbItems
};
}
}
}
From the client side I get:
The ObjectContext instance has been disposed and can no longer be used for "operations"(name of db) that require a connection.
So with this error I can verify that the problem as to do with the list that acts like a "virtual" list, and its objects are not returned to the client side, but are passed as a reference or something like that.
So how can I deep copy this list and retrieve a clone of it?
Thanks anyways
It looks like the list is no longer accessible when the context gets disposed, possibly because the variable was defined within the scope of the context. Try defining dbItems outside of the using statement:
public object Any(GetInterventions request)
{
List<Intervention> dbItems;
using (var dbConnection = new operationsContext())
{
dbItems = dbConnection.Interventions.ToList();
}
return new GetInterventionsResponse{
interventions = dbItems
};
}
Also, you may run into this issue if you are expecting navigation properties of Interventions to be loaded, which they will not be with your code because EF uses lazy loading. For example, if Intervention has a Person navigation property, you would need to include that to have it be available. Like this:
dbItems = dbConnection.Interventions.Include(x => x.Persons).ToList();
Edit based on comment below:
You can also have includes multiple levels deep like this:
dbItems = dbConnection.Interventions.Include(x => x.Persons.Car).ToList();
or for a nested list...
dbItems = dbConnection.Interventions.Include(x => x.Persons.Select(y => y.Cars).ToList();
or for multiple navigation properties...
dbItems = dbConnection.Interventions.Include(x => x.Persons)
.Include(x => x.Cars).ToList();