How to loop through dbcontext all dbset in Entity Framework Core to get count? - entity-framework-core

I have 20 Dbsets in my context which I want to get the row count for each dbset to make sure all dbset row count is 0. To get the count for one dbset, this is my code:
var person = context.Persons.Count();
Is there a way to loop through the context, get the count for each dbset dynamically?

There is solution. Usage is simple:
var tablesinfo = ctx.GetTablesInfo();
if (tablesinfo != null)
{
var withRecords = tablesinfo
.IgnoreQueryFilters()
.Where(ti => ti.RecordCount > 0)
.ToArray();
}
Extension returns IQueryable<TableInfo> and you can reuse this query later. Probably you will need to filter out Views, but I think you can handle that. Note that IgnoreQueryFilters can be important if you have Global Query Filters defined.
What extension do:
It scans Model for entity types registered for particular DbContext and generates big Concat of Count queries. Here we have to do that via grouping by constant value.
Schematically it will generate the following LINQ query:
var tablesinfo =
ctx.Set<Entity1>.GroupBy(e => 1).Select(g => new TableInfo { TableName = "Entity1", RecordCount = g.Count()})
.Concat(ctx.Set<Entity2>.GroupBy(e => 1).Select(g => new TableInfo { TableName = "Entity2", RecordCount = g.Count()}))
.Concat(ctx.Set<Entity3>.GroupBy(e => 1).Select(g => new TableInfo { TableName = "Entity3", RecordCount = g.Count()}))
...
Which wll be converted to the following SQL:
SELECT "Entity1" AS TableName, COUNT(*) AS RecordCount FROM Entity1
UNION ALL
SELECT "Entity2" AS TableName, COUNT(*) AS RecordCount FROM Entity2
UNION ALL
SELECT "Entity3" AS TableName, COUNT(*) AS RecordCount FROM Entity3
...
Implementation:
public static class QueryableExtensions
{
public class TableInfo
{
public string TableName { get; set; } = null!;
public int RecordCount { get; set; }
}
public static IQueryable<TableInfo> GetTablesInfo(this DbContext ctx)
{
Expression query = null;
IQueryProvider provider = null;
var ctxConst = Expression.Constant(ctx);
var groupingKey = Expression.Constant(1);
// gathering information for MemberInit creation
var newExpression = Expression.New(typeof(TableInfo).GetConstructor(Type.EmptyTypes));
var tableNameProperty = typeof(TableInfo).GetProperty(nameof(TableInfo.TableName));
var recordCountProperty = typeof(TableInfo).GetProperty(nameof(TableInfo.RecordCount));
foreach (var entityType in ctx.Model.GetEntityTypes())
{
var entityParam = Expression.Parameter(entityType.ClrType, "e");
var tableName = entityType.GetTableName();
// ctx.Set<entityType>()
var setQuery = Expression.Call(ctxConst, nameof(DbContext.Set), new[] {entityType.ClrType});
// here we initialize IQueryProvider, which is needed for creating final query
provider ??= ((IQueryable) Expression.Lambda(setQuery).Compile().DynamicInvoke()).Provider;
// grouping paraneter has generic type, we have to specify it
var groupingParameter = Expression.Parameter(typeof(IGrouping<,>).MakeGenericType(typeof(int), entityParam.Type), "g");
// g => new TableInfo { TableName = "tableName", RecordCount = g.Count() }
var selector = Expression.MemberInit(newExpression,
Expression.Bind(tableNameProperty, Expression.Constant(tableName)),
Expression.Bind(recordCountProperty,
Expression.Call(typeof(Enumerable), nameof(Enumerable.Count), new[] {entityParam.Type}, groupingParameter)));
// ctx.Set<entityType>.GroupBy(e => 1)
var groupByCall = Expression.Call(typeof(Queryable), nameof(Queryable.GroupBy), new[]
{
entityParam.Type,
typeof(int)
},
setQuery,
Expression.Lambda(groupingKey, entityParam)
);
// ctx.Set<entityType>.GroupBy(e => 1).Select(g => new TableInfo { TableName = "tableName", RecordCount = g.Count()}))
groupByCall = Expression.Call(typeof(Queryable), nameof(Queryable.Select),
new[] {groupingParameter.Type, typeof(TableInfo)},
groupByCall,
Expression.Lambda(selector, groupingParameter));
// generate Concat if needed
if (query != null)
query = Expression.Call(typeof(Queryable), nameof(Queryable.Concat), new[] {typeof(TableInfo)}, query,
groupByCall);
else
query = groupByCall;
}
// unusual situation, but Model can have no registered entities
if (query == null)
return null;
return provider.CreateQuery<TableInfo>(query);
}
}

Related

How do check for duplicate before AddRange?

This is my sample model,
Name
Age
I have a list of records for which I will use Entity Framework Core AddRange. This is my sample data, say list 1
John, 21
Mike, 18
Rick, 19
Alex, 20
I have another query which is a list of the existing records, say list 2.
John, 21
Alex, 20
I want to compare these 2 lists to get what exists in list 1 but not in list 2.
Mike, 18
Rick, 19
I tried using LINQ except but it can't be used for complex types. I also tried to use .Any like below but it returns nothing.
var result = list1.Where(x=> !list2.Any(y=>x.Name == y.Name));
I do not want to check and insert one record at one time. I want to insert by list of objects using AddRange.
Is there any Nuget package that can easily get this?
For not so big amount of records, you can use the following extension:
var nonExistent = await ctx.Some.GetNonExistentAsync(list2, x => x.Name);
Or with complex key:
var nonExistent = await ctx.Some.GetNonExistentAsync(list2,
x => new { x.Name, x.Other });
It will generate effective SQL to check which records do not exist.
And implementation:
public static class QueryableExtensions
{
public static async Task<List<T>> GetNonExistentAsync<T, TKey>(this IQueryable<T> query, IEnumerable<T> records,
Expression<Func<T, TKey>> keySelector, CancellationToken cancellationToken = default)
{
var recordsList = records.ToList();
var keySelectorCompiled = keySelector.Compile();
var predicate = BuildPredicate(recordsList, keySelector, keySelectorCompiled);
var existent = (await query
.Where(predicate)
.Select(keySelector)
.ToListAsync(cancellationToken))
.ToHashSet();
var result = recordsList.Where(r => !existent.Contains(keySelectorCompiled(r))).ToList();
return result;
}
private static Expression<Func<T, bool>> BuildPredicate<T, TKey>(IEnumerable<T> records, Expression<Func<T, TKey>> keySelector, Func<T, TKey> keySelectorCompiled)
{
var members = CollectMembers(keySelector.Body).ToList();
if (members.Count == 0)
throw new InvalidOperationException("No key found");
Expression? predicate = null;
if (members.Count == 1 && keySelector.Body.NodeType == ExpressionType.MemberAccess)
{
// we can use Contains
var recordsSequence = Expression.Constant(records.Select(keySelectorCompiled));
predicate = Expression.Call(typeof(Enumerable), nameof(Enumerable.Contains), new[] { typeof(TKey) },
recordsSequence, keySelector.Body);
}
else
{
foreach (var record in records)
{
Expression? subPredicate = null;
var recordExpression = Expression.Constant(record);
foreach (var m in members)
{
var memberExpression =
ReplacingExpressionVisitor.Replace(keySelector.Parameters[0], recordExpression, m);
var equality = Expression.Equal(m, memberExpression);
subPredicate = subPredicate == null ? equality : Expression.AndAlso(subPredicate, equality);
}
if (subPredicate == null)
throw new InvalidOperationException(); // should never happen
predicate = predicate == null ? subPredicate : Expression.OrElse(predicate, subPredicate);
}
}
predicate ??= Expression.Constant(false);
return Expression.Lambda<Func<T, bool>>(predicate, keySelector.Parameters);
}
private static IEnumerable<Expression> CollectMembers(Expression expr)
{
switch (expr.NodeType)
{
case ExpressionType.New:
{
var ne = (NewExpression)expr;
foreach (var a in ne.Arguments)
foreach (var m in CollectMembers(a))
{
yield return m;
}
break;
}
case ExpressionType.MemberAccess:
yield return expr;
break;
default:
throw new InvalidOperationException();
}
}
}

Filters not working in my repository in ASP.NET Core

I have these parameters in a class:
public class UserParams
{
public string Gender {get; set;}
public int MinAge {get; set;} = 1;
public int MaxAge {get; set;} = 19;
}
The query is done in the repository as shown below. First is to query for the child sex or gender and the second is to query for the child sex or gender
var query = _context.Children.AsQueryable();
query = query.Where(c => c.Sex == userParams.Gender);
var minchildDob = DateTime.Today.AddYears(-userParams.MaxAge - 1);
var maxchildDob = DateTime.Today.AddYears(-userParams.MinAge);
query = query.Where(u => u.DateOfBirth >= minchildDob && u.DateOfBirth <= maxchildDob);
return await PagedList<Child>.CreateAsync(query.AsNoTracking(), userParams.PageNumber, userParams.PageSize);
The gender filter returns empty array of children and the minchildDob and maxchildDob too not working
if (!string.IsNullOrEmpty(temp.Gender))
{
all = all.Where(u => new[] { "men", "women" }.Contains(u.sex));
//all = all.Where(t => t.sex == temp.Gender);
}
=======================Update=======================
var temp = new UserParams();
temp.Gender = "men";
var minchildDob = DateTime.Today.AddYears(-temp.MaxAge - 1);
var maxchildDob = DateTime.Today.AddYears(-temp.MinAge);
IEnumerable<Table> all = from m in _context.data
select m;
_logger.LogError("all data");
foreach (var item in all)
{
_logger.LogError(item.name);
}
_logger.LogError("============================================");
if (!string.IsNullOrEmpty(temp.Gender)) {
all = all.Where(t => t.sex == temp.Gender);
}
_logger.LogError("filter gender");
foreach (var item in all) {
_logger.LogError(item.name);
}
_logger.LogError("============================================");
if (temp.MaxAge > 0) {
all = all.Where(t => t.birthday >= minchildDob && t.birthday <= maxchildDob);
}
_logger.LogError("filter age");
foreach (var item in all)
{
_logger.LogError(item.name);
}
_logger.LogError("============================================");

EF Core - how to select count from child table based on foreign key

TableB has a field TableAId which is linked to the Id of TableA. I want to select the count from TableB based on TableAId like -
SELECT *,
(SELECT COUNT(*) FROM tableB where tableB.TableAId = tableA.Id) as count
FROM tableA
So far I have the code:
var data = _context.TableA.AsQueryable();
...
data = data.Select(l => l.TableAId= p.Id).Count();
But on the last line, p is not recognized as a variable.
How do I make this work?
EDIT :
my original query is quiet complex and already filtering data
var data = _context.TableA.AsQueryable();
data = data.Include(p => p.SomeClassA)
.Include(p => p.SomeClassB);
data = data.Where(p => p.Id == somevalue);
data = data.Where(p => p.SomeClassA.Name.Contains(someothervalue));
data = data.Where(p => p.SomeClassA.SomeField.Contains(yetanothervalue));
I tried adding this but it cannot compile
(TableAId & Count do not exist):
data = data.Join(
_context.TableB,
groupByQuery => groupByQuery.TableAId ,
TableA => TableA.Id,
(groupByQuery, TableAItem) => new
{
TableAId = groupByQuery.Id,
Count = groupByQuery.Count,
TableAItem = TableAItem
}
);
If you are just interested in count, then do the following:
var data = _context.TableB.AsQueryable();
var groupByCountQuery = data.GroupBy(a=>a.TableAId, (tableAId, tableBItems) => new
{
TableAId = tableAId,
Count = tableBItems.Count()
});
var result = groupByCountQuery.ToList(); // or change to ToListAsync()
This will give you the count based on TableAId.
If you need the tableA items as well in the result, following ca be done:
var groupByCountQuery = data.GroupBy(a=>a.TableAId, (tableAId, tableBItems) => new
{
TableAId = tableAId,
Count = tableBItems.Count()
}).Join(_context.TableA,
groupByQuery => groupByQuery.TableAId,
tableA => tableA.Id,
(groupByQuery , tableA) => new {
TableAId = groupByQuery.TableAId,
Count = groupByQuery.Count,
TableAItem = tableA
} );
Let's say TableA has properties - Id, FieldA1 and FieldA2.
First, you have to include TableB in your query so that you can take its count, like -
var data = _context.TableA.Include(p=> p.TableB).AsQueryable();
Then in the Select method you have to create a new object with TableA's properties and TableB's count, like -
var list = data.Select(p =>
new
{
Id = p.Id,
A1 = p.FieldA1,
A2 = p.FieldA2,
Count = p.OrderLines.Count
}).ToList();
Notice, this is assigned to a new variable list. That is because it does not return a list of TableA, it returns a list of an anonymous object with properties - Id, A1, A2 and Count. Therefore, you cannot assign it to the previously declared data variable, because data is of type IQueryable<TableA>.
Alternatively, you can declare a class to hold the data values, like -
public class MyData
{
public int Id { get; set; }
public string A1 { get; set; }
public string A2 { get; set; }
public int Count { get; set; }
}
and use it like -
var list = data.Select(p =>
new MyData
{
Id = p.Id,
A1 = p.FieldA1,
A2 = p.FieldA2,
Count = p.OrderLines.Count
}).ToList();
If you want to start from TableA, you can use the following linq query:
var data = _context.TableAs.AsQueryable();
var x = (from a in data
join b in _context.TableBs on a.Id equals b.TableAId
group a.TableB by a.Id into g
select new
{
TableAId = g.Key,
TableBItem = g.FirstOrDefault(),
Count = g.Count()
}).ToList();
Result:

Reuse DTO mappings across both single and list variants

We regularily write extension methods like this that convert from Database objects to DTO objects for use elsewhere in our system.
As you can see in the example below, the actual mapping code is repeated. Is it possible to write a reusable select mapping that can be used in both of these methods?
public static async Task<List<Group>> ToCommonListAsync(this IQueryable<DataLayer.Models.Group> entityGroups)
{
var groups =
await entityGroups.Select(
g =>
new Group()
{
Id = g.Id,
AccountId = g.AccountId,
Name = g.Name,
ParentId = g.ParentId,
UserIds = g.GroupUserMappings.Select(d => d.UserId).ToList()
}).ToListAsync();
return groups;
}
public static async Task<Group> ToCommonFirstAsync(this IQueryable<DataLayer.Models.Group> entityGroups)
{
var group =
await entityGroups.Select(
g =>
new Group()
{
Id = g.Id,
AccountId = g.AccountId,
Name = g.Name,
ParentId = g.ParentId,
UserIds = g.GroupUserMappings.Select(d => d.UserId).ToList()
}).FirstOrDefaultAsync();
return group;
}
You could move your mapping/projection code out into a variable like this:
public static class Extensions
{
private static readonly Expression<Func<DataLayer.Models.Group, Group>> Projection = g =>
new Group
{
Id = g.Id,
AccountId = g.AccountId,
Name = g.Name,
ParentId = g.ParentId,
UserIds = g.GroupUserMappings.Select(d => d.UserId).ToList()
};
public static async Task<List<Group>> ToCommonListAsync(this IQueryable<DataLayer.Models.Group> entityGroups)
{
return await entityGroups.Select(Projection).ToListAsync();
}
public static async Task<Group> ToCommonFirstAsync(this IQueryable<DataLayer.Models.Group> entityGroups)
{
return await entityGroups.Select(Projection).FirstOrDefaultAsync();
}
}

LINQ to Entity cannot use System.Object.GetValue

Below find a method that does not work. We fail on the line query.Select(...
Below that find a method with hard coded object property names which does work. But, this method is obviously not dynamic, nor flexible. There may be many properties of a Customer I may wish to search on.
The error string is at bottom. I get it that somehow the LINQ to Entity is unable to deal with conversion of GetValue to some sort of TSQL. Would anyone know of how I might code this up?
public List<Customer> GetForQuery(params Tuple<string, string>[] keyValuePairs) {
using (var db = new DBEntities()) {
var availableProperties = typeof(Customer).GetTypeInfo().DeclaredProperties.ToList();
var query = db.Customers.Select(c => c);
foreach (Tuple<string, string> pair in keyValuePairs) {
PropertyInfo pi = availableProperties.First(p => p.Name.Equals(pair.Item1));
if (pi == null)
continue;
query = query.Where(u => pi.GetValue(u, null).ToString().StartsWith(pair.Item2));
}
var results = query.Select(c => c).ToList();
return results;
}
}
How I might call the above:
CustomerController custController = new CustomerController();
List<Customer> results = custController.GetForQuery(Tuple.Create<string, string>("FName", "Bob" ));
The working fixed method:
public List<Customer> GetForQuery(string firstName = "", string lastName = "", string phoneNumber = "") {
using (var db = new DBEntities()) {
var query = db.Customers.Select(c => c);
if (firstName.HasContent())
query = query.Where(u => u.FName.StartsWith(firstName));
if (lastName.HasContent())
query = query.Where(u => u.LName.StartsWith(lastName));
if (phoneNumber.HasContent())
query = query.Where(u => u.EveningPhone.StartsWith(phoneNumber));
var results = query.Select(c => c).ToList();
return results;
}
}
ERROR:
LINQ to Entities does not recognize the method 'System.Object GetValue(System.Object, System.Object[])' method, and this method cannot be translated into a store expression.