I'm trying to retrieve a list of missing Ids in database from another list of Ids in Entity Framework Core.
Is there a way to get this call in one line?
public static async Task<IEnumerable<TKey>> GetMissingIds<T, TKey>(
this IQueryable<T> db, IEnumerable<TKey> ids)
where T : BaseEntity<TKey>
{
var existingIds = await db
.AsNoTracking()
.Where(entity => ids.Contains(entity.Id))
.Select(entity => entity.Id)
.ToListAsync();
return ids.Except(existingIds);
}
EF Core supports only Contains with local collections (with small exceptions), so there is no effective way to retrieve Ids which are not present in database via LINQ Query.
Anyway there is third-party extensions which can do that linq2db.EntityFrameworkCore (note that I'm one of the creators).
Using this extension you can join local collection to LINQ query:
public static Task<IEnumerable<TKey>> GetMissingIds<T, TKey>(
this IQueryable<T> query, IEnumerable<TKey> ids, CabcellationToken cancellationToken = default)
where T : BaseEntity<TKey>
{
// we need context to retrieve options and mapping information from EF Core
var context = LinqToDBForEFTools.GetCurrentContext(query) ?? throw new InvalidOperationException();
// create linq2db connection
using var db = context.CreateLinqToDbConnection();
var resultQuery =
from id in ids.AsQueryable(db) // transform Ids to queryable
join e in query on id equals e.Id into gj
from e in gj.DefaultIfEmpty()
where e == null
select id;
// there can be collision with EF Core async extensions, so use ToListAsyncLinqToDB
return resultQuery.ToListAsyncLinqToDB(cancellationToken);
}
This is the sample of generated query:
SELECT
[id].[item]
FROM
(VALUES
(10248), (10249), (10250), (10251), (10252), (10253), (10254),
(10255), (10256), (10257), (10023)
) [id]([item])
LEFT JOIN (
SELECT
[e].[OrderID] as [e]
FROM
[Orders] [e]
) [t1] ON [id].[item] = [t1].[e]
WHERE
[t1].[e] IS NULL
Related
In my application I try to execute a Join query using EntityFramework, Repository Pattern, It is throwing bellow error. What is the problem in the link query? Let me explain in details
Error Description
Unable to create a constant value of type 'Anonymous type'. Only
primitive types or enumeration types are supported in this context
Initialization
_repository = new GenericRepository<WeeklyEntry>();
_repositoryGroup = new GenericRepository<Group>();
_repositoryGroupMember = new GenericRepository<GroupMember>();
Fetching Logic
var groups = _repositoryGroup.GetAll().OrderBy(o => o.ID)
.Select(s => new { s.ID, s.Name }).ToList();
var groupMembers = _repositoryGroupMember.GetAll().OrderBy(o => o.ID)
.Select(s => new { s.GroupID, s.ID, s.Name })
.ToList();
Main Query [Not Working]
var results = (from we in _repository.GetAll()
join g in groups on we.GroupID equals g.ID into grpjoin
from g in grpjoin.DefaultIfEmpty()
join gm in groupMembers on we.DepositedByMemberID equals gm.ID into gmjoin
from gm in gmjoin.DefaultIfEmpty()
where gm.GroupID == g.ID
select new
{
GroupID = g.ID,
GroupName = g.Name,
MemberID = grpmresult.ID,
grpmresult.Name,
we.ID
}).ToList();
To try to achieve bellow SQL Query
select w.GroupID, g.Name, gm.Name, w.ID
from [dbo].[WeeklyEntry] as w
left outer join [dbo].[Group] as g on g.ID = w.GroupID
left outer join [dbo].[GroupMember] as gm on gm.GroupID = g.ID
AND gm.ID = w.DepositedByMemberID
order by w.GroupID
Strange Findings
If I include .ToList(); with each query like from we in _repository.GetAll().ToList() the entire query will work & give expected result without Any ERROR!!!
So if I convert each query return type to In-memory Or IEnumerable<> it is working as expected without any error but IQueryable<> query not working as expected.
New Code Snippet [Working]
var results = (from we in _repository.GetAll().ToList()
join g in groups on we.GroupID equals g.ID into grpjoin
from g in grpjoin.DefaultIfEmpty()
join gm in groupMembers on we.DepositedByMemberID equals gm.ID into gmjoin
from gm in gmjoin.DefaultIfEmpty()
where gm.GroupID == g.ID
select new {...}.ToList();
You can't join a database table with an in-memory collection (in your case, List):
Unable to create a constant value of type Only primitive types or enumeration types are supported in this context
You're converting these to in-memory collections by calling ToList:
var groups = _repositoryGroup.GetAll().OrderBy(o => o.ID)
.Select(s => new { s.ID, s.Name }).ToList();
var groupMembers = _repositoryGroupMember.GetAll().OrderBy(o => o.ID)
.Select(s => new { s.GroupID, s.ID, s.Name })
.ToList();
which you then try and join in the next bit of code.
If you simply remove those ToList calls, the join should work (you can keep the one for the final result, if you prefer it).
Remember that IEnumerable is lazy and will only actually run a SQL query if you "execute it" by iterating (usually via a foreach loop or some function like ToList).
Let me explain!
You have a GenericRepository<T> class, that is something like this:
public class GenericRepository<T>
{
MyDbContext dbContext;
public GenericRepository()
{
dbContext = new MyDbContext();
}
public IQueryable<T> GetAll()
{
// whatever
}
}
and then you have:
_repository = new GenericRepository<WeeklyEntry>();
_repositoryGroup = new GenericRepository<Group>();
_repositoryGroupMember = new GenericRepository<GroupMember>();
var groups = _repositoryGroup.GetAll().ToList(); // other query operators are irrelevant and removed
var groupMembers = _repositoryGroupMember.GetAll().ToList();
Calling ToList() runs your queries and bring the data to memory. Now you have two in-memory collection (List<T>).
When you write:
var results = from we in _repository.GetAll() // <-- this is IQueryable<T>
join g in groups // <-- this is List<T> (IEnumerable<T>)
on we.GroupID equals g.ID into grpjoin
...
you are joining an IQueryable, with an in-memory list. When running this query, EF has no way to know that your in-memory lists (groups and groupMembers) are actually queries from the database. It only sees two lists, containing some data. It has no way to translate that into SQL and hence throws an error.
To fix this, you should remove calls to ToList(). That way, you have three IQueryables that are joined together. EF can translate that into SQL, only if they are from a single DbContext. And since they are not, it throws another error, telling you exactly that.
You create an instance DbContext for each of the GenericRepository<T> instances. So, groups, groupMembers and we come from three different DbContexts.
To solve this error, you should somehow manage to use a single DbContext for all your GenericRepository<T>s.
For example:
using (var dbContext = new MyDbContext())
{
var groups = dbContext.Set<Group>();
var groupMembers = dbContext.Set<GroupMember>();
var results = from we in dbContext.Set<WeeklyEntry>()
join g in groups
on we.GroupID equals g.ID into grpjoin
...
}
I'm using entity framework to connect to database from my application. I have table in SQL, named Orders. It contains such fields as: TransactionId, ParticipantId and is linked to Transactions table which has one to many connection to Participants table. I need to get data from it using List of classes with such properties: TransactionId, ParticipantId, OrganizationId. Linq must meet such conditions: (orders.TransactionId == TransactionId && orders.ParticipantId == ParticipantId && orders.Transaction.Participants.Any(x=> x.Id == OrganizationId)). This should be done by one query, not by multiple, so, please don't recommend foreach or smth like that.
Like #NetMage said, generally we need examples. Assuming that you've got a dbcontext set up, the ask is pretty simple:
public static void GetData(int transactionId, int participantId, int organizationId)
{
using (var db = new MyDbContext())
{
var query =
(
from t in db.Transactions
from o in db.Orders
.Where(w => w.TransactionId == t.TransactionId)
from p in db.Participants
.Where(w => w.TransactionId == t.TransactionId)
where t.TransactionId = transactionId &&
o.ParticipantId = participantId
select new { Order = o, Transaction = t, Participant = p}
);
}
}
Again since we don't have a lot of information here it's hard to do more. You should be able to take it from there. I know I didn't use the organizationId filter, but since I don't know the target shape of the data I'm not sure what the best path would be
I'm developing a client app that uses breezejs and Entity Framework 6 on the back end. I've got a statement like this:
var country = 'Mexico';
var customers = EntityQuery.from('customers')
.where('country', '==', country)
.expand('order')
I want to use There may be hundreds of orders that each customer has made. For the purposes of performance, I only want to retrieve the latest order for each customer. This will be based on the created date for the order. In SQL, I could write something like this:
SELECT c.customerId, companyName, ContactName, City, Country, max(o.OrderDate) as LatestOrder FROM Customers c
inner join Orders o on c.CustomerID = o.CustomerID
group by c.customerId, companyName, ContactName, City, Country
If this was run against the northwind database, only the most recent order row is returned for each customer.
How can I write a similar query in breeze, so that it runs on the server side and therefore returns less data to the client. I know I could handle this all on the client but writing some javascript in a querysucceeded method that could be run by the client - but that's not the goal here.
thanks
For a case like this, you should create a special endpoint method that will perform your query.
Then you can use an Entity Framework query to do what you want, using the LINQ syntax.
Here are two Web API examples:
[HttpGet]
public IQueryable<Object> CustomersLatestOrderEntities()
{
// IQueryable<Object> containing Customer and Order entity
var entities = ContextProvider.Context.Customers.Select(c => new { Customer = c, LatestOrder = c.Orders.OrderByDescending(o => o.OrderDate).FirstOrDefault() });
return entities;
}
[HttpGet]
public IQueryable<Object> CustomersLatestOrderProjections()
{
// IQueryable<Object> containing Customer and Order entity
var entities = ContextProvider.Context.Customers.Select(c => new { Customer = c, LatestOrder = c.Orders.OrderByDescending(o => o.OrderDate).FirstOrDefault() });
// IQueryable<Object> containing just data fields, no entities
var projections = entities.Select(e => new { e.Customer.CustomerID, e.Customer.ContactName, e.LatestOrder.OrderDate });
return projections;
}
Note that you have a choice here. You can return actual entities, or you can return just some data fields. Which is right for you depends upon how you are going to use them on the client. If they are just for display in a
non-editable list, you can just return the plain data (CustomersLatestOrderProjections above). If they can potentially
be edited, then return the object containing the entities (CustomersLatestOrderEntities). Breeze will merge the entities
into its cache, even though they are contained inside this anonymous object.
Either way, because it returns IQueryable, you can use the Breeze filtering syntax from the client to further qualify the query.
var projectionQuery = breeze.EntityQuery.from("CustomersLatestOrderProjections")
.skip(20)
.take(10);
var entityQuery = breeze.EntityQuery.from("CustomersLatestOrderEntities")
.where('customer.countryName', 'startsWith', 'C');
.take(10);
I am in the process of converting an application that uses LINQ to SQL over to LINQ to Entities. I use a repository pattern and I have run in a problem that works in LINQ to SQL but not Entities.
In my data layer, I use LINQ statements to fill my object graph so that none of my database entities are exposed anywhere else. In this example, I have a Lookup Respository that returns a list of Categories. It looks like this:
public IQueryable<Entities.DomainModels.Category> getCategories()
{
return (from c in Categories
where !c.inactive
orderby c.categoryName
select new Entities.DomainModels.Category
{
id = c.categoryID,
category = c.categoryName,
inactive = c.inactive
});
}
Later, I want to put the categories into a sub query and it looks like this:
var d = from p in Programs
let categories = (from pc in p.Categories
join c in getCategories() on pc.categoryID equals c.id
select c)
select new
{
id = p.id,
title = p.title
categories = categories.ToList()
};
When I run this, I get the following error:
LINQ to Entities does not recognize the method 'System.Linq.IQueryable`1[Entities.DomainModels.Category] getCategories()' method, and this method cannot be translated into a store expression.
For reference, the following works though it doesn't return the data I need (it's basically a join):
var q = from p in Programs
from pc in p.Categories
join c in getCategories() on pc.categoryID equals c.id
select new
{
id = p.id,
category = c
};
I understand what the error means in concept however LINQ to SQL would make it work. I have this pattern throughout my data layer and I really want to keep it. Should this be working? If not, how can I modify it without mixing my layers.
You cant pass getCategories() to EF.
The query must be destructible to expression tree.
Calculate getCategories() first.
eg
var simpleList = getCategories().Select(id).Tolist;
then use a contains
where(t=> simpleList.Contains(t.CatId) // or the query syntax equivalent
I have an extension method defined like so:
public static TSource MaxBy<TSource, TResult>(this IEnumerable<TSource> collection, Func<TSource, TResult> func) where TSource : class
{
var comparer = Comparer<TSource>.Default;
TSource maxItem = null;
foreach (var item in collection)
{
if (comparer.Compare(item, maxItem) > 0)
maxItem = item;
}
return maxItem;
}
which I then use in the following LINQ-to-Entities query:
var balancesQuery = from t in db.Transactions
where t.UserId == userId
group t by t.CurrencyCode into tg
let tMaxDate = tg.MaxBy(i => i.TsCreate)
join c in db.Currencies on tg.Key equals c.CurrencyCode
select new { Currency = c, Balance = tMaxDate.Balance }
So what I'm doing is - get the newest transaction (MaxBy TsCreate) in each currency (group by CurrencyCode) and then select the balance against each of those transactions.
My problem is - this does not work with Entity Framework (LINQ-to-Entities; I get:
LINQ to Entities does not recognize the method 'Transaction MaxBy[Transaction,DateTime](System.Collections.Generic.IEnumerable'[Transaction], System.Func'2[Transaction,System.DateTime])' method, and this method cannot be translated into a store expression.
The same query works with LINQ to SQL.
My questions are:
Is there a way to make it work with Entity Framework?
Or maybe there is a better way of querying for the same information, which would work with Entity Framework?
Thanks in advance for any help!
Isn't the query same as:
var balancesQuery = from t in db.Transactions
where t.UserId == userId
group t by t.CurrencyCode into tg
join c in db.Currencies on tg.Key equals c.CurrencyCode
select new {
Currency = c,
Balance = tg.OrderByDescending(i => i.TsCreate).Take(1).Balance
};
You can't use extensions with LINQ queries that actually pull data from the database as it's impossible for this to be turned into SQL. However, you can do this by returning the results into memory using ToArray() or ToList() to trigger the database query executing and then call your extension functions on resultant real data in memory.