Group By with Entity Framework - entity-framework

enter image description hereI have a code. And there you need to make a grouping by name.
//<date,<partid,amount>>
Dictionary<string, Dictionary<int, double>> emSpending = new Dictionary<string, Dictionary<int, double>>();
foreach (Orders order in db.Orders.ToList())
{
foreach (OrderItems orderitem in order.OrderItems.ToList())
{
if (!emSpending.ContainsKey(order.Date.ToString("yyyy-MM"))) emSpending.Add(order.Date.ToString("yyyy-MM"), new Dictionary<int, double>());
if (!emSpending[order.Date.ToString("yyyy-MM")].ContainsKey(Convert.ToInt32(orderitem.PartID))) emSpending[order.Date.ToString("yyyy-MM")].Add(Convert.ToInt32(orderitem.PartID), 0);
emSpending[order.Date.ToString("yyyy-MM")][Convert.ToInt32(orderitem.PartID)] += Convert.ToDouble(orderitem.Amount);
}
}
DataGridViewColumn col1 = new DataGridViewColumn();
col1.CellTemplate = new DataGridViewTextBoxCell();
col1.Name = "Department";
col1.AutoSizeMode = DataGridViewAutoSizeColumnMode.Fill;
col1.HeaderText = "Department";
dgvEMSpending.Columns.Add(col1);
foreach (string date in emSpending.Keys)
{
DataGridViewColumn col = new DataGridViewColumn();
col.Name = date;
col.HeaderText = date;
col.AutoSizeMode = DataGridViewAutoSizeColumnMode.Fill;
col.CellTemplate = new DataGridViewTextBoxCell();
dgvEMSpending.Columns.Add(col);
}
List<string> allKey = emSpending.Keys.ToList();
foreach (string date in allKey)
if (date == "Department") continue;
else
{
dgvEMSpending.Rows.Add();
foreach (int partid in emSpending[date].Keys)
{
dgvEMSpending.Rows[dgvEMSpending.Rows.Count - 1].Cells[0].Value = db.Parts.Where(x => x.ID == partid).SingleOrDefault().Name.GroupBy(Name);
for (int i = 1; i < dgvEMSpending.Columns.Count; i++)
{
if (!emSpending.ContainsKey(dgvEMSpending.Columns[i].Name)) emSpending.Add(dgvEMSpending.Columns[i].Name, new Dictionary<int, double>());
if (!emSpending[dgvEMSpending.Columns[i].Name].ContainsKey(partid)) emSpending[dgvEMSpending.Columns[i].Name].Add(partid, 0);
double val = emSpending[dgvEMSpending.Columns[i].Name][partid];
dgvEMSpending.Rows[dgvEMSpending.RowCount - 1].Cells[i].Value = val;
}
}
}
I tried to use group by myself, but something doesn't work. It just outputs the same names, and I want to group them so that there is a grouping. Pls helped to me.

Ok, a few issues to help you out first. This code:
foreach (Orders order in db.Orders.ToList())
{
foreach (OrderItems orderitem in order.OrderItems.ToList())
{
if (!emSpending.ContainsKey(order.Date.ToString("yyyy-MM"))) emSpending.Add(order.Date.ToString("yyyy-MM"), new Dictionary<int, double>());
if (!emSpending[order.Date.ToString("yyyy-MM")].ContainsKey(Convert.ToInt32(orderitem.PartID))) emSpending[order.Date.ToString("yyyy-MM")].Add(Convert.ToInt32(orderitem.PartID), 0);
emSpending[order.Date.ToString("yyyy-MM")][Convert.ToInt32(orderitem.PartID)] += Convert.ToDouble(orderitem.Amount);
}
}
Right off the bat this is going to trip lazy loading on OrderItems. If you have 10 orders 1-10 you're going to be running 11 queries against the database:
SELECT * FROM Orders;
SELECT * FROM OrderItems WHERE OrderId = 1;
SELECT * FROM OrderItems WHERE OrderId = 2;
// ...
SELECT * FROM OrderItems WHERE OrderId = 10;
Now if you have 100 orders or 1000 orders, you should see the problem. At a minimum ensure that if you are touching a collection or reference on entities you are loading, eager load it with Include:
foreach (Orders order in db.Orders.Include(x => x.OrderItems).ToList())
This will run a single query that fetches the Orders and their OrderItems. However, if you have a LOT of rows this is going to take a while and consume a LOT of memory.
The next tip is "only load what you need". You need 1 field from Order and 2 fields from OrderItem. So why load everything from both tables??
var orderItemDetails = db.Orders
.SelectMany(o => o.OrderItems.Select(oi => new { o.Date, oi.PartId, oi.Amount })
.ToList();
This would give us just the Order date, and each Part ID and Amount. Now that this data is in memory we can group it to populate your desired dictionary structure without having to iterate over it row by row.
var emSpending = orderItemDetails.GroupBy(x => x.Date.ToString("yyyy-MM"))
.ToDictionary(g => g.Key,
g => g.GroupBy(y => y.PartId)
.ToDictionary(g2 => g2.Key, g2 => g2.Sum(z => z.Amount)));
Depending on the Types in your entities you may need to insert casts. This first groups the outer dictionary of the yyyy-MM of the order dates, then it groups the remaining data for each date by part ID, and sums the Amount.
Now relating to your question, from your code example I'm guessing the problem area you are facing is this line:
dgvEMSpending.Rows[dgvEMSpending.Rows.Count - 1].Cells[0].Value = db.Parts
.Where(x => x.ID == partid)
.SingleOrDefault().Name.GroupBy(Name);
Now the question would be to explain what exactly you are expecting from this? You are fetching a single Part by ID. How would you expect this to be "grouped"?
If you want to display the Part name instead of the PartId then I believe you would just want to Select the Part Name:
dgvEMSpending.Rows[dgvEMSpending.Rows.Count - 1].Cells[0].Value = db.Parts
.Where(x => x.ID == partid)
.Select(x => x.Name)
.SingleOrDefault();
We can go one better to fetch the Part names for each used product in one hit using our loaded order details:
var partIds = orderItemDetails
.Select(x=> x.PartId)
.Distinct()
.ToList();
var partDetails = db.Parts
.Where(x => partIds.Contains(x.ID))
.ToDictionary(x => x.ID, x => x.Name);
This fetches us a dictionary set indexed by ID for the part names, it would be done outside of the loop after we had loaded the orderItemDetails. Now we don't have to go to the DB with every row:
dgvEMSpending.Rows[dgvEMSpending.Rows.Count - 1].Cells[0].Value = partDetails[partId];

Related

Is my Linq Query Optimal and High performance in EF Core 3.1?

can i remove vg.First().Voucher ? and replace the beter code? what is the optimal and best practice?
is convertable this code to another method? like chain method?
var query = from v in _journalLineRepository.TableNoTracking
.Include(j => j.Voucher).AsEnumerable()
group v by v.AccountId into vg
select new // <-- temporary projection with group by fields needed
{
AccountId = vg.Key,
Credit = vg.Sum(v => v.Credit),
Debit = vg.Sum(v => v.Debit),
Voucher = vg.First().Voucher
} into vg
join p in _partyRepository.TableNoTracking.Include(p => p.PartyPhones).AsEnumerable() on vg.AccountId equals p.AccountId // <-- additional join(s)
select new PartyDeptorAndCreditorViewModel
{
PartyId = p.Id,
FullName = p.FullName,
PhoneNo = p.PartyPhones.FirstOrDefault(p => p.IsActive)?.Phone,
ProjectId = vg.Voucher.ProjectId,
AccountId = vg.AccountId.Value,
Creditor = vg.Credit,
Deptor = vg.Debit,
Balance = vg.Credit - vg.Debit,
VoucherDate = vg.Voucher.VoucherDate,
VoucherRegisterDate = vg.Voucher.VoucherDate,
BalanceType =
vg.Debit > vg.Credit ? AccountingComplexEnum.ShowPartyBalanceParamSearch.Deptor.ToDisplay(DisplayProperty.Name) :
vg.Debit < vg.Credit ? AccountingComplexEnum.ShowPartyBalanceParamSearch.Creditor.ToDisplay(DisplayProperty.Name) :
AccountingComplexEnum.ShowPartyBalanceParamSearch.ZeroBalance.ToDisplay(DisplayProperty.Name),
};
I'd certainly be looking at the SQL query generated. At face value I see a few warning flags that it may not be composing a query but possibly pre-executing to in-memory processing which would be inefficient. It would firstly depend on what these .TableNoTracking methods/properties return, and the use of .AsEnumerable on the eager load joins.
Firstly, when projecting with Select, eager load joins (.Include) are not necessary. The projections will take care of the joins for you, provided it is projecting down to SQL. If you take out the .Include().AsEnumerable() calls and your query still works then it is likely projecting down to SQL. If it is no longer working then it's processing in memory and not efficiently.
Edit: Nope, the inner projection won't resolve: Regarding the .Voucher, your final projection is using 2 fields from this entity, so it stands you could replace this in the initial projection:
select new // <-- temporary projection with group by fields needed
{
AccountId = vg.Key,
Credit = vg.Sum(v => v.Credit),
Debit = vg.Sum(v => v.Debit),
Voucher = vg.Select(v => new { v.ProjectId, v.VoucherDate }).First()
} into vg
When it comes to transformations like this:
BalanceType = vg.Debit > vg.Credit
? AccountingComplexEnum.ShowPartyBalanceParamSearch.Deptor.ToDisplay(DisplayProperty.Name)
: vg.Debit < vg.Credit
? AccountingComplexEnum.ShowPartyBalanceParamSearch.Creditor.ToDisplay(DisplayProperty.Name)
: AccountingComplexEnum.ShowPartyBalanceParamSearch.ZeroBalance.ToDisplay(DisplayProperty.Name),
... inside a projection, this sends off warning flags as Linq2EF needs to compose projections down to SQL so methods/extensions like ToDisplay won't be understood. Instead, since this is based solely on the Credit/Debit amounts, I'd move this to be computed by the property in the view model:
select new PartyDeptorAndCreditorViewModel
{
PartyId = p.Id,
FullName = p.FullName,
PhoneNo = p.PartyPhones
.Where(p => p.IsActive)
.Select(p => p.Phone)
.FirstOrDefault(),
ProjectId = vg.Voucher.ProjectId,
AccountId = vg.AccountId.Value,
Creditor = vg.Credit,
Deptor = vg.Debit,
Balance = vg.Credit - vg.Debit,
VoucherDate = vg.Voucher.VoucherDate,
VoucherRegisterDate = vg.Voucher.VoucherDate
};
Then in the view model:
[Serializable]
public class PartyDebtorAndCreditorViewModel
{
// ...
public decimal Balance { get; set; }
public string BalanceType
{
get
{
return Balance < 0
? AccountingComplexEnum.ShowPartyBalanceParamSearch.Deptor.ToDisplay(DisplayProperty.Name)
: Balance > 0
? AccountingComplexEnum.ShowPartyBalanceParamSearch.Creditor.ToDisplay(DisplayProperty.Name)
: AccountingComplexEnum.ShowPartyBalanceParamSearch.ZeroBalance.ToDisplay(DisplayProperty.Name);
}
}
}

MongoDB .NET Group by with list result of whole objects

I was digging into the documentation of MongoDB http://mongodb.github.io/mongo-csharp-driver/2.7/reference/driver/crud/linq/ and I saw, using .NET drive is possible to make group by on the database.
For 1 ProductId I can have few elements in a database. I want to get whole Last element.
So I am trying to do something like:
var query = _collection.AsQueryable()
.GroupBy(p => p.ProductId, (k, s) =>
new { Name = k, Result = s.First() })
.Select(x => new { Name = x.Name, Result = x.Result }).First();
The problem is that I see an error message like:
System.NotSupportedException: Specified method is not supported.
at
MongoDB.Driver.Linq.Processors.AccumulatorBinder.GetAccumulatorArgument(Expression node)
I know that for now in my example i didnt order by the result.. But this will be my second step. For now i see that i cannot group by. Is it possible to do this kind of group by?
My solution for that is:
var query = _collection.Aggregate(new AggregateOptions { AllowDiskUse = true })
.Match(x => ElementIds.Contains(x.ElementId))
.SortByDescending(x => x.StartDate).ThenByDescending(x => x.CreatedAt)
.Group(x => x.ElementId, x => new
{
StartDate = x.First().StartDate,
Grades = x.First().Grades,
SellingId = x.First().SellingId,
CreatedAt = x.First().CreatedAt,
ModifiedAt = x.First().ModifiedAt,
Id = x.First().Id,
ElementId = x.First().ElementId
})
.ToEnumerable(token);
After that, I parsed it into my model.
AllowDiskUse = true, because in my case MongoDB's memory is not enough to handle this operation.

How to do aggregate functions such as count in Entity Framework Core

I have this SQL that I would like to execute in Entity Framework Core 2.1:
Select ItemTypeId, Count(ItemTypeId) as Count from Items i
where i.ParentId = 2
group by ItemTypeId
How do I do that?
This is what I came up with, but it returns zero:
var query = this._context.Items.Where(a => a.ParentId == 2)
.GroupBy(i => new { i.ItemTypeId })
.Select(g => new { g.Key.ItemTypeId, Count = g.Count(i=> i.ItemTypeId == g.Key.ItemTypeId) });
var items = query.ToList();
The only example I could find was here
You don't need Count = g.Count(i=> i.ItemTypeId == g.Key.ItemTypeId), instead use g.Count().

Can't add/sum string values in IEnumerable

The ultimate goal is to display a GrandTotal column using Highcharts. The GrandTotal should be the sum of TotalAmount for a given Offer id. TotalAmount is a string and the values are like $10.00 or 10.00. GrandTotal is an int, but could easily be changed. Here is what I have done so far.
Step 1) Convert the two IEnumerable lists into their ViewModel counterparts. I set GrandTotal to 0 here because I don't know the amount.
var offersConvert = offers
.Select(o => new OfferSummaryViewModel
{
Id = o.Id,
Name = o.Name,
Created = o.Created,
Shares = o.Shares,
Redemptions = o.Redemptions,
GrandTotal = 0
})
.ToList();
var sharedOffersConvert = sharedOffers
.Select(s => new SharedOfferViewModel
{
OfferId = s.OfferId,
//TotalAmount = s.TotalAmount.Replace("$", string.Empty).Replace(",", string.Empty).Trim()
TotalAmount = s.TotalAmount
})
//.Where(i => i.TotalAmount != null)
.ToList();
Step 2) Join the two lists on the Id of the Offer.
var data = offersConvert
.Join(sharedOffersConvert,
o => o.Id,
s => s.OfferId,
(o, s) => new { offersConvert = o, sharedOffersConvert = s })
.Select(o => new
{
Id = o.offersConvert.Id,
Created = o.offersConvert.Created,
Shares = o.offersConvert.Shares,
Redemptions = o.offersConvert.Redemptions,
Name = o.offersConvert.Name,
OfferId = o.sharedOffersConvert.OfferId,
TotalAmount = o.sharedOffersConvert.TotalAmount,
//GrandTotal = Convert.ToInt32(o.sharedOffersConvert.TotalAmount.Replace("$", string.Empty).Replace(",", string.Empty).Trim())
//GrandTotal = Convert.ToInt32(Convert.ToDouble(o.sharedOffersConvert.TotalAmount, CultureInfo.InvariantCulture))
})
//.Where(o => o.Id == o.OfferId)
.OrderBy(o => o.Created.Add(offset))
.ToList();
As you can tell, I've tried to remove any dollar signs and commas. I've even tried to trim white space in order to get clean data. I am then trying to convert the strings to int values, so I can sum them. Nothing seems to work. I've even tried .GroupBy and other methods (see below). At least with .GroupBy I can get to the .Sum operator. With the other method I run into issues when I can't convert int into ToList(), so I have to try and convert ToString().
.Where(o => o.Id == o.OfferId)
.GroupBy(g => g.Id)
.Select(x => new { GrandTotal = x.Sum(o => o.TotalAmount) })
ERROR in above: can't convert TotalAmount to decimal
.Where(i => i.Id == i.OfferId)
.Sum(i => Convert.ToInt32(Convert.ToDouble(i.TotalAmount, CultureInfo.InvariantCulture)
)).ToString()
Does anyone know how I can add/sum the string values in TotalAmount to get a GrandTotal per Offer id?
Any help us much appreciated. Thanks!
UPDATE: This works, but I really don't understand why and I don't think it is very clean. I really couldn't find many examples where people were joining two lists together and summing one of the columns. This seems pretty common to me, but perhaps it is not.
var data = (from o in offersConvert
join s in sharedOffersConvert on o.Id equals s.OfferId
orderby o.Created.Add(offset)
let k = new
{
Id = o.Id,
Name = o.Name,
Created = o.Created,
Shares = o.Shares,
Redemptions = o.Redemptions
}
group s by k into totals
select new
{
OfferId = totals.Key.Id,
Name = totals.Key.Name,
Created = totals.Key.Created,
Shares = totals.Key.Shares,
Id = totals.Key.Id,
Redemptions = totals.Key.Redemptions,
GrandTotal = totals.Sum((s => s.TotalAmount == null ? Decimal.Zero : Decimal.Parse(s.TotalAmount, NumberStyles.Currency)))
})
.ToList();
You can use
Decimal.TryParse("$10.00", NumberStyles.Currency, CultureInfo.CurrentCulture, out var res);
Or in the context of your LINQ,
GrandTotal = Decimal.Parse(o.sharedOffersConvert.TotalAmount, NumberStyles.Currency)
if o.sahredOffersConvert.TotalAmount may be null,
GrandTotal = (o.sharedOffersConvert.TotalAmount ==null) ? Decimal.Zero : Decimal.Parse(o.sharedOffersConvert.TotalAmount, NumberStyles.Currency)
I'm still puzzled why everyone lately has this odd fascination with the Lambda syntax. And that seems to have been only one of several means you've used to make this more complicated than necessary:
var data = (from o in offers
join s in sharedOffers on o.Id equals s.OfferId
orderby o.Created.Add(offset)
select new
{
Id = o.Id,
Created = o.Created,
Shares = o.Shares,
Redemptions = o.Redemptions,
Name = o.Name,
OfferId = o.OfferId,
TotalAmount = Decimal.Parse(o.TotalAmount, NumberStyles.Currency)
})
.ToList();
Further, as offset seems to be a constant (for the life of this query), adding it to Created isn't going to affect the ordering, and that bit can be removed.
And, since it appears your final output is just the grand totals, it can be reduced further to:
var data = (from o in offers
join s in sharedOffers on o.Id equals s.OfferId
orderby o.Created
group Decimal.Parse(o.TotalAmount, NumberStyles.Currency) by o.id into totals
select new
{
Id = totals.Key,
GrandAmount = totals.Sum()
})
.ToList();
UPDATE: Putting back what I took out... This should work (I don't have your tables, so I can't test it)
var data = (from o in offers
join s in sharedOffers on o.Id equals s.OfferId
orderby o.Created.Add
group o by o.Id into totals
let item = totals.First()
select new
{
Id = item.Id,
Created = item.Created,
Shares = item.Shares,
Redemptions = o.Redemptions,
Name = item.Name,
OfferId = item.OfferId,
GrandTotal = totals.Sum(t=>Decimal.Parse(t.TotalAmount, NumberStyles.Currency))
})
.ToList();

Best way to order by children collection with Entity

I am seeking for the best solution for this simple problem.
Run in C#/Entity the following SQL:
select user.name, userstat.point from user, userstat where userstat.user_id = user.id order by userstat.point desc
There is a User table [Id, Name, ...] and Statistic table [Id, UserId, Point. ...], where it's connected to User by Statistic.UserId. It's a 1:1 relation, so there is (max) 1 Statistic record for each User record.
I want to have a list User+Point, ordered by Point desc, and select a range, let's say 1000-1100.
Currently I have this:
public List<PointItem> Get(int startPos, int count)
{
using (DB.Database db = new DB.Database())
{
var dbList = db.Users.Where(user => .... ).ToList();
List<PointItem> temp = new List<PointItem>(count);
foreach (DB.User user in db.Users)
{
//should be always 1 stat for the user, but just to be sure check it...
if (user.Stats != null && user.Stats.Count > 0)
temp.Add(new PointItem { Name = user.Name, Point = user.Stats.First().Point });
} <--- this foreach takes forever
return temp.OrderByDescending(item => item.Point).Skip(startPos).Take(count).ToList();
}
}
It works fine, but when I have 10000 User (with 10000 UserStat) it runs for 100sec, which is only 1000x slower than I want it to be.
Is there more efficient solution than this?
If I run SQL, it takes 0 sec basically for 10K record.
EDIT
I made it faster, now 100sec -> 1 sec, but still I want it faster (if possible).
var userPoint = db.Users
.Where(u => u.UserStats.Count > 0 && ....)
.Select(up => new
{
User = up,
Point = up.UserStats.FirstOrDefault().Point
})
.OrderByDescending(up => up.Point)
.ToList();
var region = userPoint.Skip(0).Take(100);
Ok, I found the solution, the following code is 0.05 sec. Just need to go from child to parent:
using (DB.Database db = new DB.Database())
{
var userPoint = db.UserStats
.Where(s => s.User.xxx .....)
.Select(userpoint => new
{
User = userpoint.User.Name,
Point = userpoint.Point
})
.OrderByDescending(userpoint => userpoint.Point)
.ToList().Skip(startPos).Take(count);
}