I fail to translate a sql query to a linq query that could calculate some stock.
This is my test query that I'm trying to convert to a linq query.
SELECT
i.*,
(SELECT COUNT(t.*) FROM tickets t
WHERE t.starttime::time = i.sessionstarttime::time
AND t.starttime::date = '2018-04-06'::date)
as stock
FROM items I
-- note that the hardcoded date ('2018-04-06') is a function parameter
( tl;dr; how would you convert this PostgreSQL query to LINQ? )
My attempts so far are the variations of the following query:
var items = await _context.Items.Select(x => new Item
{
Id = x.Id,
IsTicket = x.IsTicket,
Name = x.Name,
Price = x.Price,
SaleItems = x.SaleItems,
SessionStartTime = x.SessionStartTime,
DateCreated = x.DateCreated,
DateEdit = x.DateEdit,
UserIdCreated = x.UserIdCreated,
UserIdEdited = x.UserIdEdited,
// calculate stock in subquery
Stock = _context.Tickets.Count(
t => t.StartTime.Date == ticketDate
&& x.SessionStartTime.HasValue
&& t.StartTime.Hour == x.SessionStartTime.Value.Hours // this is the part that is failing
&& t.State != TicketState.Canceled)
}).ToListAsync();
t.StartTime is Datetime and x.SessionStartTime is Nullable Timespan
So when I comment the line && t.StartTime.Hour == x.SessionStartTime.Value.Hours everything is fine, but with it I get warnings that it could not be translated and will be evaluated locally. But I don't want to download the whole ticket table just to count them.
The t.StartTime.Hour part is fine, I tried to perform static comparisons with both parameters. t.StartTime.Hour == 5 was translated without any problems, but x.SessionStartTime.Value.Hours == 5 failed to translate.
Also the problematic part in the application output:
([t].StartTime.Hour == Convert([x].SessionStartTime, TimeSpan).Hours))
So I guess that convert part is failing.
So what I'm missing and how I could work around this problem. Any help will be appreciated.
Update:
After experimenting a bit I have found two workarounds, that I wouldn't call the answers.
First I noticed that EF is trying to convert Nullable<TimeSpan> to a regular TimeSpan from the mentioned output: ([t].StartTime.Hour == Convert([x].SessionStartTime, TimeSpan).Hours))
I thought I could prevent that conversion by converting to a string and comparing the strings (I have a feeling this will bite me in the future):
t.StartTime.ToString().Contains(x.SessionStartTime.ToString())
The second workaround is only viable for my scenario since I know the items query is final and I can materialise it without calculated Stock, and then loop through the results and calculate it on a separate query. But this seems to add additional calls to the database and sacrifice some performance.
foreach(var x in items.Where(x=>x.SessionStartTime.HasValue))
{
// accessing the t.StartTime.TimeOfDay property seems to fail the LINQ to SQL as well
var hours = x.SessionStartTime.Value.Hours;
var minutes = x.SessionStartTime.Value.Minutes;
x.Stock = _context.Tickets.Count(t => t.StartTime.Date == ticketDate
&& t.StartTime.Hour == hours
&& t.StartTime.Minute == minutes);
}
Related
My Linq Query keeps returning the null error on FirstOrDefault
The cast to value type 'System.Int32' failed because the materialized value is null
because it can't find any records to match on the ClinicalAssetID form the ClinicalReading Table, fair enough!
But I want the fields in my details page just to appear blank if the table does not have matching entry.
But how can I handle the null issue when using the order by function ?
Current Code:
var ClinicalASSPATINCVM = (from s in db.ClinicalAssets
join cp in db.ClinicalPATs on s.ClinicalAssetID equals cp.ClinicalAssetID into AP
from subASSPAT in AP.DefaultIfEmpty()
join ci in db.ClinicalINSs on s.ClinicalAssetID equals ci.ClinicalAssetID into AI
from subASSINC in AI.DefaultIfEmpty()
join co in db.ClinicalReadings on s.ClinicalAssetID equals co.ClinicalAssetID into AR
let subASSRED = AR.OrderByDescending(subASSRED => subASSRED.MeterReadingDone).FirstOrDefault()
select new ClinicalASSPATINCVM
{
ClinicalAssetID = s.ClinicalAssetID,
AssetTypeName = s.AssetTypeName,
ProductName = s.ProductName,
ModelName = s.ModelName,
SupplierName = s.SupplierName,
ManufacturerName = s.ManufacturerName,
SerialNo = s.SerialNo,
PurchaseDate = s.PurchaseDate,
PoNo = s.PoNo,
Costing = s.Costing,
TeamName = s.TeamName,
StaffName = s.StaffName,
WarrantyEndDate = subASSPAT.WarrantyEndDate,
InspectionDate = subASSPAT.InspectionDate,
InspectionOutcomeResult = subASSPAT.InspectionOutcomeResult,
InspectionDocumnets = subASSPAT.InspectionDocumnets,
LastTypeofInspection = subASSINC.LastTypeofInspection,
NextInspectionDate = subASSINC.NextInspectionDate,
NextInspectionType = subASSINC.NextInspectionType,
MeterReadingDone = subASSRED.MeterReadingDone,
MeterReadingDue = subASSRED.MeterReadingDue,
MeterReading = subASSRED.MeterReading,
MeterUnitsUsed = subASSRED.MeterUnitsUsed,
FilterReplaced = subASSRED.FilterReplaced
}).FirstOrDefault(x => x.ClinicalAssetID == id);
Tried this but doesn't work
.DefaultIfEmpty(new ClinicalASSPATINCVM())
.FirstOrDefault()
Error was:
CS1929 'IOrderedEnumerable<ClinicalReading>' does not contain a definition for 'DefaultIfEmpty' and the best extension method overload 'Queryable.DefaultIfEmpty<ClinicalASSPATINCVM>(IQueryable<ClinicalASSPATINCVM>, ClinicalASSPATINCVM)' requires a receiver of type 'IQueryable<ClinicalASSPATINCVM>'
Feel a little closer with this but still errors
let subASSRED = AR.OrderByDescending(subASSRED => (subASSRED.MeterReadingDone != null) ? subASSRED.MeterReadingDone : String.Empty).FirstOrDefault()
Error:
CS0173 Type of conditional expression cannot be determined because there is no implicit conversion between 'System.DateTime?' and 'string'
The original error means that some of the following properties of the ClinicalASSPATINCVM class - MeterReadingDone, MeterReadingDue, MeterReading, MeterUnitsUsed, or FilterReplaced is of type int.
Remember that subASSRED here
let subASSRED = AR.OrderByDescending(subASSRED => subASSRED.MeterReadingDone).FirstOrDefault()
might be null (no corresponding record).
Now look at this part of the projection:
MeterReadingDone = subASSRED.MeterReadingDone,
MeterReadingDue = subASSRED.MeterReadingDue,
MeterReading = subASSRED.MeterReading,
MeterUnitsUsed = subASSRED.MeterUnitsUsed,
FilterReplaced = subASSRED.FilterReplaced
If that was LINQ to Objects, all these would generate NRE (Null Reference Exception) at runtime. In LINQ to Entities this is converted and executed as SQL. SQL has no issues with expression like subASSRED.SomeProperty because SQL supports NULL naturally even if SomeProperty normally does not allow NULL. So the SQL query executes normally, but now EF must materialize the result into objects, and the C# object property is not nullable, hence the error in question.
To solve it, find the int property(es) and use the following pattern inside query:
SomeIntProperty = (int?)subASSRED.SomeIntProperty ?? 0 // or other meaningful default
or change receiving object property type to int? and leave the original query as is.
Do the same for any non nullable type property, e.g. DateTime, double, decimal, Guid etc.
You're problem is because your DefaultIfEmpty is executed AsQueryable. Perform it AsEnumerable and it will work:
// create the default element only once!
static readonly ClinicalAssPatInVcm defaultElement = new ClinicalAssPatInVcm ();
var result = <my big linq query>
.Where(x => x.ClinicalAssetID == id)
.AsEnumerable()
.DefaultIfEmpty(defaultElement)
.FirstOrDefault();
This won't lead to a performance penalty!
Database management systems are extremely optimized for selecting data. One of the slower parts of a database query is the transport of the selected data to your local process. Hence it is wise to let the DBMS do most of the selecting, and only after you know that you only have the data that you really plan to use, move the data to your local process.
In your case, you need at utmost one element from your DBMS, and if there is nothing, you want to use a default object instead.
AsQueryable will move the selected data to your local process in a smart way, probably per "page" of selected data.
The page size is a good compromise: not too small, so you don't have to ask for the next page too often; not too large, so that you don't transfer much more items than you actually use.
Besides, because of the Where statement you expect at utmost one element anyway. So that a full "page" is fetched is no problem, the page will contain only one element.
After the page is fetched, DefaultIfEmpty checks if the page is empty, and if so, returns a sequence containing the defaultElement. If not, it returns the complete page.
After the DefaultIfEmpty you only take the first element, which is what you want.
If I try to get parts from a machine into the machines list of parts from the db, I execute this:
Machine ma = new Machine();
ma = dbcontext.Machine.Where(s => s.Guid == guid).ToList()[0];
IQueryable<Part> PartsQuery = from m in db.Machines
where m.Guid == guid
from p in m.Parts
select p;
ma.parts.AddRange(PartsQuery.ToList());
I get double the Parts into my parts list of the machine than actually are in the database!
If I do this instead of the last line:
List<parts> partsFromDb = PartsQuery.ToList();
ma.parts.AddRange(partsFromDb);
the amount of parts in the ma.parts list is correct. Can someone explain that to me please?
You can achieve what you trying to do in one round trip to your database:
Machine mab=context.Machine.Include(m=>m.Parts).FirstOrDefault(m=> m.Guid == guid);
About your issue, that's probably is due to the caching policy of EF and maybe Lazy Loading is involve too. I don't know how you are testing your code, but if your do the following:
Machine ma = context.Machine.FirstOrDefault(m=> m.Guid == guid);
IQueryable<Part> PartsQuery = from m in db.Machines
where m.Guid == guid
from p in m.Parts
select p;
PartsQuery.ToList(); //materialize your query but don't save the result;
var parts=ma.parts;// now take a look here and you will see the related parts were loaded
That should be the reason why the data is duplicated, because when you materialize your query and consult later the navigation property (m.parts), the related entities are already there. But anyways the best way to get what you need is using the query that I show at the beginning of my answer.
Machine ma = new Machine();
ma = dbcontext.Machine.Where(s => s.Guid == guid).ToList()[0];
IQueryable<Part> PartsQuery = from m in db.Machines
where m.Guid == guid
from p in m.Parts
select p;
ma.parts.AddRange(PartsQuery.ToList());
Is 100% equivalent to:
// 1. Find and retrieve the first machine with the given GUID
Machine machine = dbcontext.Machine.First(s => s.Guid == guid);
// 2. Again, find and retrieve the machines with the given GUID, select the parts of each machine that matches and flatten it down to a single list.
IList<Part> machineParts = db.Machines
.Where(m => m.Guid == guid)
.SelectMany(m => m.Parts)
.ToList();
// 3. Add.. all of the parts to that machine again?
machine.parts.AddRange(machineParts);
So it makes sense that you end up with double the parts inside the retrieved machine.
To be honest, I don't believe that the last change you speak about, i.e. capturing the 'PartsQuery' into a temporary variable, makes any difference with regards to the end result of your machine.
Something else must be going on there.
I have a problem that has taken me weeks to resolve and I have not been able to.
I have a class where I have two methods. The following is supposed to take the latest date from database. That date represents the latest payment that a customer has done to "something":
public DateTime getLatestPaymentDate(int? idCustomer)
{
DateTime lastDate;
lastDate = (from fp in ge.Payments
from cst in ge.Customers
from brs in ge.Records.AsEnumerable()
where (cst.idCustomer == brs.idCustomer && brs.idHardBox == fp.idHardbox
&& cst.idCustomer == idCustomer)
select fp.datePayment).AsEnumerable().Max();
return lastDate;
}//getLatestPaymentDate
And here I have the other method, which is supposed to call the previous one to complete a Linq query and pass it to a Crystal Report:
//Linq query to retrieve all those customers'data who have not paid their safebox(es) annuity in the last year.
public List<ReportObject> GetPendingPayers()
{
List<ReportObject> defaulterCustomers;
defaulterCustomers = (from c in ge.Customer
from br in ge.Records
from p in ge.Payments
where (c.idCustomer == br.idCustomer
&& br.idHardBox == p.idHardBox)
select new ReportObject
{
CustomerId = c.idCustomer,
CustomerName = c.nameCustomer,
HardBoxDateRecord = br.idHardRecord,
PaymentDate = getLatestPaymentDate(c.idCustomer),
}).Distinct().ToList();
}//GetPendingPayers
No compile error is thrown here, but when I run the application and the second method tries to call the first one in the field PaymentDate the error mentioned in the header occurs:
Linq to Entities does not recognize the method System.DateTime.. and cannot translate this into a store expression
Please anybody with an useful input that put me off from this messy error? Any help will be appreciated !
Thanks a lot !
Have a look at these other questions :
LINQ to Entities does not recognize the method
LINQ to Entities does not recognize the method 'System.DateTime Parse(System.String)' method
Basically, you cannot use a value on the C# side and translate it into SQL. The first question offers a more thorough explanation ; the second offers a simple solution to your problem.
EDIT :
Simply put : the EF is asking the SQL server to perform the getLatestPaymentDate method, which it has no clue about. You need to execute it on the program side.
Simply perform your query first, put the results into a list and then do your Select on the in-memory list :
List<ReportObject> defaulterCustomers;
var queryResult = (from c in ge.Customer
from br in ge.Records
from p in ge.Payments
where (c.idCustomer == br.idCustomer
&& br.idHardBox == p.idHardBox)).Distinct().ToList();
defaulterCustomers = from r in queryResult
select new ReportObject
{
CustomerId = r.idCustomer,
CustomerName = r.nameCustomer,
HardBoxDateRecord = r.idHardRecord,
PaymentDate = getLatestPaymentDate(r.idCustomer),
}).Distinct().ToList();
I don't have access to your code, obviously, so try it out and tell me if it works for you!
You'll end up with an in-memory list
I see that the question's been asked several times of how to set the command timeout for long-running queries with Entity Framework. The problem I'm running into now is that the query that gets run against the server doesn't really take that long to execute and return.
Here's the code that runs the query:
var records = (from c in _context.Set<CompletedQuiz>()
where c.FarmId == _entityId && c.ToolboxId == _toolboxId
group c by new { c.UserId, c.LessonId } into g
select g).ToList()
.Select(c => new {
UserId = c.Key.UserId,
LessonId = c.Key.LessonId,
NumQuestions = c.Max(n => n.TotalNumQuestions),
NumLessons = c.Select(l => l.LessonId).Distinct().Count(),
Start = c.Min(s => s.LogonDateTime),
End = c.Max(e => e.LogoffDateTime),
MaxScore = c.Max(s => s.Score),
Passed = c.Any(p => p.Passed)
});
I'm selecting from a fairly simple view called CompletedQuizzes, and grouping on the record ids for users and lessons. I've run this with SQL Profiler running to capture the actual query that's executed; if I run that exact same query in SSMS, it runs almost instantly (<0 seconds). However, running from my application will often exceed the default command timeout of 30 seconds. I put a breakpoint on the line that's shown above, and I added the call to .ToList() to make sure that the query is executed immediately.
What else should I be checking as a possible culprit here?
EDIT:
I still don't understand why the code above takes so long to execute, but I reworked it using using Linq extension methods, and now it runs as fast as I would expect. Here's what it looks like now:
var records = _context.Set<CompletedQuiz>()
.Where(c => c.FarmId == _entityId && c.ToolboxId == _toolboxId)
.GroupBy(c => new { c.UserId, c.LessonId })
.Select(c => new {
UserId = c.Key.UserId,
LessonId = c.Key.LessonId,
NumQuestions = c.Max(n => n.TotalNumQuestions),
NumLessons = c.Select(l => l.LessonId).Distinct().Count(),
Start = c.Min(s => s.LogonDateTime),
End = c.Max(e => e.LogoffDateTime),
MaxScore = c.Max(s => s.Score),
Passed = c.Any(p => p.Passed)
});
I guess at this point I would adjust my question to why is the query generated by the second block of code executed so much more quickly from my application?
I think it is qutie obvious your first query is using .ToList() befor projection. It means that you load all CompleteQuiz instances satisfying your condition to your application and execute all aggregations in your application. It also means that there must be a lot of subsequent queries to lazy load your relations for aggregation computations.
In your second query there is no ToList and thus whole query including all aggregations is performed in the database.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
The community reviewed whether to reopen this question 1 year ago and left it closed:
Original close reason(s) were not resolved
Improve this question
What is the best way to check if an object exists in the database from a performance point of view? I'm using Entity Framework 1.0 (ASP.NET 3.5 SP1).
If you don't want to execute SQL directly, the best way is to use Any(). This is because Any() will return as soon as it finds a match. Another option is Count(), but this might need to check every row before returning.
Here's an example of how to use it:
if (context.MyEntity.Any(o => o.Id == idToMatch))
{
// Match!
}
And in vb.net
If context.MyEntity.Any(function(o) o.Id = idToMatch) Then
' Match!
End If
From a performance point of view, I guess that a direct SQL query using the EXISTS command would be appropriate. See here for how to execute SQL directly in Entity Framework: http://blogs.microsoft.co.il/blogs/gilf/archive/2009/11/25/execute-t-sql-statements-in-entity-framework-4.aspx
I had to manage a scenario where the percentage of duplicates being provided in the new data records was very high, and so many thousands of database calls were being made to check for duplicates (so the CPU sent a lot of time at 100%). In the end I decided to keep the last 100,000 records cached in memory. This way I could check for duplicates against the cached records which was extremely fast when compared to a LINQ query against the SQL database, and then write any genuinely new records to the database (as well as add them to the data cache, which I also sorted and trimmed to keep its length manageable).
Note that the raw data was a CSV file that contained many individual records that had to be parsed. The records in each consecutive file (which came at a rate of about 1 every 5 minutes) overlapped considerably, hence the high percentage of duplicates.
In short, if you have timestamped raw data coming in, pretty much in order, then using a memory cache might help with the record duplication check.
I know this is a very old thread but just incase someone like myself needs this solution but in VB.NET here's what I used base on the answers above.
Private Function ValidateUniquePayroll(PropertyToCheck As String) As Boolean
// Return true if Username is Unique
Dim rtnValue = False
Dim context = New CPMModel.CPMEntities
If (context.Employees.Any()) Then ' Check if there are "any" records in the Employee table
Dim employee = From c In context.Employees Select c.PayrollNumber ' Select just the PayrollNumber column to work with
For Each item As Object In employee ' Loop through each employee in the Employees entity
If (item = PropertyToCheck) Then ' Check if PayrollNumber in current row matches PropertyToCheck
// Found a match, throw exception and return False
rtnValue = False
Exit For
Else
// No matches, return True (Unique)
rtnValue = True
End If
Next
Else
// The is currently no employees in the person entity so return True (Unqiue)
rtnValue = True
End If
Return rtnValue
End Function
I had some trouble with this - my EntityKey consists of three properties (PK with 3 columns) and I didn't want to check each of the columns because that would be ugly.
I thought about a solution that works all time with all entities.
Another reason for this is I don't like to catch UpdateExceptions every time.
A little bit of Reflection is needed to get the values of the key properties.
The code is implemented as an extension to simplify the usage as:
context.EntityExists<MyEntityType>(item);
Have a look:
public static bool EntityExists<T>(this ObjectContext context, T entity)
where T : EntityObject
{
object value;
var entityKeyValues = new List<KeyValuePair<string, object>>();
var objectSet = context.CreateObjectSet<T>().EntitySet;
foreach (var member in objectSet.ElementType.KeyMembers)
{
var info = entity.GetType().GetProperty(member.Name);
var tempValue = info.GetValue(entity, null);
var pair = new KeyValuePair<string, object>(member.Name, tempValue);
entityKeyValues.Add(pair);
}
var key = new EntityKey(objectSet.EntityContainer.Name + "." + objectSet.Name, entityKeyValues);
if (context.TryGetObjectByKey(key, out value))
{
return value != null;
}
return false;
}
I just check if object is null , it works 100% for me
try
{
var ID = Convert.ToInt32(Request.Params["ID"]);
var Cert = (from cert in db.TblCompCertUploads where cert.CertID == ID select cert).FirstOrDefault();
if (Cert != null)
{
db.TblCompCertUploads.DeleteObject(Cert);
db.SaveChanges();
ViewBag.Msg = "Deleted Successfully";
}
else
{
ViewBag.Msg = "Not Found !!";
}
}
catch
{
ViewBag.Msg = "Something Went wrong";
}
Why not do it?
var result= ctx.table.Where(x => x.UserName == "Value").FirstOrDefault();
if(result?.field == value)
{
// Match!
}
Best way to do it
Regardless of what your object is and for what table in the database the only thing you need to have is the primary key in the object.
C# Code
var dbValue = EntityObject.Entry(obj).GetDatabaseValues();
if (dbValue == null)
{
Don't exist
}
VB.NET Code
Dim dbValue = EntityObject.Entry(obj).GetDatabaseValues()
If dbValue Is Nothing Then
Don't exist
End If