I am trying to understand what is the best way to create a dynamic query.
I have a requirement where I will be writing an API to retrieve data from DB. the API has lot of filter paramters. eg. I need to retrieve movies that can be filtered on following properties.
MovieName, Genere, Rating, Language, Category
I can give these parameters in any combination.. so in my Data layer I started framing my dynamic query like this.
IQueryable<Movie> qryContext = null;
if(!string.isnullorEmpty(request.MovieName))
qryContext = context.Movies.Where(x => x.MovieName == request.MovieName)
if(!string.isnullorEmpty(request.Genere))
qryContext = context.Movies.Where(x => x.Genere == request.Genere)
if(!string.isnullorEmpty(request.Language))
qryContext = context.Movies.Where(x => x.Language == request.Language)
if(!string.isnullorEmpty(request.Category))
qryContext = context.Movies.Where(x => x.Category == request.Category)
if(qryContext!= null)
return qryContext.ToList();
else
return null;
Based on the given parameters, the sql query is framed..
But When I search in google reg dynamic queries in EF, most of the links refer to using Expression. Do I need to make use of Expression or can I proeeed with the above method.
Also let me know what advantage I get on using expressions.
Basically what you see inside the Where(...) is expression.
In your example this would be x => x.Language == request.Language
if(!string.isnullorEmpty(request.Language)) qryContext = context.Movies.Where(x => x.Language == request.Language)
Also, I would recommend you to take a look into the Dynamic expression library from EF Plus team. Here https://dynamic-linq.net/basic-simple-query
This allows you to pass expression as a string. And you can construct your filters in the Frontend, and pass them as a string, which helps you to write a much cleaner implementation of filters.
Related
I'm trying to use LinQ Intersect (or equivalent) into an IQueryable method but it seems like I'm doing it wrong.
I have some PRODUCTS that match some SPECIFITY (like colors, materials, height...), those specifications have different values, for example:
color : blue, red, yellow
height : 128cm, 152cm...
I need to get the products that match ALL the list of couple specifityId / specifityValue I provide.
Here what I'm trying to do:
// The list of couple SpecifityID (color, material..) / SpecifityValue (red, yellow, wood...)
List<string> SpecId_SpecValue = new List<string>();
SpecId_SpecValue.Add("3535a444-1139-4a1e-989f-795eb9be43be_BEA");
SpecId_SpecValue.Add("35ad6162-a885-4a6a-8044-78b68f6b2c4b_Purple");
int filterCOunt = SpecId_SpecValue.Count;
var query =
Products
.Include(pd => pd.ProductsSpecifity)
.Where(z => SpecId_SpecValue
.Intersect(z.ProductsSpecifity.Select(x => (x.SpecifityID.ToString() + "_" + x.SpecifityValue)).ToList()).Count() == filterCOunt);
I got the error : InvalidOperationException: The LINQ expression 'DbSet() could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information. which mean it can't be translated to SQL and I need to ToList before my filter.
The problem is, I don't want to call ToList() because I got huge number of products in my Database and I don't want to load them in memory before filtering them.
Is there an other way to achieve what I need to do?
I ended up using a solution found in the link #Gert Arnold provide here.
I used BlazarTech.QueryableValues.SqlServer #yv989c's answers
Here's what is now working like a charm :
// The list of couple SpecifityID (color, material..) / SpecifityValue (red, yellow, wood...)
Dictionary<Guid, string> SpecId_SpecValue = new Dictionary<Guid, string>();
SpecId_SpecValue.Add(new Guid("3535a444-1139-4a1e-989f-795eb9be43be"), "BEA");
SpecId_SpecValue.Add(new Guid("35ad6162-a885-4a6a-8044-78b68f6b2c4b"), "Purple");
// BlazarTech.QueryableValues.SqlServer
var queryableValues = DbContext.AsQueryableValues(SpecId_SpecValue);
var query = Products.Include(pd => pd.ProductsSpecifity)
.Where(x => x.ProductsSpecifity
.Where(e => queryableValues
.Where(v =>
v.Key == e.SpecifityID &&
v.Value == e.SpecifityValue
)
.Any()
).Count() == dynamicFilter.Count);
The query expresses "products of which all x.SpecifityID.ToString() + "_" + x.SpecifityValue combinations exactly match some given combinations".
Set combination operators like Except often don't play nice with EF for various reasons I'm not going into here. Fortunately, in many of these cases a work-around can be found by using Contains, which EF does support well. In your case:
var query = Products.Include(pd => pd.ProductsSpecifity)
.Where(z => z.ProductsSpecifity
.Select(x => x.SpecifityID.ToString() + "_" + x.SpecifityValue)
.Count(s => SpecId_SpecValue.Contains(s)) == filterCount);
Please note that the comparison is not efficient. Transforming database values before comparison disables any use of indexes (is not sargable). But doing this more efficiently isn't trivial in EF, see this.
My Linq Query keeps returning the null error on FirstOrDefault
The cast to value type 'System.Int32' failed because the materialized value is null
because it can't find any records to match on the ClinicalAssetID form the ClinicalReading Table, fair enough!
But I want the fields in my details page just to appear blank if the table does not have matching entry.
But how can I handle the null issue when using the order by function ?
Current Code:
var ClinicalASSPATINCVM = (from s in db.ClinicalAssets
join cp in db.ClinicalPATs on s.ClinicalAssetID equals cp.ClinicalAssetID into AP
from subASSPAT in AP.DefaultIfEmpty()
join ci in db.ClinicalINSs on s.ClinicalAssetID equals ci.ClinicalAssetID into AI
from subASSINC in AI.DefaultIfEmpty()
join co in db.ClinicalReadings on s.ClinicalAssetID equals co.ClinicalAssetID into AR
let subASSRED = AR.OrderByDescending(subASSRED => subASSRED.MeterReadingDone).FirstOrDefault()
select new ClinicalASSPATINCVM
{
ClinicalAssetID = s.ClinicalAssetID,
AssetTypeName = s.AssetTypeName,
ProductName = s.ProductName,
ModelName = s.ModelName,
SupplierName = s.SupplierName,
ManufacturerName = s.ManufacturerName,
SerialNo = s.SerialNo,
PurchaseDate = s.PurchaseDate,
PoNo = s.PoNo,
Costing = s.Costing,
TeamName = s.TeamName,
StaffName = s.StaffName,
WarrantyEndDate = subASSPAT.WarrantyEndDate,
InspectionDate = subASSPAT.InspectionDate,
InspectionOutcomeResult = subASSPAT.InspectionOutcomeResult,
InspectionDocumnets = subASSPAT.InspectionDocumnets,
LastTypeofInspection = subASSINC.LastTypeofInspection,
NextInspectionDate = subASSINC.NextInspectionDate,
NextInspectionType = subASSINC.NextInspectionType,
MeterReadingDone = subASSRED.MeterReadingDone,
MeterReadingDue = subASSRED.MeterReadingDue,
MeterReading = subASSRED.MeterReading,
MeterUnitsUsed = subASSRED.MeterUnitsUsed,
FilterReplaced = subASSRED.FilterReplaced
}).FirstOrDefault(x => x.ClinicalAssetID == id);
Tried this but doesn't work
.DefaultIfEmpty(new ClinicalASSPATINCVM())
.FirstOrDefault()
Error was:
CS1929 'IOrderedEnumerable<ClinicalReading>' does not contain a definition for 'DefaultIfEmpty' and the best extension method overload 'Queryable.DefaultIfEmpty<ClinicalASSPATINCVM>(IQueryable<ClinicalASSPATINCVM>, ClinicalASSPATINCVM)' requires a receiver of type 'IQueryable<ClinicalASSPATINCVM>'
Feel a little closer with this but still errors
let subASSRED = AR.OrderByDescending(subASSRED => (subASSRED.MeterReadingDone != null) ? subASSRED.MeterReadingDone : String.Empty).FirstOrDefault()
Error:
CS0173 Type of conditional expression cannot be determined because there is no implicit conversion between 'System.DateTime?' and 'string'
The original error means that some of the following properties of the ClinicalASSPATINCVM class - MeterReadingDone, MeterReadingDue, MeterReading, MeterUnitsUsed, or FilterReplaced is of type int.
Remember that subASSRED here
let subASSRED = AR.OrderByDescending(subASSRED => subASSRED.MeterReadingDone).FirstOrDefault()
might be null (no corresponding record).
Now look at this part of the projection:
MeterReadingDone = subASSRED.MeterReadingDone,
MeterReadingDue = subASSRED.MeterReadingDue,
MeterReading = subASSRED.MeterReading,
MeterUnitsUsed = subASSRED.MeterUnitsUsed,
FilterReplaced = subASSRED.FilterReplaced
If that was LINQ to Objects, all these would generate NRE (Null Reference Exception) at runtime. In LINQ to Entities this is converted and executed as SQL. SQL has no issues with expression like subASSRED.SomeProperty because SQL supports NULL naturally even if SomeProperty normally does not allow NULL. So the SQL query executes normally, but now EF must materialize the result into objects, and the C# object property is not nullable, hence the error in question.
To solve it, find the int property(es) and use the following pattern inside query:
SomeIntProperty = (int?)subASSRED.SomeIntProperty ?? 0 // or other meaningful default
or change receiving object property type to int? and leave the original query as is.
Do the same for any non nullable type property, e.g. DateTime, double, decimal, Guid etc.
You're problem is because your DefaultIfEmpty is executed AsQueryable. Perform it AsEnumerable and it will work:
// create the default element only once!
static readonly ClinicalAssPatInVcm defaultElement = new ClinicalAssPatInVcm ();
var result = <my big linq query>
.Where(x => x.ClinicalAssetID == id)
.AsEnumerable()
.DefaultIfEmpty(defaultElement)
.FirstOrDefault();
This won't lead to a performance penalty!
Database management systems are extremely optimized for selecting data. One of the slower parts of a database query is the transport of the selected data to your local process. Hence it is wise to let the DBMS do most of the selecting, and only after you know that you only have the data that you really plan to use, move the data to your local process.
In your case, you need at utmost one element from your DBMS, and if there is nothing, you want to use a default object instead.
AsQueryable will move the selected data to your local process in a smart way, probably per "page" of selected data.
The page size is a good compromise: not too small, so you don't have to ask for the next page too often; not too large, so that you don't transfer much more items than you actually use.
Besides, because of the Where statement you expect at utmost one element anyway. So that a full "page" is fetched is no problem, the page will contain only one element.
After the page is fetched, DefaultIfEmpty checks if the page is empty, and if so, returns a sequence containing the defaultElement. If not, it returns the complete page.
After the DefaultIfEmpty you only take the first element, which is what you want.
I am new to jayData and am trying to filter on an entity set. The filter needs to perform an complex evaluation beyond what I saw in the samples.
Here is a working sample of what I am trying to accomplish (the listView line isn't and is just there to show what I plan to do with the data):
function () {
var weekday = moment().isoWeekday()-1;
console.log(weekday);
var de = leagueDB.DailyEvents.toArray(function (events) {
console.log(events);
var filtered = [];
for (var e = 0; e < events.length;e++) {
console.log(events[e]);
console.log(events[e].RecurrenceRule);
var rule = RRule.fromString(events[e].RecurrenceRule);
var ruleOptions = rule.options.byweekday;
var isDay = ruleOptions.indexOf(weekday);
console.log(ruleOptions, isDay);
if(isDay =! -1)
{
filtered.push(events[e]);
}
}
$("#listView").kendoListView({dataSource:filtered});
});
Basically it is just evaluating a recurring rule string to see if the current day meets that criteria, if so add that event to the list for viewing.
But it blows up when I try to do this:
eventListLocal:leagueDB.DailyEvents.filter(function(e){
console.log("The Weekday is:"+viewModel.weekday);
console.log(e);
console.log("The recurrence rule is:"+e.RecurrenceRule);
var rruleOptions = viewModel.rruleOptions(e.RecurrenceRule);
if (rruleOptions !== -1) {
return true;
}
}).asKendoDataSource()
The error that is generating is:
Exception: Unable to resolve type:undefined
The thing is it seems to be occurring on "e" and the console logs like the event is not being passed in. However, I am not seeing a list either. In short I am really confused as to what is going on.
Any help would be appreciated.
Thanks,
You can't write filter expressions such as this.
When you write .filter(...), jaydata will parse your expression and then it will generate filter for underlying provider, for example where for webSql and $filter for oDataProvider.
Both JayData expression parser and the data provider itself should understand your filter.
Your filter is not suitable for this approach, because most of your codes are not familiar for jaydata expression parser and the underlying data provider, for example your console.log etc.
You can simplify your filter, or you should load all your data into an array, and then you can use filter method of array itself, there, you can write any filter you like, and your filter will work. Of course this has performance issue in some scenarios when your data set is large.
Read more on http://jaydata.org/tutorials/entityexpressions-the-heart-of-jaydata
In WordPress as you must already known, when using get_posts() or query_posts() or even WP_Query, it is not possible to order the returned posts by specifying a list of post ID in the order we want.
Instead we have to loop through the results and re-order them on the PHP side. This is a performance hit and a bad practice. Instead we should use built-in MySQL functions to retrieve the posts in the desired order upfront.
Thankfully there is the posts_orderby which can be used to specify a custom ORDERBY statement, like this:
// My list of post IDs in my custom order
$my_post_ids = array(1,3,2);
// Apply filter to the ORDERBY SQL statement
add_filter('posts_orderby', 'my_custom_orderby');
function my_custom_orderby($orderby_statement) {
global $my_post_ids;
$orderby_statement = 'FIELD(ID, '.implode(',',$my_post_ids).')';
return $orderby_statement;
}
// My custom query
$my_custom_query = new WP_Query(array('post_type' => 'post', 'post__in' => $my_post_ids);
However there is a problem with the above code, is that it will affect the order of all queries on the page! Including queries made by plugins, shortcodes, and so on.
Easy fix!
The simple way to fix this, is to apply the filter only one time, and remove it as soon as it is called, by putting a remove_filter() within the filter itself, so it is run only once:
// My list of post IDs in my custom order
$my_post_ids = array(1,3,2);
// Apply filter to the ORDERBY SQL statement
add_filter('posts_orderby', 'my_custom_orderby');
function my_custom_orderby($orderby_statement) {
// Disable this filter for future queries!
remove_filter(current_filter(), __FUNCTION__);
global $my_post_ids;
$orderby_statement = 'FIELD(ID, '.implode(',',$my_post_ids).')';
return $orderby_statement;
}
// My custom query
$my_custom_query = new WP_Query(array('post_type' => 'post', 'post__in' => $my_post_ids);
Because I set this filter just before my custom query, once I execute my custom query it should be filtered by the posts_orderby filter set above, which is then immediately disabled so it won't affect any future queries.
In theory, that's great, and it works great in most case scenarios!
An issue with WPML
However I have encountered a case when using the WPML plugin where this filter affects other queries than mine and causes errors. I believe the WPML plugin is creating a query of its own that is executed just before my own custom query, making my filter applies to the WPML query instead of mine!
Is there any possible way to add a check within the filter to make sure that it affects the correct query?
Thank you very much
Edit:
The fix for WPML
For information, while the accepted answer for this question is correct, it didn't solve the problem I was having with WPML. Here is how I fixed the WPML conflict:
// My list of post IDs in my custom order
$my_post_ids = array(1,3,2);
// Apply filter to the ORDERBY SQL statement
add_filter('posts_orderby', 'my_custom_orderby');
function my_custom_orderby($orderby_statement) {
// Disable this filter for future queries!
remove_filter(current_filter(), __FUNCTION__);
global $my_post_ids, $wpdb;
$orderby_statement = 'FIELD('.$wpdb->base_prefix.'posts.ID, '.implode(',',$my_post_ids).')';
return $orderby_statement;
}
// My custom query
$my_custom_query = new WP_Query(array('post_type' => 'post', 'post__in' => $my_post_ids);
This filter takes two parameters, $orderby and &$this. "this" being the WP_Query object. I'm not sure how to detect that WPML is making the call, but we can check that your call is the one being
made.
$my_post_ids = array(1,3,2);
add_filter( 'posts_orderby', 'my_custom_orderby', 10, 2 );
function my_custom_orderby( $orderby_statement, $object )
{
global $my_post_ids;
if( $my_post_ids != $object->query['post__in'] )
return $orderby_statement;
// Disable this filter for future queries!
remove_filter( current_filter(), __FUNCTION__ );
$orderby_statement = 'FIELD(ID, ' . implode( ',', $my_post_ids ) . ')';
return $orderby_statement;
}
I'm creating a query to search the db using EF. TdsDb being the EF context.
string searchValue = "Widget";
TdsDb tdsDb = new TdsDb();
IQueryable<Counterparty> counterparties;
I can do exact match:
counterparties = tdsDb.Counterparties.Where(x => x.CounterpartyName == searchValue);
or wildcard match:
counterparties = tdsDb.Counterparties.Where(x => x.CounterpartyName.Contains(searchValue));
But I want to be able to do both i.e. (psudo code)
counterparties = tdsDb.Counterparties.Where(x =>
if (searchValue.EndsWith("%"))
{
if (searchValue.StartsWith("%"))
{x.CounterpartyName.Contains(searchValue)}
else
{x.CounterpartyName.StartsWith(searchValue)}
}
else
{x => x.CounterpartyName == searchValue}
);
Now clearly I can't put an if statement in the where clause like that. But I also can't duplicate the queries: shown here they are hugely dumbed down. The production query is far longer, so having multiple versions of the same long query that vary on only one clause seems very unhealthy and unmaintainable.
Any ideas?
You should be able to use the ternary operator:
bool startsWithWildCard = searchValue.StartsWith("%");
bool endsWithWildCard = searchValue.EndsWith("%");
counterparties = tdsDb.Counterparties.Where(x =>
endsWithWildCard
? (startsWithWildCard
? x.CounterpartyName.Contains(searchValue)
: (x.CounterpartyName.StartsWith(searchValue)))
: (x.CounterpartyName == searchValue));
Did you test btw if querying by a searchValue that has an % at the beginning or end works as you expect? It might be possible that % will be escaped as a character to query for because StartsWith and Contains will prepend/append % wildcards to the generated SQL search term anyway. In that case you need to cut off the % from the searchValue before you pass it into StartsWith or Contains.