I'm in the process of building the capability for a user to perform ad-hoc queries on a SQL Server database. The resulting query will take on the following basic form:
SELECT <ONE TO MANY USER SELECTED FIELDS>
FROM <ONE TO MANY TABLES DETERMINED BY FIELDS SELECTED BY USER>
WHERE <ZERO TO MANY CRITERIA FOR THE SELECTED FIELDS>
It's a guarantee that the selection will most likely span more than one table.
Some (not all) of the fields may have 0 or more filter criteria for a particular field.
My application is using the default EF4 classes within ASP.NET MVC 2 using C#. I am currently passing in an object called QueryItem that contains all the information for a particular criteria.
My question(s) are:
What is the best approach for coding this? (Code samples of what I have to date below).
Can this be done with Linq2SQL or should I use ADO.NET(My current approach)
If ADO.NET is the best way, how do you access the DBConnection within EF4?
Note: I intend to refactor this into SQLParameter objects, to protect against SQL injection. My goal right now is best practice in developing the query first.
QueryItem class:
public class QueryItem
{
public bool IsIncluded { get; set; }
public bool IsRequired { get; set; }
public string LabelText { get; set; }
public string DatabaseLoc { get; set; }
public List<string> SelectedValue { get; set; }
public List<SelectListItem> SelectList { get; set; }
}
Query Parsing Code
foreach(QueryItem qi in viewModel.StandardQueryItems)
{
string[] dLoc = qi.DatabaseLoc.Split(new Char[] { '.' }); //Split the table.fieldname value into a string array
if(qi.IsIncluded == true) //Check if the field is marked for inclusion in the final query
{
fields.Append(qi.DatabaseLoc + ","); //Append table.fieldname to SELECT statement
if(!tables.ToString().Contains(dLoc[0])) // Confirm that the table name has not already been added to the FROM statement
{
tables.Append(dLoc[0] + ","); //Append the table value to the FROM statement
}
}
if(qi.SelectedValue != null)
{
if(qi.SelectedValue.Count == 1)
{
query.Append(qi.DatabaseLoc + " = '" + qi.SelectedValue[0].ToString() + "'");
}
else
{
foreach(string s in qi.SelectedValue)
{
//Needs to handle "IN" case properly
query.Append(qi.DatabaseLoc + " IN " + qi.SelectedValue.ToString());
}
}
}
}
I have built a similar system to what you are describing in the past by passing in a single parameter to a stored procedure of type xml. By doing so, you can actually specify(in xml), what all you would like to report off of and build the SQL necessary to return the results you want.
This also makes your C# code easier, as all you have to do is generate some xml that your procedure will read. Generating Dynamic SQL is definitely not something you should use unless you have to, but when you want to allow users to dynamically select what they want to report off of, it's pretty much the only way to go about doing it.
Another option for you might be to look into Reporting Services - that will allow the user to pick what fields they want to view and save that particular 'report' in their own section where they can then go back and run it again at any time.. You could also create the reports for them if they aren't computer savvy(which is a lot easier to do with report builder, provided that all they need is data and no special features).
Either way you go about it, their are pros and cons to both solutions.. You'll have to determine which option is best for you.
xml/dynamic sql: Hard to maintain/make changes to.(I feel sorry for anyone who has to come behind someone who is generating dynamic sql and try to understand the logic behind the mess).
reporting services: very easy to spit out reports that look good, but it's a little less flexible and it's not free.
Related
With its recent improvements, I'm looking to move from Dapper back to EF (Core).
The majority of our code currently uses the standard patterns of mapping entities to tables, however we'd also like to be able to make simple ad-hoc queries that map to a simple POCO.
For example, say I have a SQL statement which returns a result set of strings. I created a class as follows...
public class SimpleStringDTO
{
public string Result { get; set; }
}
.. and called it as such.
public DbSet<SimpleStringDTO> SingleStringResults { get; set; }
public IQueryable<SimpleStringDTO> Names()
{
var sql = $"select name [result] from names";
var result = this.SingleStringResults.FromSql(sql);
return result;
}
My thoughts are that I could use the same DBSet and POCO for other simple queries to other tables.
When I execute it, EF throws an error "The entity type 'SimpleStringDTO' requires a primary key to be defined.".
Do I really need to define another field as a PK? There'll be cases where there isn't a PK defined. I just want something simple and flexible. Ideally, I'd rather not define a DBSet or POCO at all, just return the results straight to an IEnumerable<string>.
Can someone please point me towards best practises here?
While I wait for EF Core 2.1 I've ended up adding a fake key to my model
[Key]
public Guid Id { get; set; }
and then returning a fake Guid from SQL.
var sql = $"select newid(), name [result] from names";
I have the following in Entity Framework Core:
public class Book {
public Int32 Id { get; set; }
public String Title { get; set; }
public virtual Theme Theme { get; set; }
}
public class Theme {
public Int32 Id { get; set; }
public String Name { get; set; }
public Byte[] Illustration { get; set; }
public virtual ICollection<Ebook> Ebooks { get; set; }
}
And I have the following linq query:
List<BookModel> books = await context.Books.Select(x =>
new BookModel {
Id = x.Id,
Name = x.Name,
Theme = new ThemeModel {
Id = x.Theme.Id,
Name = x.Theme.Name
}
}).ToListAsync();
I didn't need to include the Theme to make this work, e.g:
List<BookModel> books = await context.Books.Include(x => x.Theme).Select(x => ...
When will I need to use Include in Entity Framework?
UPDATE
I added a column of type Byte[] Illustration in Theme. In my projection I am not including that column so will it be loaded if I use Include? Or is never loaded unless I have it in the projection?
In search for an official answer to your question from Microsoft's side, I found this quote from Diego Vega (part of the Entity Framework and .NET team) made at the aspnet/Announcements github
repository:
A very common issue we see when looking at user LINQ queries is the use of Include() where it is unnecessary and cannot be honored. The typical pattern usually looks something like this:
var pids = context.Orders
.Include(o => o.Product)
.Where(o => o.Product.Name == "Baked Beans")
.Select(o =>o.ProductId)
.ToList();
One might assume that the Include operation here is required because of the reference to the Product navigation property in the Where and Select operations. However, in EF Core, these two things are orthogonal: Include controls which navigation properties are loaded in entities returned in the final results, and our LINQ translator can directly translate expressions involving navigation properties.
You didn't need Include because you were working inside EF context. When you reference Theme inside the anonymous object you are creating, that's not using lazy loading, that's telling EF to do a join.
If you return a list of books and you don't include the themes, then when you try to get the theme you'll notice that it's null. If the EF connection is open and you have lazy loading, it will go to the DB and grab it for you. But, if the connection is not opened, then you have to get it explicitely.
On the other hand, if you use Include, you get the data right away. Under the hood it's gonna do a JOIN to the necessary table and get the data right there.
You can check the SQL query that EF is generating for you and that's gonna make things clearer for you. You'll see only one SQL query.
If you Include a child, it is loaded as part of the original query, which makes it larger.
If you don't Include or reference the child in some other way in the query, the initial resultset is smaller, but each child you later reference will lazy load through a new request to the database.
If you loop through 1000 users in one request and then ask for their 10 photos each, you will make 1001 database requests if you don't Include the child...
Also, lazy loading requires the context hasn't been disposed. Always an unpleasant surprise when you pass an Entity to a view for UI rendering for example.
update
Try this for example and see it fail:
var book = await context.Books.First();
var theme = book.Theme;
Then try this:
var book = await context.Books.Include(b => b.Theme).First();
var theme = book.Theme;
I have two relevant tables here:
public partial class List
{...
public int RegionId { get; set; }
[ForeignKey("RegionId")]
public virtual Region Region { get; set; }
...}
public partial class Region
{
public Region()
{
Lists = new HashSet<List>();
}
public int RegionId { get; set; }
[Required]
[StringLength(255)]
public string Name { get; set; }
public DateTime Added { get; set; }
public virtual ICollection<List> Lists { get; set; }
}
Here's the contents of the Regions table:
RegionId Name
1 Global
2 China
3 USA
4 UK
8 Canada
9 Spain
10 France
On the breeze side of things, I pull down the Regions:
var query = new breeze.EntityQuery().from("Regions").select("RegionId,Name").orderBy("RegionId");
return $rootScope.breezeManager.executeQuery(query).then(function (data) {
service.regions = data.results;
It seems to work:
service.regions[2]; // Name: "USA", RegionId: 3
However, when I try to create a new entity:
var newList = $rootScope.breezeManager.createEntity('List', listValues);
And in listValues, I specify { RegionId: 3, ...}:
newList.RegionId // 3
newList.Region // null
That would be strange already perhaps, but the really frustrating thing is, if I specify another value, like 1, it works. Same for 2, and 4:
newList.RegionId // 1
newList.Region.Name // "Global"
I've been poring through the code (and the internet) for hours trying to figure this out, but it's eluded me, and thus qualifies for my first ever SO question!
Update
I'm now even more confused. I planned to workaround this by manually setting the Region after the createEntity call, so I added this line of code right above createEntity:
var region = Enumerable.From(service.regions).Single("$.RegionId == " + listValues.RegionId);
However, after doing so, and with no other changes, newList now correctly gets a populated Region, even for 3 (USA)! I then commented out that line and tried again, and it still worked. Tried a few other times, various combinations of things, and now it's not working again, even with that line. Not sure if that helps. I'll keep experimenting.
I don't know what you're trying to do exactly but I do see that you have a fundamental misunderstanding.
I believe you are expecting your first query to return Region entities and you think that service.regions is populated with Region entities in the success callback.
Neither is true. Your query contains a select() clause which makes it what we call a projection. A projection returns data objects, not entities. There are no Region entities in cache either.
You may have seen elsewhere - perhaps a Pluralsight video - where a projection query did return entities. That happens when the query includes a toType clause which actually casts the projected data into instances of the targeted type. That's a trick you should only perform with great care, knowing what you are doing, why, and the limitations. But you're not casting in this query so this digression is besides the point.
It follows that the service.regions array holds some data objects that contain Region data but it does not hold Region entities.
This also explains why, when you created a new List entity with RegionId:3, the new entity's Region property returned null. Of course it is null. Based only on what you've told us, there are no Region entities in cache at all, let alone a Region with id=3.
I can't explain how you're able to get a Region with id=1 or id=2. I'm guessing there is something you haven't told us ... like you acquired these regions by way of some other query.
I don't understand why you're using a projection query in the first place. Why not just query for all regions and be done with it?
breeze.EntityQuery.from("Regions").orderBy("RegionId")
.using(manager).execute(function(data) {
service.regions = data.results;
});
I don't understand your update at all; that doesn't look like a valid EF query to me.
Tangential issues
First, why are you using a HashSet<List> to initialize the Lists property in your server-side Region model class? That's a specialized collection type that only confuses matters here. It doesn't do what the casual reader might think it does. While it prevents someone from adding the same object reference twice, it doesn't do the more important job of preventing someone from adding two different List objects with the same RegionId. The simple, obvious, and correct thing to do is ...
Lists = new System.Collections.Generic.List<List>();
// the full type name is only necessary because you burned the name "List"
// when defining your model class.
Second, on the client-side, please don't extend the Angular $rootScope with anything. That kind of global variable "pollution" is widely regarded as "bad practice". Keep your Breeze stuff inside a proper Angular service and inject that service when you need something.
I'll try and be as descriptive as I can in this post. I've read a dozen or more SO questions that were peripherally related to my issue, but so far none have matched up with what's going on.
So, for performing audit-logging on our database transactions (create, update, delete), our design uses an IAuditable interface, like so:
public interface IAuditable
{
Guid AuditTargetId { get; set; }
int? ContextId1 { get; }
int? ContextId2 { get; }
int? ContextId3 { get; }
}
The three contextual IDs are related to how the domain model is laid out, and as noted, some or all of them may be null, depending on the entity being audited (they're used for filtering purposes for when admins only want to see the audit logs for a specific scope of the application). Any model that needs to be audited upon a CUD action just needs to implement this interface.
The way that the audit tables themselves are being populated is through an AuditableContext that sits between the base DbContext and our domain's context. It contains the audit table DbSets, and it overrides the SaveChanges method using the EF ChangeTracker, like so:
foreach (var entry in ChangeTracker.Entries<IAuditable>())
{
if (entry.State != EntityState.Modified &&
entry.State != EntityState.Added &&
entry.State != EntityState.Deleted)
{
continue;
}
// Otherwise, make audit records!
}
base.SaveChanges();
The "make audit records" process is a slightly-complex bit of code using reflection and other fun things to extract out fields that need to be audited (there are ways for auditable models to have some of their fields "opt out" of auditing) and all that.
So that logic is all well and good. The issues comes when I have an auditable model like this:
public class Foo: Model, IAuditable
{
public int FooId { get; set; }
// other fields, blah blah blah...
public virtual Bar Bar { get; set; }
#region IAuditable members
// most of the auditable members are just pulling from the right fields
public int? ContextId3
{
get { return Bar.BarId; }
}
#endregion
}
As is pointed out, for the most part, those contextual audit fields are just standard properties from the models. But there are some cases, like here, where the context id needs to be pulled from a virtual complex property.
This ends up resulting in a NullReferenceException when trying to get that property out from within the SaveChanges() method - it says that the virtual Bar property does not exist. I've read some about how ChangeTracker is built to allow lazy-loading of complex properties, but I can't find the syntax to get it right. The fields don't exist in the "original values" list, and the object state manager doesn't have those fields, I guess because they come from the interface and not the entities directly being audited.
So does anyone know how to get around this weird issue? Can I just force eager-loading of the entire object, virtual properties included, instead of the lazy loading that is apparently being stubborn?
Sorry for the long-ish post, I feel like this is a really specific problem and the detail is probably needed.
TIA! :)
I have a ServiceOperation to query items available to a certain user at a certain time.
In short this methode does:
var fullResult = from i in Items where ... select i; //get ALL possible items where...,
Lets say this returns items {A, B, C, D}.
A second query filters out which of those items the calling user has access to.
var clientResult = from ci in fullResult where (privilege's and schedule's are true)
This mite result in {A, C } and is returned.
The result on the client side is: only the list of items the client has access to is displayed. This can be annoying since you don't know if you made a mistake in searching, or the item is just not available right now.
What I would like to be able to do is show all possible results to the client {A, B, C, D} yet FLAG B and D in this case as unavailable.
My entity has already a property isReadOnly I could use.
Can I write a query to not just filter out, but also flag any remaining results as read only? An ideal result would be {A, B.isREadOnly=true, C, D.isReadOnly=true}
Or did I reach the limit of what is doable and do I have to write a traditional WCF web method, creating a separate class, returning a list of results?
PS: this 'isReadOnly' property is only used for this, I don't mind it being changed in the DB at all
Thanx for any pointers
Andreas
If I were you I would consider not returning the entity directly out of your service and instead map it to something that has the ReadOnly property. For example, if your entity is:
public class A
{
public string Name { get; set; }
}
Then you could have a data contract like this:
[DataContract]
public class AExtra
{
[DataMember]
public string Name { get; set; }
[DataMember]
public bool IsReadOnly { get; set; }
}
what this means is that you could do this in your query:
var res = from a
in Items
where (...whatever your original logic is)
select new AExtra
{
Name = a.Name,
IsReadOnly = (...put your logic for determining ReadOnly in here)
};
And then return res from your service operation.
Just an opinion really but I like to do things like this rather than send the entities directly out of the service - it always gives me a bit more freedom to change things without having too many knock-on effects.