Data Model
As part of re-writing a legacy I have come across the above data model that we want to model using Spring Data JPA.
The section entity contains subsections stored in the same table with a parent child relation
Each section contains many questions
Each question contains many profiles.
With the below code in spring data I encounter N + 1 issue.
There was a question in the past for a similar issue (JPA/Hibernate load recursive object graph without N+1/cartesian products)
I would like to know if there are new and better ways to solve this N+1 issue?
Related
I am using JPA/JFreeChart to display data I collected with a microcontroller, however, I measure 14 sensors every 10 seconds. I have been measuring for over 2 months and I have over 7000000 sets of data.
Now to my actual problem, since I don't want to load 7000000 rows every time I start my program, I only want to use average values by minutes/hours. I have thought of using a NamedQuery however I don't know how to keep the relationship within it and make JPA use it since up until now the loading of the data has been done by JPA itself. Maybe I can just solve this by adding more annotations to this?
#OneToMany(mappedBy="sensor")
#OrderBy("timestamp ASC")
public List<Value> getValues() {
return this.values;
}
Thanks in advance!
Best Regards
Straight JPA does not allow filtering results, since this means that the entity's relationship no longer reflects exactly what is in the database, and it would have to standardize behavior on what is done when adding an entity to the relationship that isn't in the collection, but already exists in the database.
The easiest way for this mapping though would be to mark the attribute as #Transient. You can then use the get method to read the values from the database using when needed, and cache them in the entity if you want.
Many providers do allow adding filters to the queries used to bring in mappings, for instance EclipseLink allows setting #AdditionalCriteria on the mapping as described here: http://wiki.eclipse.org/EclipseLink/Development/AdditionalCriteria Or you can modify the mapping directly as shown here: http://wiki.eclipse.org/EclipseLink/Examples/JPA/MappingSelectionCriteria
I'm developing an Lighswitch application in VS2012 with an external SQL Server 2008. I'm using a WCF RIA Service to fetch my data, using these techniques: http://www.c-sharpcorner.com/UploadFile/raj1979/how-to-use-wcf-ria-service-in-lightswitch-2012/.
My main table contains some columns with static data and about 5 columns with foreign keys. The static data columns loads instant but all the foreign properties takes 3-5 secs to load on a screen with ~100 rows. This is not acceptable..
The problem for me is that I don't know where the bottleneck might be. Any ideas where I could start looking? The only thing I could think of so far is that my ADO.NET model uses Lazy Loading, and it does.
All the foreign key properties only has two columns with a PK and some text, so they're not data heavy.
Any help appreciated, thanks!
I don't know if this is the exact answer to your speed problem, but that article you quoted suggests that you add a new ADO.NET Data Model to the class library containing the RIA Service.
There's absolutely no reason to generate a second data model, as LightSwitch already generates one when you attach the external data source (your SQL Server database). You may as well re-use it. An EF data model takes time to construct, so I suppose that using two EF data models could explain the slowness you're experiencing, but it could also be something else entirely.
Have a look at this article (the original RIA Service/LightSwitch article, & still the best), which among several other techniques, Eric will show you how to re-use the data model that LightSwitch creates for your data source.
How Do I: Display a Chart Built On Aggregated Data
In this article, the RIA service is created to specifically aggregate data to display in a chart, but you can do whatever you need to do in it. It doesn't have to aggregate data. I use RIA services with an attached data source (a SQL database) extensively, to provide data for lists, reports, as well as lookup data for combo boxes.
I am almost a beginer at this new art, but as Yan says in his book, instead a very good one, lightswitch is directed to the screen. I dare to ask you if you need to have 100 lines in your screen. You can't even see them all. If you reduce that number to 20 you will reduce your waiting time to 1/5th. I've had the same problem and now I use the least possible number of rows in screen. For that to be possible without disturbing the business performance, I choose to open screens with the maximum number of parameters, not required, which I send when I open them. If one of the parameters is not the option in the case I mention it as "Nothing". And so, I can have a minimum number of lines in the grid, fullfiling the purpous of teh screen, the program performance and the business performance.
I'm developing a data access layer for a database with over 700 tables. I created the model including all the tables, which generated a huge model. I then changed the model to use DBContext from 4.1 which seemed to improve how it compiled and worked. The designer didnt seem to work at all.
I then created a test app which just added two records to the table, but the processor went 100% in the db.SaveChanges method. Being a black box it was difficult to accertain what went wrong.
So my questions are
Is the entity framework the best approach to a large database
If so, should the model be broken down into logical areas. I did note that you cant have the same sql table in multiple models
I have read that the code only approach is best in these large cases. What is that.
Any guidance would be truly appreciated
Thanks
Large database is always something special. Any technology has some pros and cons when working with a large database.
The problem you have encountered is the most probably related to building the model. When you start the application and use EF related stuff for the first time EF must build the model description and compile it - this is the most time consuming operation you can find in EF. Complexity of this operation grows with number of entities in the model. Once the model is compiled it is reused for the whole lifetime of the application (if you restart the application or unload application domain the model must be compiled again). You can avoid this by precompiling the model. It is done at design time where you use some tool to generate code from the model and you include that code into your project (it must be done again after each change in the model). For EDMX based models you can use EdmGen.exe to generate views and for code first based models you can use EF Power Tools CTP1.
EDMX (the designer) was improved in VS 2010 SP1 to be able to work with large models but I still think the large in this case is around 100 entities / tables. In the same time you rarely need 715 tables in the same model. I believe that these 715 tables indeed model several domains so you can divide them into multiple models.
The same is true when you are using DbContext and code first. If you model a class do you think that it is correct design when the class exposes 715 properties? I don't think so but that is exactly what your derived DbContext looks like - it has a public property for each exposed entity set (in the simplest mapping it means one property per table).
Same entity can be used in multiple models but you should try to avoid it as much as possible because it can introduce some complexities when loading entity in one context type and using it in other context type.
Code only = code first = Entity framework when you define mapping in the code without using EDMX.
take a look this post.
http://blogs.msdn.com/b/adonet/archive/2008/11/24/working-with-large-models-in-entity-framework-part-1.aspx
Setup: I have a simple web app that has a handfull of forms, each on a separate page. These forms represent patient data. There is a one-to-one relationship between patient and all these forms/entities. Each form maps directly to a db table and a JPA entity, maybe not the best architecture but it works and is simple.
Question: If form/entity A and form/entity B share a common chunk of data (one of more fields), what is the best way to handle that in JPA. I.E. - If the data gets inserted via form A, I need it to show up in form B as existing data and vice versa. In other words its logical for both entities to contain that data. I believe I will have to move the common data into its own entity and define the relationships that way, but I have tried many different ways and none gets me all the way, at least with basic JPA. Can this be done through pure JPA relationships or will I have to write a bunch of code to make this happen manually. Not looking for code specifically, just the correct way to model this data. Thanks.
If the forms have separate tables with duplicate columns for the common data, then you cannot directly share the data. You will need to copy the data from one Entity to the other in your application. You could use a Embeddable to define the common data, but would still need to copy this Embeddable from one form to the other.
If you put the common data in a 3rd table, then you can share the data. Form A and Form B would define a OneToOne relationship to the common data.
I'm working with a large hierarchical data set in sql server - modelled using the standard "EntityID, ParentID" kind of approach. There are about 25,000 nodes in the whole tree.
I often need to access subtrees of the tree, and then access related data that hangs off the nodes of the subtree. I built a data access layer a few years ago based on table-valued functions, using recursive queries to fetch an arbitrary subtree, given the root node of the subtree.
I'm thinking of using Entity Framework, but I can't see how to query hierarchical data like
this. AFAIK there is no recursive querying in Linq, and I can't expose a TVF in my entity data model.
Is the only solution to keep using stored procs? Has anyone else solved this?
Clarification: By 25,000 nodes in the tree I'm referring to the size of the hierarchical dataset, not to anything to do with objects or the Entity Framework.
It may the best to use a pattern called "Nested Set", which allows you to get an arbitrary subtree within one query. This is especially useful if the nodes aren't manipulated very often: Managing hierarchical data in MySQL.
In a perfect world the entity framework would provide possibilities to save and query data using this data pattern.
Everything IS possible with Entity Framework but you have to hack and slash your way in to it. The database I am currently working against has too many "holder tables" since Points for instance is shared with both teams and users. Both users and teams can also have a blog.
When you say 25 000 nodes do you mean navigational properties? If so I think it could be tricky to get the data access in place. It's not hard to navigate, search etc with entity framework but I tend to model on paper then create the database based on how I want to navigate while using entity framework. Sounds like you don't have that option.
Thanks for these suggestions.
I'm beginning to realise that the answer is to remodel the data in the database - either along the lines of nested sets as Georg suggests, or maybe a transitive closure table, which I've just come across.
That way, I'm hoping to get two key benefits:
a) faster querying aginst arbitrary subtrees
b) a data model which no longer requires recursive querying - so perhaps bringing it within easy reach of the Entity Framework!
It's always amazing how so often the right answer to a difficult problem is not to answer it, but to do something else instead!