I am using JPA/JFreeChart to display data I collected with a microcontroller, however, I measure 14 sensors every 10 seconds. I have been measuring for over 2 months and I have over 7000000 sets of data.
Now to my actual problem, since I don't want to load 7000000 rows every time I start my program, I only want to use average values by minutes/hours. I have thought of using a NamedQuery however I don't know how to keep the relationship within it and make JPA use it since up until now the loading of the data has been done by JPA itself. Maybe I can just solve this by adding more annotations to this?
#OneToMany(mappedBy="sensor")
#OrderBy("timestamp ASC")
public List<Value> getValues() {
return this.values;
}
Thanks in advance!
Best Regards
Straight JPA does not allow filtering results, since this means that the entity's relationship no longer reflects exactly what is in the database, and it would have to standardize behavior on what is done when adding an entity to the relationship that isn't in the collection, but already exists in the database.
The easiest way for this mapping though would be to mark the attribute as #Transient. You can then use the get method to read the values from the database using when needed, and cache them in the entity if you want.
Many providers do allow adding filters to the queries used to bring in mappings, for instance EclipseLink allows setting #AdditionalCriteria on the mapping as described here: http://wiki.eclipse.org/EclipseLink/Development/AdditionalCriteria Or you can modify the mapping directly as shown here: http://wiki.eclipse.org/EclipseLink/Examples/JPA/MappingSelectionCriteria
Related
We have an entity and a corresponding table in the database with one additional column which contains digested hash of the entity fields, calculated each time programmatically in application. Entity has associations with two additional tables/entities which fields also take part in hashing.
Now a decision was made to get rid of one of the fields from the main entity (boolean flag) and exclude it from hashing, since it makes two otherwise identical entities get different hashes when one entity has its flag set to true, while other is false. Since hashes are different both entities get stored in the database, which is not what we want.
Removing the field is simple, but we also need to re-calculate hashes for entities which have been already stored in the database. Since there might be duplicates, we also need to get rid of one of two duplicated entries. This whole operation must be done once after migration.
The stack we use is Quarkus, Flyway, Hibernate with Panache, and PostgreSQL. I have tried to use Flyway callbacks with Event.AFTER_MIGRATE to get all existing entities from the db, but I can't use Panache since its not initialised yet by the time callback hits. Using plain java.sql.* Connection and Statement is pretty cumbersome, cause I need to fetch data from 3 tables, create entity from all of the fields, re-calculate hash and put it back, while taking care of possible conflicts. Another option would be to create a new REST API endpoint specifically for the job which the client will have to call after app has booted, but somehow I don't feel that that is the best solution.
How do you tackle this kind of a situation?
I am quite new to JPA. I have a particular repository that uses the keys that have parts that are set by the caller and some values that are automatically calculated using these values. There is a need for this :)
Since the keys and entities are simple Java classes it appears to me that I need to put my code that modifies the key (or substitutes it with an internal one with additional values) is the repository implementation. However I do not think that copying the code from SimpleJpaRepository to my custom repositories is a good idea...I think that something should be possible with the entity manager. Basically what I need is proxy that gets called every time something like find() or delete() is called, takes the entity, updates its key, passes the call over to the real repository implementation.
Could someone point me to the right direction or an example that does something similar?
Thanks!
In JPA, you have a bunch of events for this, just chose the one that suits you best. It looks like you are looking for #PrePersist.
http://www.objectdb.com/api/java/jpa/annotations/callback
That said, if the data of these fields is calculated based only in the data of the other fields, it goes against database normalization. A more sensate approach would be make the calculated field #Transient and provide only the getters, that will calculate the values based in the persistent fields.
I'm using EF (dll version is 4.4) to query against a database. The database contains several tables with course information. When having a look what actually is sent to the db I see a massive, almost 1300 line SQL query (which I'm not going to paste here because of it's size). The query I'm running on the context looks like:
entities.Plans
.Include("program")
.Include("program.offers")
.Include("program.fees")
.Include("program.intakes")
.Include("program.requirements")
.Include("program.codes")
.Include("focuses")
.Include("codes")
.Include("exceptions")
.Include("requirements")
where plans.Code == planCode
select plans).SingleOrDefault();
I want to avoid having to go back to the server when collecting information from each of the related tables but with such a large query I'm wondering if there is there a better way of doing this?
Thanks.
A bit late but, as your data is only changing once a day, look at putting everything you need into an indexed view and place this view in your EF model.
You can usually add the .Include() after the where clause. This means that you're only pulling out information that matches what you want, see if that reduces your query any.
As you are performing an eager loading, so if you are choosing the required entities then its fine. Otherwise you can go with Lazy Loading, but as you specified that you don't want database round trip, so you can avoid it.
I would suggest, If this query is used multiple times then you can use compiled query. So that it will increase the performance.
Go through this link, if you want..
http://msdn.microsoft.com/en-us/library/bb896297.aspx
If you're using DbContext, you can use the .Local property on the context to look if your entity is already retrieved and therefore attached to the context.
If the query had run before and your root Plan entities are already attached based on Plan.Code == planId, presumably its sub-entities are also already attached since you did eager loading, so referring to them via the navigation properties won't hit the DB for them again during the lifetime of that context.
This article may be helpful in using .Local.
You may be able to get a slightly more concise SQL query by using projection rather than Include to pull back your referenced entities:
var planAggregate =
(from plan in entities.Plans
let program = plan.Program
let offers = program.Offers
let fees = program.Fees
//...
where plan.Code == planCode
select new {
plan
program,
offers,
fees,
//...
})
.SingleOrDefault();
If you disable lazy loading on your context this kind of query will result in the navigation properties of your entities being populated with the entities which were included in your query.
(I've only tested this on EF.dll v5.0, but it should behave the same on EF.dll v4.4, which is just EF5 on .NET 4.0. When I tested using this pattern rather than Include on a similarly shaped query it cut about 70 lines off of 500 lines of SQL. Your mileage may vary.)
I'm building my EF (v4.0.30319) data model from my SQL Server database. Each table has Created and Updated fields populated via database triggers.
This database is the backend of a ASP.NET Web API application and I recently discovered a problem. The problem is, since the Created and Updated fields are not populated in the model passed into the api endpoint, they are written to the database as NULL. This overwrites any values already in the database for those properties.
I discovered I can edit the EF data model and just delete those two columns from the entity. It works and the datetimes are not overwritten with NULL. But this leads to another, less serious but more annoying, problem... my data model has a bunch of tables that contain these properties and all the tables need to be updated by removing these two columns.
Is there a way to tell EF to ignore certain columns in entities across the entire data model without manually deleting them?
As far as I know, no. Generating the model from the database is going to always create all of the fields from the database table. Well, as long as it has a primary key it will.
It is possible to only update the fields that you want i.e. don't include the "Created" and "Updated" fields in your create and update methods. I'd have to say though, that I'd think it'd be better if those fields didn't even exist on the model at that point. You may at one point see those fields on the model and not remember that they won't get persisted to the DB.
You may want to look into just inserting the datetimes into those fields when you call your create() and update() methods. Then you could just ditch the triggers. You'd obviously want to use a class library for all of your database operations so this functionality would be in one place. To keep it nice and neat, you know?
I am having a model "Events" (Zend_Db_Table_Abstract) that's got various relationships to other models. Usually I think I would do something like this to find it and its relationships:
$events = new Events();
$event = $events->find($id)->current();
$eventsRelationship1 = $event->findDependentRowset('Relationship1');
As the relationship is already set up I am wondering if there's any sort of automatic join available or something. Every time I fetch my event I need to have all the relationships, too. Currently I see only two ways to achieve that:
Build the query myself, hard coded. Don't like this, because it's working around the already set up relationship and "model method convenience".
Fetch every related object with a single query. This one's ugly, too, as I have to trigger too many queries.
This goes even a step further when thinking about getting a set of multiple rows. For a single event I may query the database multiple times, but when fetching 100 rows joins are just elementary.
So, does anyone know a way to create joins by using those relationships or is there no other way than hardcoding the query?
Thanks in advance
Arne
The way to solve this challenge is to 'upgrade' your database access to use the dataMapper pattern.
You are essentially adding an extra layer between the model in your application an their representation in the db. This mapper layer allows you read/write data from different tables - rather than a direct link between one model and one table.
Here is a good tutorial to follow. (There are some bits you can skip - I left out all the getters and setters as its just me using the code).
It takes a little while to get your head round the way it works, when you've just been using Zend_Db_Table_Abstract, but it is worth it.