I am using Java JPA-Eclipselink persistence (v2.6.9) in combination with Apache Tomcat Web Server. I am fetching from database large amount of data from nested tables on each web service invocation. Data in the database is constantly being expanded with new data.
We predict that number of fetches are going to increase, so we ran stress test and find out that expected traffic is going to cause database clogging. In order to avoid clogging, I would like to implement temporary caching in a way that eclipse link should fetch data and hold it for a minute in a cache and fetch new data after a minute. If I understood right I am looking for L2 caching in Eclipselink.
I have tried several options, but eclipse link is always fetching data from database on each call of web service method.
Could you please help me make eclipselink use caching in intervals of 60 seconds?
Please find below my attempts.
Simplified enities
#Entity
#Cache(
type=CacheType.SOFT, // Cache everything until the JVM decides memory is low.
size=64000, // Use 64,000 as the initial cache size.
expiry=36000000 // 10 minutes
)
#Table(name="SITUATION_DATA")
#NamedQuery(name="SituationData.findAll", query="SELECT s FROM SituationData s")
public class DatexSituationData implements Serializable {
private static final long serialVersionUID = 1L;
//bi-directional many-to-one association to SituationRecord
#OneToMany(mappedBy="datexSituationData", cascade = CascadeType.PERSIST, fetch =
FetchType.EAGER)
#JoinFetch(value=JoinFetchType.OUTER)
private List<SituationRecord> situationRecords;
}
#Entity
#Table(name="SituationRecord")
#NamedQuery(name="SituationRecord.findAll", query="SELECT s FROM SituationRecord s")
public class SituationRecord implements Serializable {
private static final long serialVersionUID = 1L;
#OneToMany(mappedBy="situationRecord", cascade = CascadeType.PERSIST, fetch = FetchType.EAGER)
#JoinFetch(value=JoinFetchType.OUTER)
private List<SituationRecordComment> situationRecordComment;
#OneToMany(mappedBy="situationRecord", cascade = CascadeType.PERSIST, fetch = FetchType.EAGER)
#JoinFetch(value=JoinFetchType.OUTER)
private List<SituationRecordTypeElement> situationRecordTypeElements;
//bi-directional many-to-one association to SituationLocation
#ManyToOne(cascade = CascadeType.PERSIST, fetch = FetchType.EAGER)
#JoinFetch(value=JoinFetchType.OUTER)
#JoinColumn(name="ID_LOKACIJE")
private SituationLocation situationLocation;
//bi-directional many-to-one association to DatexSituationData
#ManyToOne()
#JoinColumns({
#JoinColumn(name="SITUATION_ID", referencedColumnName="ID", nullable=false),
#JoinColumn(name="SITUATION_VERSION", referencedColumnName="VERSION", nullable=false)
})
private DatexSituationData datexSituationData;
}
#Entity
#Table(name="SITUATION_LOCATIONS")
#NamedQuery(name="SituationLocation.findAll", query="SELECT s FROM SituationLocation s")
public class SituationLocation implements Serializable {
private static final long serialVersionUID = 1L;
#Id
GeneratedValue(strategy=GenerationType.SEQUENCE, generator="situation_location_seq")
#SequenceGenerator(name="situation_location_seq", sequenceName="SEQ_SITUATION_LOCATION",
allocationSize=1)
#Column(name="ID_LOKACIJE", unique=true, nullable=false)
private long idLokacije;
//bi-directional many-to-one association to SituationRecord
#OneToMany(mappedBy="situationLocation", cascade = CascadeType.PERSIST)
private List<SituationRecord> situationRecords;
}
Fetching from database
String sQuery =
"SELECT ds FROM SituationData ds where CONCAT(ds.id.version, ds.id.id) in (select
CONCAT(max(FUNC('TO_NUMBER', ds1.id.version)), ds1.id.id) from SituationData ds1 group by
ds1.id.id)";
EntityManager em = Emf.getInstance().getFactory().createEntityManager();
Query q = em.createQuery(sQuery, DatexSituationData.class);
q.setHint("eclipselink.join-fetch", "ds.situationRecords");
q.setHint("eclipselink.join-fetch", "ds.situationRecords.situationLocation");
q.setHint("eclipselink.join-fetch", "ds.situationRecords.situationRecordTypeElements");
q.setHint("eclipselink.join-fetch",
"ds.situationRecords.situationRecordGeneralPublicCommentMappings.
situationRecordGeneralPublicComments");
q.setHint("eclipselink.sql.hint", "/*+ leading(t1 t0) */");
q.setHint("eclipselink.query-results-cache", "true");
q.setHint("eclipselink.query-results-cache.expiry", "120000");
lResult = q.getResultList();
Persistence.xml
I have tried following properties
<!--property name="eclipselink.query-results-cache" value="true"/>
<property name="eclipselink.query-results-cache.expiry" value="120000"/>
<property name="eclipselink.cache.shared.default" value="true"/-->
<property name="eclipselink.cache.shared.default" value="false"/>
<property name="eclipselink.cache.shared.SituationData" value="true"/>
<property name="eclipselink.cache.type.SituationData" value="SOFT"/>
<property name="eclipselink.cache.size.SituationData" value="9400000"/>
The settings you have put on your entity are for entity caching, but complex queries still need to use the database to figure out which entities match read all/list queries. You should still see improvements though as executing this query multiple times will use that entity cache - while the data is still 'fetched' from the DB, you are avoiding the overhead of building/refreshing instances. A better test might be to check the overheads involved in:
EntityManager em1 = Emf.getInstance().getFactory().createEntityManager();
EntityManager em2 = Emf.getInstance().getFactory().createEntityManager();
DatexSituationData data1 = em1.find(DatexSituationData.class, 1L);
DatexSituationData data2 = em2.find(DatexSituationData.class, 1L);
The first find will go to the database and create instances to populate the shared and local caches, while the second should just use the shared cache to create a copy for its local EntityManager cache. This is a test that shows if your shared cache and its settings are working without other complexities.
Some issues with your settings and what you've shown though:
Entity class (and name) is "DatexSituationData" while you have named queries using "SituationData"
"eclipselink.cache.shared.SituationData" again is for SituationData
"eclipselink.cache.shared.default" is false. This turns off the cache for all entities, which will cause some confusion in your app as SituationData/DatexSituationData might be referenced in the shared cache, but its referenced SituationRecords won't forcing DB hits to load.
You've also set "query-results-cache" options as a query hint passed to the query. This is described in the docs and is what you may want, as it will allow the query results, based on the parameters used, to be cached. Limitations from the documentation:
Only named queries can have their results cached, dynamic queries
cannot use the query results cache. As well, if you modify a named
query before execution, such as setting hints or properties, then it
cannot use the cached results.
So to test query caching, you should use only a named query with no changes to it:
EntityManager em1 = Emf.getInstance().getFactory().createEntityManager();
Query namedQuery = em.createNamedQuery("DatexSituationData.findAll", DatexSituationData.class);
namedQuery.getResultList();
em.clear();
namedQuery = em.createNamedQuery("DatexSituationData.findAll", DatexSituationData.class);
namedQuery.getResultList();
If you need to add query hints, such as for fetch joins and the query result caching itself, add them in the query definition:
#NamedQuery(name="DatexSituationData.complexFind",
query="SELECT ds fetch join ds.situationRecords FROM DatexSituationData ds where CONCAT(ds.id.version, ds.id.id) in (select
CONCAT(max(FUNC('TO_NUMBER', ds1.id.version)), ds1.id.id) from SituationData ds1 group by
ds1.id.id)",
hints={#QueryHint(name="eclipselink.join-fetch", value="ds.situationRecords.situationLocation"),
#QueryHint(name="eclipselink.join-fetch", value="ds.situationRecords.situationRecordTypeElements"),
#QueryHint(name="eclipselink.join-fetch", value="ds.situationRecords.situationRecordGeneralPublicCommentMappings.situationRecordGeneralPublicComments"),
#QueryHint(name="eclipselink.sql.hint", value="/*+ leading(t1 t0) */"),
#QueryHint(name="eclipselink.query-results-cache", value="true"),
#QueryHint(name="eclipselink.query-results-cache.expiry", value="120000")})//2 minutes
or add named queries in the runtime:
Query namedQueryToSave = em.createQuery(yourQueryString, DatexSituationData.class);
//add all hints
em.getEntityManagerFactory().addNamedQuery("DatexSituationData.complexFind", namedQueryToSave);
em.close();
Either way, then you can execute the query. Using the previous test in a single em using "DatexSituationData.complexFind" should then show a hit to the database on the first run, but not the second - as long as they are less than 2 minutes apart.
Related
How can I avoid unnecessary queries to the DB?
I have LoadEntity with two nested entity - CarrierEntity and DriverEntity. Java class:
#Entity
public class LoadEntity {
...
#ManyToOne
#JoinColumn(name="carrier_id", nullable=false)
private CarrierEntity carrierEntity;
#ManyToOne
#JoinColumn(name="driver_id", nullable=false)
private DriverEntity driverEntity;
}
But API send me carrierId and driverId. I make it:
DriverEntity driverEntity = driverService.getDriverEntityById(request.getDriverId());
loadEntity.setDriverEntity(driverEntity);
loadRepository.save(loadEntity);
How can I write only driverId with JPA?
With Spring Data JPA you can always fall back on plain SQL.
Of course, this will side step all the great/annoying logic JPA gives you.
This means you won't get any events and the entities in memory might be out of sync with the database.
For this reason you might also increase the version column, if you are using optimistic locking.
That said you could update a sing field like this:
interface LoadRepository extends CrudRepository<LoadEntity, Long> {
#Query(query="update load_entity set driver_id = :driverId where carrier_id=:carrier_id", nativeQuery=true)
#Modifying
void updateDriverId(Long carrierId, Long driverId);
}
If you just want to avoid the loading of the DriverEntity you may also use JpaRepository.getById
I'm writing some entity relationships using Spring Data and Java. I have this pair of classes (edited):
Subject:
#Entity
#Table(name = "SUBJECT")
// Lombok, etc., attributes removed
public class Subject {
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE)
#Column(name = "ID", updatable = false, nullable = false)
#JsonProperty("id")
private Long id;
#OneToMany(targetEntity = SubjectResource.class, mappedBy = "subject", fetch = FetchType.LAZY, cascade = CascadeType.ALL)
private List<SubjectResource> resources;
}
SubjectResource:
#Entity
#Table(name = "SUBJECT_RESOURCE")
// Lombok, etc., attributes removed
public class SubjectResource {
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE)
#Column(name = "ID", updatable = false, nullable = false)
private Long id;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "SUBJECT_ID")
private Subject subject;
}
I'm trying to solve these issues:
Question 1: Can I manipulate #OneToMany or #ManyToOne to NOT have the child class recurse its parent?
Fetch of resources returns subject data:
/subject/101:
{"id":101,"resources":[{"id":1001,"subject":101},{"id":1002,"subject":101},{"id":1003,"subject":101},{"id":1004,"subject":101}]}
/subjectResource/1001:
{id:1001,subject:{"id":101,"resources":[{"id":1001,"subject":101},{"id":1002,"subject":101},{"id":1003,"subject":101},{"id":1004,"subject":101}]}}
That is, /subjectResource/1001 returns its ID and the entire /subject/101 query.
How can I have just the subjectResource data, without its parent?
Question 2: Through #OneToMany or #ManyToOne can I get Hibernate to fetch on a "1" (O(1)) basis?
When /subjects does its thing, it works with Hibernate on a "n+1" (O(n)) basis: 1 fetch of subjects, n fetches of resources, one for each subject ID.
I could force a single fetch through a fancy repository #Query annotation ("select s from subject s left join fetch s.resources"). But that means putting the subject : subject_resource definitions in two places, etc.
Can JPA implementation / Hibernate be forced to do a join, and thus make only one database call, through annotation within an entity class?
Question 3: How do I get my Spring Data / Spring Repository to cooperate with Ignite, and have the cache return the data it already had on the first call?
I'm usng FetchType.LAZY, as all good pupils do. I'm also storing things in Apache Ignite. For /subject/101 the initial call fetches everything OK, returning it in JSON. But the second call gets from the Ignite cache, which complains about being out of transaction.
How do get my LAZY fetches to cooperate with Ignite?
Thanks,
Jerome.
Here are my entities:
#Entity
public class Actor {
private List<Film> films;
#ManyToMany
#JoinTable(name="film_actor",
joinColumns =#JoinColumn(name="actor_id"),
inverseJoinColumns = #JoinColumn(name="film_id"))
public List<Film> getFilms(){
return films;
}
//... more in here
Moving on:
#Entity
public class Film {
private List actors;
#ManyToMany
#JoinTable(name="film_actor",
joinColumns =#JoinColumn(name="film_id"),
inverseJoinColumns = #JoinColumn(name="actor_id"))
public List<Actor> getActors(){
return actors;
}
//... more in here
And the join table:
#javax.persistence.IdClass(com.tugay.sakkillaa.model.FilmActorPK.class)
#javax.persistence.Table(name = "film_actor", schema = "", catalog = "sakila")
#Entity
public class FilmActor {
private short actorId;
private short filmId;
private Timestamp lastUpdate;
So my problem is:
When I remove a Film from an Actor and merge that Actor, and check the database, I see that everything is fine. Say the actor id is 5 and the film id is 3, I see that these id 's are removed from film_actor table..
The problem is, in my JSF project, altough my beans are request scoped and they are supposed to be fetching the new information, for the Film part, they do not. They still bring me Actor with id = 3 for Film with id = 5. Here is a sample code:
#RequestScoped
#Named
public class FilmTableBackingBean {
#Inject
FilmDao filmDao;
List<Film> allFilms;
public List<Film> getAllFilms(){
if(allFilms == null || allFilms.isEmpty()){
allFilms = filmDao.getAll();
}
return allFilms;
}
}
So as you can see this is a request scoped bean. And everytime I access this bean, allFilms is initially is null. So new data is fetched from the database. However, this fetched data does not match with the data in the database. It still brings the Actor.
So I am guessing this is something like a cache issue.
Any help?
Edit: Only after I restart the Server, the fetched information by JPA is correct.
Edit: This does not help either:
#Entity
public class Film {
private short filmId;
#ManyToMany(mappedBy = "films", fetch = FetchType.EAGER)
public List<Actor> getActors(){
return actors;
}
The mapping is wrong.
The join table is mapped twice: once as the join table of the many-to-many association, and once as an entity. It's one or the other, but not both.
And the many-to-many is wrong as well. One side MUST be the inverse side and use the mappedBy attribute (and thus not define a join table, which is already defined at the other, owning side of the association). See example 7.24, and its preceeding text, in the Hibernate documentation (which also applies to other JPA implementations)
Side note: why use a short for an ID? A Long would be a wiser choice.
JB Nizet is correct, but you also need to maintain both sides of relationships as there is caching in JPA. The EntityManager itself caches managed entities, so make sure your JSF project is closing and re obtaining EntityManagers, clearing them if they are long lived or refreshing entities that might be stale. Providers like EclipseLink also have a second level cache http://wiki.eclipse.org/EclipseLink/Examples/JPA/Caching
the question and problem is pretty simple, though annoying and I am looking for a global solution, because it's application-wide problem for us.
The code below is really not interesting but I post it for clarification!
We use PostgreSQL database with JPA 2.0 and we generated all the facades and entities, of course we did some editing but not much really.
The problem is that every entity contains a Collection of its children, which however (for us only?) is NOT updated after creation a children element.
The objects are written to database, you can select them easily, but what we really would like to see is the refreshed collection of children in parent object.
Why is this happening? If we (manually) refresh the entity of parent em.refresh(parent) it does the trick but it would mean for us a lot of work in Facades I guess. But maybe there is no other way?
Thanks for support!
/* EDIT */
I guess it has to be some annotation problem or cache or something, but I've already tried
#OneToMany(mappedBy = "idquestion", orphanRemoval=true, fetch= FetchType.EAGER)
and
#Cacheable(false)
didn't work properly.
/* EDIT */
Some sample code for understanding.
Database level:
CREATE TABLE Question (
idQuestion SERIAL,
questionContent VARCHAR,
CONSTRAINT Question_idQuestion_PK PRIMARY KEY (idQuestion)
);
CREATE TABLE Answer (
idAnswer SERIAL,
answerContent VARCHAR,
idQuestion INTEGER,
CONSTRAINT Answer_idAnswer_PK PRIMARY KEY (idAnswer),
CONSTRAINT Answer_idQuestion_FK FOREIGN KEY (idQuestion) REFERENCES Question(idQuestion)
);
Than we have generated some Entities in Netbeans 7.1, all of them look similar to:
#Entity
#Table(name = "question", catalog = "jobfairdb", schema = "public")
#XmlRootElement
#NamedQueries({ BLAH BLAH BLAH...})
public class Question implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#Basic(optional = false)
#NotNull
#GeneratedValue(strategy= GenerationType.IDENTITY)
#Column(name = "idquestion", nullable = false)
private Integer idquestion;
#Size(max = 2147483647)
#Column(name = "questioncontent", length = 2147483647)
private String questioncontent;
#OneToMany(mappedBy = "idquestion", orphanRemoval=true)
private Collection<Answer> answerCollection;
Getters... setters...
We use (again) generated facades for them, all implementing AbstractFacade like:
public abstract class CCAbstractFacade<T> {
private Class<T> entityClass;
public CCAbstractFacade(Class<T> entityClass) {
this.entityClass = entityClass;
}
protected abstract EntityManager getEntityManager();
public void create(T entity) {
getEntityManager().persist(entity);
}
The father entity is updated automatically if you use container managed transactions and you fetch the collection after the transaction is complete. Otherwise, you have to update yourself the collection.
This article explains in detail this behaviour: JPA implementation patterns: Bidirectional associations
EDIT:
The simplest way to use Container Managed Transactions is to have transaction-type="JTA" in persistence.xml and use Container-Managed Entity Managers.
You seem to be setting the ManyToOne side, but not adding to the OneToMany, you have to do both.
In JPA, and in Java in general you must update both sides of a bi-directional relationship, otherwise the state of your objects will not be in sync. Not doing so, would be wrong in any Java code, not just JPA.
There is no magic in JPA that will do this for you. EclipseLink does have a magic option for this that you could set through a customizer (mapping.setRelationshipPartnerAttributeName()), but it is not recommended, fixing your code to be correct is the best solution.
See,
http://en.wikibooks.org/wiki/Java_Persistence/Relationships#Object_corruption.2C_one_side_of_the_relationship_is_not_updated_after_updating_the_other_side
I have an entity VM with a relationship to another entity BP. The relationship is eagerly fetched. First I load a VM. After loading the VM is detached, serialized and changed at the client side. Now I want to update the changed entity so I use the EntityManager.merge() method from JPA. Now I run into the following error from OpenJPA:
"Encountered new object in persistent field "Vm.bp" during attach. However, this field does not allow cascade attach. Set the cascade attribute for this field to CascadeType.MERGE or CascadeType.ALL (JPA annotations) or "merge" or "all" (JPA orm.xml). You cannot attach a reference to a new object without cascading."
Why do I have to add a Cascade.MERGE to a relationship to another entity that will never change? And why does JPA think that BP is a new object ("...cannot attach reference to a new object...")?
When using ManyToOne relationships do I always have to add Cascade.MERGE in order to update the entity or is this because of the EAGER fetch type?
Here's my entity:
#Entity
#Table(name = "VM")
public class Vm extends BaseEntity implements Serializable {
public static final long serialVersionUID = -8495541781540291902L;
#Id
#SequenceGenerator(name = "SeqVm", sequenceName = "SEQ_VM")
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "SeqVm")
#Column(name = "ID")
private Long id;
// lots of other fields and relations
#ManyToOne(fetch = FetchType.EAGER)
#JoinColumn(name = "BP_ID")
private Bp bp;
// ...
}
I found the reason why this error message comes up: The #Version annotated database field of the related Bp entity was initialized with "0". Apparently OpenJPA (1.2.3) is not able to cope with entity versions of zero.
Setting the version to 1 solved my issue.