I am trying to cache this service
#Cacheable(value = "productCache", key = "{#prodSearch.field1, #prodSearch.field2, #prodSearch.field3, #prodSearch.field4}", sync = true)
#Override
public Product identifyProduct(ProductSearch prodSearch)
any one of the field is mandatory for searching, so keys can be stored like this for example
field1,null,field3,null
null,field2,null,null
field1,null,null,null
......
I want to remove this cache when the cached product was updated or deleted by product id, since this id is not part of cache key in the above service, how to evict those cache properly?
Related
Update
Ockham's razor sliced through this problem.
Turns out the issue was:
Document did not have an _id field
#Id
private String id;
So the update() method would insert a new record rather than update existing record.
the 'find' code used 'findOne()'
for some reason findOne() appeared to behave differently in different environemts (i.e. in local dev environments it would retrieve 'most recent', but on our server environment , retrieved the 'oldest'.). Whatever something in local env masked the problem.
TLDR
'save and immediate retrieve' mongodb document with embedded document (i.e. field w/ java object/json) does not show expected updates
problem appears intermittently and only on some environments
Background
I'm new to mongo so don't know the tricks/techniques/gotchas/etc.. I'm versed in relationaldb and transactions, so this error threw me.
for better or worse, I designed a mongo collection that looks like this:
#Document
public class BidOffers {
...
// note 'Suppliers' here is plural, ie. offers from multiple suppliers
SuppliersOffers offers;
where SuppliersOffers object simply has a list of SupplierOffer objects
public class SuppliersOffers {
List<SupplierOffer> offers = new ArrayList<>();
and SupplierOffer just has a 'supplier code' and a price
public class SupplierOffer {
String supplierCode
BigDecimal price
}
Usecase
In this usecase:
Retrieve document BidOffers from mongodb
Note that document has only one offer
Add a new offer to the list
Save document
Print saved document to log, i.e. and note/confirm "saved document now has two offers in list, not just one"
Some (short) time later (i.e. in the same thread) retrieve the document
Expected Results
retrieved document has two offers
Actual Result
retrieve document has only one offer
More details
Java 8
Spring Boot 2.x
Problem appears on Mongodb 4.0 version (AWS Managed Service)
Problem appears regularly, but not 100% consistently
I do not see this problem when testing locally (Mongdb 3.6.8)
Two (JSF + JPA + EclipseLink + MySQL) applications share the same database. One application runs a scheduled task where the other one creates tasks for schedules. The tasks created by the first application is collected by queries in the second one without any issue. The second application updates fields in the task, but the changes done by the second application is not refreshed when queried by JPQL.
I have added QueryHints.CACHE_USAGE as CacheUsage.DoNotCheckCache, still, the latest updates are not reflected in the query results.
The code is given below.
How can I get the latest updates done to the database from a JPQL query?
public List<T> findByJpql(String jpql, Map<String, Object> parameters, boolean withoutCache) {
TypedQuery<T> qry = getEntityManager().createQuery(jpql, entityClass);
Set s = parameters.entrySet();
Iterator it = s.iterator();
while (it.hasNext()) {
Map.Entry m = (Map.Entry) it.next();
String pPara = (String) m.getKey();
if (m.getValue() instanceof Date) {
Date pVal = (Date) m.getValue();
qry.setParameter(pPara, pVal, TemporalType.DATE);
} else {
Object pVal = (Object) m.getValue();
qry.setParameter(pPara, pVal);
}
}
if(withoutCache){
qry.setHint(QueryHints.CACHE_USAGE, CacheUsage.DoNotCheckCache);
}
return qry.getResultList();
}
The CacheUsage settings affect what EclipseLink can query using what is in memory, but not what happens after it goes to the database for results.
It seems you don't want to out right avoid the cache, but refresh it I assume so the latest changes can be visible. This is a very common situation when multiple apps and levels of caching are involved, so there are many different solutions you might want to look into such as manual invalidation or even if both apps are JPA based, cache coordination (so one app can send an invalidation even to the other). Or you can control this on specific queries with the "eclipselink.refresh" query hint, which will force the query to reload the data within the cached object with what is returned from the database. Please take care with it, as if used in a local EntityManager, any modified entities that would be returned by the query will also be refreshed and changes lost
References for caching:
https://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Basic_JPA_Development/Caching
https://www.eclipse.org/eclipselink/documentation/2.6/concepts/cache010.htm
Make the Entity not to depend on cache by adding the following lines.
#Cache(
type=CacheType.NONE, // Cache nothing
expiry=0,
alwaysRefresh=true
)
I have a Sidekiq worker, which makes an API call, parses the returned json and creates ActiveRecord objects (products). Since the products belong to a brand and the product json also contains the data for the brand, the worker does the following before actually saving the product to the database:
check if the brand exists in the database by checking its unique api_id (the id, with which the brand comes from the api, in the db, there is a unique index on this column);
if it does - fetch its primary id;
if is doesn't - create it and get its primary id
I have implemented this like so:
def brand_id(brand_json)
Brand.where(api_id: brand_json[:api_id]).pluck(:id).first.presence ||
Brand.create!(name: brand_json[:name], api_id: brand_json[:api_id]).id
end
After that the worker creates the product with the brand_id set to the fetched id.
Now I am thinking of the following scenario:
two workers simultaneously fetch data for two products that belong to the same brand that doesn't yes exist in the database;
worker 1 one checks for the brand and doesn't find it;
shortly after that worker 2 checks for the brand and doesn't find it;
worker 1 creates the brand;
Now what happens with worker 2? My assumption - it tries to create the brand, but an error at the database level occurs, as there is already a record with the same api_id? (probably ActiveRecord::RecordNotUnique error is raised?)
Also, how do I handle this case and this type of errors in the context of Sidekiq and ActiveRecord? Should I somehow implement a table-wide lock on the brands table to prevent such things? If yes - than I will not be able to concurrently create products, as at any given time only one worker will have access to the brands table, which is required for creating a product.
Or maybe I should wrap my brand_id(brand_json) method in transaction like so:
def brand_id(brand_json)
ActiveRecord::Base.transaction do
Brand.where(api_id: brand_json[:api_id]).pluck(:id).first.presence ||
Brand.create!(name: brand_json[:name], api_id: brand_json[:api_id]).id
end
end
Please, advise.
Put the unique index constraints (possibly in the form of a multi column index) in the DB.
Just try to create the object. The database will prevent you from making more than one.
Only the thread that succeeded in creating the initial object (no exception occurred) is allowed to proceed with extra processing.
When I query the metadata using RetrieveMetadataChangesRequest, the RetrieveMetadataChangesResponse returns EntityMetadata and DeletedMetadata. The DeletedMetadata only returns the MetadataId.
Is there a way to get the metadata for the attribute without knowing the entity? Even just the attribute name would be fine.
RetrieveAttributeRequest I think only works if the attribute exists and if you have the entitylogicalname.
No, the only infomration available is the MetadataId.
Quoting from the SDK:
This collection is a dictionary of GUID values using a
DeletedMetadataFilters as a key. The GUID values represent MetadataId
values of the metadata items.
Looking at another part of the SDK specifically addresses this question:
You will also use DeletedMetadataFilters enumeration as a key to the
RetrieveMetadataChangesResponse.DeletedMetadata to filter the GUID
values found in the RetrieveMetadataChangesResponse.DeletedMetadata
property. When you design a metadata cache you will want to use the
MetadataId for each item so that you can identify deleted metadata
items and remove them.
So as a developer you are expected to populate a cache of metadata of interest to your application. You can query the CRM Metadata to find changes and deletes - but in the case of a delete you are responsible for having collected the metadata in your cache.
I have a common Repository with Add, Update, Delete.
We'll name it CustomerRepository.
I have a entity (POCO) named Customer, which is an aggregate root, with Addresses.
public class Customer
{
public Address Addresses { get; set; }
}
I am in a detached entity framework 5 scenario.
Now, let's say that after getting the customer, I choose to delete a client address.
I submit the Customer aggregate root to the repository, by the Update method.
How can I save the modifications made on the addresses ?
If the address id is 0, I can suppose that the address is new.
For the rest of the address, I can chose to attach all the addresses, and mark it as updated no matter what.
For deleted addresses I can see no workaround...
We could say this solution is incomplete and inefficient.
So how the updates of aggregate root childs should be done ?
Do I have to complete the CustomerRepository with methods like AddAddress, UpdateAddress, DeleteAddress ?
It seems like it would kind of break the pattern though...
Do I put a Persistence state on each POCO:
public enum PersistanceState
{
Unchanged,
New,
Updated,
Deleted
}
And then have only one method in my CustomerRepository, Save ?
In this case it seems that I am reinventing the Entity "Non-POCO" objects, and adding data access related attribute to a business object...
First, you should keep your repository with Add, Update, and Delete methods, although I personally prefer Add, indexer set, and Remove so that the repository looks like an in memory collection to the application code.
Secondly, the repository should be responsible for tracking persistence states. I don't even clutter up my domain objects with
object ID { get; }
like some people do. Instead, my repositories look like this:
public class ConcreteRepository : List<AggregateRootDataModel>, IAggregateRootRepository
The AggregateRootDataModel class is what I use to track the IDs of my in-memory objects as well as track any persistence information. In your case, I would put a property of
List<AddressDataModel> Addresses { get; }
on my CustomerDataModel class which would also hold the Customer domain object as well as the database ID for the customer. Then, when a customer is updated, I would have code like:
public class ConcreteRepository : List<AggregateRootDataModel>, IAggregateRootRepository
{
public Customer this[int index]
{
set
{
//Lookup the data model
AggregateRootDataModel model = (from AggregateRootDataModel dm in this
where dm.Customer == value
select dm).SingleOrDefault();
//Inside the setter for this property, run your comparison
//and mark addresses as needing to be added, updated, or deleted.
model.Customer = value;
SaveModel(model); //Run your EF code to save the model back to the database.
}
}
}
The main caveat with this approach is that your Domain Model must be a reference type and you shouldn't be overriding GetHashCode(). The main reason for this is that when you perform the lookup for the matching data model, the hash code can't be dependent upon the values of any changeable properties because it needs to remain the same even if the application code has modified the values of properties on the instance of the domain model. Using this approach, the application code becomes:
IAggregateRootRepository rep = new ConcreteRepository([arguments that load the repository from the db]);
Customer customer = rep[0]; //or however you choose to select your Customer.
customer.Addresses = newAddresses; //change the addresses
rep[0] = customer;
The easy way is using Self Tracking entities What is the purpose of self tracking entities? (I don't like it, because tracking is different responsability).
The hard way, you take the original collection and you compare :-/
Update relationships when saving changes of EF4 POCO objects
Other way may be, event tracking ?