Creating a project with SpringData using Reactive Cassandra repository. I have sample Book application where I wrote custom query.
#GetMapping("/books2")
public Flux<Book2> getBooks(#Valid #RequestBody Book2 book ){
MapId id1 = id( "id", book.getId()).with("isbn", book.getIsbn());
if(Objects.nonNull(book.getName()))
id1.with( "name", book.getName());
if(Objects.nonNull(book.getLocalDate()))
id1.with( "localDate", book.getLocalDate());
return book2Repository.findAllById( Collections.singletonList(id1));
}
I have many rows but return result is only one.
Looking into code SimpleReactiveCassandraRepository.java,
public Flux<T> findAllById(Iterable<ID> ids) {
Assert.notNull(ids, "The given Iterable of ids must not be null");
if (FindByIdQuery.hasCompositeKeys(ids)) {
return this.findAllById((Publisher)Flux.fromIterable(ids));
} else {
FindByIdQuery query = FindByIdQuery.forIds(ids);
List<Object> idCollection = query.getIdCollection();
....
....
public Flux<T> findAllById(Publisher<ID> idStream) {
Assert.notNull(idStream, "The given Publisher of ids must not be null");
return Flux.from(idStream).flatMap(this::findById);
}
The findAllById seem to check if Query has composite key and calls "findAllById" which seem to call findById , which return single record.
How do i get to return multiple rows ?
Is this a bug ?
I tried with 2.2.7 and 3.0.1 spring-data-cassandara and results seems same.
Related
I am using JPA Specification , need to select specific columns only.
This is the code:
Specification<Item> spec = (root, query, builder) -> {
query.select(root.get("Id"));
Predicate predicate = builder.equal(root.get("Id"), "12345");
return predicate;
};
In log I see that all columns from Item Entity are selected from database.
Is it a bug?
usage:
interface:
public interface Repo extends PagingAndSortingRepository<Item,String>, JpaSpecificationExecutor<Item> {
}
call:
repo.findAll(spec );
JpaSpecificationExecutor is specifically defined to return the entity type. I suspect it is ignoring the .select(root.get("Id")).
Normally you would use Specifications if you have an extremely dynamic set of conditions you are querying by. If you have just a few parameters you need to search by, I would use a derived query, or a named query.
public interface Repo extends PagingAndSortingRepository<Item,String>, JpaSpecificationExecutor<Item> {
#Query("Select i.id from Item i where name=:name")
Long getIdforName(String name);
}
The application is syncing data records between devices via an online Mongo DB collection. Multiple devices can send batches of new or modified records to the server Mongo collection at any time. Devices get all record updates for them that they don't already have, by requesting records added or modified since their last get request.
Approach 1 - was to add a Date object field (called stored1) to the records before saving to MongoDb. When a device requests records , mongoDb paging is used to skip entries up to the current page, and then limit to 1000. Now that the data set is large, each page request is taking a long time, and mongo hit a memory error.
https://docs.mongodb.com/manual/reference/limits/#operations
Setting allowDiskUse(true) as shown in the posted code in my current configuration isn't fixing the memory error for some reason. If that can be fixed, it still wouldn't be a long term solution as the query times with the paging are already too long.
Approach 2:
What is the best way for pagination on mongodb using java
https://arpitbhayani.me/blogs/benchmark-and-compare-pagination-approach-in-mongodb
The 2nd approach considered is to change from Mongo paging skipping returned records, to just asking for stored time > largest stored time last received, until the number of records in a return is less than the limit. This requires the stored timestamp to be unique between all records matching the query, or it could miss records or get duplicate records etc.
In the example code, using the stored2 field, there's still a chance of duplicate timestamps, even if the probability is low.
Mongo has a BSON timestamp that guarantees unique values per collection, but I don't see a way to use it with document save(), or query on it in Spring Boot. It would need to be set on each record newly inserted, or replaced, or updated.
https://docs.mongodb.com/manual/reference/bson-types/#timestamps
Any suggestions on how to do this?
#Getter
#Setter
public abstract class DataModel {
private Map<String, Object> data;
#Id // maps this field name to the database _id field, automatically indexed
private String uid;
/** Time this entry is written to the db (new or modified), to support querying for changes since last query */
private Date stored1; //APPROCAH 1
private long stored2; //APPROACH 2
}
/** SpringBoot+MongoDb database interface implementation */
#Component
#Scope("prototype")
public class SpringDb implements DbInterface {
#Autowired
public MongoTemplate db; // the database
#Override
public boolean set(Collection<?> newRecords, Collection<?> updatedRecords) {
// get current time for this set
Date date = new Date();
int randomOffset = ThreadLocalRandom.current().nextInt(0, 500000);
long startingNanoSeconds = Instant.now().getEpochSecond() * 1000000000L + instant.getNano() + randomOffset;
int ns = 0;
if (updatedRecords != null && updatedRecords.size() > 0) {
for (Object entry : updatedRecords) {
entry.setStored1(date); //APPROACH 1
entry.setStored2(startingNs + ns++); //APPROCH 2
db.save(entry, repoName);
}
}
// for new documents only
if (newRecords != null && newRecords.size() > 0) {
for (DataModel entry : newRecords) {
entry.setStored1(date); //APPROACH 1
entry.setStored2(startingNs + ns++); // APPROACH 2
}
//multi record insert
db.insert(newRecords, repoName);
}
return true;
}
#Override
public List<DataModel> get(Map<String, String> params, int maxResults, int page, String sortParameter) {
// generate query
Query query = buildQuery(params);
//APPROACH 1
// do a paged query
Pageable pageable = PageRequest.of(page, maxResults, Direction.ASC, sortParameter);
List<T> queryResults = db.find(query.allowDiskUse(true).with(pageable), DataModel.class, repoName); //allowDiskUse(true) not working, still get memory error
// count total results
Page<T> pageQuery = PageableExecutionUtils.getPage(queryResults, pageable,
() -> db.count(Query.of(query).limit(-1).skip(-1), clazz, getRepoName(clazz)));
// return the query results
queryResults = pageQuery.getContent();
//APPROACH 2
List<T> queryResults = db.find(query.allowDiskUse(true), DataModel.class, repoName);
return queryResults;
}
#Override
public boolean update(Map<String, String> params, Map<String, Object> data) {
// generate query
Query query = buildQuery(params);
//This applies the same changes to every entry
Update update = new Update();
for (Map.Entry<String, Object> entry : data.entrySet()) {
update.set(entry.getKey(), entry.getValue());
}
db.updateMulti(query, update, DataModel.class, repoName);
return true;
}
private Query buildQuery(Map<String, String> params) {
//...
}
}
The solution I ended up using was to define, and index on, another field called storedId, which is a string concatenation of the modified record storedTime, and the _id. This guarantees all these storedId record fields are unique, because _id is unique.
Here's an example to show how indexing and querying on the concatenated storedTime+_id field works, while indexing and querying on the separate storedTime and _id fields fails:
public abstract class DataModel {
private Map<String, Object> data;
#Indexed
private String _id; // Unique id
#Indexed
private String storedTime; // Time this entry is written to the db (new or modified)
#Indexed
String storedId; // String concatenation of storedTime and _id field
}
//Querying on separate fields and indexes:
{
//storedTime, _id
"time1", "id2"
"time1", "id3"
"time1", "id4"
"time2", "id1"
"time2", "id5"
}
get (storedTime>"time0", _id>"id0", limit=2) // returns _id's 2,3 (next query needs to check for more at storedTime="time1" but skip _id’s <="id3")
get (storedTime>="time1", _id>"id3", limit=2) // returns _id's 4,5
//FAILS because this second query MISSES _id 1 (Note any existing _id record can be modified at any time, so the _id fields are not in storedTime order)
//Querying on the combined field and index:
{
//storedId
"time1-id2"
"time1-id3"
"time1-id4"
"time2-id1"
"time2-id5"
}
get (storedId>"time0", limit=2) // returns _id's 2,3 (next query for values greater than the greatest last value returned)
get (storedId>"time1-id3", limit=2) // returns _id's 4,1 (next query for values greater than the greatest last value returned)
get (storedId>"time2-id1", limit=2) //: returns _id 5
//WORKS, this doesn't miss or duplicate any records
So I have two Database Tables
Club
- ClubId
- ClubName
ClubMembers
ClubMemberId
ClubId
FirstName
LastName
in my api controller I have
private ClubsEntities db = new ClubsEntities();
// GET: api/Clubs
public IQueryable<Club> GetClub()
{
return db.Club;
}
But when I hit it i get data returned from both tables
[{"ClubMember":[{"ClubMemberId":1,"ClubId":1,"FirstName":"John","LastName":"Smith"}],"ClubId":1,"ClubName":"Test"},{"ClubMember":[],"ClubId":2,"ClubName":"Test 2"}]
How can I get it to just return from club this is baffling me
Ok so ive figured out that if I just specify a new class just with the values I want returned and select those values, it will keep my returned data cleaner.
return db.Club.Select(c => new myClubs { ClubId = c.ClubId, ClubName = c.ClubName }).ToList();
I created a a query to only get 4 items from a row in a table which does not include the column cloth style, so i understand why i get the error, but how can i tell Spring Jpa or JPA it is on purpose. and i just want the id, name and color table ?
this is my code:
#RequestMapping(value = "/query/material",method = RequestMethod.GET)
public String QueryMaterialTable(HttpServletRequest request){
DataTableRequest<Material> dataTableInRQ = new DataTableRequest<Material>(request);
PaginationCriteria pagination = dataTableInRQ.getPaginationRequest();
String baseQuery = "SELECT id as id, time as time, name as name, color as color, price as price, (SELECT COUNT(1) FROM MATERIAL) AS totalrecords FROM MATERIAL";
String paginatedQuery = AppUtil.buildPaginatedQuery(baseQuery, pagination);
System.out.println(paginatedQuery);
Query query = entityManager.createNativeQuery(paginatedQuery, Material.class);
#SuppressWarnings("unchecked")
List<Material> materialList = query.getResultList();
DataTableResults<Material> dataTableResult = new DataTableResults<Material>();
dataTableResult.setDraw(dataTableInRQ.getDraw());
dataTableResult.setListOfDataObjects(materialList);
if (!AppUtil.isObjectEmpty(materialList)) {
dataTableResult.setRecordsTotal(String.valueOf(materialList.size())
);
if (dataTableInRQ.getPaginationRequest().isFilterByEmpty()) {
dataTableResult.setRecordsFiltered(String.valueOf(materialList.size()));
} else {
dataTableResult.setRecordsFiltered(String.valueOf(materialList.size()));
}
}
return new Gson().toJson(dataTableResult);
}
If I got the question right, your problem is with the following two lines:
Query query = entityManager.createNativeQuery(paginatedQuery, Material.class);
List<Material> materialList = query.getResultList();
You have various options to fix this:
provide a complete column list, i.e. provide the missing column in the SQL statement and just make them NULL;
Don't use Material but a new class that has the matching attributes.
Don't use a native query but JPQL and a constructor expression.
Use a ResultTransformer.
Use Spring Data and a Projection.
Use a Spring JdbcTemplate.
I am using SpringBatch version 3.0.7, Hibernate 4.3.11, and H2 database. When using the JpaPagingItemReader, does the JPQL require a unique sort order? I see that it is required for the JdbcPagingItemReader (see BATCH-2465).
In a Step I am using a JpaPagingItemReader to load entities from the database and then write them to a flat file. I expect the flat file to contain unique entities sorted in the order specified by the JPQL. If I set the page size to something small, like 1, and then provide a JPQL statement that sorts entities with a non unique key, I am seeing the same entity repeat multiple times in the output file. If I sort by a unique key, there are no "duplicates". If I set the page size >= total number of entities, so there is only 1 page, there are no "duplicates".
Empirically it would seem that the JpaPagingItemReader requires the JPQL to have a unique sort key.
Having a look at the implementation of JpaPagingItemReader, you'll find the method doReadPage():
#Override
#SuppressWarnings("unchecked")
protected void doReadPage() {
EntityTransaction tx = null;
if (transacted) {
tx = entityManager.getTransaction();
tx.begin();
entityManager.flush();
entityManager.clear();
}//end if
Query query = createQuery().setFirstResult(getPage() * getPageSize()).setMaxResults(getPageSize());
if (parameterValues != null) {
for (Map.Entry<String, Object> me : parameterValues.entrySet()) {
query.setParameter(me.getKey(), me.getValue());
}
}
if (results == null) {
results = new CopyOnWriteArrayList<T>();
}
else {
results.clear();
}
if (!transacted) {
List<T> queryResult = query.getResultList();
for (T entity : queryResult) {
entityManager.detach(entity);
results.add(entity);
}//end if
} else {
results.addAll(query.getResultList());
tx.commit();
}//end if
}
As you can see, for every page that is read, an new query is created for every Page. Therefore, it must be ensured that your query returns always the same amount of elements in the exact same order, and hence, it needs a 'unique sort key'. Otherwise you will have duplicates and missing entries (there will be a missing entry for every duplicate, since the total number of rows will be identical).