I use webflux + spring data and in the same transaction, I'm trying to fetch data and update in the same transaction. On userPrivilegesRepository::saveAll execution stops. That works on H2 database, but does not work on postgres database.
#Transactional
public Flux<Void> syncUsersWithPrivileges() {
return userDetailsRepository.findUserWithoutPrivileges()
.doOnNext(it -> log.info("Syncing users without privileges: id {}", it.getId()))
.concatMap(this::createUserPrivileges)
.buffer(BATCH_SIZE)
.doOnNext(it -> log.info("Update complete for batch users size {}", it.size()))
.flatMap(it -> Mono.empty());
}
public Mono<UserEntity> createUserPrivileges(UserEntity user) {
return Flux.fromIterable(Privileges.jobSeeker())
.map(it -> UserPrivilegesEntity.builder()
.privilegesId(it)
.userId(user.getId())
.id(user.getId() + "-" + it.name())
.isNew(true).build())
.collectList()
.flatMapMany(userPrivilegesRepository::saveAll)
.then(Mono.just(user));
}
the repository
#Query("select u.* from users as u left join user_privileges as up ON up.user_id = u.id\n"
+ "where up.user_id is null")
Flux<UserEntity> findUserWithoutPrivileges();
If I remove #Transactional annotation everything works fine. Any thoughts on that?
Related
i have scenario like:
i have to check the table if entry is available in DB then if available i need to call the same external api n times using webclient, collect all the response and save them in DB. if entry is not available in DB call the old flow.
here is my implementation. need suggestions to improve it. without for-each
public Mono<List<ResponseObject>> getdata(String id, Req obj) {
return isEntryInDB(id) //checking the entry in DB
.flatMap(
x -> {
final List<Mono<ResponseObject>> responseList = new ArrayList<>();
IntStream.range(0, obj.getQuantity()) // quantity decides how many times api call t happen
.forEach(
i -> {
Mono<ResponseObject> responseMono =
webClientCall(
id,
req.getType())
.map(
res ->
MapperForMappingDataToDesriedFormat(res));
responseList.add(responseMono);
});
return saveToDb(responseList);
})
.switchIfEmpty(oldFlow(id, req)); //if DB entry is not there take this existing flow.
need some suggestions to improve it without using foreach.
I would avoid using IntStream and rather use native operator to reactor called Flux in this case.
You can replace, InsStream.range with Flux.range. Something like this:
return isEntryPresent("123")
.flatMapMany(s -> Flux.range(0, obj.getQuantity())
.flatMap(this::callApi))
.collectList()
.flatMap(this::saveToDb)
.switchIfEmpty(Mono.defer(() ->oldFlow(id, req)));
private Mono<Object> saveToDb(List<String> stringList){
return Mono.just("done");
}
private Mono<String> callApi(int id) {
return Mono.just("iterating" + id);
}
private Mono<String> isEntryPresent(String id) {
return Mono.just("string");
}
As part of migrating our service to Kotlin, we wanted to keep our Postgres DB in pure sql and decided to use Ktor(as Native SQL) along with HikariCP as the DataSource for connection pooling.
(For a few reasons, e.g - we're creating our own DOMAINS, TRIGGER FUNCTIONS and using JSON/JSONB, which is not all compatible with ORMs, and we also don't want to be framework-dependant cause we might switch to different ones in the future)
I'm using :
HikariCP for connection pool
PreparedStatements for all the queries (via the Ktorm library)
I've made these functions :
fun <T : Any> String.select(db: Database, vararg values: Any, transform: (ResultSet) -> T): List<T> {
val result = arrayListOf<T>()
db.useConnection { conn ->
conn.prepareStatement(this).use { st ->
values.forEachIndexed { idx, value ->
st.bind(idx + 1, value)
}
st.executeQuery().asIterable().map { rs ->
result += transform(rs)
}
}
}
return result
}
private fun PreparedStatement.bind(index: Int, value: Any) {
when (value) {
is Int -> setInt(index, value)
is Long -> setLong(index, value)
is Float -> setFloat(index, value)
is String -> setString(index, value)
}
}
The usage is then like this :
val preparedStatement = "Select * from users where name = ?"
val users: List<User> = preparedStatement.select(dbClient.db, "John") { row ->
User(
id = row.getInt(1),
name = row.getString(2),
lastName = row.getString(3)
)
}
But when I run this I noticed this message in my logs :
com.zaxxer.hikari.pool.ProxyConnection - HikariPool-1 - Executed rollback on connection org.postgresql.jdbc.PgConnection#32eebfca due to dirty commit state on close().
When I tried to use autoCommit = true , I started getting then :
HikariPool-0 - Reset (autoCommit) on connection
When I tried simply using conn.commit() at the end of the statement execution, it seemed there were no suspicious logs.
But should I really COMMIT in SELECT queries ?
In https://quarkus.io/guides/reactive-sql-clients page we have code snippet to execute query changes using transaction :
SqlClientHelper.inTransactionUni(client, tx -> tx
.preparedQuery("INSERT INTO person (firstname,lastname) VALUES ($1,$2) RETURNING id").execute(Tuple.of(person.getFirstName(), person.getLastName()))
.onItem().transformToUni(id -> tx.preparedQuery("INSERT INTO addr (person_id,addrline1) VALUES ($1,$2)")
.execute(Tuple.of(id.iterator().next().getLong("id"), person.getLastName()))).onItem().ignore().andContinueWithNull());
so here SqlClientHelper will begin the transaction,commit and rollback if any failure but is there any way to find out the root cause of the failure and print it in logs ?
In the documentation its not mentioned how we can do that.
You can use Mutiny's onFailure to get the exception class and act on it. See this for more details.
based on the link as given in the accepted Answer this is working for me :
return SqlClientHelper.inTransactionUni(mysqlPool, tx -> {
return tx.query(query).execute().onItem().transformToUni(
id -> tx.query("SELECT TRAN_ID FROM " + tableName + "
ORDER BY TO_DB_TS DESC LIMIT 1").execute())
.onItem().transform(rows ->
rows.iterator().next().getString(0)).onFailure().invoke(f -> {
LOG.error("Error while inserting data to " +
tableName + " table::"+f.getMessage());
});
});
I want to perform the following select using JPA:
select * from permissions_table where permissions.role in ("Role1", "Role2")
What I have so far looks like this:
protected Set<String> getPermissions(Connection conn, String username, Collection<String> roleNames) throws SQLException {
PreparedStatement ps = null;
Set<String> permissions = new LinkedHashSet<String>();
try {
EntityManager em = entityManagerFactory.createEntityManager();
CriteriaBuilder builder = em.getCriteriaBuilder();
CriteriaQuery<HierarchicalPermission> criteria = builder.createQuery( HierarchicalPermission.class );
Root<HierarchicalPermission> permission = criteria.from(HierarchicalPermission.class);
criteria.select(permission).where(permission.get("Role").in(roleNames));
List<HierarchicalPermission> hPermissions = em.createQuery(criteria).getResultList();
for ( HierarchicalPermission p : hPermissions ) {
System.out.println( "Permission (" + p.getRole() +")");
}
}
catch(Exception ex)
{
System.out.println( ex.getMessage());
}
finally {
JdbcUtils.closeStatement(ps);
}
return permissions;
}
When I step over this line:
List<HierarchicalPermission> hPermissions = em.createQuery(criteria).getResultList();
I see the following in my Eclipse output window:
Hibernate: select hierarchic0_.iIdentity as iIdentity0_, hierarchic0_.timestamp as timestamp0_, hierarchic0_.szRole as szRole0_, hierarchic0_.szDescription as szDescri4_0_, hierarchic0_.iResource as iResource0_ from occ.ROLE_PERMISSIONS hierarchic0_ where hierarchic0_.szRole in (?)
and Eclipse debugger appears to stall. At this point, I can only pause or stop execution as shown in this screen shot.
What is this supposed to mean? Is this not a valid representation of the above query?
Database was locked by Sybase Interactive SQL on another machine so Hibernate was stalling while attempting to execute query. One would think that Hibernate would throw some sort of exception instead of simply stalling when it encounters resource contention but I guess this is not the case.
I'm using the IBM implentation of Open JPA on WebSphere 7 and I'm having an issue when I'm trying to reference an object that is #ManyToOne and keep getting the following error from DB2:
com.ibm.db2.jcc.b.SqlException: [jcc][t4][10120][10898][3.50.152] Invalid operation: result set is closed. ERRORCODE=-4470, SQLSTATE=null
I'm pulling my hair out as to why this doesn't work and hope that somebody can help.
Here is a simplified view the database schemas:
Table Report
record_id - integer - (primary key - generated by DB2)
agency - integer not null (foreign key to Dropdown table)
Table Dropdown
record_id - integer - (primary key - generated by DB2)
Here is the JPA entity for the Report which references the agency
#ManyToOne(fetch=FetchType.EAGER)
#JoinColumn(name="AGENCY")
private Dropdown agency;
Here is the code where I'm running a named query to get the data and then just iterating over the result set to print out the report id and the agnency. Whenever report.getAgency() is called, I get the "result set is closed" error from DB2:
#SuppressWarnings("unchecked")
public List<Report> getOpenIncidentsForUser(String aceId) throws Exception
{
List<Report> results = null;
EntityManager em = getEntityManager();
try
{
Query query = em.createNamedQuery("getOpenIncidentsForUser");
query.setParameter(1, aceId);
results = (List<Report>) query.getResultList();
Iterator<Report> it = results.iterator();
while(it.hasNext())
{
Report report = it.next();
System.out.println("Report [" + report.getRecordId() + "] Agency: [" + report.getAgency() + "]");
}
}
catch (Exception e)
{
log.fatal("Fatal error getting incidents for user", e);
throw e;
}
finally
{
em.close();
}
return (List<Report>) results;
}
if I don't ever refer to the getAgency method, I can print out anything else about the report with no problems. It only seems to be with the reference to the 2nd table. Any ideas?
I had answered this in responses to my original comment, but realized that I never marked the question as answered, so I wanted to do that officially.
The fix is documented here: https://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.base.doc/ae/tejb_jpatroubleshoot.html
The fix ended up being the resulSetHoldability setting needed to be 1 instead of 2
For XA data sources you have to set downgradeHoldCursorsUnderXa to true, otherwise you could get a persistence exception with this message:
An SQL OPEN for a held cursor was issued on a XA connection
Setting DB2 resultSetHoldability=1 will only work if you are using a non-XA datasource. If you need to keep 2PC, then this is not a solution.
I had this exact problem and finally solved it by hard-coding a transaction around the offending code. This is what I have:
public class RequeueRuleList_back {
/*
* Injected resources ...
*/
#Resource UserTransaction txn;
#PersistenceUnit EntityManagerFactory emf;
:
public List<RequeueRuleBean> getRequeueRules() {
/*
* We need a hard transaction around this code even though it is just a query
* otherwise we cannot use a DB2 XA datasource to do this:
*
* com.ibm.db2.jcc.am.SqlException: [jcc][t4][10120][10898][3.63.75] Invalid operation: result set is closed. ERRORCODE=-4470, SQLSTATE=null
*/
try {
txn.begin();
} catch (Exception e) {
FacesContext.getCurrentInstance().addMessage(null,
new FacesMessage("Error starting transaction: " + e.getMessage()));
return null;
}
EntityManager em = emf.createEntityManager();
:
Query q = em.createQuery("SELECT rr FROM RequeueRule rr");
// Do useful things ...
em.close();
try {
txn.commit();
} catch (Exception e) {
FacesContext.getCurrentInstance().addMessage(null,
new FacesMessage("Error committing transaction: " + e.getMessage()));
}
:
}
}
If you just use jdbc to connect DB2 and didn't use Hibernate etc, you also have got this error. Because in the new JDBC version with DB2 9.7, many functions you shouldn't support in new vesion,althought there are no error running on old version jdbc.
These function include.
1: PreparedStatement
old version
pt.executeUpdate(sql);
new version
pt.executeUpdate();
2: Connection Iteration
old version：
try{
conn = ConnectionFactory.getConnection(ApplicationConstants.LOCAL_DATASOURCE_JNDI_NAME);
sql="select role_id,role_sname,role_sdesc from db2admin.mng_roles "+sql_condition+" order by role_id asc";
pt = conn.prepareStatement(sql.toString());
System.out.println("sql ="+sql);
rs = pt.executeQuery();
while(rs.next()){
i++;
role_id=rs.getInt(1);
role_sname=PubFunction.DoNull(rs.getString(2)).trim();
role_sdesc=PubFunction.DoNull(rs.getString(3)).trim();
role_right=PubFunction.DoNull(newright.getRightsbyRole(conn,role_id)).trim();}
new version
try{
conn = ConnectionFactory.getConnection(ApplicationConstants.LOCAL_DATASOURCE_JNDI_NAME);
sql="select role_id,role_sname,role_sdesc from db2admin.mng_roles "+sql_condition+" order by role_id asc";
pt = conn.prepareStatement(sql.toString());
System.out.println("sql ="+sql);
rs = pt.executeQuery();
while(rs.next()){
i++;
role_id=rs.getInt(1);
role_sname=PubFunction.DoNull(rs.getString(2)).trim();
role_sdesc=PubFunction.DoNull(rs.getString(3)).trim();
role_right=PubFunction.DoNull(newright.getRightsbyRole(null,role_id)).trim();}