Hibernate Postgres Slow Insert - postgresql

I try to insert data into Postgres using JPA/Hibernate/Postgres.
The data is parsed from a CSV File and then should be saved into a postgres database. The code that persists the data looks as follows:
#Autowired
KundeRepository repo;
#Transactional
public void safe(Kunde kd) {
repo.save(kd);
}
public void safeAll(Iterable<Kunde> kt) {
repo.save(kt);
repo.flush();
}
The entity look as follows
public class account implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Long id;
}
#ManyToOne
Kunde kunde;
#OneToMany
List<TransaktionsGruppe> gruppen;
#Entity
#Table(name = "kunde")
#NoArgsConstructor
public class Kunde implements Serializable {
public static final String kundennummerKey = "KUNDENNUMMER";
private static final long serialVersionUID = 1L;
#Id
#Getter
#Setter
private String id;
#OneToMany
List<Account> accounts;
#Entity
#Table(name = "transaktionsgruppe")
public class Transaktionsgruppe implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Long id;
#ManyToOne
Account acc;
private String bezeichnung;
When I now pass a Collection to the safeAll method the inserts are really slow. Especially it seems to call the hibernate sequence for every insert. Is there a way to speed things up?
My configuration looks as follows:
datasource:
type: com.zaxxer.hikari.HikariDataSource
url: dburl
username: user
password: pw
hikari:
validation-timeout: 10000
health-check-properties: {"connectivityCheckTimeoutMs","1000"}
jpa:
show-sql: true
properties:
hibernate.cache.use_second_level_cache: false
hibernate.cache.use_query_cache: false
hibernate.generate_statistics: false
hibernate.jdbc.batch.size: 100
hibernate.order_inserts: true+
Currently show-sql is enabled. It is about 60000 Entities overall and it takes more than 20 Minutes. the entitties are rather small in size

Your safeAll() method is not annotated with #Transactional so Spring opens and closes a transaction for each item of your list. By annotating it, Spring will open and close one single transaction for the whole list.

Related

Spring batch JdbcCursorItemReader : reading from tables having FK relation

Here's my Reader :
private static final String SELECT_ALL_BOOKS = "SELECT * FROM BOOKS WHERE COLOR = 'yellow'";
#Bean
public JdbcCursorItemReader<BookEntity> itemReader(final DataSource dataSource) {
return new JdbcCursorItemReaderBuilder<BookEntity>()
.name("book_reader")
.sql(SELECT_ALL_BOOKS)
.dataSource(dataSource)
.rowMapper(new BeanPropertyRowMapper<>(BookEntity.class))
.build();
}
And my entity :
#Entity
#Getter
#Setter
#Table(name = "book")
#AllArgsConstructor
#NoArgsConstructor
public class BookEntity implements java.io.Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "id_book")
private Integer idBook;
#Column(name = "color")
private String color;
#Column(name = "page_number")
private Integer pageNumber;
#Column(name = "read")
private Boolean read;
#ManyToOne(fetch = FetchType.EAGER, cascade = {CascadeType.ALL})
#JoinColumn(name = "id_author")
private Author author;
...
...
}
Problem is : with my job executing my step, I'm having the books but with an author = null. So the foreign key is not mapped correctly, all the other String/Integer/Boolean fields are retrieved correctly.
I'm new with Spring batch so maybe I'm using a wrong kind of reader.
Any one has an idea about the problem ? Thanks
Please refer this- You are using JDBC Item reader that is mapped to native columns and properties are binded by BeanPropertyRowMapper
https://docs.spring.io/spring-batch/docs/current/reference/html/readersAndWriters.html#JdbcCursorItemReader
Change the query join with AUTHOR tables as a native query or use JPA readers to support ORM
Below reference will give repository reader
https://docs.spring.io/spring-batch/docs/current/api/org/springframework/batch/item/data/RepositoryItemReader.html
Below example give some idea to implement and its SO references
https://github.com/gpassini/Spring-Batch-RepositoryItemReader-NativeQuery-Example/tree/master/src/main

Query for joins in Spring JPA

I have the below entities
#Entity
#Getter
#Setter
public class Aggregate {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Long id;
#OneToMany(mappedBy = "aggregate")
private Set<Single> singleSet;
}
#Entity
#Getter
#Setter
public class Single {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private String id;
private Integer number;
#ManyToOne
#JoinColumn(name = "agg_id")
private Aggregate aggregate;
}
I also have the below repository
public interface AggregateRepo extends CrudRepository<Aggregate, Long> {
}
I want to return all associated Single records where number in object Single is equal to some random number
I am assuming that the query will be something like this
public interface AggregateRepo extends CrudRepository<Aggregate, Long> {
public List<Single> findBySingleSet_Number(Integer number);
}
However when I try to use Intellij to complete my named query it always populates like this
public interface AggregateRepo extends CrudRepository<Aggregate, Long> {
public List<Single> findBySingleSet_Empty_Number(Integer number);
}
I am wondering what the Empty stands for ?
Also should I create another Single repository since the query is related to returning Single records.

SpringBoot 2 JPA relationship between different schemas in postgres

Need help with this problem:
I have two tables in different schemas in the same db:
DB_test
schema_1
shema_2
in each schema i have different tables:
schema_1 -> client
schema_2 -> phone.
my code
table 1
#Entity #Table(schema = "schema_1")
public class Client implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Integer id;
private String name;
#OneToMany(mappedBy = "client")
private List<Phone> phones = new ArrayList<>();
public PessoaFisica() {
}
}
table 2
#Entity
#Table(schema = "schema_2")
public class Phone implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Integer id;
private String number;
#ManyToOne
#JoinColumn(name = "fk_client")
private Client client;
public Client() {
}
}
So the relationship does not work.
But if the two tables are in the same schema, it works.
What should I do for a schema table to see the table of another schema.

How can I load only specific child object to a mapping place holder

I have following object mapping in our project. One contact can have multiple quotes from typeA and typeB.
#Entity
#Table(name = "t_ind_contact")
public class Contact implements Serializable {
#Id
#Column(name = "id", unique = true)
private Long id;
#OneToMany
private List<Quote> quotes;
}
Abstract Quote class.
#Entity
#Inheritance
#DiscriminatorColumn(name = "app_code")
#Table(name = "t_ind_quote")
public abstract class Quote implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#Column(name = "id")
private long id;
#OneToMany
#JoinColumn(name = "contact_id")
private Contact contact;
#Column(name = "app_code", insertable = false, updatable = false)
private int appCode;
}
TypeA Quote class
#Entity
#DiscriminatorValue("1")
public class TypeAQuote implements Serializable {
private static final long serialVersionUID = 1L;
}
TypeB Quote class
#Entity
#DiscriminatorValue("2")
public class TypeBQuote implements Serializable {
private static final long serialVersionUID = 1L;
}
This mapping is working fine for all operations. But for some operations we need the contact object with typeA quotes only. As of now, we will filter typeA quotes after system loads data from the database.
Is there a way that we can populate a contact object only with typeA quotes? I mean even the generated queries will load only typeA quotes.

JPA transaction/rollback behaviour with objects persisted via cascade

I have two objects Antrag (application) and Anlage (facility). An application can be made for multiple facilities. The application is persisted directly in the DAO. The facilities are persisted via cascade.
#Entity
#Table(name = "EEG_ANTRAG")
public class Antrag implements Serializable {
private static final long serialVersionUID = -2440344011443487714L;
#Id
#Column(name = "ANT_ID", nullable = false)
#SequenceGenerator(name = "sequenceGeneratorAntrag", sequenceName = "EEG_ANTRAG_SEQ", allocationSize = 1)
#GeneratedValue(generator = "sequenceGeneratorAntrag")
#Getter #Setter private Long id;
#OneToMany(mappedBy = "antrag", cascade = { CascadeType.ALL }, orphanRemoval = true)
#OrderBy("id ASC")
#Getter private List<Anlage> anlageList = new ArrayList<Anlage>();
public Anlage addAnlage(Anlage anlage)
anlageList.add(anlage);
anlage.setApplication(this);
return anlage;
}
/* some more simple attributes; just Strings, boolean, .. */
}
#Entity
#Table(name = "EEG_ANLAGE")
public class Anlage implements Serializable {
private static final long serialVersionUID = -3940344011443487741L;
#Id
#Column(name = "ANL_ID")
#SequenceGenerator(name = "sequenceGeneratorAnlage", sequenceName = "EEG_ANLAGE_SEQ", allocationSize = 1)
#GeneratedValue(generator = "sequenceGeneratorAnlage")
#Getter #Setter private Long id;
#ManyToOne
#JoinColumn(name = "ANL_ANT_ID")
#Getter #Setter private Antrag antrag;
/* some more simple attributes; just Strings, boolean, .. */
}
#Stateless
public class AntragDaoBean implements AntragDaoLocal {
#PersistenceContext(unitName = "ejb-model")
private EntityManager em;
#Override
public void persistAntrag(Antrag antrag) {
em.persist(antrag);
}
}
When an error occurs on inserting the facilities, e.g. some column name is misspelled in the entity, an exception is thrown. The stacktrace indicates, that a rollback was performed. The problem is, that the application is still persisted. Shouldn't the insertion of the application be rolled back as well?
We are using EclipseLink 2.4.1. The EclipseLink debug output states, that all inserts are performed in one single transaction. The database is Oracle 11g.
Is my ecpectation of the transactional behaviour wrong? How do I get the behaviour I want?
/* shortened exemplary stacktrace for rollback */
EvaluationException:
javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:101)
EJBTransactionRolledbackException:
org.jboss.as.ejb3.tx.CMTTxInterceptor.handleEndTransactionException(CMTTxInterceptor.java:115)
RollbackException:
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1177)
DatabaseException:
org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:324)
SQLSyntaxErrorException:
oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
Your expectation is correct: everything should be made in a single transaction, and the insertion of Antrag should be rolled back as well.
I think your persistence-unit is simply not JTA: test in the persistence.xml file that you have something like:
<persistence-unit name="ejb-model" transaction-type="JTA">
<jta-data-source>java:/someNameDB</jta-data-source>