I'm trying to configure my Micronaut project to insert/update using batch operations (create one single insert with 50 items included instead of 50 different inserts).
I cannot find a more detailed manual so the configuration that I'm using is this one:
datasources:
default:
url: jdbc:postgresql:///postgres?cloudSqlInstance=uri&socketFactory=com.google.cloud.sql.postgres.SocketFactory&ipTypes=PRIVATE
username: ****
password: ****
driverClassName: org.postgresql.Driver
jpa:
default:
properties:
hibernate:
hbm2ddl:
auto: none
jdbc:
batch_size: 50
order_inserts: true
order_updates: true
batch_versioned_data: true
show_sql: false
This is the repo:
import io.micronaut.data.annotation.Repository;
import io.micronaut.data.repository.GenericRepository;
#Repository
public interface ArticleExportPort extends GenericRepository<ArticleExport, Long> {
void insertMany(Iterable<ArticleExport> articleExports);
List<ArticleExport> findAll();
}
And this is the Entity:
#Data
#Entity
#Table(name = "PRODUCT_EXPORT")
public class ArticleExport {
#Id
private Long Id;
#NotNull
#Column(name = "PRODUCT_ID",nullable = false)
private Long productId;
}
The database is a PostgreSQL located in Google Cloud but also testing in local it doesn't work.
Any suggestion?
Thanks!
Related
I'm trying to run batch operations in my Micronaut app.
Currently, I can run insert operations in batch (with reWriteBatchedInserts option activated) but the configuration is not working for updates.
Something special about my app is that I want to set the id manually.
This is my entity:
#Data
#Entity
#SuperBuilder
#NoArgsConstructor
#Table(name = "PRODUCT")
public class ArticleItem implements Cloneable, Serializable {
#Id
#Column
private Long id;
The service:
#Singleton
#Slf4j
public class ArticleServiceImpl implements ArticleService {
#Transactional
public List<GenericSingleResponse> internalSaveArticles(
List<ArticleItem> articleItemList,
Integer erp, String traceId, String json
) {
articlePort.saveAll(insertList);
The port:
#Singleton
public class ArticlePortImpl implements ArticlePort {
private final EntityManager entityManager;
#Transactional
public void updateAll(List<ArticleItem> articleItemList) {
articleItemList.forEach(f -> entityManager.merge(f));
}
The configuration:
datasources:
default:
url: jdbc:postgresql://localhost:5432/postgres?reWriteBatchedInserts=true
jpa:
default:
properties:
hibernate:
hbm2ddl:
auto: none
jdbc:
batch_size: 100
batch_versioned_data: true
order_inserts: true
order_updates: true
batch_versioned_data: true
show_sql: false
id:
optimizer:
pooled:
preferred: pooled-lo
I am rather new to Spring boot and I am trying to write a very simple program that can perform post, get and delete on a postgreSQL data base. the database is named "recipes" schema "public" and table "recipe"
The problem that I ran into is that when I make the get request through postman, it simply returns null despite the data base being initialized with data.
like so
I did my best to try and narrow down the problem and the furthest I got is that line from the service layer is returning nothing when evaluated
jdbcTemplate.query(sql, new RecipeRowMapper())
The database is initialized with the following SQL
INSERT INTO recipe(id, name, ingredients, instructions, date_added)
values (1, 'ini test1', '10 cows 20 rabbits', 'cook ingredients with salt', '2004-01-02'),
(2, 'ini test2', '30 apples 20 pears', 'peel then boil', '2004-01-13');
I know the database is not empty because when I run the following SQL
SELECT * from recipe
i get
And the data base is connected as seen below (one thing I do find strange is that the table "recipe" isn't showing up in the DB browser but I don't know what to make of it)
application.yml
app:
datasource:
main:
driver-class-name: org.postgresql.Driver
jdbc-url: jdbc:postgresql://localhost:5432/recipes?currentSchema=public
username: postgres
password: password
server:
error:
include-binding-errors: always
include-message: always
spring.jpa:
database: POSTGRESQL
hibernate.ddl-auto: create
show-sql: true
dialect: org.hibernate.dialect.PostgreSQL9Dialect
format_sql: true
spring.flyway:
baseline-on-migrate: true
this is the service layer
public List<Recipe> getRecipes(){
var sql = """
SELECT id, name, ingredients, instructions, date_added
FROM public.recipe
LIMIT 50
""";
return jdbcTemplate.query(sql, new RecipeRowMapper());
}
and this is the controller
#GetMapping(path = "/test")
public String testRecipe(){
return recipeService.test();
}
and rowmapper
public class RecipeRowMapper implements RowMapper<Recipe> {
#Override
public Recipe mapRow(ResultSet rs, int rowNum) throws SQLException {
return new Recipe(
rs.getLong("id"),
rs.getString("name"),
rs.getString("ingredients"),
rs.getString("instructions"),
LocalDate.parse(rs.getString("date_added"))
);
}
}
finally recipe entity looks like this
#Data
#Entity
#Table
public class Recipe {
#Id
#GeneratedValue(
strategy = GenerationType.IDENTITY
)
#Column(name = "id", updatable = false, nullable = false)
private long id;
#Column(name = "name")
private String name;
#Column(name = "ingredients")
private String ingredients;
#Column(name = "instructions")
private String instructions;
#Column(name = "date_added")
private LocalDate dateAdded;
public Recipe(){};
public Recipe(long id, String name, String ingredients, String instructions, LocalDate date){}
public Recipe(String name,
String ingredients,
String instructions,
LocalDate dateAdded
) {
this.name = name;
this.ingredients = ingredients;
this.instructions = instructions;
this.dateAdded = dateAdded;
}
}
As it turns out the problem is caused by the LocalDate not being converted correctly and was being posted as null. That caused the
LocalDate.parse(rs.getString("date_added"))
to throw a null pointer exception which is what has been causing all the problems...
I have a simple spring boot application with two entities, CustomUser and Role, the application connects to the PostgreSQL database without any issues but no tables are created.
application.yml:
server:
port: 8082
spring:
application:
name: user-service
datasource:
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://localhost:5432/sdm-user-db
username: root
password: password
# JPA properties
jpa:
hibernate:
ddl-auto: create
show-sql: true
database: postgresql
database-platform: org.hibernate.dialect.PostgreSQLDialect
open-in-view: false
generate-ddl: true
zipkin:
base-url: http://localhost:9411/
eureka:
client:
register-with-eureka: true
fetch-registry: true
service-url:
defaultZone: http://localhost:8761/eureka/
instance:
hostname: localhost
CustomUser:
#AllArgsConstructor
#NoArgsConstructor
#Getter
#Setter
#Entity
public class CustomUser implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(generator = "UUID")
#GenericGenerator(
name = "UUID",
strategy = "org.hibernate.id.UUIDGenerator"
)
#Column(name = "id", updatable = false, nullable = false, columnDefinition = "VARCHAR(36)")
#Type(type = "uuid-char")
private UUID id;
private String name;
#Column(unique=true)
private String email;
private String password;
private String hashKey;
#ManyToMany(fetch = FetchType.EAGER)
private Collection<Role> roles = new ArrayList<Role>();
}
Role:
#AllArgsConstructor
#NoArgsConstructor
#Getter
#Setter
#Entity
public class Role implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy= GenerationType.IDENTITY)
private Long id;
private String name;
}
pom.xml database and jpa dependencies are included
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
The application runs without any issues so I have not much to debug with.
pg admin showing empty tables for the database
here I even try the connection with the database with the intelij plugin
successful connection test with intelij plugin
then I tried a simple commandLineRunner where I save one role into the database, it fails and here is the caused by part of the error log
Caused by: org.postgresql.util.PSQLException: ERROR: relation "role" does not exist
Position : 13
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675) ~[postgresql-42.3.5.jar:42.3.5]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365) ~[postgresql-42.3.5.jar:42.3.5]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355) ~[postgresql-42.3.5.jar:42.3.5]
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490) ~[postgresql-42.3.5.jar:42.3.5]
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408) ~[postgresql-42.3.5.jar:42.3.5]
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:167) ~[postgresql-42.3.5.jar:42.3.5]
at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:135) ~[postgresql-42.3.5.jar:42.3.5]
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) ~[HikariCP-4.0.3.jar:na]
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) ~[HikariCP-4.0.3.jar:na]
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:197) ~[hibernate-core-5.6.9.Final.jar:5.6.9.Final]
... 67 common frames omitted
If you are using Hibernate, setting spring.jpa.hibernate.ddl-auto to update is likely to help.
My Environment
mysql 8.0.25
hibernate-core:5.4.32
hibernate-spatial:5.4.32
spring-boot2.5.4
java 8
What I did
application.yml
spring:
datasource:
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://localhost:3306/database?serverTimezone=UTC&characterEncoding=UTF-8
username: root
password: password
jpa:
hibernate.ddl-auto: create
generate-ddl: true
database: mysql
properties:
hibernate.dialect: org.hibernate.spatial.dialect.mysql.MySQL56SpatialDialect
logging:
level:
org:
hibernate:
SQL: debug
type: trace
Entity class
import com.example.mypackage.domain.BaseTimeEntity;
import lombok.Builder;
import lombok.Getter;
import lombok.NoArgsConstructor;
import org.springframework.data.geo.Point;
import javax.persistence.*;
#Getter
#NoArgsConstructor
#Entity
public class Party extends BaseTimeEntity { // BaseTimeEntity adds modifiedAt, createdAt columns
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#Column(columnDefinition = "TEXT")
private String title;
#Column(columnDefinition = "POINT")
private Point coordinate;
#Builder
public Party(Point coordinate, String title, String body) {
this.coordinate = coordinate;
this.title = title;
this.body = body;
}
}
Test
#SpringBootTest
class PartyRepositoryTest {
#Autowired
PartyRepository partyRepository;
#Test
public void register_party() {
// Given
Double x = 127.02558;
Double y = 37.30160;
Point coordinate = new Point(x, y);
partyRepository.save(
Party.builder()
.coordinate(coordinate)
.title("test title")
.build()
);
// When
List<Party> partyList = partyRepository.findAll();
// Then
Party party = partyList.get(0);
assertEquals(x, party.getCoordinate().getX());
assertEquals(y, party.getCoordinate().getY());
}
What I expected
insert row in 'party' table successfully
What actually happened
I got error. Log is as below.
insert into party (created_at, modified_at, body, coordinate, title) values (?, ?, ?, ?, ?)
binding parameter [1] as [TIMESTAMP] - [2021-09-12T14:45:31.018]
binding parameter [2] as [TIMESTAMP] - [2021-09-12T14:45:31.018]
binding parameter [3] as [VARCHAR] - []
binding parameter [4] as [VARBINARY] - [Point [x=127.025580, y=37.301600]]
binding parameter [5] as [VARCHAR] - [test title]
SQL Error: 1416, SQLState: 22001
Data truncation: Cannot get geometry object from data you send to the GEOMETRY field
Question
Please Let me know what am I doing wrong?
does hibernate-spatial supports mysql point?
You are using the wrong spatial type: org.springframework.data.geo.Point is not supported by Hibernate Spatial. Use either org.locationtech.jts.geom.* or org.geolatte.geom.* in your entity class and it should be fine.
After a lot of research, that's how it worked for me:
Language: Kotlin
MySQL version: 8.0.23
build.gradle.kts
...
implementation("org.hibernate:hibernate-spatial:5.6.2.Final")
...
properties.yml
...
spring:
jpa:
properties:
hibernate.dialect: org.hibernate.spatial.dialect.mysql.MySQL8SpatialDialect
database-platform: org.hibernate.spatial.dialect.mysql.MySQL8SpatialDialect
...
entity:
package svns.mono.fad.springcore.data.entity
...
import org.locationtech.jts.geom.Point
...
#Entity
#Table(name = "issue")
data class IssueEntity(
...
val location: Point,
...
)
I try to insert data into Postgres using JPA/Hibernate/Postgres.
The data is parsed from a CSV File and then should be saved into a postgres database. The code that persists the data looks as follows:
#Autowired
KundeRepository repo;
#Transactional
public void safe(Kunde kd) {
repo.save(kd);
}
public void safeAll(Iterable<Kunde> kt) {
repo.save(kt);
repo.flush();
}
The entity look as follows
public class account implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Long id;
}
#ManyToOne
Kunde kunde;
#OneToMany
List<TransaktionsGruppe> gruppen;
#Entity
#Table(name = "kunde")
#NoArgsConstructor
public class Kunde implements Serializable {
public static final String kundennummerKey = "KUNDENNUMMER";
private static final long serialVersionUID = 1L;
#Id
#Getter
#Setter
private String id;
#OneToMany
List<Account> accounts;
#Entity
#Table(name = "transaktionsgruppe")
public class Transaktionsgruppe implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Long id;
#ManyToOne
Account acc;
private String bezeichnung;
When I now pass a Collection to the safeAll method the inserts are really slow. Especially it seems to call the hibernate sequence for every insert. Is there a way to speed things up?
My configuration looks as follows:
datasource:
type: com.zaxxer.hikari.HikariDataSource
url: dburl
username: user
password: pw
hikari:
validation-timeout: 10000
health-check-properties: {"connectivityCheckTimeoutMs","1000"}
jpa:
show-sql: true
properties:
hibernate.cache.use_second_level_cache: false
hibernate.cache.use_query_cache: false
hibernate.generate_statistics: false
hibernate.jdbc.batch.size: 100
hibernate.order_inserts: true+
Currently show-sql is enabled. It is about 60000 Entities overall and it takes more than 20 Minutes. the entitties are rather small in size
Your safeAll() method is not annotated with #Transactional so Spring opens and closes a transaction for each item of your list. By annotating it, Spring will open and close one single transaction for the whole list.