Not able to set keepAliveTime config for Hikari in SpringBoot - postgresql

I have an spring-boot based application that connects to postgres DB. It uses the default HikariCP as JDBC connection pool. From the documentation I see that we can configure a property keepALiveTime which I think is useful to us but I am not able to set it in my yml file. It looks like this
datasource:
jdbc-url: jdbc:postgresql://x.x.x.x:5433/xxx?sslmode=require&sslcert=/file.crt&sslkey=/tmp/file.key
username: ${JDBC_USERNAME}
password: ${JDBC_PASSWORD}
keepaliveTime: 80000
jpa:
hibernate:
ddl-auto: update
show-sql: true
properties:
hibernate.format_sql: true
hibernate.dialect: org.hibernate.dialect.PostgreSQL81Dialect
hikari:
idle-timeout: 500000
minimum-idle: 10
maximum-pool-size: 15
keepaliveTime: 80000
max-lifetime: 48000
connection-timeout: 180000
All other configs are auto suggested and are able to be used. But keepAliveTime option is not available for me to populate. Although I am providing it in the configs Hikari doesn't use it.

keepAliveTime parameter arrived in HikariCP 4.0.0. In order to use it with older version of spring boot - we can exclude the HikariCP dependency from where it is being read and explicitly the version that we want. Its not recommended but I found no issues using 4.0.0 version with spring boot version 2.3.0-Release
To remove the dependency
implementation('org.springframework.boot:spring-boot-starter-data-jpa') {
exclude group: 'com.zaxxer', module: 'HikariCP'
}
To add the dependency
implementation('com.zaxxer:HikariCP:4.0.3') {
force=true
}
I added the keepAliveTime in DataSource Configuration file
public HikariDataSource dataSource() {
HikariDataSource h = DataSourceBuilder.create().type(HikariDataSource.class).build();
System.out.println(h.getJdbcUrl());
h.setKeepaliveTime(100000l);
h.setJdbcUrl("yufuyfuyfu");
System.out.println(h.getJdbcUrl());
return h;
}

please notice that keepAliveTime should be less than maxLifeTime, from HikariCP docs:
This property controls how frequently HikariCP will attempt to keep a connection alive, in order to prevent it from being timed out by the database or network infrastructure. This value must be less than the maxLifetime value

Related

Akka Projection configuration for a JDBC database connection

In Akka Projection’s documentation under offsetting in a relational database with JDBC, there is no information about how and where the configuration for establishing a connection to the relational database used should be included. I mean configs such as the username, password, or the url.
In the documentation under offset in a relational database with Slick, the following configuration is provided for the database connection, which is unclear whether it can be used for JDBC as well:
# add here your Slick db settings
db {
# url = "jdbc:h2:mem:test1"
# driver = org.h2.Driver
# connectionPool = disabled
# keepAliveConnection = true
}
How and where should I specify the JDBC connection parameters?
https://doc.akka.io/docs/akka-projection/current/jdbc.html#defining-a-jdbcsession
The following line in the Scala code snippet is where you can specify connection parameters:
val c = DriverManager.getConnection("jdbc:h2:mem:test;DB_CLOSE_DELAY=-1")

Spring Batch / Postgres : ERROR: relation "batch_job_instance" does not exist

I am trying to configure Spring Batch to use PostGres DB. I have included the following dependencies in my build.gradle.kts file:
implementation("org.springframework.boot:spring-boot-starter-data-jpa")
implementation("org.postgresql:postgresql")
My application.yml for my SpringBatch module has the following included:
spring:
datasource:
url: jdbc:postgresql://postgres:5432/springbatchdb
username: postgres
password: root
driverClassName: org.postgresql.Driver
docker-compose.yml
postgres:
restart: always
image: postgres:12-alpine
container_name: postgres
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=root
- POSTGRES_DB=springbatchdb
ports:
- "5432:5432"
volumes:
- postgresql:/var/lib/postgresql
- postgresql_data:/var/lib/postgresql/data
However, when I try to add a data file I see the following error in the logs of both my SpringBatch Docker container, and the PostGres container:
Spring Batch:
<<< Exception in method: org.meanwhileinhell.spring.batch.server.SpringBatchController.handle Error Message: PreparedStatementCallback; bad SQL grammar [SELECT JOB_INSTANCE_ID, JOB_NAME from BATCH_JOB_INSTANCE where JOB_NAME = ? and JOB_KEY = ?]; nested exception is org.postgresql.util.PSQLException: ERROR: relation "batch_job_instance" does not exist
PostGres:
LOG: database system is ready to accept connections
2021-01-08 09:54:56.778 UTC [56] ERROR: relation "batch_job_instance" does not exist at character 39
2021-01-08 09:54:56.778 UTC [56] STATEMENT: SELECT JOB_INSTANCE_ID, JOB_NAME from BATCH_JOB_INSTANCE where JOB_NAME = $1 and JOB_KEY = $2
2021-01-08 09:55:27.033 UTC [56] ERROR: relation "batch_job_instance" does not exist at character 39
2021-01-08 09:55:27.033 UTC [56] STATEMENT: SELECT JOB_INSTANCE_ID, JOB_NAME from BATCH_JOB_INSTANCE where JOB_NAME = $1 and JOB_KEY = $2
I can see that the SB server is picking up POSTGRES from my metadata ok.
JobRepositoryFactoryBean : No database type set, using meta data indicating: POSTGRES
What am I missing to get the initial db configured during the server start?
Edit: I've tried adding spring.datasource.initialize=true explicitly, but no change.
Please check below added in application.yml
spring.batch.initialize-schema: always
Please check below dependencies are added
<artifactId>spring-boot-starter-batch</artifactId>
yaml file is
spring.datasource.url=jdbc:postgresql://localhost:5432/postgres
spring.datasource.username=postgres
spring.datasource.password=1234
spring.datasource.driver-class-name=org.postgresql.Driver
spring.batch.jdbc.initialize-schema=always
gradle dependencies
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-jdbc'
implementation 'org.springframework.boot:spring-boot-starter-batch'
implementation 'org.projectlombok:lombok-maven-plugin:1.18.6.0'
implementation group: 'org.postgresql', name: 'postgresql', version: '42.3.1'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
testImplementation 'org.springframework.batch:spring-batch-test'
}
You need to set spring.batch.initialize-schema=always property to tell Spring Boot to create Spring Batch tables automatically. Please refer to the Initialize a Spring Batch Database section of Spring Boot's reference documentation for more details.
For anyone who has spring.batch.initialize-schema=always set already and it's still not working, also verify that you are connecting to the database with a user that has sufficient privileges, including to create the necessary tables.
in application.properties,
Prior Spring Boot 2.5 we can use
spring.batch.initialize-schema=ALWAYS
Later version of Spring Boot 2.5 use below
spring.batch.jdbc.initialize-schema=ALWAYS
Solution that worked for me in Spring 5.0!
I spent a lot of time resolving issues like ERROR: relation "X" does not exist when using the latest Spring Boot Starter 3.0 and Spring Batch 5.0.
spring.batch.jdbc.initialize-schema=always
However, it didn't create the necessary tables for me. Though as per the documentation, it should have created tables.
After a lot of research, I found, that in the latest Spring Batch 5.0, there are a lot of improvements. And I was doing a lot of things wrong when migrating to the new Spring 5.
Remove #EnableBatchProcessing from your configurations. As you don't need that anymore with the latest Spring Batch 5.
Example:
#Configuration
#AllArgsConstructor
#EnableBatchProcessing //please remove it.
public class SpringBatchConfiguration {}
change it to:
#Configuration
#AllArgsConstructor
public class SpringBatchConfiguration {}
PlatformTransaction Manager: The second thing I was doing wrong was using an incorrect Transaction Manager, if you are using JPA for persisting entities you need a corresponding Transaction Manager.
I was using the ResourcelessTransactionManager() which was wrong in my case and was creating a lot of headaches while running.
For JPA you need a JpaTransactionManager()
Something like:
#Bean
public PlatformTransactionManager transactionManager() {
return new JpaTransactionManager();
}
The third thing I learned after mistakes; we don't need to create a bean of Datasource unless we are doing something complex like having 2 Datasource one for writing Spring Batch associated tables and another for persisting our business data.
Wherever we are required to use JobRepository just inject it.
Something like:
#Bean
#Autowired
Job job(JobRepository jobRepository) {
JobBuilder jobBuilderFactory = new JobBuilder("somename", jobRepository );
return jobBuilderFactory.flow(step1(jobRepository)).end()
.build();
}
For more details on migrations: Spring Migration 3 Guide.

How to disable Postgres auto-commit in Spring Data R2DBC

I see that currently, you can disable auto-commit on Connection object. But when you work with Spring Data you don't have access to Connection object.
Also, Postgres doesn't have this feature as part of the connection URL.
Does anyone know how to turn off it permanently?
I do not know if there is a way to disable it permanently but you could go a bit low-level and use Connection object to set auto-commit feature:
#Autowired
private ConnectionFactory connectionFactory;
public Flux<Integer> insertSmth() {
return Mono.from(connectionFactory.create())
.flatMapMany(connection ->
Flux.from(connection.setAutoCommit(false))
.thenMany(connection.beginTransaction())
.thenMany(connection.createStatement("INSERT INTO ...").execute())
...
}
Edit:
I seems propagation and reset of Auto-Commit and Isolation Level control through TransactionDefinition is available in Spring Data R2DBC 1.0.0 RC1.
See this thread: https://github.com/r2dbc/r2dbc-spi/pull/95

Spring Boot JPA Schema Initialization

I want to initialise my datasource with a DLL script in my spring boot project (only during dev of course). As mentioned in the docs here I set the spring.datasource.schema property to the DLL scripts which is in src/main/resources/postgresql/define-schema.sql.
spring:
profiles: dev
datasource:
platform: postgresql
driver-class-name: org.postgresql.Driver
url: jdbc:postgresql://localhost:5432/postgres
username: postgres
password: ****
initialize: true
schema: ./postgresql/define-schema.sql
continue-on-error: false
jpa:
hibernate:
ddl-auto: validate
generate-ddl: false
show-sql: true
But the script won't be executed. I also tried to put it on the class path root and call it schema.sql ... Nothing happens.
The dev profile is selected, at least I see it in the log: The following profiles are active: dev
The application than fails on the JPA schema validation.
The only warning I get from hibernate:
Found use of deprecated [org.hibernate.id.SequenceGenerator] sequence-based id generator; use org.hibernate.id.enhanced.SequenceStyleGenerator instead. See Hibernate Domain Model Mapping Guide for details.
But I don't think this has any to do with the initialise problem.
I've got spring-boot-security-starter in my dependencies but not configured yet, could that be a problem source?
Does anybody recognise an obivous typo, mistake anything else?
Looking forward hearing from you!
Amp
Prefix the path to your sql script with classpath
Example:
spring.datasource.schema=classpath:/postgresql/define-schema.sql

Grails 3 Setup application.yml production mongodb

How to set the use connectionstring which usess mongodb in production.
development:
grails:
mongodb:
connectionString: "mongodb://localhost:27017/fanfest"
production:
dataSource:
dbCreate: update
url: jdbc:h2:./prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
properties:
jmxEnabled: true
initialSize: 5
maxActive: 50
minIdle: 5
maxIdle: 25
maxWait: 10000
maxAge: 600000
timeBetweenEvictionRunsMillis: 5000
minEvictableIdleTimeMillis: 60000
validationQuery: SELECT 1
validationQueryTimeout: 3
validationInterval: 15000
testOnBorrow: true
testWhileIdle: true
testOnReturn: false
jdbcInterceptors: ConnectionState
defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
Working with Grails 3 version. Able to connect with mongodb in development environment. Kindly provide some suggestions to set the mongodb in production environment.
You currently have development pointing at a mongo instance. The production configuration is pointed at the in memory H2 database. If you would like to configure a mongo database for your production environment may I suggest you take a look at Getting Started guide for Mongo and GORM.
In the production section of your configuration file you can use the connection string parameter as follows:
production {
grails {
mongodb {
connectionString = "mongodb://localhost:27017/PROD_fanfest"
}
}
}
Please note I have used your development URL but changed the table name since I recommend you keep the development database and production database separate. Configuring production datasource to use Mongo is that easy. There are more configuration options described in the Getting Started documentation.
Relevant information on configuring options below:
options {
connectionsPerHost = 10 // The maximum number of connections allowed per host
threadsAllowedToBlockForConnectionMultiplier = 5
maxWaitTime = 120000 // Max wait time of a blocking thread for a connection.
connectTimeout = 0 // The connect timeout in milliseconds. 0 == infinite
socketTimeout = 0 // The socket timeout. 0 == infinite
socketKeepAlive = false // Whether or not to have socket keep alive turned on
writeConcern = new com.mongodb.WriteConcern(0, 0, false) // Specifies the number of servers to wait for on the write operation, and exception raising behavior
sslEnabled = false // Specifies if the driver should use an SSL connection to Mongo
socketFactory = … // Specifies the SocketFactory to use for creating connections
}