How to set the use connectionstring which usess mongodb in production.
development:
grails:
mongodb:
connectionString: "mongodb://localhost:27017/fanfest"
production:
dataSource:
dbCreate: update
url: jdbc:h2:./prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
properties:
jmxEnabled: true
initialSize: 5
maxActive: 50
minIdle: 5
maxIdle: 25
maxWait: 10000
maxAge: 600000
timeBetweenEvictionRunsMillis: 5000
minEvictableIdleTimeMillis: 60000
validationQuery: SELECT 1
validationQueryTimeout: 3
validationInterval: 15000
testOnBorrow: true
testWhileIdle: true
testOnReturn: false
jdbcInterceptors: ConnectionState
defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
Working with Grails 3 version. Able to connect with mongodb in development environment. Kindly provide some suggestions to set the mongodb in production environment.
You currently have development pointing at a mongo instance. The production configuration is pointed at the in memory H2 database. If you would like to configure a mongo database for your production environment may I suggest you take a look at Getting Started guide for Mongo and GORM.
In the production section of your configuration file you can use the connection string parameter as follows:
production {
grails {
mongodb {
connectionString = "mongodb://localhost:27017/PROD_fanfest"
}
}
}
Please note I have used your development URL but changed the table name since I recommend you keep the development database and production database separate. Configuring production datasource to use Mongo is that easy. There are more configuration options described in the Getting Started documentation.
Relevant information on configuring options below:
options {
connectionsPerHost = 10 // The maximum number of connections allowed per host
threadsAllowedToBlockForConnectionMultiplier = 5
maxWaitTime = 120000 // Max wait time of a blocking thread for a connection.
connectTimeout = 0 // The connect timeout in milliseconds. 0 == infinite
socketTimeout = 0 // The socket timeout. 0 == infinite
socketKeepAlive = false // Whether or not to have socket keep alive turned on
writeConcern = new com.mongodb.WriteConcern(0, 0, false) // Specifies the number of servers to wait for on the write operation, and exception raising behavior
sslEnabled = false // Specifies if the driver should use an SSL connection to Mongo
socketFactory = … // Specifies the SocketFactory to use for creating connections
}
Related
I have an spring-boot based application that connects to postgres DB. It uses the default HikariCP as JDBC connection pool. From the documentation I see that we can configure a property keepALiveTime which I think is useful to us but I am not able to set it in my yml file. It looks like this
datasource:
jdbc-url: jdbc:postgresql://x.x.x.x:5433/xxx?sslmode=require&sslcert=/file.crt&sslkey=/tmp/file.key
username: ${JDBC_USERNAME}
password: ${JDBC_PASSWORD}
keepaliveTime: 80000
jpa:
hibernate:
ddl-auto: update
show-sql: true
properties:
hibernate.format_sql: true
hibernate.dialect: org.hibernate.dialect.PostgreSQL81Dialect
hikari:
idle-timeout: 500000
minimum-idle: 10
maximum-pool-size: 15
keepaliveTime: 80000
max-lifetime: 48000
connection-timeout: 180000
All other configs are auto suggested and are able to be used. But keepAliveTime option is not available for me to populate. Although I am providing it in the configs Hikari doesn't use it.
keepAliveTime parameter arrived in HikariCP 4.0.0. In order to use it with older version of spring boot - we can exclude the HikariCP dependency from where it is being read and explicitly the version that we want. Its not recommended but I found no issues using 4.0.0 version with spring boot version 2.3.0-Release
To remove the dependency
implementation('org.springframework.boot:spring-boot-starter-data-jpa') {
exclude group: 'com.zaxxer', module: 'HikariCP'
}
To add the dependency
implementation('com.zaxxer:HikariCP:4.0.3') {
force=true
}
I added the keepAliveTime in DataSource Configuration file
public HikariDataSource dataSource() {
HikariDataSource h = DataSourceBuilder.create().type(HikariDataSource.class).build();
System.out.println(h.getJdbcUrl());
h.setKeepaliveTime(100000l);
h.setJdbcUrl("yufuyfuyfu");
System.out.println(h.getJdbcUrl());
return h;
}
please notice that keepAliveTime should be less than maxLifeTime, from HikariCP docs:
This property controls how frequently HikariCP will attempt to keep a connection alive, in order to prevent it from being timed out by the database or network infrastructure. This value must be less than the maxLifetime value
we are using Grails 4 with Hibernate 5 and Postgresql and I ran into a strange problem which I don't know how to troubleshoot.
I have created a test which creates a Family (a large object graph having around 20 INSERT and UPDATE queries) and then it tries to retrieve it using its id.
So there is a FamilyController extending grails.rest.RestfulControllerhaving static responseFormats = ['json'] and the following methods:
#Override
protected Family createResource() {
def instance = new NewFamilyCommand()
bindData(instance, getObjectToBind())
instance.validate()
//Check for binding errors
if(instance.hasErrors()) {
throw new ValidationException('Unable to bind new family', instance.errors)
}
//Send the command to the service, which will return us an actual Family domain object
return familyService.addFamilyToUser(instance)
}
and
#Override
protected Family queryForResource(Serializable id) {
def family = familyService.safeGetFamily(Long.parseLong(id))
...
return family
}
Running this in a loop using a Cypress test, and it works fine most of the time.
Problem is that from time to time (seems to be a multiple of 50 which coincidentally is the number of maxConnections configured in Tomcat) querying the Family by the returned id does not find it.
Here is the Tomcat configuration for the datasource (default recommended by Grails):
dataSource:
pooled: true
jmxExport: true
driverClassName: org.postgresql.Driver
dialect: "net.kaleidos.hibernate.PostgresqlExtensionsDialect"
username: "test"
password: "test"
properties:
initialSize: 2
maxActive: 50
minIdle: 2
maxIdle: 2
maxWait: 10000
maxAge: 10 * 60000
timeBetweenEvictionRunsMillis: 5000
minEvictableIdleTimeMillis: 60000
validationQuery: "SELECT 1"
validationQueryTimeout: 3
validationInterval: 15000
testOnBorrow: true
testWhileIdle: true
testOnReturn: false
jdbcInterceptors: "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation: 2 # TRANSACTION_READ_COMMITTED
removeAbandoned: true
removeAbandonedTimeout: 300
Postgresql runs in a docker container with 2 PIDS for this database, let's say 73 and 74.
I looked in the Postgresql logs and I noticed a difference between a successful test and a failed one.
In a successful scenario, both the Family creation and the Family retrieval run in the same PID.
In a failed scenario, the Family creation is performed by pid 74 and Family retrieval is performed by PID 73.
What is more curious is that PID 73 is idle most of the time, it runs it's first query around the creation of Family 50 (presumably when the connections are started to being reused) and then at Family 101 is getting used in the Family retrieval query, whose transaction begins before PID 74 Family creation transaction commits (at least that is shown in Postgres logs but maybe the logs are not printed chronologically).
Checking the database right after the test fails, I see the Family saved in the database and also the tests pass if I add a little wait time before querying for the result.
I am wondering how could that be, I assume that Postgresl return the ID only after the transaction is being committed, and then why would the other PID not see it?
Any help in troubleshooting this would be greatly appreciated.
UPDATE: seems to be related to Hibernate.FLUSH_MODE. If I set sessionFactory.currentSession.setFlushMode(FlushMode.MANUAL) in familyService.addFamilyToUser(instance) the problem goes away and the statements are properly ordered. It seems that Hibernate flushes the session too early returning the family id even if Postgres does not commit all the insert/update statements.
Looks like it was our fault, we messed around with the DirtyCheckable#hasChanged implementation and the code generated a lot of UPDATE statements after the initial insert and this caused the session to be flushed several times and the underlying Posgresql transactions were out of sync with the Hibernate transactions
I am confused about pgbouncer pool size configuration and ORM(like sequelize.js), query builder(like knex.js) library pool size configuration. The architecture like this:
Application code => pgbouncer => postgresql
pgbouncer.ini:
;; ...
;; Default pool size. 20 is good number when transaction pooling
;; is in use, in session pooling it needs to be the number of
;; max clients you want to handle at any moment
;default_pool_size = 20
;; ...
sequelize connection pool configuration:
const sequelize = new Sequelize(/* ... */, {
// ...
pool: {
max: 5,
min: 0,
acquire: 30000,
idle: 10000
}
});
knex.js connection pool configuration:
var knex = require('knex')({
client: 'mysql',
connection: {
host : '127.0.0.1',
user : 'your_database_user',
password : 'your_database_password',
database : 'myapp_test'
},
pool: { min: 0, max: 7 }
});
What happened if I use sequelize.js connection pool configuration and pgbouncer connection pool size configuration together? Which configuration does the database server use? Should I use only one of them? Thanks.
If you have for example 3 application processes running knex or sequelize, then you should setup pgbouncer poolsize to be 3 times bigger than single knex / sequelize pool uses.
Then you also need to make sure that postgres server also has enough connections configured to handle pgbouncer connections.
Though as #jjanes said. There is no reason to use pgbouncer with knex / sequelize, because they already provide pooling. I suppose pgbouncer is meant to be used with frameworks, which doesn't support pooling. For example if PHP or cgi stript reinitializes on every page load and makes calls to database.
It rarely makes sense to daisy chain connection pools together. So there is probably no point in using pgbouncer in addition to the built-in ones. The database server doesn't know about your connection poolers except to the extent the pool manager sends its own explicit commands to the database, and it has its own configuration file which it uses (postgresql.conf).
Currently I am using Grails and I am running several servers connecting to a single mongo server.
options {
autoConnectRetry = true
connectTimeout = 3000
connectionsPerHost = 100
socketTimeout = 60000
threadsAllowedToBlockForConnectionMultiplier = 10
maxAutoConnectRetryTime=5
maxWaitTime=120000
}
Unfortunately, when I run 50 servers, total number of connections goes up by 5k. After a bit of research I found that this was a simple config in the DataSource.groovy
I am sure that my programs do not need 100 mongo connections.
But I am unsure what value should I set this to.
I have 2 doubts.
First, how to determine the optimal value for the connectionsPerHost.
Second, whether all these 100 connections are created once and then pooled?
I'm using Grails 2.4.4, mongo plugin 3.0.2, MongoDB 2.4.10 using a remote database connection.
grails {
mongo {
host = "11.12.13.14" // A remote server IP
port = 27017
databaseName = "blogger"
username = "blog"
password = "xyz"
options {
autoConnectRetry = true
connectTimeout = 3000
connectionsPerHost = 40
socketTimeout = 120000
threadsAllowedToBlockForConnectionMultiplier = 5
maxAutoConnectRetryTime=5
maxWaitTime=120000
}
}
}
In a part of our application, a service method iterates over a 20,000 user's and sends them an email:
Person.withCriteria { // Line 323
eq("active", true)
order("dateJoined", "asc")
}.each { personInstance ->
// Code to send an email which takes an average of 1 second
}
After executing this for some 6000 user, I'm getting a MongoDB cursor exception:
2015-04-11 07:31:14,218 [quartzScheduler_Worker-1] ERROR listeners.ExceptionPrinterJobListener - Exception occurred in job: Grails Job
org.quartz.JobExecutionException: com.mongodb.MongoException$CursorNotFound: Cursor 1337814790631604331 not found on server 11.12.13.14:27017 [See nested exception: com.mongodb.MongoException$CursorNotFound: Cursor 1337814790631604331 not found on server 11.12.13.14:27017]
at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:111)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: com.mongodb.MongoException$CursorNotFound: Cursor 1337814790631604331 not found on server 11.12.13.14:27017
at com.mongodb.QueryResultIterator.throwOnQueryFailure(QueryResultIterator.java:218)
at com.mongodb.QueryResultIterator.init(QueryResultIterator.java:198)
at com.mongodb.QueryResultIterator.initFromQueryResponse(QueryResultIterator.java:176)
at com.mongodb.QueryResultIterator.getMore(QueryResultIterator.java:141)
at com.mongodb.QueryResultIterator.hasNext(QueryResultIterator.java:127)
at com.mongodb.DBCursor._hasNext(DBCursor.java:551)
at com.mongodb.DBCursor.hasNext(DBCursor.java:571)
at org.grails.datastore.mapping.mongo.query.MongoQuery$MongoResultList$1.hasNext(MongoQuery.java:1893)
at com.test.person.PersonService.sendWeeklyEmail(PersonService.groovy:323)
at com.test.WeeklyJob.execute(WeeklyJob.groovy:41)
at grails.plugins.quartz.GrailsJobFactory$GrailsJob.execute(GrailsJobFactory.java:104)
... 2 more
I looked for documentation and found that cursor automatically get closed in 20 minutes and when I confirmed it with the logs, this exception came exactly after 20 minutes.
But this behaviour of auto-close in 20 minutes is applicable for inactive cursor but here the cursor is active.
UPDATE:
I read some articles and found that, this could be an issue of TCP keepalive timeout. So we changed the TCP keepalive timeout to 2 minutes from default 2 hours, but it still doesn't solve the problem.
Looks like a compatibility issue with MonoDB on this server. Read more on the Jira for details https://jira.mongodb.org/browse/SERVER-18439
Hope this helps to someone else!