I had an heroku app has been running on there for the pass 5 years, and I have not doing any deployment for a while.
Only recently I saw there were two auto update by heroku postgres addon itself
Then from today, I try to go to my app as usual, try to do login, but failed as the app error log is saying not able to connect to postgres DB anymore. Please see these error, and my spring bean setting for DB connection (this setting has been there for 5 years unchanged, suddenly all stop working...)
I am wondering has anyone else encountered this issue recently? is it due to heroku major upgrade?
Please advise an solution if possible
Thanks
=================================================================
I have attached error log here:
2021-02-26T17:18:50.688282+00:00 app[web.1]: Feb 26, 2021 5:18:50 PM org.apache.catalina.core.StandardWrapperValve invoke
2021-02-26T17:18:50.688302+00:00 app[web.1]: SEVERE: Servlet.service() for servlet [mvc-dispatcher] in context with path [/] threw exception [Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Cannot open connection] with root cause
2021-02-26T17:18:50.688304+00:00 app[web.1]: org.postgresql.util.PSQLException: FATAL: no pg_hba.conf entry for host "35.175.150.172", user "hazpvnhqxtodzl", database "d7fhtbev11bt6m", SSL off
2021-02-26T17:18:50.688305+00:00 app[web.1]: at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:293)
2021-02-26T17:18:50.688306+00:00 app[web.1]: at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
2021-02-26T17:18:50.688307+00:00 app[web.1]: at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
2021-02-26T17:18:50.688307+00:00 app[web.1]: at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
2021-02-26T17:18:50.688308+00:00 app[web.1]: at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
2021-02-26T17:18:50.688308+00:00 app[web.1]: at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
2021-02-26T17:18:50.688308+00:00 app[web.1]: at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
2021-02-26T17:18:50.688309+00:00 app[web.1]: at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
2021-02-26T17:18:50.688310+00:00 app[web.1]: at org.postgresql.Driver.makeConnection(Driver.java:393)
2021-02-26T17:18:50.688311+00:00 app[web.1]: at org.postgresql.Driver.connect(Driver.java:267)
2021-02-26T17:18:50.688311+00:00 app[web.1]: at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
2021-02-26T17:18:50.688312+00:00 app[web.1]: at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
2021-02-26T17:18:50.688312+00:00 app[web.1]: at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
2021-02-26T17:18:50.688313+00:00 app[web.1]: at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
2021-02-26T17:18:50.688313+00:00 app[web.1]: at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
2021-02-26T17:18:50.688314+00:00 app[web.1]: at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
2021-02-26T17:18:50.688316+00:00 app[web.1]: at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:71)
2021-02-26T17:18:50.688316+00:00 app[web.1]: at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
2021-02-26T17:18:50.688317+00:00 app[web.1]: at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
2021-02-26T17:18:50.688317+00:00 app[web.1]: at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
2021-02-26T17:18:50.688318+00:00 app[web.1]: at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
2021-02-26T17:18:50.688318+00:00 app[web.1]: at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
2021-02-26T17:18:50.688318+00:00 app[web.1]: at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:60)
2021-02-26T17:18:50.688319+00:00 app[web.1]: at org.springframework.orm.jpa.DefaultJpaDialect.beginTransaction(DefaultJpaDialect.java:70)
2021-02-26T17:18:50.688320+00:00 app[web.1]: at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:61)
2021-02-26T17:18:50.688320+00:00 app[web.1]: at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:377)
2021-02-26T17:18:50.688321+00:00 app[web.1]: at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
2021-02-26T17:18:50.688321+00:00 app[web.1]: at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)
2021-02-26T17:18:50.688322+00:00 app[web.1]: at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)
2021-02-26T17:18:50.688322+00:00 app[web.1]: at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
2021-02-26T17:18:50.688323+00:00 app[web.1]: at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
2021-02-26T17:18:50.688323+00:00 app[web.1]: at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
2021-02-26T17:18:50.688324+00:00 app[web.1]: at com.sun.proxy.$Proxy34.listTillReportHistory(Unknown Source)
2021-02-26T17:18:50.688324+00:00 app[web.1]: at com.lsinternal.restfulservice.ApiTill.reportTillHistory(ApiTill.java:75)
2021-02-26T17:18:50.688325+00:00 app[web.1]: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-02-26T17:18:50.688325+00:00 app[web.1]: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2021-02-26T17:18:50.688326+00:00 app[web.1]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-02-26T17:18:50.688326+00:00 app[web.1]: at java.lang.reflect.Method.invoke(Method.java:607)
2021-02-26T17:18:50.688333+00:00 app[web.1]: at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
2021-02-26T17:18:50.688335+00:00 app[web.1]: at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
2021-02-26T17:18:50.688336+00:00 app[web.1]: at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
2021-02-26T17:18:50.688337+00:00 app[web.1]: at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743)
2021-02-26T17:18:50.688337+00:00 app[web.1]: at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672)
2021-02-26T17:18:50.688338+00:00 app[web.1]: at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82)
2021-02-26T17:18:50.688338+00:00 app[web.1]: at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:919)
2021-02-26T17:18:50.688339+00:00 app[web.1]: at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:851)
2021-02-26T17:18:50.688339+00:00 app[web.1]: at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)
2021-02-26T17:18:50.688340+00:00 app[web.1]: at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:844)
2021-02-26T17:18:50.688340+00:00 app[web.1]: at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
2021-02-26T17:18:50.688341+00:00 app[web.1]: at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)
2021-02-26T17:18:50.688341+00:00 app[web.1]: at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
2021-02-26T17:18:50.688342+00:00 app[web.1]: at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
2021-02-26T17:18:50.688342+00:00 app[web.1]: at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
2021-02-26T17:18:50.688343+00:00 app[web.1]: at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
Also, the config of my DB connection in spring xml config file:
<bean class="java.net.URI" id="dbUrl">
<constructor-arg value="#{systemEnvironment['DATABASE_URL']}"/>
</bean>
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
<property name="url" value="#{ 'jdbc:postgresql://' + #dbUrl.getHost() + ':' + #dbUrl.getPort() + #dbUrl.getPath() + '?sslmode=require' }"/>
<property name="username" value="#{ #dbUrl.getUserInfo().split(':')[0] }"/>
<property name="password" value="#{ #dbUrl.getUserInfo().split(':')[1] }"/>
</bean>
<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"/>
</property>
<property name="jpaProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.PostgreSQLDialect</prop>
<prop key="hibernate.show_sql">true</prop>
<!-- change this to 'verify' before running as a production app -->
<prop key="hibernate.hbm2ddl.auto">update</prop>
</props>
</property>
</bean>
it is absolutely identical to this tutorial -> https://devcenter.heroku.com/articles/heroku-postgresql#spring-xml
I have been trying to solve the issue in the pass 3 days, really ran out of idea now... please advise if possible
Thanks
Looks like this one:
Seeing "FATAL: no pg_hba.conf entry" errors in Postgres
Issue
I'm seeing "no pg_hba.conf entry for host" errors:
sql_error_code = 28000 FATAL: no pg_hba.conf entry for host "122.180.247.11", user "u3idolso5k2v83", database "dc85788d13v9ej", SSL off
Resolution
"FATAL: no pg_hba.conf entry for host" errors indicate that there was a failed authentication attempt to the database, so the connection couldn't be established. This can happen because of different reasons:
The authentication failed because the user/password credentials were invalid: ((user "xxxx", database "yyyy")). This could happen if you're trying to connect to the database using wrong or revoked credentials. (See also: Why am I seeing connection errors for my Heroku Postgres database from an unexpected IP address?).
The authentication failed because the connection didn't use SSL encryption: (SSL off). All Heroku Postgres production databases require using SSL connections to ensure that communications between applications and the database remain secure. If your client is not using SSL to connect to your database, you would see these errors even if you're using the right credentials to connect to it.
"I think" here SSL OFF is the case/(at least also) true. (Please rather copy&paste logs, than screen-shoting 'em!)
-> (2):
Most clients will connect over SSL by default, but on occasion it is necessary to set the sslmode=require parameter on a Postgres connection. Please add this parameter in code rather than editing the config var directly. Please check you are enforcing use of SSL especially if you are using Java or Node.js clients.
...
It is also important that you use a version of the Postgres JDBC driver verion 9.2 or greater. For example, in Maven add this to your pom.xml:
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.2.1</version>
</dependency>
...
The DATABASE_URL for the Heroku Postgres add-on follows this naming convention:
postgres://<username>:<password>#<host>/<dbname>
However the Postgres JDBC driver uses the following convention:
jdbc:postgresql://<host>:<port>/<dbname>?sslmode=require&user=<username>&password=<password>
If you navigate to your Database Credentials in your panel, Heroku says:
Please note that these credentials are not permanent. Heroku rotates
credentials periodically and updates applications where this database
is attached.
I recommend you check your credentials again, maybe it has changed.
Related
I deployed a NodeJS app to Heroku and provisioned an instance of MongoDB with MLab. Given that the password for the default database user is unclear (it never asks to set one up), I created a new user with admin privileges called mknerr. I set the MONGOLAB_URI variable like so:
heroku config:set MONGOLAB_URI=mongodb://mknerr:{password}#ds159631.mlab.com:59631/heroku_ws1jw516?authSource=heroku_ws1jw516&w=1
I tried to test the connection locally using mongo ds159631.mlab.com:59631/heroku_ws1jw516 -u mknerr -p {password} but I received the error "-bash: mongo: command not found" despite having already run npm mongodb. Rather than waste time trying to figure out where my mongo installation is, I switched to an instance of Cloud9 I know has mongo installed and ran the same command as above.
It appeared to confirm my credentials were accurate:
MongoDB shell version v3.6.5
connecting to: mongodb://ds159631.mlab.com:59631/heroku_ws1jw5l6
MongoDB server version: 3.6.6
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
rs-ds159631:PRIMARY>
However when I try to load my application, it throws a 503 error. After running heroku logs --tail I see:
2018-07-30T15:04:41.171411+00:00 heroku[web.1]: State changed from up to starting
2018-07-30T15:04:42.187835+00:00 heroku[web.1]: Stopping all processes with SIGTERM
2018-07-30T15:04:42.324261+00:00 heroku[web.1]: Process exited with status 143
2018-07-30T15:04:43.802715+00:00 heroku[web.1]: Starting process with command `npm start`
2018-07-30T15:04:40.930637+00:00 app[api]: Set MONGOLAB_URI config vars by user {me}
2018-07-30T15:04:46.562425+00:00 app[web.1]:
2018-07-30T15:04:46.562447+00:00 app[web.1]: > todos_api#1.0.0 start /app
2018-07-30T15:04:46.562449+00:00 app[web.1]: > node index.js
2018-07-30T15:04:46.562450+00:00 app[web.1]:
2018-07-30T15:04:47.302228+00:00 app[web.1]: App is running on Port 21448
2018-07-30T15:04:47.339759+00:00 app[web.1]: ERROR connecting to: mongodb://mknerr:{password}#ds159631.mlab.com:59631/heroku_ws1jw516?authsSource=heroku_ws1jw516. MongoError: Authentication failed.
2018-07-30T15:04:47.341632+00:00 app[web.1]: (node:20) UnhandledPromiseRejectionWarning: MongoError: Authentication failed.
2018-07-30T15:04:47.341635+00:00 app[web.1]: at /app/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:598:61
2018-07-30T15:04:47.341637+00:00 app[web.1]: at authenticateStragglers (/app/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:516:16)
2018-07-30T15:04:47.341638+00:00 app[web.1]: at Connection.messageHandler (/app/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:552:5)
2018-07-30T15:04:47.341640+00:00 app[web.1]: at emitMessageHandler (/app/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connection.js:309:10)
2018-07-30T15:04:47.341642+00:00 app[web.1]: at Socket.<anonymous> (/app/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connection.js:452:17)
2018-07-30T15:04:47.341644+00:00 app[web.1]: at emitOne (events.js:116:13)
2018-07-30T15:04:47.341646+00:00 app[web.1]: at Socket.emit (events.js:211:7)
2018-07-30T15:04:47.341647+00:00 app[web.1]: at addChunk (_stream_readable.js:263:12)
2018-07-30T15:04:47.341649+00:00 app[web.1]: at readableAddChunk (_stream_readable.js:250:11)
2018-07-30T15:04:47.341650+00:00 app[web.1]: at Socket.Readable.push (_stream_readable.js:208:10)
2018-07-30T15:04:47.341652+00:00 app[web.1]: at TCP.onread (net.js:597:20)
2018-07-30T15:04:47.341703+00:00 app[web.1]: (node:20) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
2018-07-30T15:04:47.341773+00:00 app[web.1]: (node:20) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Why is it failing to connect and throwing an authentication error despite my already having confirmed my credentials are accurate?
I keep getting errors like this...
app[web.1]: Configuration error: Configuration error[Cannot connect to database [default]]
app[web.1]: at scala.collection.immutable.List.foreach(List.scala:381)
app[web.1]: Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "lcnjgrwbmbffwv"
app[web.1]: at play.core.StaticApplication.<init>(ApplicationProvider.scala:55)
app[web.1]: at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:408)
app[web.1]: at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
app[web.1]: at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
app[web.1]: at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
app[web.1]: at org.postgresql.Driver.makeConnection(Driver.java:410)
app[web.1]: at java.sql.DriverManager.getConnection(DriverManager.java:664)
app[web.1]: at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:363)
app[web.1]: at play.api.db.BoneCPPlugin$$anonfun$onStart$1.apply(DB.scala:240)
app[web.1]: at java.sql.DriverManager.getConnection(DriverManager.java:247)
app[web.1]: at org.postgresql.Driver.connect(Driver.java:280)
app[web.1]: at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
app[web.1]: at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:120)
app[web.1]: at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
app[web.1]: ... 18 more
heroku[web.1]: Process exited with status 255
Not just in Heroku, but when I run my Play app with a user whose name is different than the default user name in the database.
My Play Framework "application.conf" is like this...
db.default.username="nameOfDatabaseUser"
db.default.password="passOfDatabaseUser"
db.default.driver=org.postgresql.Driver
db.default.url="jdbc:postgresql://urlofdatabase:5432/name_of_database"
Here's the thing. When my terminal user is named "nameOfDatabaseUser", it works. But when my terminal user has a different name, it connects to the database as the name of the current Linux terminal user.
In the above case it appears that Heroku tried to connect as a user lcnjgrwbmbffwv and there was no user lcnjgrwbmbffwv in the database.
How do I specify the default database username so that it doesn't use my terminal username?
The correct key to specify the user is db.default.user, not db.default.username.
I have deployed a play scala/java web app on heroku that uses an embeded netty server. The Procfile command is web: target/universal/stage/bin/mmbu-timesheets -Dhttp.port=80
Taken from heroku logs:
2015-04-27T13:49:27.160198+00:00 heroku[web.1]: Starting process with command `target/universal/stage/bin/mmbu-timesheets -Dhttp.port=80`
2015-04-27T13:49:29.321552+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx384m -Xss512k -Dfile.encoding=UTF-8 -Djava.rmi.server.useCodebaseOnly=true
2015-04-27T13:49:29.898379+00:00 app[web.1]: Play server process ID is 3
2015-04-27T13:49:32.549840+00:00 app[web.1]: [[37minfo[0m] application - mongodb connection ds031701.mongolab.com:31701 db->heroku_app36286493
2015-04-27T13:49:33.374954+00:00 app[web.1]: [[37minfo[0m] play - Application started (Prod)
2015-04-27T13:49:33.628067+00:00 app[web.1]: Oops, cannot start the server.
2015-04-27T13:49:33.630568+00:00 app[web.1]: at play.core.server.NettyServer$$anonfun$8.apply(NettyServer.scala:89)
2015-04-27T13:49:33.630523+00:00 app[web.1]: at play.core.server.NettyServer$$anonfun$8.apply(NettyServer.scala:92)
2015-04-27T13:49:33.630859+00:00 app[web.1]: at play.core.server.NettyServer$.createServer(NettyServer.scala:206)
2015-04-27T13:49:33.630358+00:00 app[web.1]: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:80
2015-04-27T13:49:33.630483+00:00 app[web.1]: at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
2015-04-27T13:49:33.630606+00:00 app[web.1]: at scala.Option.map(Option.scala:146)
2015-04-27T13:49:33.630802+00:00 app[web.1]: at play.core.server.NettyServer.<init>(NettyServer.scala:89)
2015-04-27T13:49:33.630898+00:00 app[web.1]: at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:243)
2015-04-27T13:49:33.630941+00:00 app[web.1]: at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:238)
2015-04-27T13:49:33.631096+00:00 app[web.1]: at play.core.server.NettyServer$.main(NettyServer.scala:238)
2015-04-27T13:49:33.630975+00:00 app[web.1]: at scala.Option.map(Option.scala:146)
2015-04-27T13:49:33.631150+00:00 app[web.1]: at play.core.server.NettyServer.main(NettyServer.scala)
2015-04-27T13:49:33.633017+00:00 app[web.1]: Caused by: java.net.SocketException: Permission denied
2015-04-27T13:49:33.633065+00:00 app[web.1]: at sun.nio.ch.Net.bind0(Native Method)
2015-04-27T13:49:33.633136+00:00 app[web.1]: at sun.nio.ch.Net.bind(Net.java:437)
2015-04-27T13:49:33.633179+00:00 app[web.1]: at sun.nio.ch.Net.bind(Net.java:429)
2015-04-27T13:49:33.633235+00:00 app[web.1]: at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
2015-04-27T13:49:33.633396+00:00 app[web.1]: at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
2015-04-27T13:49:33.633444+00:00 app[web.1]: at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372)
2015-04-27T13:49:33.633550+00:00 app[web.1]: at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
2015-04-27T13:49:33.633600+00:00 app[web.1]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2015-04-27T13:49:33.633643+00:00 app[web.1]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2015-04-27T13:49:33.633305+00:00 app[web.1]: at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
2015-04-27T13:49:33.633525+00:00 app[web.1]: at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296)
2015-04-27T13:49:33.633703+00:00 app[web.1]: at java.lang.Thread.run(Thread.java:745)
2015-04-27T13:49:34.926567+00:00 heroku[web.1]: Process exited with status 255
2015-04-27T13:49:34.945535+00:00 heroku[web.1]: State changed from starting to crashed
I get "Permission denied" exception. Is there a way to start the netty server in heroku?
I had to let heroky specify the port with
web: target/universal/stage/bin/mmbu-timesheets -Dhttp.port=$PORT
in Procfile
I have an Akka thread in my Play Framework application (hosted on Heroku) that gets called every 10 minutes. In this thread I access many models in my database in quick succession. After compiling and starting the application this thread runs fine for a few hours. Eventually (and for reasons unknown) the logger returns this:
c.j.b.PoolWatchThread - Error in trying to obtain a connection
My DB configuration (Heroku's PostgreSQL Basic plan) looks like this:
db.default.user=USER
db.default.password=PASS
db.default.jndiName=DefaultDS
db.default.driver=org.postgresql.Driver
db.default.url=URL
db.default.partitionCount=2
db.default.maxConnectionsPerPartition=5
db.default.minConnectionsPerPartition=5
db.default.acquireIncrement=1
db.default.acquireRetryAttempts=1
db.default.acquireRetryDelay=5 seconds
db.default.maxConnectionAge=1 hour
db.default.idleMaxAge=10 minute
I have adjusted these variables without solving the problem.
Here is the stack trace
2013-08-11T21:53:29.304108+00:00 app[web.2]: [error] c.j.b.PoolWatchThread - Error in trying to obtain a connection. Retrying in 5000ms
2013-08-11T21:53:29.304108+00:00 app[web.2]: org.postgresql.util.PSQLException: FATAL: too many connections for role "[DB USER NAME]"
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:469) ~[postgresql-9.1-901.jdbc4.jar:na]
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112) ~[postgresql-9.1-901.jdbc4.jar:na]
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) ~[postgresql-9.1-901.jdbc4.jar:na]
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) ~[postgresql-9.1-901.jdbc4.jar:na]
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) ~[postgresql-9.1-901.jdbc4.jar:na]
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) ~[postgresql-9.1-901.jdbc4.jar:na]
I have a hunch it could be something to do with how and when I query the DB for the models. Like, maybe the connections are not released? Help is appreciated.
Edit:
More from the logger
2013-08-13T14:43:49.257852+00:00 app[web.1]: javax.persistence.PersistenceException: java.sql.SQLException: Timed out waiting for a free available connection.
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:368)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.server.core.DefaultServer.createQueryTransaction(DefaultServer.java:2060)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.server.core.OrmQueryRequest.initTransIfRequired(OrmQueryRequest.java:181)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1501)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:904)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.util.DefaultExpressionList.findList(DefaultExpressionList.java:177)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at models.Person.findByHandle(Person.java:85)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at models.NapalmJob.run(NapalmJob.java:41)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at models.NapalmActor.onReceive(NapalmActor.java:9)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:159)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.actor.ActorCell.receiveMessage(ActorCell.scala:425)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.actor.ActorCell.invoke(ActorCell.scala:386)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:230)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.dispatch.Mailbox.run(Mailbox.scala:212)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:502)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
2013-08-13T14:43:49.258200+00:00 app[web.1]: at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
2013-08-13T14:43:49.258200+00:00 app[web.1]: Caused by: java.sql.SQLException: Timed out waiting for a free available connection.
2013-08-13T14:43:49.258200+00:00 app[web.1]: at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:503)
2013-08-13T14:43:49.258200+00:00 app[web.1]: at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:114)
2013-08-13T14:43:49.258200+00:00 app[web.1]: at play.db.ebean.EbeanPlugin$WrappingDatasource.getConnection(EbeanPlugin.java:146)
2013-08-13T14:43:49.258200+00:00 app[web.1]: at com.avaje.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:338)
2013-08-13T14:43:49.258200+00:00 app[web.1]: ... 18 more
Queries are being formed but are not executed... Could some queries be taking longer to finish then others and thus not releasing the resources?
This answer may help:
Heroku POSTGRESQL - "Too many connections for role" error
Also, are you overriding Play's default settings?
I tried pushing some evolutions to Heroku:
2012-08-30T10:58:00+00:00 heroku[slugc]: Slug compilation finished
2012-08-30T10:58:02+00:00 heroku[web.1]: Starting process with command `target/start -Dhttp.port=32436 -Xmx384m -Xss512k
-XX:+UseCompressedOops`
2012-08-30T10:58:03+00:00 app[web.1]: Play server process ID is 2
2012-08-30T10:58:05+00:00 app[web.1]: [←[37minfo←[0m] play - database [default] connected at [Database-address]
2012-08-30T10:58:05+00:00 app[web.1]: CREATE TABLE `unapprovedteaminfo` (
---
Copy of 1.sql
---
2012-08-30T10:58:05+00:00 app[web.1]: VALUES (1, 2, 2, '2012-01-01 00:00:00', 'k');
2012-08-30T10:58:05+00:00 app[web.1]: ERROR: syntax error at or near "`"
2012-08-30T10:58:05+00:00 app[web.1]: Position: 14 [ERROR:0, SQLSTATE:42601]
2012-08-30T10:58:05+00:00 app[web.1]: Oops, cannot start the server.
2012-08-30T10:58:05+00:00 app[web.1]: PlayException: Database 'default' is in inconsistent state! [An evolution has not
been applied properly. Please check the problem and resolve it manually before marking it as resolved.]
2012-08-30T10:58:05+00:00 app[web.1]: at play.api.db.evolutions.Evolutions$.checkEvolutionsState(Evolutions.scala:155)
2012-08-30T10:58:05+00:00 app[web.1]: at play.api.db.evolutions.Evolutions$.databaseEvolutions(Evolutions.scala:308)
2012-08-30T10:58:05+00:00 app[web.1]: at play.api.db.evolutions.Evolutions$.evolutionScript(Evolutions.scala:284)
2012-08-30T10:58:05+00:00 app[web.1]: at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1.apply(Evolutions.s
cala:412)
2012-08-30T10:58:05+00:00 app[web.1]: at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1.apply(Evolutions.s
cala:410)
2012-08-30T10:58:05+00:00 app[web.1]: at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59
)
2012-08-30T10:58:05+00:00 app[web.1]: at scala.collection.immutable.List.foreach(List.scala:45)
2012-08-30T10:58:05+00:00 app[web.1]: at play.api.db.evolutions.EvolutionsPlugin.onStart(Evolutions.scala:410)
2012-08-30T10:58:05+00:00 app[web.1]: at play.api.Play$$anonfun$start$1.apply(Play.scala:60)
2012-08-30T10:58:05+00:00 app[web.1]: at play.api.Play$$anonfun$start$1.apply(Play.scala:60)
2012-08-30T10:58:05+00:00 app[web.1]: at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59
)
2012-08-30T10:58:05+00:00 app[web.1]: at scala.collection.immutable.List.foreach(List.scala:45)
2012-08-30T10:58:05+00:00 app[web.1]: at play.api.Play$.start(Play.scala:60)
2012-08-30T10:58:05+00:00 app[web.1]: at play.core.StaticApplication.<init>(ApplicationProvider.scala:51)
2012-08-30T10:58:05+00:00 app[web.1]: at play.core.server.NettyServer$.createServer(NettyServer.scala:132)
2012-08-30T10:58:05+00:00 app[web.1]: at play.core.server.NettyServer$$anonfun$main$5.apply(NettyServer.scala:153)
2012-08-30T10:58:05+00:00 app[web.1]: at play.core.server.NettyServer$$anonfun$main$5.apply(NettyServer.scala:152)
2012-08-30T10:58:05+00:00 app[web.1]: at scala.Option.map(Option.scala:133)
2012-08-30T10:58:05+00:00 app[web.1]: at play.core.server.NettyServer$.main(NettyServer.scala:152)
2012-08-30T10:58:05+00:00 app[web.1]: at play.core.server.NettyServer.main(NettyServer.scala)
2012-08-30T10:58:06+00:00 heroku[web.1]: Process exited with status 255
2012-08-30T10:58:06+00:00 heroku[web.1]: State changed from starting to crashed
These evolutions would work just fine with H2, I guess there must be some issues with POSTGRESQL syntax. Anyway, the problem I am having now is that, even if I change 1.sql and push the new version on Heroku, the same error will show up in the logs - the copied 1.sql does not change according to the new pushed version.
If you're sending to Heroku 1.sql file generated for H2, there is big chance that they will conflict, instead, set the Heroku's url in application.conf as db.default.url and run it in dev mode on localhost first to generate valid Postres SQL. Also you will need to comment out the db.default.user and db.default.password settings, as Heroku's URL contains this information already.
If you don't know current connection's URL you will find it on:
https://postgres.heroku.com/ -> YourDBs -> db-name -> Connection settings -> JDBC URL
To connect from localhost (or other machine, not in the Heroku's space) you need to add this param to the URL:
&ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory
Thanks to this approach you can test your application locally with established connection to the remote Heroku's DB, (don't be surprised - that combination will be much slower than working with local app + local DB or Heroku app + Heroku DB) anyway for testing it's better than repeatedly pushing small fixes to the cloud.
Also you'll be able to perform evolutions from localhost - without pushing the code to Heroku, and it's really shorter approach if you have problems in your SQL.
Additionally it's good occasion to turn on SQL logging in local dev mode and hunt for redundant SQL queries, if you'll remove all not necessary queries and will be satisfied on localhost, after deploying to the Heroku, you will be just much more satisfied :)
Reset
If you'll create proper 1.sql file than most probably you'll need to reset your DB containing invalid structure (as I assume that's not a problem yet as you just moving from local to Heroku). First find a name of the DB for the current app (with bash, in app's folder):
heroku pg
And then reset it with (it will destroy all data so consider backup first if you don't want to loose your data!):
heroku pg:reset HEROKU_DATABASE_SOMEBASE
I think you have some problem with the definition in VALUES with the data time field. I posted my working evolutions config setup for play 2.4, Slick 3.0.3 & postgres. I got automatically generated the *.sql files placed in the evolution folder.
Regenerate evolution scripts in play 2
Cheers