I have deployed a play scala/java web app on heroku that uses an embeded netty server. The Procfile command is web: target/universal/stage/bin/mmbu-timesheets -Dhttp.port=80
Taken from heroku logs:
2015-04-27T13:49:27.160198+00:00 heroku[web.1]: Starting process with command `target/universal/stage/bin/mmbu-timesheets -Dhttp.port=80`
2015-04-27T13:49:29.321552+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx384m -Xss512k -Dfile.encoding=UTF-8 -Djava.rmi.server.useCodebaseOnly=true
2015-04-27T13:49:29.898379+00:00 app[web.1]: Play server process ID is 3
2015-04-27T13:49:32.549840+00:00 app[web.1]: [[37minfo[0m] application - mongodb connection ds031701.mongolab.com:31701 db->heroku_app36286493
2015-04-27T13:49:33.374954+00:00 app[web.1]: [[37minfo[0m] play - Application started (Prod)
2015-04-27T13:49:33.628067+00:00 app[web.1]: Oops, cannot start the server.
2015-04-27T13:49:33.630568+00:00 app[web.1]: at play.core.server.NettyServer$$anonfun$8.apply(NettyServer.scala:89)
2015-04-27T13:49:33.630523+00:00 app[web.1]: at play.core.server.NettyServer$$anonfun$8.apply(NettyServer.scala:92)
2015-04-27T13:49:33.630859+00:00 app[web.1]: at play.core.server.NettyServer$.createServer(NettyServer.scala:206)
2015-04-27T13:49:33.630358+00:00 app[web.1]: org.jboss.netty.channel.ChannelException: Failed to bind to: /0.0.0.0:80
2015-04-27T13:49:33.630483+00:00 app[web.1]: at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
2015-04-27T13:49:33.630606+00:00 app[web.1]: at scala.Option.map(Option.scala:146)
2015-04-27T13:49:33.630802+00:00 app[web.1]: at play.core.server.NettyServer.<init>(NettyServer.scala:89)
2015-04-27T13:49:33.630898+00:00 app[web.1]: at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:243)
2015-04-27T13:49:33.630941+00:00 app[web.1]: at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:238)
2015-04-27T13:49:33.631096+00:00 app[web.1]: at play.core.server.NettyServer$.main(NettyServer.scala:238)
2015-04-27T13:49:33.630975+00:00 app[web.1]: at scala.Option.map(Option.scala:146)
2015-04-27T13:49:33.631150+00:00 app[web.1]: at play.core.server.NettyServer.main(NettyServer.scala)
2015-04-27T13:49:33.633017+00:00 app[web.1]: Caused by: java.net.SocketException: Permission denied
2015-04-27T13:49:33.633065+00:00 app[web.1]: at sun.nio.ch.Net.bind0(Native Method)
2015-04-27T13:49:33.633136+00:00 app[web.1]: at sun.nio.ch.Net.bind(Net.java:437)
2015-04-27T13:49:33.633179+00:00 app[web.1]: at sun.nio.ch.Net.bind(Net.java:429)
2015-04-27T13:49:33.633235+00:00 app[web.1]: at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
2015-04-27T13:49:33.633396+00:00 app[web.1]: at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
2015-04-27T13:49:33.633444+00:00 app[web.1]: at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372)
2015-04-27T13:49:33.633550+00:00 app[web.1]: at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
2015-04-27T13:49:33.633600+00:00 app[web.1]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2015-04-27T13:49:33.633643+00:00 app[web.1]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2015-04-27T13:49:33.633305+00:00 app[web.1]: at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
2015-04-27T13:49:33.633525+00:00 app[web.1]: at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296)
2015-04-27T13:49:33.633703+00:00 app[web.1]: at java.lang.Thread.run(Thread.java:745)
2015-04-27T13:49:34.926567+00:00 heroku[web.1]: Process exited with status 255
2015-04-27T13:49:34.945535+00:00 heroku[web.1]: State changed from starting to crashed
I get "Permission denied" exception. Is there a way to start the netty server in heroku?
I had to let heroky specify the port with
web: target/universal/stage/bin/mmbu-timesheets -Dhttp.port=$PORT
in Procfile
Related
I’m upgrading to latest version with heroku deployed app using the binary from https://github.com/metabase/metabase-deploy per the instructions, but am receiving ERROR: permission denied for relation databasechangelog. Tailed logs are below
2018-04-13T19:34:59.959056+00:00 heroku[web.1]: Starting process with command `HEROKU=true ./bin/start`
2018-04-13T19:35:01.482994+00:00 app[web.1]: Setting JAVA_TOOL_OPTIONS defaults based on dyno size. Custom settings will override them.
2018-04-13T19:35:01.486299+00:00 app[web.1]: Using these JAVA_OPTS: -Xmx300m -Xss512k -Dfile.encoding=UTF-8 -Xmx248m -XX:-UseGCOverheadLimit -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -XX:+UseCompressedClassPointers -XX:+IgnoreUnrecognizedVMOptions -Djava.awt.headless=true -Dfile.encoding=UTF-8 --add-opens=java.base/java.net=ALL-UNNAMED --add-modules=java.xml.bind
2018-04-13T19:35:01.539500+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx300m -Xss512k -Dfile.encoding=UTF-8
2018-04-13T19:35:07.709376+00:00 app[web.1]: 04-13 19:35:07 INFO metabase.util :: Loading Metabase...
2018-04-13T19:35:15.571783+00:00 app[web.1]: 04-13 19:35:15 INFO util.encryption :: DB details encryption is DISABLED for this Metabase instance. 🔓
2018-04-13T19:35:15.571795+00:00 app[web.1]: See http://www.metabase.com/docs/latest/operations-guide/start.html#encrypting-your-database-connection-details-at-rest for more information.
2018-04-13T19:35:23.403505+00:00 app[web.1]: 04-13 19:35:23 INFO metabase.core :: Starting Metabase in STANDALONE mode
2018-04-13T19:35:23.479490+00:00 app[web.1]: 04-13 19:35:23 INFO metabase.core :: Launching Embedded Jetty Webserver with config:
2018-04-13T19:35:23.479494+00:00 app[web.1]: {:port 22224}
2018-04-13T19:35:23.479496+00:00 app[web.1]:
2018-04-13T19:35:23.547747+00:00 app[web.1]: 04-13 19:35:23 INFO metabase.core :: Starting Metabase version v0.28.6 (83e3c0d release-0.28.6) ...
2018-04-13T19:35:23.548089+00:00 app[web.1]: 04-13 19:35:23 INFO metabase.core :: System timezone is 'Etc/UTC' ...
2018-04-13T19:35:24.159004+00:00 heroku[web.1]: State changed from starting to up
2018-04-13T19:35:27.859258+00:00 app[web.1]: 04-13 19:35:27 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
2018-04-13T19:35:27.861664+00:00 app[web.1]: 04-13 19:35:27 INFO metabase.db :: Verifying postgres Database Connection ...
2018-04-13T19:35:28.044132+00:00 app[web.1]: 04-13 19:35:28 INFO metabase.db :: Verify Database Connection ... ✅
2018-04-13T19:35:28.044416+00:00 app[web.1]: 04-13 19:35:28 INFO metabase.db :: Running Database Migrations...
2018-04-13T19:35:28.079543+00:00 app[web.1]: 04-13 19:35:28 INFO metabase.db :: Setting up Liquibase...
2018-04-13T19:35:28.201705+00:00 app[web.1]: 04-13 19:35:28 INFO metabase.db :: Liquibase is ready.
2018-04-13T19:35:28.201963+00:00 app[web.1]: 04-13 19:35:28 INFO metabase.db :: Checking if Database has unrun migrations...
2018-04-13T19:35:44.048862+00:00 app[web.1]: 04-13 19:35:44 INFO metabase.db :: Database has unrun migrations. Waiting for migration lock to be cleared...
2018-04-13T19:35:44.052917+00:00 app[web.1]: 04-13 19:35:44 INFO metabase.db :: Migration lock is cleared. Running migrations...
2018-04-13T19:35:57.677299+00:00 app[web.1]: org.postgresql.util.PSQLException: ERROR: permission denied for relation databasechangelog
2018-04-13T19:35:57.677543+00:00 app[web.1]: at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2477)
2018-04-13T19:35:57.677591+00:00 app[web.1]: at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2190)
2018-04-13T19:35:57.677695+00:00 app[web.1]: at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:300)
2018-04-13T19:35:57.677725+00:00 app[web.1]: at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428)
2018-04-13T19:35:57.677754+00:00 app[web.1]: at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)
2018-04-13T19:35:57.677790+00:00 app[web.1]: at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169)
2018-04-13T19:35:57.677822+00:00 app[web.1]: at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:136)
2018-04-13T19:35:57.677852+00:00 app[web.1]: at clojure.java.jdbc$db_do_execute_prepared_statement$fn__1839.invoke(jdbc.clj:952)
2018-04-13T19:35:57.677883+00:00 app[web.1]: at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:784)
2018-04-13T19:35:57.678002+00:00 app[web.1]: at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:711)
2018-04-13T19:35:57.678030+00:00 app[web.1]: at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:724)
2018-04-13T19:35:57.678069+00:00 app[web.1]: at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:711)
2018-04-13T19:35:57.678097+00:00 app[web.1]: at clojure.java.jdbc$db_do_execute_prepared_statement.invokeStatic(jdbc.clj:951)
2018-04-13T19:35:57.678134+00:00 app[web.1]: at clojure.java.jdbc$db_do_execute_prepared_statement.invoke(jdbc.clj:945)
2018-04-13T19:35:57.678163+00:00 app[web.1]: at clojure.java.jdbc$db_do_prepared.invokeStatic(jdbc.clj:983)
2018-04-13T19:35:57.678192+00:00 app[web.1]: at clojure.java.jdbc$db_do_prepared.invoke(jdbc.clj:963)
2018-04-13T19:35:57.678223+00:00 app[web.1]: at clojure.java.jdbc$execute_BANG_$execute_helper__1910.invoke(jdbc.clj:1364)
2018-04-13T19:35:57.678253+00:00 app[web.1]: at clojure.java.jdbc$execute_BANG_.invokeStatic(jdbc.clj:1366)
2018-04-13T19:35:57.678331+00:00 app[web.1]: at clojure.java.jdbc$execute_BANG_.invoke(jdbc.clj:1337)
2018-04-13T19:35:57.678368+00:00 app[web.1]: at clojure.java.jdbc$execute_BANG_.invokeStatic(jdbc.clj:1356)
2018-04-13T19:35:57.678402+00:00 app[web.1]: at clojure.java.jdbc$execute_BANG_.invoke(jdbc.clj:1337)
2018-04-13T19:35:57.678443+00:00 app[web.1]: at metabase.db$migrate_up_if_needed_BANG_.invokeStatic(db.clj:169)
2018-04-13T19:35:57.678473+00:00 app[web.1]: at metabase.db$migrate_up_if_needed_BANG_.invoke(db.clj:155)
2018-04-13T19:35:57.678506+00:00 app[web.1]: at metabase.db$migrate_BANG_$fn__4414.invoke(db.clj:258)
2018-04-13T19:35:57.678537+00:00 app[web.1]: at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:741)
2018-04-13T19:35:57.678576+00:00 app[web.1]: at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:711)
2018-04-13T19:35:57.678604+00:00 app[web.1]: at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:776)
2018-04-13T19:35:57.678628+00:00 app[web.1]: at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:711)
2018-04-13T19:35:57.678662+00:00 app[web.1]: at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:724)
2018-04-13T19:35:57.678692+00:00 app[web.1]: at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:711)
2018-04-13T19:35:57.678734+00:00 app[web.1]: at metabase.db$migrate_BANG_.invokeStatic(db.clj:246)
2018-04-13T19:35:57.678763+00:00 app[web.1]: at metabase.db$migrate_BANG_.invoke(db.clj:227)
2018-04-13T19:35:57.678794+00:00 app[web.1]: at metabase.db$run_schema_migrations_BANG_.invokeStatic(db.clj:381)
2018-04-13T19:35:57.678826+00:00 app[web.1]: at metabase.db$run_schema_migrations_BANG_.invoke(db.clj:376)
2018-04-13T19:35:57.678859+00:00 app[web.1]: at metabase.db$setup_db_BANG_.invokeStatic(db.clj:399)
2018-04-13T19:35:57.678889+00:00 app[web.1]: at metabase.db$setup_db_BANG_.doInvoke(db.clj:392)
2018-04-13T19:35:57.678924+00:00 app[web.1]: at clojure.lang.RestFn.invoke(RestFn.java:421)
2018-04-13T19:35:57.679001+00:00 app[web.1]: at metabase.core$init_BANG_.invokeStatic(core.clj:154)
2018-04-13T19:35:57.679031+00:00 app[web.1]: at metabase.core$init_BANG_.invoke(core.clj:133)
2018-04-13T19:35:57.679059+00:00 app[web.1]: at metabase.core$start_normally.invokeStatic(core.clj:258)
2018-04-13T19:35:57.679088+00:00 app[web.1]: at metabase.core$start_normally.invoke(core.clj:251)
2018-04-13T19:35:57.679118+00:00 app[web.1]: at metabase.core$_main.invokeStatic(core.clj:279)
2018-04-13T19:35:57.679293+00:00 app[web.1]: at metabase.core$_main.doInvoke(core.clj:274)
2018-04-13T19:35:57.679314+00:00 app[web.1]: at clojure.lang.RestFn.invoke(RestFn.java:397)
2018-04-13T19:35:57.679349+00:00 app[web.1]: at clojure.lang.AFn.applyToHelper(AFn.java:152)
2018-04-13T19:35:57.679378+00:00 app[web.1]: at clojure.lang.RestFn.applyTo(RestFn.java:132)
2018-04-13T19:35:57.679413+00:00 app[web.1]: at metabase.core.main(Unknown Source)
2018-04-13T19:35:57.679817+00:00 app[web.1]: 04-13 19:35:57 ERROR metabase.core :: Metabase Initialization FAILED: ERROR: permission denied for relation databasechangelog
2018-04-13T19:35:57.683116+00:00 app[web.1]: 04-13 19:35:57 INFO metabase.core :: Metabase Shutting Down ...
2018-04-13T19:35:57.683752+00:00 app[web.1]: 04-13 19:35:57 INFO metabase.core :: Metabase Shutdown COMPLETE
2018-04-13T19:35:58.184227+00:00 heroku[web.1]: State changed from up to crashed
2018-04-13T19:35:58.075189+00:00 heroku[web.1]: Process exited with status 1
It’s the same issue as http://discourse.metabase.com/t/migration-to-new-server-and-docker-instance/3139 but it’s deployed using Heroku, so I can’t add databasechangelog as a role. Migrating questions and dashboards manually isn’t really an option. Thoughts?
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO databasechangelog;
> ERROR: role "databasechangelog" does not exist.
CREATE ROLE databasechangelog;
> ERROR: permission denied to create role
The error is
ERROR: permission denied for relation databasechangelog
So I think you want to do:
grant all privileges on databasechangelog to <username-metabase-is-using-to-connect>
I have a deployed Springboot server on heroku that runs like its supposed to except mlab will not create a connection. The weird thing is that it worked one time, but I had to restart the server, and it never connected again. I recreated the issue locally by trying to start the server without starting mongodb. The error I get is
2017-05-24T13:10:56.522858+00:00 app[web.1]:
2017-05-24T13:10:56.522859+00:00 app[web.1]: com.mongodb.MongoSocketOpenException: Exception opening socket
2017-05-24T13:10:56.522860+00:00 app[web.1]: at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[mongodb-driver-core-3.4.2.jar!/:na]
2017-05-24T13:10:56.522861+00:00 app[web.1]: at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115) ~[mongodb-driver-core-3.4.2.jar!/:na]
2017-05-24T13:10:56.522862+00:00 app[web.1]: at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:113) ~[mongodb-driver-core-3.4.2.jar!/:na]
2017-05-24T13:10:56.522863+00:00 app[web.1]: at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121-heroku16]
2017-05-24T13:10:56.522863+00:00 app[web.1]: Caused by: java.net.ConnectException: Connection refused (Connection refused)
2017-05-24T13:10:56.522865+00:00 app[web.1]: at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_121-heroku16]
2017-05-24T13:10:56.522865+00:00 app[web.1]: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_121-heroku16]
2017-05-24T13:10:56.522866+00:00 app[web.1]: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_121-heroku16]
2017-05-24T13:10:56.522867+00:00 app[web.1]: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_121-heroku16]
2017-05-24T13:10:56.522867+00:00 app[web.1]: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_121-heroku16]
2017-05-24T13:10:56.522868+00:00 app[web.1]: at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_121-heroku16]
2017-05-24T13:10:56.522869+00:00 app[web.1]: at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:57) ~[mongodb-driver-core-3.4.2.jar!/:na]
2017-05-24T13:10:56.522869+00:00 app[web.1]: at com.mongodb.connection.SocketStream.open(SocketStream.java:58) ~[mongodb-driver-core-3.4.2.jar!/:na]
2017-05-24T13:10:56.522871+00:00 app[web.1]: ... 3 common frames omitted
I am using spring.data.mongodb.uri=${MONGODB_URI} to pull the mongo uri, but I am having no luck at all. I've been trying to get this to connect for two days not with only one successful connection for some weird reason. Any recommendations would be greatly appreciated.
I keep getting errors like this...
app[web.1]: Configuration error: Configuration error[Cannot connect to database [default]]
app[web.1]: at scala.collection.immutable.List.foreach(List.scala:381)
app[web.1]: Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "lcnjgrwbmbffwv"
app[web.1]: at play.core.StaticApplication.<init>(ApplicationProvider.scala:55)
app[web.1]: at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:408)
app[web.1]: at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
app[web.1]: at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
app[web.1]: at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
app[web.1]: at org.postgresql.Driver.makeConnection(Driver.java:410)
app[web.1]: at java.sql.DriverManager.getConnection(DriverManager.java:664)
app[web.1]: at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:363)
app[web.1]: at play.api.db.BoneCPPlugin$$anonfun$onStart$1.apply(DB.scala:240)
app[web.1]: at java.sql.DriverManager.getConnection(DriverManager.java:247)
app[web.1]: at org.postgresql.Driver.connect(Driver.java:280)
app[web.1]: at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
app[web.1]: at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:120)
app[web.1]: at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
app[web.1]: ... 18 more
heroku[web.1]: Process exited with status 255
Not just in Heroku, but when I run my Play app with a user whose name is different than the default user name in the database.
My Play Framework "application.conf" is like this...
db.default.username="nameOfDatabaseUser"
db.default.password="passOfDatabaseUser"
db.default.driver=org.postgresql.Driver
db.default.url="jdbc:postgresql://urlofdatabase:5432/name_of_database"
Here's the thing. When my terminal user is named "nameOfDatabaseUser", it works. But when my terminal user has a different name, it connects to the database as the name of the current Linux terminal user.
In the above case it appears that Heroku tried to connect as a user lcnjgrwbmbffwv and there was no user lcnjgrwbmbffwv in the database.
How do I specify the default database username so that it doesn't use my terminal username?
The correct key to specify the user is db.default.user, not db.default.username.
I have an Akka thread in my Play Framework application (hosted on Heroku) that gets called every 10 minutes. In this thread I access many models in my database in quick succession. After compiling and starting the application this thread runs fine for a few hours. Eventually (and for reasons unknown) the logger returns this:
c.j.b.PoolWatchThread - Error in trying to obtain a connection
My DB configuration (Heroku's PostgreSQL Basic plan) looks like this:
db.default.user=USER
db.default.password=PASS
db.default.jndiName=DefaultDS
db.default.driver=org.postgresql.Driver
db.default.url=URL
db.default.partitionCount=2
db.default.maxConnectionsPerPartition=5
db.default.minConnectionsPerPartition=5
db.default.acquireIncrement=1
db.default.acquireRetryAttempts=1
db.default.acquireRetryDelay=5 seconds
db.default.maxConnectionAge=1 hour
db.default.idleMaxAge=10 minute
I have adjusted these variables without solving the problem.
Here is the stack trace
2013-08-11T21:53:29.304108+00:00 app[web.2]: [error] c.j.b.PoolWatchThread - Error in trying to obtain a connection. Retrying in 5000ms
2013-08-11T21:53:29.304108+00:00 app[web.2]: org.postgresql.util.PSQLException: FATAL: too many connections for role "[DB USER NAME]"
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:469) ~[postgresql-9.1-901.jdbc4.jar:na]
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112) ~[postgresql-9.1-901.jdbc4.jar:na]
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) ~[postgresql-9.1-901.jdbc4.jar:na]
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) ~[postgresql-9.1-901.jdbc4.jar:na]
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) ~[postgresql-9.1-901.jdbc4.jar:na]
2013-08-11T21:53:29.304108+00:00 app[web.2]: at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) ~[postgresql-9.1-901.jdbc4.jar:na]
I have a hunch it could be something to do with how and when I query the DB for the models. Like, maybe the connections are not released? Help is appreciated.
Edit:
More from the logger
2013-08-13T14:43:49.257852+00:00 app[web.1]: javax.persistence.PersistenceException: java.sql.SQLException: Timed out waiting for a free available connection.
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:368)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.server.core.DefaultServer.createQueryTransaction(DefaultServer.java:2060)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.server.core.OrmQueryRequest.initTransIfRequired(OrmQueryRequest.java:181)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1501)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:904)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at com.avaje.ebeaninternal.util.DefaultExpressionList.findList(DefaultExpressionList.java:177)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at models.Person.findByHandle(Person.java:85)
2013-08-13T14:43:49.257852+00:00 app[web.1]: at models.NapalmJob.run(NapalmJob.java:41)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at models.NapalmActor.onReceive(NapalmActor.java:9)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:159)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.actor.ActorCell.receiveMessage(ActorCell.scala:425)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.actor.ActorCell.invoke(ActorCell.scala:386)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:230)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.dispatch.Mailbox.run(Mailbox.scala:212)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:502)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
2013-08-13T14:43:49.258049+00:00 app[web.1]: at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
2013-08-13T14:43:49.258200+00:00 app[web.1]: at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
2013-08-13T14:43:49.258200+00:00 app[web.1]: Caused by: java.sql.SQLException: Timed out waiting for a free available connection.
2013-08-13T14:43:49.258200+00:00 app[web.1]: at com.jolbox.bonecp.BoneCP.getConnection(BoneCP.java:503)
2013-08-13T14:43:49.258200+00:00 app[web.1]: at com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:114)
2013-08-13T14:43:49.258200+00:00 app[web.1]: at play.db.ebean.EbeanPlugin$WrappingDatasource.getConnection(EbeanPlugin.java:146)
2013-08-13T14:43:49.258200+00:00 app[web.1]: at com.avaje.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:338)
2013-08-13T14:43:49.258200+00:00 app[web.1]: ... 18 more
Queries are being formed but are not executed... Could some queries be taking longer to finish then others and thus not releasing the resources?
This answer may help:
Heroku POSTGRESQL - "Too many connections for role" error
Also, are you overriding Play's default settings?
I just tried to deployed a play 1.2.4 app to heroku.
The App works great locally, but on heroku i just got some strange errors, as soon as i try to save some data.
2012-07-02T21:26:04+00:00 app[web.1]: Internal Server Error (500) for request POST /players/save
2012-07-02T21:26:04+00:00 app[web.1]:
2012-07-02T21:26:04+00:00 app[web.1]: Oops: UnexpectedException
2012-07-02T21:26:04+00:00 app[web.1]: An unexpected error occured caused by exception UnexpectedException: Unexpected Error
2012-07-02T21:26:04+00:00 app[web.1]:
2012-07-02T21:26:04+00:00 app[web.1]: play.exceptions.UnexpectedException: Unexpected Error
2012-07-02T21:26:04+00:00 app[web.1]: at play.data.validation.ValidationPlugin.beforeActionInvocation(ValidationPlugin.java:59)
2012-07-02T21:26:04+00:00 app[web.1]: at play.plugins.PluginCollection.beforeActionInvocation(PluginCollection.java:594)
2012-07-02T21:26:04+00:00 app[web.1]: at play.mvc.ActionInvoker.invoke(ActionInvoker.java:134)
2012-07-02T21:26:04+00:00 app[web.1]: at Invocation.HTTP Request(Play!)
2012-07-02T21:26:04+00:00 app[web.1]: Caused by: play.exceptions.UnexpectedException: Unexpected Error
2012-07-02T21:26:04+00:00 app[web.1]: at play.db.jpa.JPAPlugin.bind(JPAPlugin.java:84)
2012-07-02T21:26:04+00:00 app[web.1]: at play.plugins.PluginCollection.bind(PluginCollection.java:534)
2012-07-02T21:26:04+00:00 app[web.1]: at play.data.binding.Binder.bind(Binder.java:111)
2012-07-02T21:26:04+00:00 app[web.1]: at play.mvc.ActionInvoker.getActionMethodArgs(ActionInvoker.java:642)
2012-07-02T21:26:04+00:00 app[web.1]: at play.data.validation.ValidationPlugin$Validator.validateAction(ValidationPlugin.java:96)
2012-07-02T21:26:04+00:00 app[web.1]: at play.data.validation.ValidationPlugin.beforeActionInvocation(ValidationPlugin.java:51)
2012-07-02T21:26:04+00:00 app[web.1]: ... 3 more
2012-07-02T21:26:04+00:00 app[web.1]: Caused by: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not execute query
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1214)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1147)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:307)
2012-07-02T21:26:04+00:00 app[web.1]: at play.db.jpa.JPAPlugin.bind(JPAPlugin.java:79)
2012-07-02T21:26:04+00:00 app[web.1]: ... 8 more
2012-07-02T21:26:04+00:00 app[web.1]: Caused by: org.hibernate.exception.SQLGrammarException: could not execute query
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:92)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.loader.Loader.doList(Loader.java:2536)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.loader.Loader.list(Loader.java:2271)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:452)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:363)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:274)
2012-07-02T21:26:04+00:00 app[web.1]: ... 9 more
2012-07-02T21:26:04+00:00 app[web.1]: Caused by: org.postgresql.util.PSQLException: ERROR: operator does not exist: bigint = bytea
2012-07-02T21:26:04+00:00 app[web.1]: Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.
2012-07-02T21:26:04+00:00 app[web.1]: Position: 130
2012-07-02T21:26:04+00:00 app[web.1]: at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
2012-07-02T21:26:04+00:00 app[web.1]: at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
2012-07-02T21:26:04+00:00 app[web.1]: at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
2012-07-02T21:26:04+00:00 app[web.1]: at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
2012-07-02T21:26:04+00:00 app[web.1]: at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
2012-07-02T21:26:04+00:00 app[web.1]: at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.loader.Loader.getResultSet(Loader.java:1953)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.loader.Loader.doQuery(Loader.java:802)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
2012-07-02T21:26:04+00:00 app[web.1]: at org.hibernate.loader.Loader.doList(Loader.java:2533)
2012-07-02T21:26:04+00:00 app[web.1]: ... 17 more
The App is public accessable here:
https://github.com/phaus/kickster
I also tried to use db evolutions first. It did not work neither.
I found several pages for
Caused by: org.postgresql.util.PSQLException: ERROR: operator does not exist: bigint = bytea
but i looks, like the problem is within the Play Framework?
This is a Play Framework bug that has been fixed in Play 1.2.5.