I'm having the same issue after upgrading to 2.3 on Ubuntu 18. Everything loads, bbb-conf --status all active, no errors on bbb but I keep getting the "404 Not found" error when trying to access the Redlight Rooms or any resource requiring the html5 client application.
After running sudo bbb-conf --debug I get some errors as shown below:
May 25 21:37:11 cloudroom sh[28344]: SignalException: SIGTERM May 25 21:37:14 cloudroom sh[28344]: Original Exception (Redis::CannotConnectError): Error connecting to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED) May 25 21:37:14 cloudroom sh[28344]: /usr/lib/ruby/vendor_ruby/rake/application.rb:186:in standard_exception_handling' May 25 21:45:37 cloudroom sh[29476]: SignalException: SIGTERM May 25 21:45:41 cloudroom sh[29476]: Original Exception (Redis::CannotConnectError): Error connecting to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED) May 25 21:45:41 cloudroom sh[29476]: /usr/lib/ruby/vendor_ruby/rake/application.rb:186:in standard_exception_handling' May 25 22:03:58 cloudroom sh[4189]: SignalException: SIGTERM May 25 22:04:02 cloudroom sh[4189]: Original Exception (Redis::CannotConnectError): Error connecting to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED) May 25 22:04:02 cloudroom sh[4189]: /usr/lib/ruby/vendor_ruby/rake/application.rb:186:in standard_exception_handling' May 25 22:30:07 cloudroom sh[6716]: SignalException: SIGTERM May 25 22:30:08 cloudroom run-prod.sh[6823]: 2021-05-25T22:30:08.231Z WARN i.l.c.protocol.ConnectionWatchdog - Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:6379 May 25 22:30:08 cloudroom run-prod.sh[6823]: 2021-05-25T22:30:08.235Z WARN i.l.c.protocol.ConnectionWatchdog - Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:6379 May 25 22:30:08 cloudroom run-prod.sh[6823]: 2021-05-25T22:30:08.256Z WARN i.l.c.protocol.ConnectionWatchdog - Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:6379 May 25 22:30:08 cloudroom run-prod.sh[6823]: 2021-05-25T22:30:08.257Z WARN i.l.c.protocol.ConnectionWatchdog - Cannot reconnect: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:6379 May 25 22:30:11 cloudroom sh[6716]: Original Exception (Redis::CannotConnectError): Error connecting to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED) May 25 22:30:11 cloudroom sh[6716]: /usr/lib/ruby/vendor_ruby/rake/application.rb:186:in standard_exception_handling'
Any assistance will be greatly appreciated, I've been trying to solce this for the past two days.
Related
I want to run a self managed Thingsboard on AWS (t2.micro).
I have installed Thingsboard CE on a t2.micro AWS instance running Ubuntu 20.04 sever.
I followed the aws setup and Ubuntu install guides(postgresql + built in queue service).
I also set up haproxy using this guide.
I was able to successfully log in to my Thingsboard. I only changed the passwords and checked the basic functionalities, but didn't create any new dashboards or made any modifications.
After this I left the computer on, running Thingsboard. Next day I could not reach Thingsboard and although the AWS instance was running I could not ssh into it anymore. After stopping and starting(reboot didn't work) the instance everything was ok (could ssh and Thingsboard was reachable).
I can reproduce this failure just by leaving the instance on, it seems that after serveral hours (5-8 hrs) Thingsboard(or something else not sure) fails which freezes the whole computer.
I have checked two things:
I checked CPU utiization on AWS monitoring.
It seems that after some hours there is a big jump in CPU load and then it drops back to almost zero. While Thingsboard is running, it is constant.See printscreen from AWS monitoring
I checked the Thingsboard logs (in /var/log/thingsboard):
There are some errors, but unfortunately most of the things are not enough for me guess what could be a problem with the fresh installation. Here are some lines from the log:
2021-11-12 00:21:59,626 [http-nio-0.0.0.0-8080-exec-13] INFO o.a.coyote.http11.Http11Processor - Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name
[0x160x030x010x00{0x010x000x00w0x030x030x170xb80xb80xe50xef0x000xb50x0a&0x930x020x00:0xde0xd70xa00xab0xb
70x8bU0xc00x92r0x9330x10O0x8c<o0xf70xf90x000x000x1a0xc0/0xc0+0xc00x110xc00x070xc00x130xc00x090xc00x140xc00x0a0x000x050x00/0x0050xc00x120x000x0a0x010x000x0040x000x050x000x050x010x000
x000x000x000x000x0a0x000x080x000x060x000x170x000x180x000x190x000x0b0x000x020x010x000x000x0d0x000x100x000x0e0x040x010x040x030x020x010x020x030x040x010x050x010x060x010xff0x010x000x010x00...].
HTTP method names must be tokens
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:417)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:261)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-11-12 00:22:01,486 [sql-queue-2-ts-4-thread-1] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate
connection org.postgresql.jdbc.PgConnection#4393afd0 (This connection
has been closed.). Possibly consider using a shorter maxLifetime
value.
2021-11-12 00:22:01,487 [sql-queue-2-ts latest-8-thread-1] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate
connection org.postgresql.jdbc.PgConnection#75b9496b (This connection
has been closed.). Possibly consider using a shorter maxLifetime
value.
2021-11-12 00:22:01,487 [sql-queue-0-ts latest-6-thread-1] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate
connection org.postgresql.jdbc.PgConnection#31849eec (This connection
has been closed.). Possibly consider using a shorter maxLifetime
value.
2021-11-12 00:22:01,487 [sql-queue-0-ts-2-thread-1] WARN com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Failed to validate
connection org.postgresql.jdbc.PgConnection#725fafe3 (This connection
has been closed.). Possibly consider using a shorter maxLifetime
value.
Some more:
2021-11-12 00:23:46,205 [sql-log-1-thread-1] INFO o.t.s.dao.sql.TbSqlBlockingQueue - Queue-2 [TS Latest] queueSize [9] totalAdded [0] totalSaved [0] totalFailed [0]
2021-11-12 00:23:47,741 [sql-queue-0-ts-2-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 08003
2021-11-12 00:23:47,742 [sql-queue-2-ts-4-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 08003
2021-11-12 00:23:47,742 [sql-queue-2-ts latest-8-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 08003
2021-11-12 00:23:47,742 [sql-queue-0-ts latest-6-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: 08003
2021-11-12 00:23:48,022 [sql-queue-0-ts-2-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - HikariPool-1 - Connection is not available, request timed out after 634223ms.
2021-11-12 00:23:48,058 [sql-queue-0-ts-2-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - This connection has been closed.
2021-11-12 00:23:48,022 [sql-queue-0-ts latest-6-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - HikariPool-1 - Connection is not available, request timed out after 634223ms.
2021-11-12 00:23:48,059 [sql-queue-0-ts latest-6-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - This connection has been closed.
2021-11-12 00:23:48,022 [sql-queue-2-ts latest-8-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - HikariPool-1 - Connection is not available, request timed out after 624177ms.
2021-11-12 00:23:48,059 [sql-queue-2-ts latest-8-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - This connection has been closed.
2021-11-12 00:23:48,023 [sql-queue-2-ts-4-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - HikariPool-1 - Connection is not available, request timed out after 627819ms.
2021-11-12 00:23:48,059 [sql-queue-2-ts-4-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - This connection has been closed.
At the last:
2021-11-12 00:33:10,919 [sql-queue-0-ts latest-6-thread-1] ERROR o.t.s.dao.sql.TbSqlBlockingQueue - [TS Latest] Failed to save 1 entities
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:448)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:574)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:361)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)
at org.thingsboard.server.dao.sqlts.insert.latest.psql.PsqlLatestInsertTsRepository$$EnhancerBySpringCGLIB$$381b448c.saveOrUpdate(<generated>)
at org.thingsboard.server.dao.sqlts.SqlTimeseriesLatestDao.lambda$init$3(SqlTimeseriesLatestDao.java:133)
at org.thingsboard.server.dao.sql.TbSqlBlockingQueue.lambda$init$2(TbSqlBlockingQueue.java:71)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:276)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:284)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:184)
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:402)
... 16 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 634223ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:695)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
... 23 common frames omitted
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:877)
at org.postgresql.jdbc.PgConnection.setNetworkTimeout(PgConnection.java:1610)
at com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(PoolBase.java:560)
at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:173)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
... 28 common frames omitted
What is interesting that the timestaps on the CPU load going to max don't precisely correlate with the error messages in the log.
I apologise for the long error messages, but right now I don't know what could be the root cause.
I haven't tried to reinstall the whole computer yet.
My question would be, how should I proceed? Does anyone have ever faced similar issues? What logs/services/etc. should I check to grasp the root cause?
Should I try using a machine with more resources? Should I try other database and queue service?
In the current form this Thingsboard instance is not stable even for tests.
Edit: Sorry I could not format properly the first part of the error code.
Edit2: First link was wrong.
Here are some points:
It looks like the operating system run out of memory and become unresponsive. To fix the issue try to manage Java heap memory
For 4Gb instance, this Java heap limit JAVA_OPTS="$JAVA_OPTS -Xms1024M -Xmx1024M" may be useful, because Java uses some non-heap memory as well, PostgreSQL and others require some memory to run.
t2 instances on AWS may slow down the whole thing by CPU throttling. Instances like c6 or m5 are better options by performance.
In-memory queues may result in out-of-memory issues and data loss in case of a heavy message rate or some processing congestion due to the third party. Consider using Kafka to make your installation much stable and solid.
After I increased the RAM to 4GB (from 1GB) the ThingsBoard server is up without problem. There are no sporadic freezes anyomore. As there were no other provable suggestions for the problem and now my system works without a problem I consider the question answered.
I am learning Symfony,
I create my project with : symfony new myproject --full
directly after this I run symfony serve
in the console I am getting thoses errors :
Tailing Web Server log file (/Users/ben/.symfony/log/cd52af540b09d661e4ffb4f5029da4bbaf3586a9.log)
Tailing PHP-FPM log file (/Users/ben/.symfony/log/cd52af540b09d661e4ffb4f5029da4bbaf3586a9/53fb8ec204547646acb3461995e4da5a54cc7575.log)
[OK] Web server listening
The Web server is using PHP FPM 7.3.11
https://127.0.0.1:8000
[Web Server ] Jun 3 23:38:48 |DEBUG | PHP Reloading PHP versions
[Web Server ] Jun 3 23:38:48 |DEBUG | PHP Using PHP version 7.3.11 (from default version in $PATH)
[Application] Jun 3 21:29:59 |CRITICA| REQUES Uncaught PHP Exception Doctrine\DBAL\Exception\ConnectionException: "An exception occurred in driver: SQLSTATE[08006] [7] could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?" at /Users/ben/Desktop/symfonytuto/demo/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractPostgreSQLDriver.php line 96
[Application] Jun 3 21:29:59 |CRITICA| REQUES Exception thrown when handling an exception (Doctrine\DBAL\Exception\ConnectionException: An exception occurred in driver: SQLSTATE[08006] [7] could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? at /Users/ben/Desktop/symfonytuto/demo/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractPostgreSQLDriver.php line 96)
[Application] Jun 3 21:29:59 |CRITICA| PHP Uncaught Exception: An exception occurred in driver: SQLSTATE[08006] [7] could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?
[Application] Jun 3 21:37:59 |ERROR | REQUES Uncaught PHP Exception Symfony\Component\HttpKernel\Exception\NotFoundHttpException: "No route found for "GET https://127.0.0.1:8000/"" at /Users/ben/Desktop/symfonytuto/demo/vendor/symfony/http-kernel/EventListener/RouterListener.php line 136
[Application] Jun 3 21:38:00 |ERROR | REQUES Uncaught PHP Exception Symfony\Component\HttpKernel\Exception\NotFoundHttpException: "No route found for "GET https://127.0.0.1:8000/favicon.ico" (from "https://127.0.0.1:8000/")" at /Users/ben/Desktop/symfonytuto/demo/vendor/symfony/http-kernel/EventListener/RouterListener.php line 136
[Application] Jun 3 21:38:00 |CRITICA| REQUES Exception thrown when handling an exception (Doctrine\DBAL\Exception\ConnectionException: An exception occurred in driver: SQLSTATE[08006] [7] could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? at /Users/ben/Desktop/symfonytuto/demo/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractPostgreSQLDriver.php line 96)
[Application] Jun 3 21:38:00 |CRITICA| PHP Uncaught Exception: An exception occurred in driver: SQLSTATE[08006] [7] could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?
[Application] Jun 3 21:38:00 |CRITICA| REQUES Uncaught PHP Exception Doctrine\DBAL\Exception\ConnectionException: "An exception occurred in driver: SQLSTATE[08006] [7] could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?" at /Users/ben/Desktop/symfonytuto/demo/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractPostgreSQLDriver.php line 96
[Application] Jun 3 21:38:00 |CRITICA| REQUES Exception thrown when handling an exception (Doctrine\DBAL\Exception\ConnectionException: An exception occurred in driver: SQLSTATE[08006] [7] could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? at /Users/ben/Desktop/symfonytuto/demo/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractPostgreSQLDriver.php line 96)
[Application] Jun 3 21:38:00 |CRITICA| PHP Uncaught Exception: An exception occurred in driver: SQLSTATE[08006] [7] could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?
[Web Server ] Jun 3 23:38:48 |INFO | PHP listening path="/usr/sbin/php-fpm" php="7.3.11" port=50170
[PHP-FPM ] Jun 3 23:38:48 |NOTICE | FPM fpm is running, pid 1240
[PHP-FPM ] Jun 3 23:38:48 |NOTICE | FPM ready to handle connections
After this the Symfony's default page is loading correctly but when I add a route to the url for example : https://127.0.0.1:8000/testtt
I'm getting this error :
An exception occurred in driver: SQLSTATE[08006] [7] could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
(it should be something like 'this route doesnt exist' no ?)
Please tell me how I can give you more info about my problem !
Thanks a lot !
This question pops up often enough that I think an answer is appropriate until the issue is resolved.
Assume you just created a new Symfony website-skeleton app, added a new controller and started the development server. You navigate to your new route and exceptions are tossed. Either get a 'driver not found' or a 'could connect to server'. Which of course is a bit unexpected since you have not yet done anything with the database.
The problem is that the Doctrine migrations bundle provides a data collector for Symfony's profiler bar that shows up at the bottom of the browser page when running in development mode. The data collector needs to connect to the database in order to collect migrations data. It does not care about the fact that there is no database. It just tries to connect and ends up tossing an exception.
I opened an issue on this. I am still not entirely sure if this is a feature or bug. Feel free to comment on the issue if you like.
This is a fairly recent problem though I have not tracked exactly when it first popped up. Hence many existing tutorials and step by step guides do not mention it. Consider it to be part of your development experience. Learning how to read exceptions is quite important.
There are several solutions. If you plan on using a database then go ahead and create yourself a database and adjust the DATABASE_URL to point to it. The database can be empty. The migration data collector just needs to be able to connect to it. On my local development machine I actually created a database called db_name just to avoid this error message.
A second solution is to just remove the migrations bundle with
composer remove doctrine/doctrine-migrations-bundle
And the problem goes away. If later on you need migrations then just use composer require to re-install it.
I have another solution that solves the problem without having to remove the package with composer remove ....
comment out Doctrine\Bundle\MigrationsBundle\DoctrineMigrationsBundle::class => ['all' => true], in /config/bundles.php
Then move /config/packages/doctrine_migrations.yaml up one directory so it's not longer trying to be included.
This may be a little less intimidating for new developers.
If the issue appears when you want to access the localhost.../lucky/number, go to .env file under the your "folder name" and uncomment this line:
DATABASE_URL="sqlite:///%kernel.project_dir%/var/data.db"
And the rest of them with mySQl and Postgres, comment them.
Hope it works for you. :)
I have a PostGIS database and a client built on top of HikariCP to read the data from a database. My client can read the data without any problem on some machines. However, on some other machines client gets stuck and is not able to read any data throwing socket timeout exceptions.
MyClass:120 - Failed to execute HikariProxyPreparedStatement#2091541230 wrapping <my-query>.
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:332)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:118)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
...
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
at org.postgresql.core.PGStream.receiveChar(PGStream.java:293)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1947)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
... 32 more
ProxyConnection:161 - HikariPool-1 - Connection org.postgresql.jdbc.PgConnection#1aafd32f marked as broken because of SQLSTATE(08006), ErrorCode(0)
org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:332)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:118)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
...
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
at org.postgresql.core.PGStream.receiveChar(PGStream.java:293)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1947)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
... 31 more
Before client throws SocketTimeoutException on the database side, I monitored pg_stat_activity table. The corresponding row for the query above had wait_event_type=Client and wait_event=ClientWrite. In addition, database server logged messages indicating connection is lost.
LOG: unexpected EOF on client connection with an open transaction
LOG: could not send data to client: Connection timed out
FATAL: connection to client lost
Versions
PostGIS-jdbc: 2.2.1 (postgresql jdbc: 9.4.1208.jre7)
HikariCP: 3.1.0
Postgres server: 10.3
PostGIS server: 2.4.4
If I don't set socketTimeout through jdbc connection string, then connection would get stuck forever. Once the connection reaches it's max life time, it would be dropped and connected again. However, it still cannot read the data. When I set socketTimeout, then exception would be thrown.
UPDATE
If socketTimeout is not set, then pg_stat_activity table would have a row for the connection with the following values: state=idle in transaction, wait_event_type=Client and wait_event=ClientRead.
My guess is that some sort of network setting is blocking the read from the server on the client side. How can I further debug this and find the root cause?
We found out that this was caused by the database server's MTU setting. MTU was set to 9000 by default and resulted in packet loss. Changing it to 1500 resolved the issue.
Trying to add a Postgres database on Metabase via SSH tunneling I ran into an error giving me the following error message: "Server error encountered"
My logs are very similar to #williamjacksn 's logs on the post:
http://discourse.metabase.com/t/ssh-tunnel-failure/2469
I am certain that my credentials are OK since I use the same for a Postico SSH connection as well as a DBeaver connection.
I already checked the following fix: https://github.com/metabase/metabase/pull/6970
Howerver I have no idea how to use it as i cannot find the ssh.clj file on my machine.
Would there be a fix for this ?
Extra info:
Install via the .jar file. Working on Firefox 58.0.2 | Mac OSX 10.10.5
Thanks in advance
Log:
03-12 10:16:48 INFO util.ssh :: creating ssh tunnel metabase#192.168.1.2:22 -L 56334:localhost:5432
03-12 10:16:53 ERROR metabase.driver :: Failed to connect to database: Timed out after 5000 milliseconds.
03-12 10:16:53 DEBUG metabase.middleware :: POST /api/setup/validate 400 (5 s) (0 DB calls)
{:errors {:dbname "Timed out after 5000 milliseconds."}}
Mar 12, 2018 10:16:53 AM org.postgresql.Driver connect
SEVERE: Connection error:
org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:275)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:194)
at org.postgresql.Driver.makeConnection(Driver.java:450)
at org.postgresql.Driver.connect(Driver.java:252)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:678)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:190)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:364)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:226)
at clojure.java.jdbc$db_query_with_resultset_STAR_.invokeStatic(jdbc.clj:1014)
at clojure.java.jdbc$db_query_with_resultset_STAR_.invoke(jdbc.clj:996)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1090)
at clojure.java.jdbc$query.invoke(jdbc.clj:1047)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1063)
at clojure.java.jdbc$query.invoke(jdbc.clj:1047)
at metabase.driver.generic_sql$can_connect_QMARK_.invokeStatic(generic_sql.clj:220)
at metabase.driver.generic_sql$can_connect_QMARK_.invoke(generic_sql.clj:217)
at metabase.driver$fn__25577$G__25354__25584.invoke(driver.clj:51)
at metabase.driver$can_connect_with_details_QMARK_$fn__25710.invoke(driver.clj:451)
at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:400)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:243)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:225)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:402)
at java.base/java.net.Socket.connect(Socket.java:591)
at org.postgresql.core.PGStream.(PGStream.java:68)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)
... 24 more
Mar 12, 2018 10:16:58 AM org.postgresql.Driver connect
SEVERE: Connection error:
org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:275)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:194)
at org.postgresql.Driver.makeConnection(Driver.java:450)
at org.postgresql.Driver.connect(Driver.java:252)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:678)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:190)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:364)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:226)
at clojure.java.jdbc$db_query_with_resultset_STAR_.invokeStatic(jdbc.clj:1014)
at clojure.java.jdbc$db_query_with_resultset_STAR_.invoke(jdbc.clj:996)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1090)
at clojure.java.jdbc$query.invoke(jdbc.clj:1047)
at clojure.java.jdbc$query.invokeStatic(jdbc.clj:1063)
at clojure.java.jdbc$query.invoke(jdbc.clj:1047)
at metabase.driver.generic_sql$can_connect_QMARK_.invokeStatic(generic_sql.clj:220)
at metabase.driver.generic_sql$can_connect_QMARK_.invoke(generic_sql.clj:217)
at metabase.driver$fn__25577$G__25354__25584.invoke(driver.clj:51)
at metabase.driver$can_connect_with_details_QMARK_$fn__25710.invoke(driver.clj:451)
at clojure.core$binding_conveyor_fn$fn__4676.invoke(core.clj:1938)
at clojure.lang.AFn.call(AFn.java:18)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:400)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:243)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:225)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:402)
at java.base/java.net.Socket.connect(Socket.java:591)
at org.postgresql.core.PGStream.(PGStream.java:68)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)
... 24 more`
I already checked the following fix: https://github.com/metabase/metabase/pull/6970 Howerver I have no idea how to use it as i cannot find the ssh.clj file on my machine.
To try out the ssh.clj changes proposed in the pull request you need to get the source code from GitHub, then build Metabase yourself from source code. I just happen to have described the steps over on the Metabase discussion forum for someone in a similar situation (though needing a different PR for a different purpose):
http://discourse.metabase.com/t/snowflake-driver-test/3110
when connection using mongo db driver with replica set it is giving warning that connecting to localhost as below mentioned. any idea why ?
Dec 21, 2013 2:47:49 PM com.mongodb.ConnectionStatus$UpdatableNode update
WARNING: Server seen down: localhost/127.0.0.1:10001 - java.io.IOException - message: couldn't connect to [localhost/127.0.0.1:10001] bc:java.net.ConnectException: Connection refused: connect
Dec 21, 2013 2:47:50 PM com.mongodb.ConnectionStatus$UpdatableNode update
WARNING: Server seen down: localhost/127.0.0.1:10003 - java.io.IOException - message: couldn't connect to [localhost/127.0.0.1:10003] bc:java.net.ConnectException: Connection refused: connect
Dec 21, 2013 2:47:52 PM com.mongodb.ConnectionStatus$UpdatableNode update
WARNING: Server seen down: localhost/127.0.0.1:10001 - java.io.IOException - message: couldn't connect to [localhost/127.0.0.1:10001] bc:java.net.ConnectException: Connection refused: connect
Dec 21, 2013 2:47:53 PM com.mongodb.ConnectionStatus$UpdatableNode update
WARNING: Server seen down: localhost/127.0.0.1:10003 - java.io.IOException - message: couldn't connect to [localhost/127.0.0.1:10003] bc:java.net.ConnectException: Connection refused: connect
Dec 21, 2013 2:47:54 PM com.mongodb.ConnectionStatus$UpdatableNode update
WARNING: Server seen down: localhost/127.0.0.1:10002 - java.io.IOException - message: couldn't connect to [localhost/127.0.0.1:10002] bc:java.net.ConnectException: Connection refused: connect
Exception in thread "main" com.mongodb.MongoException: can't find a master
at com.mongodb.DBTCPConnector.checkMaster(DBTCPConnector.java:514)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:174)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:155)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:249)
at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:205)
at com.mongodb.DBCollection.insert(DBCollection.java:57)
at com.mongodb.DBCollection.insert(DBCollection.java:100)
at mongo.Test.main(Test.java:33)