Telemetry data unable to pass through root rule chain with node save timeseries - postgresql

Things Board Version: V3.4.1 CE
OS: Window
Database: postgreSQL timescale
Queue: Rabbitmq
I discover that the telemetry data unable to pass through things board root rule chain with the node of the name save timeseries, i am not sure what is happening, i confirm there should be no problem on the connection between thingsboard and also postgreSQL...
I can see debug from here to know the problem is because failed to save to timeseries data....
2022-11-02 09:04:27,148 [sql-queue-2-ts timescale-11-thread-1] ERROR o.t.s.dao.sql.TbSqlBlockingQueue - [TS Timescale] Failed to save 2 entities
org.springframework.transaction.TransactionSystemException: Could not roll back JPA transaction; nested exception is org.hibernate.TransactionException: Unable to rollback against JDBC Connection
at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:593)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:835)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:809)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:672)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:392)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
at org.thingsboard.server.dao.sqlts.insert.timescale.TimescaleInsertTsRepository$$EnhancerBySpringCGLIB$$693764a7.saveOrUpdate()
at org.thingsboard.server.dao.sqlts.timescale.TimescaleTimeseriesDao.lambda$init$1(TimescaleTimeseriesDao.java:89)
at org.thingsboard.server.dao.sql.TbSqlBlockingQueue.lambda$init$2(TbSqlBlockingQueue.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hibernate.TransactionException: Unable to rollback against JDBC Connection
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:127)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.rollback(JdbcResourceLocalTransactionCoordinatorImpl.java:304)
at org.hibernate.engine.transaction.internal.TransactionImpl.rollback(TransactionImpl.java:142)
at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:589)
... 16 common frames omitted
Caused by: java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:515)
at com.sun.proxy.$Proxy153.rollback(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.rollback(ProxyConnection.java:396)
at com.zaxxer.hikari.pool.HikariProxyConnection.rollback(HikariProxyConnection.java)
at org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.rollback(AbstractLogicalConnectionImplementor.java:121)
... 19 common frames omitted
2022-11-02 09:04:27,148 [tb-rule-engine-consumer-37-thread-35 | QK(Main,TB_RULE_ENGINE,system)-10] INFO o.t.s.s.q.DefaultTbRuleEngineConsumerService - Failed to process 1 messages
2022-11-02 09:04:27,148 [tb-rule-engine-consumer-37-thread-35 | QK(Main,TB_RULE_ENGINE,system)-10] INFO o.t.s.s.q.DefaultTbRuleEngineConsumerService - [c1737420-58eb-11eb-808a-dfdc947dc52b] Failed to process message: TbMsg(queueName=Main, id=1318aa98-0755-49b0-9685-a71a2326ff7d, ts=1667351067141, type=POST_TELEMETRY_REQUEST, originator=354d8300-aa84-11ec-9a47-4727b3504d5d, customerId=d7094170-5c4c-11eb-b06a-c93fc5e45132, metaData=TbMsgMetaData(data={deviceType=Sensor, deviceName=RMS Voltage Sensor, ts=1667351067141}), dataType=JSON, data={"timestamp":1667351069011,"values":[{"id":"CnB Prai Gateway.RMS Shearline.Sensor5_Active","v":true,"t":1667291491472},{"id":"CnB Prai Gateway.RMS Shearline.Sensor5_Battery","v":296,"t":1667342191745},{"id":"CnB Prai Gateway.RMS Shearline.Sensor5_Signal","v":65478,"t":1667350910940},{"id":"CnB Prai Gateway.RMS Shearline.Sensor5_Voltage","v":0,"t":1667351068948}]}, ruleChainId=c1c082b0-58eb-11eb-808a-dfdc947dc52b, ruleNodeId=null, ctx=org.thingsboard.server.common.msg.TbMsgProcessingCtx#4c99aecc, callback=org.thingsboard.server.common.msg.queue.TbMsgCallback$1#415dca17), Last Rule Node: [RuleChain: Root Rule Chain|RuleNode: Save raw telemetry(71b87e70-177d-11ec-9530-3197ec48e7c5)]

I would suggest to use generator node to test where the problem is.
First you should test if you can save basic message (like the one you get when you open generator node). With this you will confirm that you can save data to database.
After that you should configure generator node to act as your device, and have same data and metadata as you would get from you device/integration.
Reach out back here with your findings from that.
Generator rule node ref: https://thingsboard.io/docs/user-guide/rule-engine-2-0/action-nodes/#generator-node

Related

Debezium: Replicating data from Oracle read-only database

My use case is to capture CDC data from Oracle read-only PDB database using Debezium. When I tried installing and run Debezium, it's throwing below error message
Can someone please help with the correct Debezium config?
ORA-16000: database or pluggable database open for read-only access
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:628)
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:562)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1145)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:726)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:291)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:492)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:144)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1034)
at oracle.jdbc.driver.OracleStatement.executeSQLStatement(OracleStatement.java:1507)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1287)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3735)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3933)
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4279)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1010)
at io.debezium.connector.oracle.logminer.LogMinerHelper.executeCallableStatement(LogMinerHelper.java:701)
at io.debezium.connector.oracle.logminer.LogMinerHelper.createFlushTable(LogMinerHelper.java:105)
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:122)
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:63)
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:159)
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:122)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: Error : 604, Position : 0, Sql = CREATE TABLE LOG_MINING_FLUSH(LAST_SCN NUMBER(19,0)), OriginalSql = CREATE TABLE LOG_MINING_FLUSH(LAST_SCN NUMBER(19,0)), Error Msg = ORA-00604: error occurred at recursive SQL level 1
ORA-16000: database or pluggable database open for read-only access
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:632)
... 24 more

Not able to create new oracle jdbc-source confluent kafka connector. It is giving error Caused by: java.net.BindException: Address already in use

I have tested default existing jdbc source connector. It is working fine for DB. I want to create another oracle jdbc source connector for different DB. For that I have created new properties file with different DB details. Then, I ran the below command.
bin/connect-standalone etc/schema-registry/connect-avro-standalone.properties source-quickstart-sqlite1.properties
The Error:
[2020-06-20 09:43:46,751] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectStandalone:130)
org.apache.kafka.connect.errors.ConnectException: Unable to initialize REST server
at org.apache.kafka.connect.runtime.rest.RestServer.initializeServer(RestServer.java:217)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:87)
Caused by: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8083
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:307)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.server.Server.doStart(Server.java:385)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.apache.kafka.connect.runtime.rest.RestServer.initializeServer(RestServer.java:215)
... 1 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
... 8 more
I have tried usuing rest.port:8089. It is also not working.
There is already an application running on you computer, who is attached (binded) to the port 8083. Probably, it's another Kafka Connect instance. Stop it before executing you command.
To find the app listening the port, execute lsof -i :8083 and you'll get the PID. You can then simply kill it, assuming it's all about local development of Konnect applications.

Postgres 11, receiving broken pipe, if stays inactive

Spring application is logging broken pipe occasionally. To deal with stale connections issue, application is already using -
testWhileIdle=true and validationQuery=SELECT 1
Stack trace -
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:333)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:118)
at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114)
at com.sun.proxy.$Proxy137.executeQuery(Unknown Source)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70)
... 133 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.postgresql.core.PGStream.flush(PGStream.java:514)
at org.postgresql.core.v3.QueryExecutorImpl.sendSync(QueryExecutorImpl.java:1363)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:304)
... 143 more
Caused by java.net.SocketException: Broken pipe (Write failed)
A vague feeling, if application stays inactive for while, then it might be giving problem connecting to backend or vice-versa. But since can not reproduce this issue locally, it is hard to point.
You can try setting the timeout of idle connection higher.
idle_in_transaction_session_timeout = 30000
in postgresql.conf
But your error seems strange to me. I would probably check if you dont have way to many connection as well.
On the spring-boot side you could try adding the following properties:
spring.datasource.test-on-borrow=true
spring.datasource.validation-query=SELECT 1
spring.datasource.validation-interval=30000
you could try reducing the validation interval. I set it to the default one there
Hope it helps

Kubernetes java.io.IOException: Broken pipe error

I am trying deploy a pod to a kubernetes 11.2 cluster and seeing this error.
Weird part is that this workers fine on another cluster (different envs) and config is same exact. Only thing, I noticed is with this failing cluster, I felt like the nodes were little slow to login. But everything seems to be deploying correctly, except for this error.
There are no changes to the code or the config to deploy, what could be the reason for this to happen only on this particular cluster and not other envs. (It works on dev, test and pre-prod) doesn't work on prod, totally dazed I am and not sure if this is infrastructure issue possibly to the kubernetes config or the application needs to be able to handle this. another thing is that i don't see the nodes going down or any errors related to the memory or lack of resources, like disk pressure and such.
any advise would be highly appreciated.
2018-09-02 18:29:51.048 INFO 29 --- [-nio-443-exec-6] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2018-09-02 18:29:56.493 ERROR 29 --- [-nio-443-exec-6] c.t.s.est.server.ServletDispatcher : An unexpected error occured while processing a request on the following uri /.well-known/est/App Service/senroll
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
...
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_171]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_171]
...
2018-09-02 18:29:56.499 ERROR 29 --- [-nio-443-exec-6] o.a.c.c.C.[.[.[/].[servletDispatcher] : Servlet.service() for servlet [servletDispatcher] in context with path [] threw exception
java.lang.IllegalStateException: Cannot call sendError() after the response has been committed
at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:472) ~[tomcat-embed-core-8.5.16.jar!/:8.5.16]
at com.trilliantnetworks.security.est.server.ServletDispatcher.unexpectedError(ServletDispatcher.java:230) ~[est-servlet-1.0.1-SNAPSHOT.jar!/:na]
at com.trilliantnetworks.security.est.server.ServletDispatcher.doPost(ServletDispatcher.java:211) ~[est-servlet-1.0.1-SNAPSHOT.jar!/:na]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) ~[tomcat-embed-core-8.5.16.jar!/:8.5.16]
...
...
2018-09-02 18:29:56.527 ERROR 29 --- [-nio-443-exec-6] o.a.c.c.C.[Tomcat].[localhost] : Exception Processing ErrorPage[errorCode=0, location=/error]
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:321) ~[tomcat-embed-core-8.5.16.jar!/:8.5.16]
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:284) ~[tomcat-embed-core-8.5.16.jar!/:8.5.16]
The nodes are fine. They are not going down or anything. I also looked at the events to see if any errors related to memory show up, but unfortunately nothing.
So, I am seeing this error in one of the pods, which installs some things to the database. This is the first step and should not fail
Mainly this is happening, the read timeout. So, I am wondering if there is some kind of timeout that  I can set in the cluster, to wait for the API response, little longer.
2018-09-02 18:33:35.818 INFO 29 --- [ main] c.t.s.c.i.r.impl.ExternalSignerService : Exception while generatePermanentKeyStore:java.net.SocketTimeoutException: Read timed out

Spring MongoDB TCP connection ends then does a query

I am having a very strange problem with Spring 3.1 and MongoDB. I am using the Spring Mongo libs. This is a pretty simple call just pulls some data and inserts some new data. But as it is in the loop when pulling new data the connection is killed. From a capture I see it pull data insert data, then do another query but after a two second delay it sends a FIN packet to the server. the server then a little bit later tries to send data back to the client, but after that the client sends RSTs as it has already killed the connection.
Please let me know what other information I can provide to help understand this issue. I can provide a download link for the a capture file as well.
Thank you for your help
Trace output
org.springframework.dao.DataAccessResourceFailureException: can't call something : id561la.ytel.com/172.31.214.55:27017/cdrstat; nested exception is com.mongodb.MongoException$Network: can't call something : id561la.ytel.com/172.31.214.55:27017/cdrstat
at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:56)
at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:1665)
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1548)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1336)
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1322)
at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:495)
at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:486)
at dao.BaseMongoSQLDAO.queryForList(BaseMongoSQLDAO.java:35)
at dao.PrefixStatSqlMapDAO.list(PrefixStatSqlMapDAO.java:59)
at services.CDRReaderService.getPrefixStat(CDRReaderService.java:705)
at services.CDRReaderService.processFile(CDRReaderService.java:659)
at services.CDRReaderService.access$100(CDRReaderService.java:42)
at services.CDRReaderService$1.run(CDRReaderService.java:150)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: com.mongodb.MongoException$Network: can't call something : id561la.ytel.com/172.31.214.55:27017/cdrstat
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:295)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:257)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:310)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:295)
at com.mongodb.DBCursor._check(DBCursor.java:368)
at com.mongodb.DBCursor._hasNext(DBCursor.java:459)
at com.mongodb.DBCursor.hasNext(DBCursor.java:484)
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1534)
... 13 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.bson.io.Bits.readFully(Bits.java:46)
at org.bson.io.Bits.readFully(Bits.java:33)
at org.bson.io.Bits.readFully(Bits.java:28)
at com.mongodb.Response.<init>(Response.java:40)
at com.mongodb.DBPort.go(DBPort.java:124)
at com.mongodb.DBPort.call(DBPort.java:74)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:286)
... 20 more