Use case
Blueprints implementation of orient graph should be able to connect to cluster and create vertices and edges on the graph concurrently (In our case there are 16 threads each with its own graph instance modifying the graph) . Client app is handling ONeedRetryException and all the server/client side configuration is unmodified. Moreover, it works fine with OrientDB v2.2.35.
Actual behavior
Three server nodes are able to join cluster and talk to each other (sync databases) but client throws a exception while creating graph.
Caused by: com.orientechnologies.orient.server.distributed.task.ODistributedRecordLockedException: Timeout (0ms) on acquiring lock on record #-1:-1 on server 'DeadLock'. It is locked by request 2.1301
DB name="apmnew1"
DB name="apmnew1"
at sun.reflect.GeneratedConstructorAccessor67.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_152]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_152]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.throwSerializedException(OChannelBinaryAsynchClient.java:318) ~[orientdb-client-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:275) ~[orientdb-client-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:191) ~[orientdb-client-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:153) ~[orientdb-client-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:1812) ~[orientdb-client-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.client.remote.OStorageRemote.lambda$networkOperationRetryTimeout$2(OStorageRemote.java:226) ~[orientdb-client-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:285) ~[orientdb-client-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetryTimeout(OStorageRemote.java:214) ~[orientdb-client-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationNoRetry(OStorageRemote.java:239) ~[orientdb-client-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.client.remote.OStorageRemote.commit(OStorageRemote.java:997) ~[orientdb-client-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.internalCommit(ODatabaseDocumentAbstract.java:2744) ~[orientdb-core-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:534) ~[orientdb-core-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:100) ~[orientdb-core-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2232) ~[orientdb-core-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2202) ~[orientdb-core-3.0.4.jar:3.0.4]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:734) ~[orientdb-core-3.0.4.jar:3.0.4]
at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.commit(OrientTransactionalGraph.java:184) ~[orientdb-graphdb-3.0.4.jar:3.0.4]
Related
I am trying to ingres data through kafka and json file and running this command:
bin/kafka-console-producer.sh --broker-list localhost:19092 --topic mytopic < $PDATA_HOME/opt_flatten_json.json
But I am getting error:
Exception while executing a state transition task mystats__0__0__20210319T0430Z
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_282]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_282]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_282]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:695) ~[?:1.8.0_282]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.8.0_282]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_282]
at org.apache.pinot.core.segment.memory.PinotByteBuffer.allocateDirect(PinotByteBuffer.java:39) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.pinot.core.segment.memory.PinotDataBuffer.allocateDirect(PinotDataBuffer.java:116) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.pinot.core.io.writer.impl.DirectMemoryManager.allocateInternal(DirectMemoryManager.java:53) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.pinot.core.io.readerwriter.RealtimeIndexOffHeapMemoryManager.allocate(RealtimeIndexOffHeapMemoryManager.java:79) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.pinot.core.realtime.impl.forward.FixedByteMVMutableForwardIndex.addDataBuffer(FixedByteMVMutableForwardIndex.java:162) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.pinot.core.realtime.impl.forward.FixedByteMVMutableForwardIndex.<init>(FixedByteMVMutableForwardIndex.java:137) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:307) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.<init>(LLRealtimeSegmentDataManager.java:1270) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:324) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:132) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:164) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:88) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-d87755899eccba3554e9cc39a1439d5ecb53aaac]
... 12 more
Default rollback method invoked on error. Error Code: ERROR
Message execution failed. msgId: eed5b297-ea20-437e-a0b5-ad4d0be75c3c, errorMsg: java.lang.reflect.InvocationTargetException
Skip internal error. errCode: ERROR, errMsg: null
Event bcbad381_DEFAULT : Unable to find a next state for resource: mystats_REALTIME partition: mystats__0__0__20210319T0430Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
Event c910d226_DEFAULT : Unable to find a next state for resource: mystats_REALTIME partition: mystats__0__0__20210319T0430Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
Event d194950f_DEFAULT : Unable to find a next state for resource: mystats_REALTIME partition: mystats__0__0__20210319T0430Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
Looks like your Kafka JVM ran out of memory:
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
This issue seems related to this ticket.
You should set the Environment Variable for Kafka appropriately to your memory needs: f.i. -XX:MaxDirectMemorySize=512m
Seems like your Pinot ran out of memory buffer for the realtime table segment. Extra info about the segment and realtime table config is needed to debug further.
i am quite new to spring-boot. I have a typical application that consists of a front-end, back-end and postgresql database (see below).
I deployed this app to a local openshift with one pod for each. The app worked fine but when i try though to replicate the spring-boot-backend-pods i noticed that only one of the replicas can access the database. I thought at first that this is happening due to RWO Access Mode of the postgresql volume so i changed it to shared acess(RWX) but i still having the problem.
error:
java.lang.IllegalArgumentException: Sensor type ST0001 already exists
at com.foo.boo.SupplyChainController.addSensorType(SupplyChainController.kt:138) ~[classes!/:na]
at com.foo.boo.SupplyChainController.initializeSupplyChain(SupplyChainController.kt:203) ~[classes!/:na]
at com.foo.boo.service.SupplyChainService.instatiateSupplyChain(SupplyChainService.java:41) ~[classes!/:na]
at com.foo.boo.websocket.SupplyChainSocketController.instatiateSupplyChain(SupplyChainSocketController.java:32) ~[classes!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:138) [spring-messaging-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Does any body know whats really happening here? I can provide more info if needed.
I am experiencing a strange error when running in Websphere Liberty which doesn't occur in unit testing code.
Unit testing is using a JpaTransactionManager and WebSphere is using WebSphereUowTransactionManager.
The application code is invoking findAll on QueryDslJpaRepository.
When I step the code I can see that it somehow ends up with metadata = null after CrudMethodMetadataPostProcessor and the proxy throws NPE.
java.lang.NullPointerException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy148.getLockModeType(Unknown Source)
at org.springframework.data.jpa.repository.support.QueryDslJpaRepository.createQuery(QueryDslJpaRepository.java:180)
Larger Stack Trace
I am using SolrJ to insert/query solr Data [solr cloud 6.5 - 3 machines]. Earlier I used below code to create the SolrClient -
HttpSolrClient server;
server = new HttpSolrClient.Builder("").build();
server.setSoTimeout(20000);
server.setConnectionTimeout(20000);
server.setDefaultMaxConnectionsPerHost(200);
server.setMaxTotalConnections(200);
server.setFollowRedirects(false);
server.setAllowCompression(true);
Now I configured 3 Zookeeper servers on this solr Cloud. Code to create solr client became -
CloudSolrClient server;
String serverURL="zkapp1,zkapp2,zkapp3";
ArrayList zkHosts = newrrayList(Arrays.asList(serverURL.split(",")));
server = new CloudSolrClient.Builder().withZkHost(zkHosts).build();
server.setSoTimeout(20000);
server.setZkConnectTimeout(20000);
server.setDefaultCollection("testSolr");
I want to know where do I need to specify other properties that were present before [DefaultMaxConnectionsPerHost, MaxTotalConnections, FollowRedirects,AllowCompression].
Please guide how to configure Zookeeper because with these settings, after some time below exception is observed [After some successful hits (read/writes)] -
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://solr6CLoudMachine:8983/solr/testSolr
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:621)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
at com.til.damcore.repository.DAMCoreSolrManager.insertContent(DAMCoreSolrManager.java:288)
at com.til.damcore.services.CMSContentUploadServiceSolrOnly.uploadCMSContent(CMSContentUploadServiceSolrOnly.java:227)
at com.til.damapi.service.InsertContentService.insertSolrOnlyContent(InsertContentService.java:136)
at com.til.damapi.controller.insert.InsertionController.insertDataSolr6(InsertionController.java:175)
at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:969)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:871)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:845)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
... 45 more
Can anybody please suggest what are the recommended production settings for Zookeeper?
I am completely stuck.
Any help is greatly appreciated.
Thanks,
Vibhav
When you're using SolrJ in conjunction with SolrCloud, Zookeeper returns the name or the IP of the Solr instance you have to connect, looking at your exception the Solr instance is solr6CLoudMachine.
First thing you have to check it is see if you can reach solr6CLoudMachine from the host where you're running the SolrJ Client.
For example, first try to see you can resolve the ip address of solr6CLoudMachine from shell:
ping solr6CLoudMachine
Then try if you can reach the port:
curl http://solr6CLoudMachine:8983/solr/
And again, given that you are facing with a SolrClould cluster, very likely you have more than one Solr instance to reach.
So, from the host where you're running your SolrJ client, you should be able to reach all the Solr instance where your target collections are located.
Been getting random "unexpected end of stream" exceptions. Load on server (AWS Elasticache) is minimal. There are bursty loads on the client side at the time but the exception doesn't always happen.
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_73]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_73]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.0.23.jar!/:8.0.23]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]
Caused by: org.springframework.data.redis.RedisConnectionFailureException: Unexpected end of stream.; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:47) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:36) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:37) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:37) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:210) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at org.springframework.data.redis.connection.jedis.JedisConnection.exists(JedisConnection.java:781) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_73]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_73]
at org.springframework.data.redis.core.CloseSuppressingInvocationHandler.invoke(CloseSuppressingInvocationHandler.java:57) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at com.sun.proxy.$Proxy126.exists(Unknown Source) ~[na:na]
at org.springframework.data.redis.cache.RedisCache$AbstractRedisCacheCallback.waitForLock(RedisCache.java:460) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at org.springframework.data.redis.cache.RedisCache$AbstractRedisCacheCallback.doInRedis(RedisCache.java:423) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:191) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:141) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at org.springframework.data.redis.cache.RedisCache.get(RedisCache.java:105) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
at org.springframework.data.redis.cache.RedisCache.get(RedisCache.java:90) ~[spring-data-redis-1.6.2.RELEASE.jar!/:1.6.2.RELEASE]
Is this something I can fix by changing the client pool config? When I have too few workers in the client pool I usually get another kind of exception, not the "end of stream" exception.
If you are using the redis pub sub mechanism then one of the possible reaosns for this error is because of the limit of data output buffer set in the configuraiton of redis.
If the data throughput is going larger than the limit set in the configuration the java redis connection will fail as it reaches maximum limit so you need to change the configureation of the client maximum output buffer value for pub sub channel and then this issue resolves
client-output-buffer-limit pubsub
similary there are config for buffer limit of normal client and replication. Normal Client has no limits where as replication has certain limits.