I am trying to connect to federated vdb via Spotfire. I keep getting following error after fetching 200K plus records. There total of 3M records in the view
TEIID session timeout has no limit.
TEIID40007 Keepalive failed for session E0NBLogYqzk3
I am not experiencing the above error when I try to read the vdb in sql client like dbeaver or squirrel and can fetch entire dataset.
Below is the snippet from server.log in Jboss:
setup [PolicyOutInterceptor]
pre-logical [ClientRequestFilterInterceptor]
prepare-send [MessageSenderInterceptor]
write [BodyWriter]
prepare-send-ending [MessageSenderEndingInterceptor]
2022-09-08 12:01:06,358 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Invoking handleMessage on interceptor org.apache.cxf.jaxrs.client.WebClient$BodyWriter#511fa86c
2022-09-08 12:01:06,358 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Invoking handleMessage on interceptor org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor#4a217e2f
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.Headers] (Worker1579_QueryProcessorQueue29181462) Accept: application/json
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.Headers] (Worker1579_QueryProcessorQueue29181462) User-Agent: Teiid Server
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.Headers] (Worker1579_QueryProcessorQueue29181462) Content-length: 0
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.Headers] (Worker1579_QueryProcessorQueue29181462) Content-Type: application/json
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.HTTPConduit] (Worker1579_QueryProcessorQueue29181462) No Trust Decider for Conduit '{http://solrserver.net:8983/solr/document/select}WebClient.http-conduit'. An afirmative Trust Decision is assumed.
2022-09-08 12:01:06,359 FINE [org.apache.cxf.transport.http.HTTPConduit] (Worker1579_QueryProcessorQueue29181462) Sending POST Message with Headers to http://solrserver.net:8983/solr/document/select Conduit :{http://solrserver.net:8983/solr/document/select}WebClient.http-conduit
2022-09-08 12:01:06,359 INFO [org.teiid.SECURITY] (SessionMonitor) TEIID40007 Keepalive failed for session E0NBLogYqzk3
2022-09-08 12:01:06,359 INFO [org.teiid.SECURITY] (SessionMonitor) TEIID40007 Keepalive failed for session E0NBLogYqzk3
2022-09-08 12:01:06,359 DEBUG [org.teiid.SECURITY] (SessionMonitor) closeSession E0NBLogYqzk3
2022-09-08 12:01:06,359 DEBUG [org.teiid.SECURITY] (SessionMonitor) closeSession E0NBLogYqzk3
2022-09-08 12:01:06,359 DEBUG [org.teiid.AUDIT_LOG] (SessionMonitor) [svcpds] <session.logoff>
2022-09-08 12:01:06,359 DEBUG [org.teiid.AUDIT_LOG] (SessionMonitor) [svcpds] <session.logoff>
2022-09-08 12:01:06,359 DEBUG [org.teiid.COMMAND_LOG] (SessionMonitor) CANCEL SRC COMMAND: endTime=2022-09-08 12:01:06.359 requestID=E0NBLogYqzk3.5 sourceCommandID=7 executionID=8950651 txID=null modelName=PDS_SOURCE_MODEL translatorName=delegate sessionID=E0NBLogYqzk3 principal=svcpds finalRowCount=-1
2022-09-08 12:01:06,359 DEBUG [org.teiid.COMMAND_LOG] (SessionMonitor) CANCEL SRC COMMAND: endTime=2022-09-08 12:01:06.359 requestID=E0NBLogYqzk3.5 sourceCommandID=7 executionID=8950651 txID=null modelName=PDS_SOURCE_MODEL translatorName=delegate sessionID=E0NBLogYqzk3 principal=svcpds finalRowCount=-1
2022-09-08 12:01:06,360 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Adding interceptor org.apache.cxf.ws.policy.PolicyInInterceptor#3404a1c0 to phase receive
2022-09-08 12:01:06,360 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Adding interceptor org.apache.cxf.jaxrs.client.WebClient$ClientAsyncResponseInterceptor#6f403813 to phase unmarshal
2022-09-08 12:01:06,360 DEBUG [org.teiid.COMMAND_LOG] (SessionMonitor) CANCEL SRC COMMAND: endTime=2022-09-08 12:01:06.36 requestID=E0NBLogYqzk3.5 sourceCommandID=3 executionID=9226341 txID=null modelName=GF translatorName=rest sessionID=E0NBLogYqzk3 principal=svcpds finalRowCount=-1
2022-09-08 12:01:06,360 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Adding interceptor org.apache.cxf.jaxrs.client.spec.ClientResponseFilterInterceptor#43b0a15f to phase pre-protocol-frontend
2022-09-08 12:01:06,360 FINE [org.apache.cxf.phase.PhaseInterceptorChain] (Worker1579_QueryProcessorQueue29181462) Chain org.apache.cxf.phase.PhaseInterceptorChain#5af24963 was created. Current flow:
receive [PolicyInInterceptor]
pre-protocol-frontend [ClientResponseFilterInterceptor]
unmarshal [ClientAsyncResponseInterceptor]
Below is the error I am getting on Spotfire client:
ImportException at Spotfire.Dxp.Data:
Failed to create DataTable (HRESULT: 80131500)
Stack Trace:
at Spotfire.Dxp.Data.ColumnFactory.CreateColumns(DataRowReader reader, String documentTitleForOrigin, IDataPropertyContainer defaultProperties, DataPropertyRegistry dataPropertyRegistry, GlobalMethodRegistry globalMethodRegistry, CxxSession session, Boolean addNewProperties, PartialDataLoadReport loadReport, ResultProperties resultProperties, PendingViewRequestsManager pendingViewRequestsManager, Boolean mangleColumnNames)
at Spotfire.Dxp.Data.Producers.SourceColumnProducer.<>c__DisplayClass75_0.<CreateView>b__0()
at Spotfire.Dxp.Framework.ApplicationModel.Progress.ExecuteSubtask(String title, ProgressOperation operation)
at Spotfire.Dxp.Data.Producers.SourceColumnProducer.CreateView(CxxSession session, DataPropertyRegistry propertyRegistry, GlobalMethodRegistry globalMethodRegistry, DataSourceConnection connection, IDataPropertyContainer defaultColumnProperties, PartialDataLoadReport& partialLoadReport)
at Spotfire.Dxp.Data.Producers.SourceColumnProducer.GetColumnsAndProperties(DataSourceConnection connection)
at Spotfire.Dxp.Data.Persistence.DataItem.PerformUpdate(SourceColumnProducer producer, DataSourceConnection connection)
at Spotfire.Dxp.Data.Persistence.DataItem.Update(SourceColumnProducer producer, DataSourceConnection connection)
at Spotfire.Dxp.Data.Persistence.DataPool.<LoadData>d__15.MoveNext()
at Spotfire.Dxp.Data.Producers.SourceColumnProducer.OnConfigure()
at Spotfire.Dxp.Framework.DocumentModel.Node.ConfigureSubTree()
at Spotfire.Dxp.Framework.DocumentModel.Node.<>c.<ConfigureSubTree>b__47_0(Node node)
at Spotfire.Dxp.Framework.DocumentModel.UndoableListAvlLeaf`1.ForEachChild(Action`1 action, Boolean includeFrozen)
at Spotfire.Dxp.Framework.DocumentModel.Node.ConfigureSubTree()
at Spotfire.Dxp.Framework.DocumentModel.Node.<>c.<ConfigureSubTree>b__47_0(Node node)
at Spotfire.Dxp.Framework.DocumentModel.UndoableList`1.ForEachChild(Action`1 action, Boolean includeFrozen)
at Spotfire.Dxp.Framework.DocumentModel.Node.ConfigureSubTree()
at Spotfire.Dxp.Framework.DocumentModel.Node.<>c.<ConfigureSubTree>b__47_0(Node node)
at Spotfire.Dxp.Framework.DocumentModel.State.NodeState.<>c__DisplayClass92_0.<ForEachManagedChild>b__0(IDocumentNodeChild documentNodeChild)
at Spotfire.Dxp.Framework.DocumentModel.State.NodeState.ForEachChild(IDocumentNodeChild[] children, Action`1 action)
at Spotfire.Dxp.Framework.DocumentModel.Node.ConfigureSubTree()
at Spotfire.Dxp.Framework.ApplicationModel.Progress.ExecuteSubtask(String title, IndeterminateProgressFormatter progressFormatter, ProgressOperation operation)
at Spotfire.Dxp.Framework.DocumentModel.DocumentNode.ConfigureAndAttachFromNew()
at Spotfire.Dxp.Framework.DocumentModel.DocumentNode.AttachSubTreeWhileExecuting(UndoableNodeBase newOwner)
at Spotfire.Dxp.Framework.DocumentModel.UndoableNode.Spotfire.Dxp.Framework.DocumentModel.IUndoableNode.AttachItemToUndoableNode(Object item)
at Spotfire.Dxp.Framework.DocumentModel.UndoableKeyedCollection`2.<>c__DisplayClass43_0.<Insert>b__0()
at Spotfire.Dxp.Framework.DocumentModel.Node.InternalTransaction(Executor executor, Boolean rollbackNestedInternalTransactionAtException, Boolean isStreamingProperty)
at Spotfire.Dxp.Framework.DocumentModel.UndoableKeyedCollection`2.Insert(Int32 index, TNode item)
at Spotfire.Dxp.Data.DataTableCollection.<>c__DisplayClass89_0.<Add>b__0()
at Spotfire.Dxp.Framework.DocumentModel.Node.InternalTransaction(Executor executor, Boolean rollbackNestedInternalTransactionAtException, Boolean isStreamingProperty)
at Spotfire.Dxp.Data.DataTableCollection.Add(DataTable dataTable)
at Spotfire.Dxp.Application.PartiallyOpenedDataSource.<LoadData>d__9.MoveNext()
at Spotfire.Dxp.Application.AnalysisApplication.<OpenPartiallyOpenedDocument>d__98.MoveNext()
at Spotfire.Dxp.Application.AnalysisApplication.<OpenDataSource>d__91.MoveNext()
at Spotfire.Dxp.Application.AnalysisApplication.ConsumeDataLoadPromptRequests(IEnumerable`1 prompts)
at Spotfire.Dxp.Application.AnalysisApplication.Open(DataSource source, DocumentOpenSettings settings)
at Spotfire.Dxp.Forms.Data.Import.DataSourceFactoryService.OpenDataSourceWithoutPrompting(DataSource dataSource, DocumentOpenSettings documentOpenSettings, IServiceProvider serviceProvider)
at Spotfire.Dxp.Forms.Data.Import.DataSourceFactoryService.OpenDataSource(InformationLinkDataSource dataSource, DocumentOpenSettings documentOpenSettings, IServiceProvider serviceProvider)
at Spotfire.Dxp.Framework.ApplicationModel.Progress.<>c__DisplayClass21_0.<Start>b__0()
at Spotfire.Dxp.Framework.ApplicationModel.MonitorableProgress.Start[T](Func`1 action)
at Spotfire.Dxp.Forms.Application.FormsProgressService.ProgressThread.DoOperationLoop()
InformationModelException at Spotfire.Dxp.Data:
Failed to get data: 57014 TEIID30160 The request /IK4StVK70Kl.1 has been cancelled. (HRESULT: 80131500)
Stack Trace:
at Spotfire.Dxp.Data.InformationModel.InternalInformationModelManager.DataStream.GetNextBlock()
at Spotfire.Dxp.Data.InformationModel.InternalInformationModelManager.DataStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at Spotfire.Dxp.Internal.Utilities.SeekableStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at Spotfire.Dxp.Framework.ApplicationModel.ProgressIncrementStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at Spotfire.Dxp.Internal.Utilities.SharedMemoryStream.CopyUnprotected(Stream inputStream, Int32 bufferSize)
at Spotfire.Dxp.Data.Cxx.CxxColumnManager.LoadSBDF(String id, Stream shm, Action streamWriter, Action abortAction, PartialTableCallback partialTableCallback, Boolean useFasterWipImplementation)
at Spotfire.Dxp.Data.Cxx.CxxDataTransfer.LoadSBDF(CxxSession session, Stream stream, PartialTableCallback partialTableCallback, Boolean useFasterSBDF)
at Spotfire.Dxp.Data.Import.SbdfDataRowReader.TryCreateCxxRepresentation(CxxSession session, Int64 exclusiveStartRowIndex, UInt64 maxRowsThatWillBeRead, PendingViewRequestsManager pendingViewRequestsManager, CxxTable& table)
at Spotfire.Dxp.Data.Cxx.CxxDataTransfer.CreateTable(DataRowReader dataRowReader, CxxSession session, PartialDataLoadReport report, Advancer rowAdvancer, Boolean needsReset, Int64 exclusiveStartRowIndex, UInt64 maxRowsThatWillBeRead, PendingViewRequestsManager partiallyLoadedConsumers)
at Spotfire.Dxp.Data.Cxx.CxxDataTransfer.CreateTable(DataRowReader dataRowReader, CxxSession session, PartialDataLoadReport report, PendingViewRequestsManager pendingViewRequestsManager, Boolean needsReset, UInt64 maxRowsToConsume)
at Spotfire.Dxp.Data.ColumnFactory.CreateColumns(DataRowReader reader, String documentTitleForOrigin, IDataPropertyContainer defaultProperties, DataPropertyRegistry dataPropertyRegistry, GlobalMethodRegistry globalMethodRegistry, CxxSession session, Boolean addNewProperties, PartialDataLoadReport loadReport, ResultProperties resultProperties, PendingViewRequestsManager pendingViewRequestsManager, Boolean mangleColumnNames)
InformationModelServiceException at Spotfire.Dxp.Services:
Failed to get data: 57014 TEIID30160 The request /IK4StVK70Kl.1 has been cancelled. (HRESULT: 80131509)
Stack Trace:
at Spotfire.Dxp.Services.WebServiceBase`1.InvokeService[T](ServiceMethod`1 serviceMethod, ExceptionFactoryMethod exceptionFactoryMethod, String customMethodNameForLogging)
at Spotfire.Dxp.Services.Data.InformationModel.QueryManagerService.InvokeService[T](ServiceMethod`1 serviceMethod)
at Spotfire.Dxp.Data.InformationModel.InternalInformationModelManager.DataStream.GetNextBlock()
Appreciate the help. Thanks
Related
I have a Spring Cloud Gateway (eureka client) app that uses Spring Cloud Load Balancer (Spring Cloud version: Hoxton.SR6) and I have an instance of a spring boot app (spring boot 2.3 with enabled graceful shutdown, (eureka client).
When I shutdown a spring boot service and perform a request through the gateway then the gateway throws 500 error (connection refused), instead of 503. 503 appears after a 1-2 minutes.
Can anyone clarify if it is an expected behavior?
It seems that the problem comes from eureka-client (1.9.21 version in my case)
AtomicReference<Applications> localRegionApps isn't frequently updated
Thanks!
UPDATE:
I decided to check deeper this 500 error. The result is that my system (ubuntu) gives this error if the port is not used:
curl -v localhost:9722
Rebuilt URL to: localhost:9722/
Trying 127.0.0.1...
TCP_NODELAY set
connect to 127.0.0.1 port 9722 failed: Connection refused
Failed to connect to localhost port 9722: Connection refused
Closing connection 0
So I put in my application.yml:
spring:
cloud:
gateway:
routes:
- id: my_route
uri: http://localhost:9722/
Then when my request is routed to my_route and none of apps uses 9722 then I get an error:
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:9722
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ HTTP GET "/internal/mail/internal/health-check" [ExceptionHandlingWebHandler]
Stack trace:
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
at io.netty.channel.unix.Socket.finishConnect(Socket.java:251)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
It seems to be an unexpected exception, since it isn't possible to handle it using a circuit breaker or any gateway filter.
Is it possible to handle this error correctly? I would like to return 503 in this case
One of the easiest ways to map particular exception to particular HTTP status code is providing a custom bean of type org.springframework.boot.web.reactive.error.ErrorAttributes. Here is an example:
#Bean
public ErrorAttributes errorAttributes() {
return new CustomErrorAttributes(httpStatusExceptionTypeMapper);
}
public class CustomErrorAttributes extends DefaultErrorAttributes {
#Override
public Map<String, Object> getErrorAttributes(ServerRequest request, ErrorAttributeOptions options) {
Map<String, Object> attributes = super.getErrorAttributes(request, options);
Throwable error = getError(request);
MergedAnnotation<ResponseStatus> responseStatusAnnotation = MergedAnnotations
.from(error.getClass(), MergedAnnotations.SearchStrategy.TYPE_HIERARCHY).get(ResponseStatus.class);
HttpStatus errorStatus = determineHttpStatus(error, responseStatusAnnotation);
attributes.put("status", errorStatus.value());
return attributes;
}
private HttpStatus determineHttpStatus(Throwable error, MergedAnnotation<ResponseStatus> responseStatusAnnotation) {
if (error instanceof ResponseStatusException) {
return ((ResponseStatusException) error).getStatus();
}
return responseStatusAnnotation.getValue("code", HttpStatus.class).orElseGet(() -> {
if (error instanceof java.net.ConnectException) {
return HttpStatus.SERVICE_UNAVAILABLE;
}
return HttpStatus.INTERNAL_SERVER_ERROR;
}
}
}
hava a try to define your custom ErrorWebExceptionHandler.
see:
org.springframework.boot.web.reactive.error.ErrorWebExceptionHandler
org.springframework.boot.autoconfigure.web.reactive.error.DefaultErrorWebExceptionHandler
You should use Cloud Circuit Breaker.
For that:
Declare corresponding starter in your pom:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-circuitbreaker-reactor-resilience4j</artifactId>
</dependency>
Declare circuit breaker in application.yaml
spring:
cloud:
gateway:
routes:
- id: my_route
uri: http://localhost:9722/
filters:
- name: CircuitBreaker
args:
name: myCircuitBreaker
fallbackUri: forward:/inCaseOfFailureUseThis
Declare the endpoint which will be called in the case of failure (a connection error, for example)
#RequestMapping("/inCaseOfFailureUseThis")
public Mono<ResponseEntity<String>> inCaseOfFailureUseThis() {
return Mono.just(ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).body("body for service failure case"));
}
I implemented curator leader election example which is given in this site
Instead of having number of curator clients I added only one curator client as follows
public void selectLeader() {
CuratorFramework client = null;
try {
client = CuratorFrameworkFactory.newClient("localhost:2181", new ExponentialBackoffRetry(1000, 3));
LeaderSelectorService service = new LeaderSelectorService(client, "/leaderSelections", "LeaderElector");
client.start();
Thread.sleep(10000);
service.start();
} catch (Exception e) {
System.out.println("error"+e);
}finally
{
System.out.println("Shutting down...");
// CloseableUtils.closeQuietly(client);
}
}
public class LeaderSelectorService extends LeaderSelectorListenerAdapter implements Closeable {
private final String name;
private final LeaderSelector leaderSelector;
public LeaderSelectorService(CuratorFramework client, String path, String name) {
this.name = name;
// create a leader selector using the given path for management
// all participants in a given leader selection must use the same path
// ExampleClient here is also a LeaderSelectorListener but this isn't required
leaderSelector = new LeaderSelector(client, path, this);
// for most cases you will want your instance to requeue when it relinquishes leadership
leaderSelector.autoRequeue();
}
public void start() throws IOException
{
// the selection for this instance doesn't start until the leader selector is started
// leader selection is done in the background so this call to leaderSelector.start() returns immediately
leaderSelector.start();
}
#Override
public void takeLeadership(CuratorFramework arg0) throws Exception {
// we are now the leader. This method should not return until we want to relinquish leadership
final int waitSeconds = (int)(5 * Math.random()) + 1;
System.out.println(name + " is now the leader. Waiting " + waitSeconds + " seconds...");
//System.out.println(name + " has been leader " + leaderCount.getAndIncrement() + " time(s) before.");
try
{
Thread.sleep(TimeUnit.SECONDS.toMillis(waitSeconds));
}
catch ( InterruptedException e )
{
System.err.println(name + " was interrupted.");
Thread.currentThread().interrupt();
}
finally
{
System.out.println(name + " relinquishing leadership.\n");
}
}
#Override
public void close() throws IOException {
leaderSelector.close();
}
}
I have only one zookeeper instance and I am using Zookeeper 3.4.6, curator-framework 4.0.0 and curator-recipes 4.0.0.
when I start the client, it connects to zookeeper and in the log I can see "State change : connected" message.
Then I wait 10s and start leader election which gives me below error repeatedly.
2017-09-06 09:34:22.727 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x15e555a719d0000, likely server has closed socket, closing socket connection and attempting reconnect
2017-09-06 09:34:22.830 INFO 1228 --- [c-1-EventThread] o.a.c.f.state.ConnectionStateManager : State change: SUSPENDED
2017-09-06 09:34:23.302 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-09-06 09:34:23.303 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established, initiating session, client: /127.0.0.1:49594, server: localhost/127.0.0.1:2181
2017-09-06 09:34:23.305 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15e555a719d0000, negotiated timeout = 120000
2017-09-06 09:34:23.305 INFO 1228 --- [c-1-EventThread] o.a.c.f.state.ConnectionStateManager : State change: RECONNECTED
2017-09-06 09:34:23.310 WARN 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session 0x15e555a719d0000 for server localhost/127.0.0.1:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_131]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_131]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_131]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_131]
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
after some time it started to give me below error message.
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:831) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:623) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.curator.framework.imps.GetConfigBuilderImpl$2.processResult(GetConfigBuilderImpl.java:222) [curator-framework-4.0.0.jar:4.0.0]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590) [zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:499) [zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
2017-09-06 09:34:31.897 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-09-06 09:34:31.898 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established, initiating session, client: /127.0.0.1:49611, server: localhost/127.0.0.1:2181
2017-09-06 09:34:31.899 INFO 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15e555a719d0000, negotiated timeout = 120000
2017-09-06 09:34:31.899 INFO 1228 --- [c-1-EventThread] o.a.c.f.state.ConnectionStateManager : State change: RECONNECTED
2017-09-06 09:34:31.907 WARN 1228 --- [localhost:2181)] org.apache.zookeeper.ClientCnxn : Session 0x15e555a719d0000 for server localhost/127.0.0.1:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Xid out of order. Got Xid 41 with err -6 expected Xid 40 for a packet with details: clientPath:/leaderSelections serverPath:/leaderSelections finished:false header:: 40,12 replyHeader:: 0,0,-4 request:: '/leaderSelections,F response:: v{}
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:892) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:101) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) ~[zookeeper-3.5.3-beta.jar:3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60]
I tried several solution in the internet but non got succeeded. Does anybody know the root cause of this issue.
I have fixed this issue. There was version number mismatch between zookeeper version and curator version. I used curator version 4.0.0 with zookeeper 3.4.6. According to apache curator site
Curator 4.0.0 - compatible with ZooKeeper 3.5.x. I changed my curator version to 2.8.0
I got a structure which can basically be summarized as:
outside user makes a rest request to akka-http server
akka-http makes a request(query?) to a (some)data source using asynchttpclient
akka-http transforms the result from asynchttpclient and serves it back to user
At some point I am getting an error from akka which tells me almost nothing. This error happens right after the asynchttpclient returns me some results. (I can infact at this point print the results on the log, they are there parsed from json etc.. but akka had already errored out)
Even in debug logging level I got no decipherable error message from akka or a stacktrace.
only message I got is:
2017-03-24 17:22:55 INFO CompanyRepository:111 - search company with name:"somecompanyname"
2017-03-24 17:22:55 INFO CompanyRepository:73 - [QUERY TIME]: 527ms
[ERROR] [03/24/2017 17:22:55.951] [company-api-system-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(company-api-system)] Error during processing of request: 'requirement failed'. Completing with 500 Internal Server Error response.
This error message is the only thing I get. Relevant parts of my config:
akka {
loglevel = "DEBUG"
# edit -- tested with sl4jlogger with no change
#loggers = ["akka.event.slf4j.Slf4jLogger"]
#logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
parsing {
max-content-length = 800m
max-chunk-size = 100m
}
server {
server-header = akka-http/${akka.http.version}
idle-timeout = 120 s
request-timeout = 120 s
bind-timeout = 10s
max-connections = 1024
pipelining-limit = 32
verbose-error-messages = on
}
client {
user-agent-header = akka-http/${akka.http.version}
}
host-connection-pool {
max-connections = 4
}
}
akka.http.routing {
verbose-error-messages = on
}
Anyone knows if I can make akka to spit out more details about what/where the error is occurring?
Edit: I realized I do NOT get this same error on resultsets which are smaller in size. <- ignore
Edit 2:
Added akka.loglevel = DEBUG, spits out a lot more noise but still not detail about the actual error.
Converted asynchttpclient to akka quickly to rule out AHC
I already had a wrapper around my query to time it, added some logging there trying to pinpoint when exactly the error is happening.
def queryTimer[ R <: Future[ Any ] ]( block: => R ): R = {
val t0 = System.currentTimeMillis()
val result = block
result.onComplete { maybeResult =>
val t1 = System.currentTimeMillis()
logger.info( "[QUERY TIME]: " + ( t1 - t0 ) + "ms" )
maybeResult match {
case Success(some) =>
logger.info( "successful feature:")
logger.info( FormattedString.prettyPrint(some))
case Failure(someFailure) =>
logger.info( "failed feature:")
logger.debug( FormattedString.prettyPrint(someFailure))
}
}
result
}
resulting log:
2017-03-28 13:19:10 INFO CompanyRepository:111 - search company with name:"some company"
[DEBUG] [03/28/2017 13:19:10.497] [company-api-system-akka.actor.default-dispatcher-2] [EventStream(akka://xca-api-actor-system)] logger log1-Logging$DefaultLogger started
[DEBUG] [03/28/2017 13:19:10.497] [company-api-system-akka.actor.default-dispatcher-2] [EventStream(akka://xca-api-actor-system)] Default Loggers started
[DEBUG] [03/28/2017 13:19:10.613] [company-api-system-akka.actor.default-dispatcher-2] [AkkaSSLConfig(akka://xca-api-actor-system)] Initializing AkkaSSLConfig extension...
[DEBUG] [03/28/2017 13:19:10.613] [company-api-system-akka.actor.default-dispatcher-2] [AkkaSSLConfig(akka://xca-api-actor-system)] buildHostnameVerifier: created hostname verifier: com.typesafe.sslconfig.ssl.DefaultHostnameVerifier#779e2339
[DEBUG] [03/28/2017 13:19:10.633] [xca-api-actor-system-akka.actor.default-dispatcher-3] [akka://xca-api-actor-system/user/pool-master/PoolInterfaceActor-0] (Re-)starting host connection pool to localhost:27474
[DEBUG] [03/28/2017 13:19:10.727] [xca-api-actor-system-akka.actor.default-dispatcher-3] [akka://xca-api-actor-system/system/IO-TCP/selectors/$a/0] Resolving localhost before connecting
[DEBUG] [03/28/2017 13:19:10.740] [xca-api-actor-system-akka.actor.default-dispatcher-4] [akka://xca-api-actor-system/system/IO-DNS] Resolution request for localhost from Actor[akka://xca-api-actor-system/system/IO-TCP/selectors/$a/0#-815754478]
[DEBUG] [03/28/2017 13:19:10.749] [xca-api-actor-system-akka.actor.default-dispatcher-4] [akka://xca-api-actor-system/system/IO-TCP/selectors/$a/0] Attempting connection to [localhost/127.0.0.1:27474]
[DEBUG] [03/28/2017 13:19:10.751] [xca-api-actor-system-akka.actor.default-dispatcher-4] [akka://xca-api-actor-system/system/IO-TCP/selectors/$a/0] Connection established to [localhost:27474]
2017-03-28 13:19:10 INFO CompanyRepository:73 - [QUERY TIME]: 376ms
2017-03-28 13:19:10 INFO CompanyRepository:77 - successful feature:
[ERROR] [03/28/2017 13:19:10.896] [company-api-system-akka.actor.default-dispatcher-7] [akka.actor.ActorSystemImpl(company-api-system)] Error during processing of request: 'requirement failed'. Completing with 500 Internal Server Error response.
2017-03-28 13:19:10 INFO CompanyRepository:78 - SearchResult(List(
( prettyprint output here!!! lots and lots of legit result, json parsed succcesfully into a bunch of case classes)
as you can see my logging format and akkas' are different, the ERROR is coming from akka with do details, while everything looks like working.
Edit 3: logs with sleep in between calls
new query timer function with sleeps
def queryTimer[ R <: Future[ Any ] ]( block: => R ): R = {
val t0 = System.currentTimeMillis()
val result = block
result.onComplete { maybeResult =>
val t1 = System.currentTimeMillis()
logger.info( "[QUERY TIME]: " + ( t1 - t0 ) + "ms" )
maybeResult match {
case Success(some) =>
Thread.sleep(500)
logger.info( "successful feature:")
Thread.sleep(500)
logger.info( FormattedString.prettyPrint(some))
Thread.sleep(500)
logger.info("we are there!")
case Failure(someFailure) =>
logger.info( "failed feature:")
logger.debug( FormattedString.prettyPrint(someFailure))
}
}
result
}
logs with sleeps
[DEBUG] [03/30/2017 11:11:58.629] [xca-api-actor-system-akka.actor.default-dispatcher-7] [akka://xca-api-actor-system/system/IO-TCP/selectors/$a/0] Attempting connection to [localhost/127.0.0.1:27474]
[DEBUG] [03/30/2017 11:11:58.631] [xca-api-actor-system-akka.actor.default-dispatcher-7] [akka://xca-api-actor-system/system/IO-TCP/selectors/$a/0] Connection established to [localhost:27474]
11:11:59.442 [pool-2-thread-1] DEBUG o.a.netty.channel.DefaultChannelPool - Closed 0 connections out of 0 in 0 ms
11:11:59.496 [pool-1-thread-1] DEBUG o.a.netty.channel.DefaultChannelPool - Closed 0 connections out of 0 in 0 ms
11:12:00.250 [ForkJoinPool-2-worker-15] INFO c.s.s.r.neo4j.CompanyRepository - [QUERY TIME]: 1880ms
[ERROR] [03/30/2017 11:12:00.265] [company-api-system-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(company-api-system)] Error during processing of request: 'requirement failed'. Completing with 500 Internal Server Error response.
11:12:00.543 [pool-2-thread-1] DEBUG o.a.netty.channel.DefaultChannelPool - Closed 0 connections out of 0 in 0 ms
11:12:00.597 [pool-1-thread-1] DEBUG o.a.netty.channel.DefaultChannelPool - Closed 0 connections out of 0 in 0 ms
11:12:00.752 [ForkJoinPool-2-worker-15] INFO c.s.s.r.neo4j.CompanyRepository - successful feature:
11:12:01.645 [pool-2-thread-1] DEBUG o.a.netty.channel.DefaultChannelPool - Closed 0 connections out of 0 in 0 ms
11:12:01.697 [pool-1-thread-1] DEBUG o.a.netty.channel.DefaultChannelPool - Closed 0 connections out of 0 in 0 ms
11:12:01.750 [ForkJoinPool-2-worker-15] INFO c.s.s.r.neo4j.CompanyRepository - SearchResult(List( "lots of legit result here"
11:12:02.281 [ForkJoinPool-2-worker-15] INFO c.s.s.r.neo4j.CompanyRepository - we are there!
Edit 4 and solution!
Apparently the default exception handler does not print a stack trace! overriding the exception handler with a very basic catch all:
implicit def myExceptionHandler: ExceptionHandler =
ExceptionHandler {
case e: Exception => {
logger.info("---------------- exception log start")
logger.error(e.getMessage, e)
logger.error("cause" , e.getCause)
logger.error("cause" , e.getStackTraceString )
logger.info( FormattedString.prettyPrint(e))
logger.info("---------------- exception log end")
Directives.complete("server made a boo boo")
}
}
results in a stack trace that befuddles the sh*t out of me!!
11:42:04.634 [company-api-system-akka.actor.default-dispatcher-2] INFO c.stepweb.scarifgate.CompanyApiApp$ - ---------------- exception log start
11:42:04.640 [company-api-system-akka.actor.default-dispatcher-2] ERROR c.stepweb.scarifgate.CompanyApiApp$ - requirement failed
java.lang.IllegalArgumentException: requirement failed
at scala.Predef$.require(Predef.scala:212) ~[scala-library-2.11.8.jar:na]
at spray.json.BasicFormats$StringJsonFormat$.write(BasicFormats.scala:121) ~[spray-json_2.11-1.3.2.jar:na]
at spray.json.BasicFormats$StringJsonFormat$.write(BasicFormats.scala:119) ~[spray-json_2.11-1.3.2.jar:na]
at spray.json.ProductFormats$class.productElement2Field(ProductFormats.scala:46) ~[spray-json_2.11-1.3.2.jar:na]
at com.stepweb.scarifgate.services.CompanyService.productElement2Field(CompanyService.scala:14) ~[classes/:na]
at spray.json.ProductFormatsInstances$$anon$3.write(ProductFormatsInstances.scala:73) ~[spray-json_2.11-1.3.2.jar:na]
at spray.json.ProductFormatsInstances$$anon$3.write(ProductFormatsInstances.scala:68) ~[spray-json_2.11-1.3.2.jar:na]
at spray.json.PimpedAny.toJson(package.scala:39) ~[spray-json_2.11-1.3.2.jar:na]
at spray.json.CollectionFormats$$anon$1$$anonfun$write$1.apply(CollectionFormats.scala:26) ~[spray-json_2.11-1.3.2.jar:na]
at spray.json.CollectionFormats$$anon$1$$anonfun$write$1.apply(CollectionFormats.scala:26) ~[spray-json_2.11-1.3.2.jar:na]
at scala.collection.immutable.List.map(List.scala:273) ~[scala-library-2.11.8.jar:na]
at spray.json.CollectionFormats$$anon$1.write(CollectionFormats.scala:26) ~[spray-json_2.11-1.3.2.jar:na]
at spray.json.CollectionFormats$$anon$1.write(CollectionFormats.scala:25) ~[spray-json_2.11-1.3.2.jar:na]
at spray.json.ProductFormats$class.productElement2Field(ProductFormats.scala:46) ~[spray-json_2.11-1.3.2.jar:na]
at com.stepweb.scarifgate.services.CompanyService.productElement2Field(CompanyService.scala:14) ~[classes/:na]
at spray.json.ProductFormatsInstances$$anon$1.write(ProductFormatsInstances.scala:30) ~[spray-json_2.11-1.3.2.jar:na]
at spray.json.ProductFormatsInstances$$anon$1.write(ProductFormatsInstances.scala:26) ~[spray-json_2.11-1.3.2.jar:na]
at akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport$$anonfun$sprayJsonMarshaller$1.apply(SprayJsonSupport.scala:62) ~[akka-http-spray-json_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport$$anonfun$sprayJsonMarshaller$1.apply(SprayJsonSupport.scala:62) ~[akka-http-spray-json_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.Marshaller$$anonfun$compose$1$$anonfun$apply$15.apply(Marshaller.scala:73) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.Marshaller$$anonfun$compose$1$$anonfun$apply$15.apply(Marshaller.scala:73) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.Marshaller$$anon$1.apply(Marshaller.scala:92) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.GenericMarshallers$$anonfun$optionMarshaller$1$$anonfun$apply$1.apply(GenericMarshallers.scala:19) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.GenericMarshallers$$anonfun$optionMarshaller$1$$anonfun$apply$1.apply(GenericMarshallers.scala:18) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.Marshaller$$anon$1.apply(Marshaller.scala:92) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.PredefinedToResponseMarshallers$$anonfun$fromStatusCodeAndHeadersAndValue$1$$anonfun$apply$5.apply(PredefinedToResponseMarshallers.scala:58) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.PredefinedToResponseMarshallers$$anonfun$fromStatusCodeAndHeadersAndValue$1$$anonfun$apply$5.apply(PredefinedToResponseMarshallers.scala:57) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.Marshaller$$anon$1.apply(Marshaller.scala:92) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.Marshaller$$anonfun$compose$1$$anonfun$apply$15.apply(Marshaller.scala:73) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.Marshaller$$anonfun$compose$1$$anonfun$apply$15.apply(Marshaller.scala:73) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.Marshaller$$anon$1.apply(Marshaller.scala:92) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.ToResponseMarshallable$$anonfun$1$$anonfun$apply$1.apply(ToResponseMarshallable.scala:29) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.ToResponseMarshallable$$anonfun$1$$anonfun$apply$1.apply(ToResponseMarshallable.scala:29) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.Marshaller$$anon$1.apply(Marshaller.scala:92) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.GenericMarshallers$$anonfun$futureMarshaller$1$$anonfun$apply$3$$anonfun$apply$4.apply(GenericMarshallers.scala:33) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.marshalling.GenericMarshallers$$anonfun$futureMarshaller$1$$anonfun$apply$3$$anonfun$apply$4.apply(GenericMarshallers.scala:33) ~[akka-http_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.util.FastFuture$.akka$http$scaladsl$util$FastFuture$$strictTransform$1(FastFuture.scala:41) ~[akka-http-core_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.util.FastFuture$$anonfun$transformWith$extension1$1.apply(FastFuture.scala:51) [akka-http-core_2.11-10.0.0.jar:10.0.0]
at akka.http.scaladsl.util.FastFuture$$anonfun$transformWith$extension1$1.apply(FastFuture.scala:50) [akka-http-core_2.11-10.0.0.jar:10.0.0]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [scala-library-2.11.8.jar:na]
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) [akka-actor_2.11-2.4.16.jar:na]
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91) [akka-actor_2.11-2.4.16.jar:na]
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91) [akka-actor_2.11-2.4.16.jar:na]
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91) [akka-actor_2.11-2.4.16.jar:na]
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) [scala-library-2.11.8.jar:na]
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90) [akka-actor_2.11-2.4.16.jar:na]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39) [akka-actor_2.11-2.4.16.jar:na]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415) [akka-actor_2.11-2.4.16.jar:na]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.8.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.8.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.8.jar:na]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.8.jar:na]
11:42:04.640 [company-api-system-akka.actor.default-dispatcher-2] ERROR c.stepweb.scarifgate.CompanyApiApp$ - cause
11:42:04.641 [company-api-system-akka.actor.default-dispatcher-2] ERROR c.stepweb.scarifgate.CompanyApiApp$ - cause
11:42:04.644 [company-api-system-akka.actor.default-dispatcher-2] INFO c.stepweb.scarifgate.CompanyApiApp$ - java.lang.IllegalArgumentException: requirement failed
11:42:04.644 [company-api-system-akka.actor.default-dispatcher-2] INFO c.stepweb.scarifgate.CompanyApiApp$ - ---------------- exception log end
so... the exception is caused here in spray.json.BasicFormats
implicit object StringJsonFormat extends JsonFormat[String] {
def write(x: String) = {
require(x ne null) // <-----------------------------------
JsString(x)
}
def read(value: JsValue) = value match {
case JsString(x) => x
case x => deserializationError("Expected String as JsString, but got " + x)
}
}
which sort of means one of the strings in this thousands of lines of response is null. Special thanks goes to the laziness of using that "require" without a message. Debugging which string is empty where will be a nightmare but I still think akka should fail in a better way.
akka-http no stack trace or details on error
Well, default akka-http ExceptionHandler doesn't print stack trace and prints only error message or its class name if the message is empty but you can provide custom exception handler that will print anything you want (i.e. stack trace in your example).
Some examples of how to make a custom exception handler are provided at GitHub ExceptionHandlerExamplesSpec.spec
The simplest way in your case seems to be to define your own custom implicit exception handler
import akka.http.scaladsl.model._
import akka.http.scaladsl.server._
import StatusCodes._
import Directives._
implicit def myExceptionHandler: ExceptionHandler =
ExceptionHandler {
case NonFatal(e) =>
logger.error(s"Exception $e at\n${e.getStackTraceString}")
complete(HttpResponse(InternalServerError, entity = "Internal Server Error"))
}
}
Try setting the loggers as well - from your configuration it seems they're not set. Something like:
akka {
loggers = ["akka.event.slf4j.Slf4jLogger"]
loglevel = "DEBUG"
logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
}
Also, consider using akka-slf4j along with their recommended logging backend logback.
This should make akka spit more details.
I am writing a selenium script using TestNG framework. i have defined the explicit wait as 20 seconds before selenium throws the NoSuchElement exception. but script does not wait for the 20 second during the execution and throws exception in 41 milliseconds.
i want this script to wait(using explicit wait only) or search for web element for 20 seconds before throwing any exception.
Below is the script followed by the execution result.
public class para {
WebDriver driver;
#BeforeClass
void InvokeFF() {
System.setProperty("webdriver.gecko.driver",
"C:/Users/Vinay/workspace_n/EGuru/drivers/geckodriver.exe");
driver = new FirefoxDriver();
// driver.get("http://seleniumpractise.blogspot.in/2016/08/bootstrap-dropdown-example-for-selenium.html");
System.out.println("Firefox invoked");
System.out.println("Firefox thread:" + Thread.currentThread().getId());
}
#Test
void Auto() throws Exception {
WebDriverWait wait = new WebDriverWait(driver, 20);
driver.get("file:///C:/Users/Vinay/Desktop/Upload1.html");
WebElement elem = driver.findElement(By.xpath(".//*[#id='1']"));
wait.until(ExpectedConditions.visibilityOfElementLocated(By
.xpath(".//*[#id='1']")));
elem.click();
Runtime.getRuntime().exec("C:\\Users\\Vinay\\Desktop\\AutoUpload.exe");
}
Firefox invoked Firefox thread:1 Execution Started [Utils] Attempting
to create C:\Users\Vinay\workspace_n\EGuru\test-output\Default
suite\Default test.xml [Utils] Directory
C:\Users\Vinay\workspace_n\EGuru\test-output\Default suite exists:
true FAILED: Auto org.openqa.selenium.NoSuchElementException: Unable
to locate element: {"method":"xpath","selector":".//*[#id='1']"}
Command duration or timeout: 41 milliseconds For documentation on this
error, please visit:
http://seleniumhq.org/exceptions/no_such_element.html Build info:
version: '2.53.0', revision: '35ae25b', time: '2016-03-15 16:57:40'
System info: host: 'Vinay-PC', ip: '192.168.1.2', os.name: 'Windows
7', os.arch: 'amd64', os.version: '6.1', java.version: '1.8.0_101'
Driver info: org.openqa.selenium.firefox.FirefoxDriver Capabilities
[{applicationCacheEnabled=true, rotatable=false, handlesAlerts=true,
databaseEnabled=true, version=45.0.2, platform=WINDOWS,
nativeEvents=false, acceptSslCerts=true, webStorageEnabled=true,
locationContextEnabled=true, browserName=firefox,
takesScreenshot=true, javascriptEnabled=true,
cssSelectorsEnabled=true}] Session ID:
d40fc001-400c-473b-8213-078e641b3c7f
*** Element info: {Using=xpath, value=.//*[#id='1']} at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
WebElement elem = driver.findElement(By.xpath(".//*[#id='1']"));
wait.until(ExpectedConditions.visibilityOfElementLocated(By
.xpath(".//*[#id='1']")));
The above two lines need to be reversed in order. You are attempting to get the element without a wait, then telling the webdriver to wait until it's there.
I have recently configured MongoDB and Sitecore 8 Analytics (rev. 150812) on a production environment and it works just fine. However, oddly, I get this error in log files for every single pageview on live website, not sure why it exactly happens and how to rectify it. It is always a pair of errors (PostSessionEndPipeline failed + PostSessionEndPipeline failed). This happens on content delivery server. MongoDB is on another server. delivery server is on one Azure VM and MongoDB is on another VM. Does it have anything to do with session timeout or something like that?
7072 00:09:19 ERROR PostSessionEndPipeline failed.
Exception: System.IO.IOException
Message: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Source: System
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MongoDB.Bson.IO.ByteBufferFactory.LoadFrom(Stream stream)
at MongoDB.Driver.Internal.MongoConnection.ReceiveMessage[TDocument](BsonBinaryReaderSettings readerSettings, IBsonSerializer serializer, IBsonSerializationOptions serializationOptions)
at MongoDB.Driver.Operations.QueryOperation`1.GetFirstBatch(IConnectionProvider connectionProvider)
at MongoDB.Driver.Operations.QueryOperation`1.Execute(IConnectionProvider connectionProvider)
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at MongoDB.Driver.MongoCollection.FindOneAs[TDocument](FindOneArgs args)
at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbCollection.<>c__DisplayClass1d`1.<FindOneByIdAs>b__1c()
at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbCollection.Execute(Action action, ExceptionBehavior exceptionBehavior)
at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbCollection.FindOneByIdAs[TDocument](BsonValue value, Nullable`1 exceptionBehavior)
at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbDriver.FindOneByIdAs[T](Object key)
at Sitecore.Analytics.Data.DataAccess.MongoDb.MongoDbDictionary.LoadAs[T](Object key)
at Sitecore.Analytics.Data.Dictionaries.TrackingDictionary`2.Get(TKey key, LookupStrategy strategy)
at Sitecore.Analytics.Pipelines.EnsureClassification.GetClassificators.Process(EnsureClassificationPipelineArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Analytics.Pipelines.PostSessionEnd.CommitSession.Process(PostSessionEndArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Web.Application.RaiseSessionEndEvent(HttpApplication context)
Nested Exception
Exception: System.Net.Sockets.SocketException
Message: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Source: System
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
1572 00:09:19 ERROR SessionEndPipeline failed.
Exception: System.IO.IOException
Message: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Source: System
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MongoDB.Bson.IO.ByteBufferFactory.LoadFrom(Stream stream)
at MongoDB.Driver.Internal.MongoConnection.ReceiveMessage[TDocument](BsonBinaryReaderSettings readerSettings, IBsonSerializer serializer, IBsonSerializationOptions serializationOptions)
at MongoDB.Driver.Operations.QueryOperation`1.GetFirstBatch(IConnectionProvider connectionProvider)
at MongoDB.Driver.Operations.QueryOperation`1.Execute(IConnectionProvider connectionProvider)
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Sitecore.Analytics.Automation.Data.MongoDbAutomationProvider.GetAutomationDocuments(ID contactId, IReadOnlyCollection`1& stateDocuments, IReadOnlyCollection`1& transitionDocuments)
at Sitecore.Analytics.Automation.Data.MongoDbAutomationProvider.GetAutomationContext(ID contactId, DateTime timestampExclusive)
at Sitecore.Analytics.Automation.Data.AutomationProvider.GetAutomationContext(ID contactId)
at Sitecore.Analytics.Automation.Data.AutomationStateManager.Create(Contact contact)
at Sitecore.Analytics.Automation.ContactAutomationEvents.Run(Session session, PageEventItem pageEventDefinitionItem)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at (Object , Object[] )
at Sitecore.Pipelines.CorePipeline.Run(PipelineArgs args)
at Sitecore.Web.Application.RaiseSessionEndEvent(HttpApplication context)
Nested Exception
Exception: System.Net.Sockets.SocketException
Message: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Source: System at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)