I get this very common zuul error, when running spring cloud microservices locally.
If I specify microservice address in application.yml it works fine, so I am wondering why it doesn't discovery the service address properly without explicitly specifying it, especially that this is a new behaviour.
zuul:
routes:
my-service:
path: /myservice/**
#url: http://localhost:8081
Any hints what could have caused this?
Here is some part of stacktrace.
Caused by: java.lang.RuntimeException: java.net.SocketException: Permission denied: connect
at rx.exceptions.Exceptions.propagate(Exceptions.java:57)
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463)
at rx.observables.BlockingObservable.single(BlockingObservable.java:340)
at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClien t.java:112)
... 137 common frames omitted
Caused by: java.net.SocketException: Permission denied: connect
Thanks!
Related
I have checked that all relevant parameters are being parsed correctly and i am able to see them in my AWS ECS service. When the service is spun up i get the below error WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 60) IJ000604: Throwable while attempting to get a new connection: null: javax.resource.ResourceException: IJ031084: Unable to create connection which then follows with this error Caused by: org.postgresql.util.PSQLException: FATAL: database "keycloak" does not exist
I am unsure if the two are related. I validated relevant parameters and have tried to imitate the same issue locally.
Any help or advice would be aprpeciated
TYIA
I'm trying to run a dockerized WildFly 26 EE 9.1 preview on a AWS Fargate cluster but it fails to start with
[31m09:15:52,203 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 76) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "webservices")]): java.lang.RuntimeException: java.net.UnknownHostException: 7a0765d6977e48a98e3b16f7533674f7-3378193245: 7a0765d6977e48a98e3b16f7533674f7-3378193245: Name does not resolve
at org.jboss.as.webservices//org.jboss.as.webservices.dmr.WSSubsystemAdd.createServerConfig(WSSubsystemAdd.java:106)
at org.jboss.as.webservices//org.jboss.as.webservices.dmr.WSSubsystemAdd.performBoottime(WSSubsystemAdd.java:93)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractBoottimeAddStepHandler.performBoottime(AbstractBoottimeAddStepHandler.java:144)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractBoottimeAddStepHandler.performRuntime(AbstractBoottimeAddStepHandler.java:119)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:164)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1045)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:777)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:466)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:384)
at org.jbos...#2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jbos...#2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jbos...#2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jbos...#2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
at java.base/java.lang.Thread.run(Thread.java:831)
at org.jbos...#2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)
Caused by: java.net.UnknownHostException: 7a0765d6977e48a98e3b16f7533674f7-3378193245: 7a0765d6977e48a98e3b16f7533674f7-3378193245: Name does not resolve
at java.base/java.net.InetAddress.getLocalHost(InetAddress.java:1644)
at org.jboss...#3.3.3.Final//org.jboss.ws.common.management.AbstractServerConfig.setWebServiceHost(AbstractServerConfig.java:133)
at org.jboss.as.webservices//org.jboss.as.webservices.config.ServerConfigImpl.setWebServiceHost(ServerConfigImpl.java:97)
at org.jboss.as.webservices//org.jboss.as.webservices.dmr.WSSubsystemAdd.createServerConfig(WSSubsystemAdd.java:104)
... 14 more
Caused by: java.net.UnknownHostException: 7a0765d6977e48a98e3b16f7533674f7-3378193245: Name does not resolve
at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:932)
at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1517)
at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:851)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
at java.base/java.net.InetAddress.getLocalHost(InetAddress.java:1639)
The cluster is deployed in a private subnet but I have a security group that allows all outgoing traffic and all traffic is routed through a NAT Gateway. I recall using the same CloudFormation scripts for WildFly 24:ish and it worked fine so it shouldn't(tm) be a networking issue.
Is the host name generally supposed to be resolvable?
I’m not sure if this issue still persists on your side, but for me it was resolved as soon as I enabled "DNS Hostnames" in VPC ("DNS resolution" was enabled already). I have no idea why this option matters, but even with disabling public IP in Service defintion of ECS the wildfly threw the error. Since the option is enabled (which only seems to enable hostnames for public ips) it is working fine.
I have tested default existing jdbc source connector. It is working fine for DB. I want to create another oracle jdbc source connector for different DB. For that I have created new properties file with different DB details. Then, I ran the below command.
bin/connect-standalone etc/schema-registry/connect-avro-standalone.properties source-quickstart-sqlite1.properties
The Error:
[2020-06-20 09:43:46,751] ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectStandalone:130)
org.apache.kafka.connect.errors.ConnectException: Unable to initialize REST server
at org.apache.kafka.connect.runtime.rest.RestServer.initializeServer(RestServer.java:217)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:87)
Caused by: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8083
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:307)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.server.Server.doStart(Server.java:385)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.apache.kafka.connect.runtime.rest.RestServer.initializeServer(RestServer.java:215)
... 1 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
... 8 more
I have tried usuing rest.port:8089. It is also not working.
There is already an application running on you computer, who is attached (binded) to the port 8083. Probably, it's another Kafka Connect instance. Stop it before executing you command.
To find the app listening the port, execute lsof -i :8083 and you'll get the PID. You can then simply kill it, assuming it's all about local development of Konnect applications.
I have been trying to switch my application to use Google Cloud logging (aka Stackdriver). I have built a custom Logback appender which uses the Google Java Cloud Logging Library.
When I run my application in the IDE it logs correctly to Google Cloud Logging. When I run it inside a Kubernetes container on GKE, I get an Authentication Error. However in both scenarios I am use the same GCP Service Account.
I have even gone to the length of outputting the Key from the Google credentials object to prove they are the same in each environment. Which they are.
Does anyone have any ideas of what I might be missing? Is there any sort of context related settings that might cause authentication to fail from inside the container? Anyone got any ideas about what to try next?
Update: Here is the stack trace:
java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.cloud.logging.LoggingImpl.flush(LoggingImpl.java:579) at at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:560) at at uk.co.processflows.platform.util.StackdriverAppender.append(StackdriverAppender.kt:135) at at uk.co.processflows.platform.util.StackdriverAppender.append(StackdriverAppender.kt:18) at at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84) at at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51) at at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270) at at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257) at at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421) at at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383) at at ch.qos.logback.classic.Logger.error(Logger.java:538) at at uk.co.processflows.platform.pushnotification.PushNotificationClient.forward(PushNotificationClient.kt:58) at at uk.co.processflows.platform.push.PushMessagingConnectionHandlerImpl.broadcast(PushMessagingConnectionHandlerImpl.kt:257) at at uk.co.processflows.platform.push.PushMessagingConnectionHandlerImpl.broadcastMessageToTenant(PushMessagingConnectionHandlerImpl.kt:153) at at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at at java.base/java.lang.reflect.Method.invoke(Method.java:567) at at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50) at at com.sun.proxy.$Proxy69.broadcastMessageToTenant(Unknown Source) at at uk.co.processflows.platform.workflow.TaskLockControllerImpl$lockTask$2.run(TaskLockControllerImpl.kt:94) at at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at at java.base/java.lang.Thread.run(Thread.java:835) Caused by: java.util.concurrent.ExecutionException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552) at at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:431) at at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:97) at at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68) at at com.google.cloud.logging.LoggingImpl.flush(LoggingImpl.java:577) at ... 25 common frames omitted Caused by: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:190) at at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:184) at at com.google.api.core.ApiFutures$GaxFunctionToGuavaFunction.apply(ApiFutures.java:204) at at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:206) at at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:194) at at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:107) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) at at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) at at com.google.api.gax.rpc.BatchedFuture.setException(BatchedFuture.java:55) at at com.google.api.gax.rpc.BatchedRequestIssuer.sendResult(BatchedRequestIssuer.java:84) at at com.google.api.gax.rpc.BatchExecutor$1.onFailure(BatchExecutor.java:98) at at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) at at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:179) at at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135) at at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) at at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) at at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) at at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) at at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:515) at at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490) at at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:699) at at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397) at at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459) at at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) at at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546) at at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467) at at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584) at at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at ... 3 common frames omitted Caused by: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:73) at at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) at at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) at ... 30 common frames omitted Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at io.grpc.Status.asRuntimeException(Status.java:532) at ... 22 common frames omitted Caused by: java.lang.NoSuchMethodError: io.grpc.MethodDescriptor.getServiceName()Ljava/lang/String; at at io.grpc.auth.GoogleAuthLibraryCallCredentials.serviceUri(GoogleAuthLibraryCallCredentials.java:162) at at io.grpc.auth.GoogleAuthLibraryCallCredentials.applyRequestMetadata(GoogleAuthLibraryCallCredentials.java:103) at at io.grpc.CallCredentials2.applyRequestMetadata(CallCredentials2.java:58) at at io.grpc.internal.CallCredentialsApplyingTransportFactory$CallCredentialsApplyingTransport.newStream(CallCredentialsApplyingTransportFactory.java:107) at at io.grpc.internal.ForwardingConnectionClientTransport.newStream(ForwardingConnectionClientTransport.java:49) at at io.grpc.internal.InternalSubchannel$CallTracingTransport.newStream(InternalSubchannel.java:690) at at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:245) at at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1.start(CensusTracingModule.java:392) at at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1.start(CensusStatsModule.java:694) at at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:32) at at com.google.api.gax.grpc.GrpcHeaderInterceptor$1.start(GrpcHeaderInterceptor.java:94) at at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:310) at at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:282) at at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:191) at at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:58) at at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64) at at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:86) at at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63) at at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41) at at com.google.api.gax.tracing.TracedBatchingCallable.futureCall(TracedBatchingCallable.java:82) at at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126) at at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87) at at com.google.api.gax.rpc.BatchExecutor.processBatch(BatchExecutor.java:82) at at com.google.api.gax.rpc.BatchExecutor.processBatch(BatchExecutor.java:53) at at com.google.api.gax.batching.ThresholdBatcher.pushCurrentBatch(ThresholdBatcher.java:233) at at com.google.api.gax.batching.ThresholdBatcher$1.run(ThresholdBatcher.java:76) at ... 6 common frames omitted
Update 2:
I have just spotted deep inside the caused by hiarchy of the stack trace there is this message.
Caused by: java.lang.NoSuchMethodError: io.grpc.MethodDescriptor.getServiceName()Ljava/lang/String; at at io.grpc.auth.GoogleAuthLibraryCallCredentials.serviceUri(GoogleAuthLibraryCallCredentials.java:162)
That doesn't sound much like an authentication problem.
Update I did not see the stack trace when I originally answered this question, and it leads me to be believe your issue is not with authentication at all. My answer may be relevant to those seeing an authentication issue with GCP APIs in GKE, so I'll leave the original here.
Original If you're getting an authentication error, it's likely you're not using the service account key correctly. You can read this tutorial to learn about how it should be done. In short, you need to export the service account key as a JSON file and mount it as a secret.
Given your issue with using the library, you might want to write your logs directly. This doc on creating structured logs might be helpful.
It turns out there was a dependancy compatibility issue with the GRPC jars. Our application uses GRPC as does apparently the Google Could Logging library. Changing our app to use the older version of GRPC that the Google cloud logging library was built against fixed the issue.
It appears when running in the container the jar files are loaded in a different order than running locally. Hence why this only happened inside Kubernetes.
I have an integration test that injects a rest client into the test class. When that rest client calls a rest service being hosted by the same pax exam test container everything stops for about a minute until I get a long list of Socket Read Timeout exceptions and connection refused exceptions. If I inject a rest client that calls a rest service in another container everything works fine. It almost seems like the rest service isn't given a thread to execute in. If I start up the pax exam container manually I can verify that the rest service has started correctly. And if I inject the OSGi service into the test class everything works correctly. Using PaxExam 3.6 and Apache ServiceMix (4.5.3)
Stack Trace:
2015-04-22 17:10:02,211 | WARN | ion(1)-127.0.0.1 | PhaseInterceptorChain | 123 - org.apache.cxf.cxf-api - 2.6.6 | Interceptor for {http://rest.mediation.velocity.randomness.com/}mediationService has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message.
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)[123:org.apache.cxf.cxf-api:2.6.6]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)[123:org.apache.cxf.cxf-api:2.6.6]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:530)[139:org.apache.cxf.cxf-rt-frontend-jaxrs:2.6.6]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:205)[139:org.apache.cxf.cxf-rt-frontend-jaxrs:2.6.6]
at com.sun.proxy.$Proxy156.getmediations(Unknown Source)
at com.randomness.velocity.mediation.rest.client.mediationClient.getmediations(mediationClient.java:90)[282:mediation-rest-client:3.1.0.SNAPSHOT]
at com.randomness.velocity.mediation.rest.client.mediationClient.getmediations(mediationClient.java:97)[282:mediation-rest-client:3.1.0.SNAPSHOT]
at com.randomness.velocity.itest.client_management.DataCleanupTests.verifyvelocityClientManagementStarts(DataCleanupTests.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_75]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_75]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_75]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_75]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.ops4j.pax.exam.invoker.junit.internal.ContainerTestRunner.runChild(ContainerTestRunner.java:67)
at org.ops4j.pax.exam.invoker.junit.internal.ContainerTestRunner.runChild(ContainerTestRunner.java:37)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.junit.runner.JUnitCore.run(JUnitCore.java:138)[313:org.ops4j.pax.tipi.junit:4.11.0.1]
at org.ops4j.pax.exam.invoker.junit.internal.JUnitProbeInvoker.invokeViaJUnit(JUnitProbeInvoker.java:124)
at org.ops4j.pax.exam.invoker.junit.internal.JUnitProbeInvoker.findAndInvoke(JUnitProbeInvoker.java:97)
at org.ops4j.pax.exam.invoker.junit.internal.JUnitProbeInvoker.call(JUnitProbeInvoker.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_75]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_75]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_75]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_75]
at org.ops4j.pax.exam.rbc.internal.RemoteBundleContextImpl.remoteCall(RemoteBundleContextImpl.java:80)[311:org.ops4j.pax.exam.rbc:3.6.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_75]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_75]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_75]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_75]
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)[:1.7.0_75]
at sun.rmi.transport.Transport$2.run(Transport.java:202)[:1.7.0_75]
at sun.rmi.transport.Transport$2.run(Transport.java:199)[:1.7.0_75]
at java.security.AccessController.doPrivileged(Native Method)[:1.7.0_75]
at sun.rmi.transport.Transport.serviceCall(Transport.java:198)[:1.7.0_75]
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567)[:1.7.0_75]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)[:1.7.0_75]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619)[:1.7.0_75]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684)[:1.7.0_75]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681)[:1.7.0_75]
at java.security.AccessController.doPrivileged(Native Method)[:1.7.0_75]
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681)[:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_75]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_75]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_75]
Caused by: java.net.SocketTimeoutException: SocketTimeoutException invoking https://127.0.0.1:7103/mediation/patient/ff8081814ce1299a014ce142cb410065?materialization=Full: Read timed out
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)[:1.7.0_75]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)[:1.7.0_75]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)[:1.7.0_75]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)[:1.7.0_75]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1469)[135:org.apache.cxf.cxf-rt-transports-http:2.6.6]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1454)[135:org.apache.cxf.cxf-rt-transports-http:2.6.6]
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)[123:org.apache.cxf.cxf-api:2.6.6]
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:659)[135:org.apache.cxf.cxf-rt-transports-http:2.6.6]
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)[123:org.apache.cxf.cxf-api:2.6.6]
... 54 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)[:1.7.0_75]
at java.net.SocketInputStream.read(SocketInputStream.java:152)[:1.7.0_75]
at java.net.SocketInputStream.read(SocketInputStream.java:122)[:1.7.0_75]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)[:1.7.0_75]
at sun.security.ssl.InputRecord.read(InputRecord.java:480)[:1.7.0_75]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934)[:1.7.0_75]
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)[:1.7.0_75]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)[:1.7.0_75]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)[:1.7.0_75]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)[:1.7.0_75]
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)[:1.7.0_75]
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)[:1.7.0_75]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)[:1.7.0_75]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1324)[:1.7.0_75]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)[:1.7.0_75]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)[:1.7.0_75]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1606)[135:org.apache.cxf.cxf-rt-transports-http:2.6.6]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1532)[135:org.apache.cxf.cxf-rt-transports-http:2.6.6]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1440)[135:org.apache.cxf.cxf-rt-transports-http:2.6.6]
I figured out what was going on. I added an osgi:service to a spring config in the integration test project that exposed the rest client as an osgi service so I could inject the client into the test class. By doing that I created a second service in the osgi container that was advertised via the same interface that the rest service was using to look up the real osgi service implementation. So the rest service was actually getting the rest client injected into it instead of the osgi service impl, this created an infinite loop that finally crashed out and threw tons of connection refused and socket read timeout exceptions to my console. Good stuff