Zuul defaulting hystrix isolation strategy to SEMAPHORE - spring-cloud
This is in continuation to my previous question where in my hystrix dashboard the Thread Pool section keeps on loading:- Hystrix Dashboard with Turbine issue
Why is zuul forcing the isolation strategy for hystrix to be SEMAPHORE. I tried adding the configuration to my application.yml but still it ignores that and uses SEMAPHORE. Is there something that I am missing here. Or how would i change it from SEMAPHORE to THREAD.
Application.yml
info:
component: Zuul Server
eureka:
client:
service-url:
defaultZone: http://localhost:8761/eureka/
endpoints:
restart:
enabled: true
shutdown:
enabled: true
health:
sensitive: false
server:
port: 8006
zuul:
routes:
UserCards-V1:
path: /user/v1/accountholders/*/cards/**
service-id: usercards-v1
strip-prefix: false
UserTransactions-V1_1:
path: /user/v1/accountholders/*/transactions
service-id: usertransactions-v1
strip-prefix: false
UserTransactions-V1_2:
path: /user/v1/accountholders/*/accounts/*/transactions
service-id: usertransactions-v1
strip-prefix: false
UserAccounts-V1:
path: /user/v1/accountholders/*/accounts/**
service-id: useraccounts-v1
strip-prefix: false
UserCardholders-V1:
path: /user/v1/accountholders/**
service-id: usercardholders-v1
strip-prefix: false
hystrix:
command.default.execution.isolation.strategy: THREAD
command.default.execution.isolation.thread.timeoutInMilliseconds: 60000
command.default.execution.timeout.enabled: false
command.default.fallback.enabled: false
threadpool.default.coreSize: 20
ribbon:
ConnectTimeout: 3000
ReadTimeout: 60000
turbine:
clusterNameExpression: zuul
Hystrix.stream
data: {"type":"HystrixCommand","name":"usercardholders-v1RibbonCommand","group":"RibbonCommand","currentTime":1451411772021,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountBadRequests":0,"rollingCountCollapsedRequests":0,"rollingCountEmit":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountEmit":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"rollingMaxConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"SEMAPHORE","propertyValue_executionIsolationThreadTimeoutInMilliseconds":60000,"propertyValue_executionTimeoutInMilliseconds":60000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":100,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1,"threadPool":"RibbonCommand"}
data: {"type":"HystrixCommand","name":"usercards-v1RibbonCommand","group":"RibbonCommand","currentTime":1451411772522,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":1,"rollingCountBadRequests":0,"rollingCountCollapsedRequests":0,"rollingCountEmit":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountEmit":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":1,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"rollingMaxConcurrentExecutionCount":1,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"SEMAPHORE","propertyValue_executionIsolationThreadTimeoutInMilliseconds":60000,"propertyValue_executionTimeoutInMilliseconds":60000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":100,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1,"threadPool":"RibbonCommand"}
zuul.ribbonIsolationStrategy=THREAD
You can use above command to force the ZUUL to use the THREAD strategy
Why is zuul forcing the isolation strategy for hystrix to be SEMAPHORE
See https://github.com/spring-cloud/spring-cloud-netflix/blob/f3deb04521c8c0a0b6d9923b0b37d8673cb7fa0b/spring-cloud-netflix-core/src/main/java/org/springframework/cloud/netflix/zuul/filters/route/support/AbstractRibbonCommand.java#L85.
// we want to default to semaphore-isolation since this wraps
// 2 others commands that are already thread isolated
Related
Netflix Eureka and Spring Boot Admin in one application, not showing clients
i'm trying to add Spring Boot Admin to my Eureka Server, but it looks like there is something wrong somewhere. I'm using Spring Boot version 2.1.5.RELEASE (starter parent) spring-cloud-starter-netflix-eureka-server spring-boot-admin-starter-server My Main Class is annotated with #SpringBootApplication #EnableAdminServer #EnableEurekaServer And this is the application.yml spring: application: name: eureka-server boot: admin: context-path: /admin security: basic: enabled: false management: security: enabled: false eureka: client: register-with-eureka: false fetch-registry: true instance: leaseRenewalIntervalInSeconds: 15 After i started the server, i try to launch a microservice The application.yaml go the client is the following: eureka: instance: leaseRenewalIntervalInSeconds: 10 leaseExpirationDurationInSeconds: 5 client: service-url: default-zone: http://localhost:8761/eureka healthcheck: enabled: true lease: duration: 5 management: endpoints: web: exposure: include: '*' security: enabled: false but i see it in eureka as down and i cannot see anything in SBA
Spring JPA with Hikari not releasing connection
I have a spring boot application which has a controller. When I try to hit an API simultaneously (35 parrallel hits) it comes to the controller but there I am using findOne of the crudrepository which checks postgres for some data. The findOne functions returns in milliseconds but the next API doesn't get hold of postgres connection. I am using HikariCp as the connection pooler with following settings hikari: idleTimeout: 10000 connectionTimeout: 60000 maximumPoolSize: 30 minimumIdle: 2 poolName: gor-srms leakDetectionThreshold: 10000 In short at any point of time only 30 API's are working in parallel. Once the API is finished then only next one gets postgres connection. Why is postgres connection not getting used for other API's since the findOne call only takes about millisecs to complete? Adding the complete application properties: server: port: ${SERVER_PORT:${PORT:8093}} contextPath: ${mdm.service.contextPath:/wms-masterdata} spring: application: name: ${mdm.service.name:mdm-service} jmx: enabled: false profiles: active: local cloud: config: discovery: enabled: false serviceId: config-server jpa: hibernate: ddl-auto: validate http: multipart: max-file-size: 20MB max-request-size: 20MB discovery: enabled: true eureka: client: enabled: ${discovery.enabled:true} serviceUrl: defaultZone: ${discovery.url:${REGISTRY_SERVICE_URL:http://localhost:8761}}/eureka/ instance: metadataMap: contextPath: ${server.contextPath} db: driver: org.postgresql.Driver url: jdbc:postgresql://${database.ip:localhost}:5432/wms_masterdata username: postgres password: postgres hikari: idleTimeout: 10000 connectionTimeout: 60000 maximumPoolSize: 10 minimumIdle: 2 poolName: gor-mdm leakDetectionThreshold: 60000 hibernate: unit_name: wms_masterdata show_sql: false generate_ddl: false entitymanager: packagesToScan: com.gor.platform.mdm.service.model butler: url: https://192.168.8.116 management: security: enabled: false health: db: enabled: false security: basic: enabled: false flyway: baseline-on-migrate: true enabled: true ignore-ignored-migrations: true
spring.datasource.TYPE=com.zaxxer.hikari.HikariDataSource spring.datasource.hikari.minimumIdle=5 spring.datasource.hikari.maximumPoolSize=20 spring.datasource.hikari.idleTimeout=30000 spring.datasource.hikari.poolName=SpringBootJPAHikariCP spring.datasource.hikari.maxLifetime=2000000 spring.datasource.hikari.testWhileIdle=true spring.datasource.hikari.validationQuery=SELECT 1 spring.datasource.hikari.connectionTimeout=30000 Having this kind of setting help to have efficient connection pooling and allowed to server more traffic.
I was finally able to solve this by setting spring.jpa.open-in-view=false. Be aware that this might lead to LazyInitialization Exception
Backend services isolation in Zuul
We are planning using Zuul in production as an API gateway. I am not sure the behavior of Zuul in this specific scenario. Suppose there are two back-end services A and B sitting behind Zuul. Service A is a slow one but with a lot of traffic. What will happen to the clients visiting Service B through Zuul? On the clients' side, Service B will be unavailable because Service A slow down the Zuul so there is no resource to handle the request for Service B. I'm not sure about this, and any advice or experience will be much appreciated.
You can configure the individual endpoints to use their own thread pools and have their own timeouts. zuul: ribbonIsolationStrategy: THREAD threadPool: useSeparateThreadPools: true host: connect-timeout-millis: 60000 socket-timeout-millis: 60000 routes: examples2: path: /something/** serviceId: examples stripPrefix: true services: path: /services/** serviceId: services stripPrefix: false sensitiveHeaders: true auth: path: /oauth/** serviceId: saapi-auth-server stripPrefix: false sensitiveHeaders: true hystrix: command: default: execution: timeout: enabled: false isolation: thread: timeoutInMilliseconds: 60000 auth: ribbon: ReadTimeout: 60000 ConnectTimeout: 60000 services: ribbon: ReadTimeout: 60000 ConnectTimeout: 60000 Here is an example config from one of our gateways that should get you started.
Springcloud: Zuul + Eureka not working
I spent all day trying to make Zuul + Eureka work together, but I hasn't been lucky with that. Zuul alone, without Eureka, works fine. I've tried a lot of different configurations like this one, which says that all I have to do is turn Zull in an Eureka client (by using #EnableDiscoveryClient at root application class). My setup is very simple: it consists of a service:8080 (service 1), Zuul:9000 and Eureka:8761. All the approaches I've tried using Eureka gave me the same error when I tried to access service 1 using Zull (http://localhost:9000/service1 in this case): com.netflix.zuul.exception.ZuulException: Forwarding error (...) com.netflix.client.ClientException: Load balancer does not have available server for client: service1 Service 1 is working fine (I can access it directly from the browser address bar http://localhost:8080) and Eureka shows both Zull and Service 1 correctly registered: The apps are configured like: Service 1 (bootstrap.yml): spring: application: name: service1 Service 1 (application.yml): eureka: instance: leaseRenewalIntervalInSeconds: 1 leaseExpirationDurationInSeconds: 2 client: serviceUrl: defaultZone: http://127.0.0.1:8761/eureka/ Zuul: ribbon: eureka: enabled: false eureka: instance: leaseRenewalIntervalInSeconds: 1 leaseExpirationDurationInSeconds: 2 client: serviceUrl: defaultZone: http://127.0.0.1:8761/eureka/ healthcheck: enabled: true server: port: 9000 Eureka: server: port: 8761 eureka: client: register-with-eureka: false fetch-registry: false logging: level: com: netflix: eureka: OFF discovery: OFF Zuul Annotations at root application class: #EnableZuulProxy #EnableDiscoveryClient Am I missing any important point?
Eureka - Can't get a response - Can't contact any eureka nodes
I have a Eureka server running on port 8761 (localhost:8761/eureka) and I have a Zuul application that I would like to register with eureka, but I keep getting the same error: Can't get a response from http://localhost:8761/eurekaapps/ZUULSERVER Can't contact any eureka nodes - possibly a security group issue? java.lang.RuntimeException: Bad status: 404 at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1155) at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1060) at com.netflix.discovery.DiscoveryClient.register(DiscoveryClient.java:606) at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1596) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Here is my application.yml for the zuul application: info: component: Zuul Server eureka: server: enabled: true client: serviceUrl: defaultZone: http://localhost:8761/eureka registerWithEureka: true fetchRegistry: false endpoints: restart: enabled: true shutdown: enabled: true health: sensitive: false zuul: route: discovery: url: http://localhost:9000/polaris/discovery path: /polaris/discovery/** sla: url: http://localhost:9000/polaris/sla path: /polaris/sla/** stores: /stores/** customers: /customers/** #remove when spring-boot 1.2.1 is out security: basic: enabled: false management: security: enabled: false server: port: 8765 logging: level: ROOT: INFO org.springframework.web: DEBUG Did I just mess up one of these values and so zuul is unable to see the eureka server? It seems like it is not registering properly. Maybe I missed a crucial parameter in the application.yml file?
The clue is "eurekaapps" in the error. You need a trailing "/" in the service URL: eureka: client: serviceUrl: defaultZone: http://localhost:8761/eureka/ (which is the default anyway I think).