Zuul defaulting hystrix isolation strategy to SEMAPHORE - spring-cloud

This is in continuation to my previous question where in my hystrix dashboard the Thread Pool section keeps on loading:- Hystrix Dashboard with Turbine issue
Why is zuul forcing the isolation strategy for hystrix to be SEMAPHORE. I tried adding the configuration to my application.yml but still it ignores that and uses SEMAPHORE. Is there something that I am missing here. Or how would i change it from SEMAPHORE to THREAD.
Application.yml
info:
component: Zuul Server
eureka:
client:
service-url:
defaultZone: http://localhost:8761/eureka/
endpoints:
restart:
enabled: true
shutdown:
enabled: true
health:
sensitive: false
server:
port: 8006
zuul:
routes:
UserCards-V1:
path: /user/v1/accountholders/*/cards/**
service-id: usercards-v1
strip-prefix: false
UserTransactions-V1_1:
path: /user/v1/accountholders/*/transactions
service-id: usertransactions-v1
strip-prefix: false
UserTransactions-V1_2:
path: /user/v1/accountholders/*/accounts/*/transactions
service-id: usertransactions-v1
strip-prefix: false
UserAccounts-V1:
path: /user/v1/accountholders/*/accounts/**
service-id: useraccounts-v1
strip-prefix: false
UserCardholders-V1:
path: /user/v1/accountholders/**
service-id: usercardholders-v1
strip-prefix: false
hystrix:
command.default.execution.isolation.strategy: THREAD
command.default.execution.isolation.thread.timeoutInMilliseconds: 60000
command.default.execution.timeout.enabled: false
command.default.fallback.enabled: false
threadpool.default.coreSize: 20
ribbon:
ConnectTimeout: 3000
ReadTimeout: 60000
turbine:
clusterNameExpression: zuul
Hystrix.stream
data: {"type":"HystrixCommand","name":"usercardholders-v1RibbonCommand","group":"RibbonCommand","currentTime":1451411772021,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountBadRequests":0,"rollingCountCollapsedRequests":0,"rollingCountEmit":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountEmit":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"rollingMaxConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"SEMAPHORE","propertyValue_executionIsolationThreadTimeoutInMilliseconds":60000,"propertyValue_executionTimeoutInMilliseconds":60000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":100,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1,"threadPool":"RibbonCommand"}
data: {"type":"HystrixCommand","name":"usercards-v1RibbonCommand","group":"RibbonCommand","currentTime":1451411772522,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":1,"rollingCountBadRequests":0,"rollingCountCollapsedRequests":0,"rollingCountEmit":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountEmit":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":1,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"rollingMaxConcurrentExecutionCount":1,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"SEMAPHORE","propertyValue_executionIsolationThreadTimeoutInMilliseconds":60000,"propertyValue_executionTimeoutInMilliseconds":60000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":100,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1,"threadPool":"RibbonCommand"}

zuul.ribbonIsolationStrategy=THREAD
You can use above command to force the ZUUL to use the THREAD strategy

Why is zuul forcing the isolation strategy for hystrix to be SEMAPHORE
See https://github.com/spring-cloud/spring-cloud-netflix/blob/f3deb04521c8c0a0b6d9923b0b37d8673cb7fa0b/spring-cloud-netflix-core/src/main/java/org/springframework/cloud/netflix/zuul/filters/route/support/AbstractRibbonCommand.java#L85.
// we want to default to semaphore-isolation since this wraps
// 2 others commands that are already thread isolated

Related

Netflix Eureka and Spring Boot Admin in one application, not showing clients

i'm trying to add Spring Boot Admin to my Eureka Server, but it looks like there is something wrong somewhere.
I'm using
Spring Boot version 2.1.5.RELEASE (starter parent)
spring-cloud-starter-netflix-eureka-server
spring-boot-admin-starter-server
My Main Class is annotated with
#SpringBootApplication
#EnableAdminServer
#EnableEurekaServer
And this is the application.yml
spring:
application:
name: eureka-server
boot:
admin:
context-path: /admin
security:
basic:
enabled: false
management:
security:
enabled: false
eureka:
client:
register-with-eureka: false
fetch-registry: true
instance:
leaseRenewalIntervalInSeconds: 15
After i started the server, i try to launch a microservice
The application.yaml go the client is the following:
eureka:
instance:
leaseRenewalIntervalInSeconds: 10
leaseExpirationDurationInSeconds: 5
client:
service-url:
default-zone: http://localhost:8761/eureka
healthcheck:
enabled: true
lease:
duration: 5
management:
endpoints:
web:
exposure:
include: '*'
security:
enabled: false
but i see it in eureka as down
and i cannot see anything in SBA

Spring JPA with Hikari not releasing connection

I have a spring boot application which has a controller. When I try to hit an API simultaneously (35 parrallel hits) it comes to the controller but there I am using findOne of the crudrepository which checks postgres for some data.
The findOne functions returns in milliseconds but the next API doesn't get hold of postgres connection.
I am using HikariCp as the connection pooler with following settings
hikari:
idleTimeout: 10000
connectionTimeout: 60000
maximumPoolSize: 30
minimumIdle: 2
poolName: gor-srms
leakDetectionThreshold: 10000
In short at any point of time only 30 API's are working in parallel. Once the API is finished then only next one gets postgres connection.
Why is postgres connection not getting used for other API's since the findOne call only takes about millisecs to complete?
Adding the complete application properties:
server:
port: ${SERVER_PORT:${PORT:8093}}
contextPath: ${mdm.service.contextPath:/wms-masterdata}
spring:
application:
name: ${mdm.service.name:mdm-service}
jmx:
enabled: false
profiles:
active: local
cloud:
config:
discovery:
enabled: false
serviceId: config-server
jpa:
hibernate:
ddl-auto: validate
http:
multipart:
max-file-size: 20MB
max-request-size: 20MB
discovery:
enabled: true
eureka:
client:
enabled: ${discovery.enabled:true}
serviceUrl:
defaultZone: ${discovery.url:${REGISTRY_SERVICE_URL:http://localhost:8761}}/eureka/
instance:
metadataMap:
contextPath: ${server.contextPath}
db:
driver: org.postgresql.Driver
url: jdbc:postgresql://${database.ip:localhost}:5432/wms_masterdata
username: postgres
password: postgres
hikari:
idleTimeout: 10000
connectionTimeout: 60000
maximumPoolSize: 10
minimumIdle: 2
poolName: gor-mdm
leakDetectionThreshold: 60000
hibernate:
unit_name: wms_masterdata
show_sql: false
generate_ddl: false
entitymanager:
packagesToScan: com.gor.platform.mdm.service.model
butler:
url: https://192.168.8.116
management:
security:
enabled: false
health:
db:
enabled: false
security:
basic:
enabled: false
flyway:
baseline-on-migrate: true
enabled: true
ignore-ignored-migrations: true
spring.datasource.TYPE=com.zaxxer.hikari.HikariDataSource
spring.datasource.hikari.minimumIdle=5
spring.datasource.hikari.maximumPoolSize=20
spring.datasource.hikari.idleTimeout=30000
spring.datasource.hikari.poolName=SpringBootJPAHikariCP
spring.datasource.hikari.maxLifetime=2000000
spring.datasource.hikari.testWhileIdle=true
spring.datasource.hikari.validationQuery=SELECT 1
spring.datasource.hikari.connectionTimeout=30000
Having this kind of setting help to have efficient connection pooling and allowed to server more traffic.
I was finally able to solve this by setting spring.jpa.open-in-view=false. Be aware that this might lead to LazyInitialization Exception

Backend services isolation in Zuul

We are planning using Zuul in production as an API gateway. I am not sure the behavior of Zuul in this specific scenario.
Suppose there are two back-end services A and B sitting behind Zuul. Service A is a slow one but with a lot of traffic. What will happen to the clients visiting Service B through Zuul?
On the clients' side, Service B will be unavailable because Service A slow down the Zuul so there is no resource to handle the request for Service B.
I'm not sure about this, and any advice or experience will be much appreciated.
You can configure the individual endpoints to use their own thread pools and have their own timeouts.
zuul:
ribbonIsolationStrategy: THREAD
threadPool:
useSeparateThreadPools: true
host:
connect-timeout-millis: 60000
socket-timeout-millis: 60000
routes:
examples2:
path: /something/**
serviceId: examples
stripPrefix: true
services:
path: /services/**
serviceId: services
stripPrefix: false
sensitiveHeaders: true
auth:
path: /oauth/**
serviceId: saapi-auth-server
stripPrefix: false
sensitiveHeaders: true
hystrix:
command:
default:
execution:
timeout:
enabled: false
isolation:
thread:
timeoutInMilliseconds: 60000
auth:
ribbon:
ReadTimeout: 60000
ConnectTimeout: 60000
services:
ribbon:
ReadTimeout: 60000
ConnectTimeout: 60000
Here is an example config from one of our gateways that should get you started.

Springcloud: Zuul + Eureka not working

I spent all day trying to make Zuul + Eureka work together, but I hasn't been lucky with that. Zuul alone, without Eureka, works fine. I've tried a lot of different configurations like this one, which says that all I have to do is turn Zull in an Eureka client (by using #EnableDiscoveryClient at root application class).
My setup is very simple: it consists of a service:8080 (service 1), Zuul:9000 and Eureka:8761.
All the approaches I've tried using Eureka gave me the same error when I tried to access service 1 using Zull (http://localhost:9000/service1 in this case):
com.netflix.zuul.exception.ZuulException: Forwarding error
(...)
com.netflix.client.ClientException: Load balancer does not have available server for client: service1
Service 1 is working fine (I can access it directly from the browser address bar http://localhost:8080) and Eureka shows both Zull and Service 1 correctly registered:
The apps are configured like:
Service 1 (bootstrap.yml):
spring:
application:
name: service1
Service 1 (application.yml):
eureka:
instance:
leaseRenewalIntervalInSeconds: 1
leaseExpirationDurationInSeconds: 2
client:
serviceUrl:
defaultZone: http://127.0.0.1:8761/eureka/
Zuul:
ribbon:
eureka:
enabled: false
eureka:
instance:
leaseRenewalIntervalInSeconds: 1
leaseExpirationDurationInSeconds: 2
client:
serviceUrl:
defaultZone: http://127.0.0.1:8761/eureka/
healthcheck:
enabled: true
server:
port: 9000
Eureka:
server:
port: 8761
eureka:
client:
register-with-eureka: false
fetch-registry: false
logging:
level:
com:
netflix:
eureka: OFF
discovery: OFF
Zuul Annotations at root application class:
#EnableZuulProxy
#EnableDiscoveryClient
Am I missing any important point?

Eureka - Can't get a response - Can't contact any eureka nodes

I have a Eureka server running on port 8761 (localhost:8761/eureka) and I have a Zuul application that I would like to register with eureka, but I keep getting the same error:
Can't get a response from http://localhost:8761/eurekaapps/ZUULSERVER
Can't contact any eureka nodes - possibly a security group issue?
java.lang.RuntimeException: Bad status: 404
at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1155)
at com.netflix.discovery.DiscoveryClient.makeRemoteCall(DiscoveryClient.java:1060)
at com.netflix.discovery.DiscoveryClient.register(DiscoveryClient.java:606)
at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1596)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Here is my application.yml for the zuul application:
info:
component: Zuul Server
eureka:
server:
enabled: true
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka
registerWithEureka: true
fetchRegistry: false
endpoints:
restart:
enabled: true
shutdown:
enabled: true
health:
sensitive: false
zuul:
route:
discovery:
url: http://localhost:9000/polaris/discovery
path: /polaris/discovery/**
sla:
url: http://localhost:9000/polaris/sla
path: /polaris/sla/**
stores: /stores/**
customers: /customers/**
#remove when spring-boot 1.2.1 is out
security:
basic:
enabled: false
management:
security:
enabled:
false
server:
port: 8765
logging:
level:
ROOT: INFO
org.springframework.web: DEBUG
Did I just mess up one of these values and so zuul is unable to see the eureka server? It seems like it is not registering properly. Maybe I missed a crucial parameter in the application.yml file?
The clue is "eurekaapps" in the error. You need a trailing "/" in the service URL:
eureka:
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka/
(which is the default anyway I think).