Setting up Eureka for non-AWS datacenter - spring-cloud

I have setup Eureka with 2 peers, after about 5 minutes I see this message appear in the UI:
EMERGENCY! EUREKA MAY BE INCORRECTLY CLAIMING INSTANCES ARE UP WHEN THEY'RE NOT. RENEWALS ARE LESSER THAN THRESHOLD AND HENCE THE INSTANCES ARE NOT BEING EXPIRED JUST TO BE SAFE.
When I look at the output from one of the running instances of Eureka, I see this:
2015-11-11 14:46:47.276 INFO 32748 --- [pool-5-thread-1] com.netflix.discovery.DiscoveryClient : The response status is 200
2015-11-11 14:56:17.427 WARN 32748 --- [eerNodesUpdater] c.n.eureka.cluster.PeerEurekaNodes : The replica size seems to be empty. Check the route 53 DNS Registry
If I'm understanding this correctly, Eureka thinks it is configured at AWS. I'm unable to find anything in the application.yml to indicate this.
Any ideas on what I'm missing in the config?
server:
port: 80
eureka:
datacenter: boston
instance:
hostname: eureka03.domain.com
client:
serviceUrl:
defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/
---
eureka:
instance:
hostname: eureka01.domain.com
client:
serviceUrl:
defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/

I found the issue with my application.yml to get peers setup. Here's the proper application.yml if anyone runs into the same issue.
application.yml on peer1
server:
port: 80
eureka:
datacenter: boston
instance:
hostname: eureka01
client:
serviceUrl:
defaultZone: http://eureka03/eureka/
application.yml on peer2
server:
port: 80
eureka:
datacenter: boston
instance:
hostname: eureka03
client:
serviceUrl:
defaultZone: http://eureka01/eureka/

It may be that you are missing instanceId which helps eureka to identifies nodes:
eureka:
instance:
hostname: app56.test.online.eniro
metadataMap:
instanceId: EUREKA01_${spring.application.name}:${spring.application.instance_id:${random.value}}
client:
serviceUrl:
defaultZone: http:///eureka01/eureka/

After about 5 minutes, I see this in the web ui. EMERGENCY! EUREKA MAY BE INCORRECTLY CLAIMING INSTANCES ARE UP WHEN THEY'RE NOT. RENEWALS ARE LESSER THAN THRESHOLD AND HENCE THE INSTANCES ARE NOT BEING EXPIRED JUST TO BE SAFE.
I have also verified the peers can talk to each other. Here's what I see in the eureka output.
2015-11-13 09:08:39.928 WARN 860 --- [p-nio-80-exec-2] c.n.e.registry.AbstractInstanceRegistry : DS: Registry: lease doesn't exist, registering resource: EUREKA - eureka03:eureka:80
2015-11-13 09:08:39.928 WARN 860 --- [p-nio-80-exec-2] c.n.eureka.resources.InstanceResource : Not Found (Renew): EUREKA - eureka03:eureka:80
2015-11-13 09:08:40.451 INFO 860 --- [p-nio-80-exec-4] c.n.e.registry.AbstractInstanceRegistry : Registered instance EUREKA/eureka03:eureka:80 with status UP (replication=true)
2015-11-13 09:08:43.066 WARN 860 --- [ka01.ctct.net-6] c.n.eureka.cluster.ReplicationTask : The replication of task EUREKA/eureka01:eureka:80:Heartbeat#eureka01 failed with response code 404
2015-11-13 09:08:43.066 WARN 860 --- [ka01.ctct.net-6] c.netflix.eureka.cluster.PeerEurekaNode : EUREKA/eureka01:eureka:80:Heartbeat#eureka01: missing entry.
2015-11-13 09:08:43.066 WARN 860 --- [ka01.ctct.net-6] c.netflix.eureka.cluster.PeerEurekaNode : EUREKA/eureka01:eureka:80:Heartbeat#p2-dev-eureka01.ctct.net: cannot find instance id eureka01:eureka:80 and hence replicating the instance with status UP
Here's my application.yml for the 2 peers
peer1
server:
port: 80
eureka:
datacenter: bedford
instance:
hostname: eureka01
metadataMap:
instanceId: EUREKA01_${spring.application.name}:${spring.application.instance_id:${random.value}}
client:
serviceUrl:
defaultZone: http://eureka03/eureka/
peer2
server:
port: 80
eureka:
datacenter: bedford
instance:
hostname: eureka03
metadataMap:
instanceId: EUREKA03_${spring.application.name}:${spring.application.instance_id:${random.value}}
client:
serviceUrl:
defaultZone: http://eureka01/eureka/

I was trying to run the 3 eureka servers on my localhost, however the hostname being localhost for all 3 caused the above noted error. The error was resolved When I changed it by creating hostname entries in hosts file

Related

Spring Eureka cluster replication issues [version eureka-Brixton.SR1]

I am trying to setup a eureka cluster which comprise of there servers. (my setup is on local machine)
Configurations for each eureka server as below:
server1:
server:
port: 8764
eureka:
instance:
metadataMap:
instanceId: ${spring.application.name}:${spring.application.instance_id:${random.value}}
client:
registerWithEureka: true
fetchRegistry: true
# availabilityZones: zone1,zone1,zone1
server:
waitTimeInMsWhenSyncEmpty: 0
serviceUrl:
defaultZone: http://localhost:8762/eureka/,http://localhost:8763/eureka/,http://localhost:8764/eureka/
server2:
server:
port: 8762
eureka:
instance:
metadataMap:
instanceId: ${spring.application.name}:${spring.application.instance_id:${random.value}}
client:
registerWithEureka: true
fetchRegistry: true
server:
waitTimeInMsWhenSyncEmpty: 0
serviceUrl:
defaultZone: http://localhost:8762/eureka/,http://localhost:8763/eureka/,http://localhost:8764/eureka/
server3:
server:
port: 8763
eureka:
instance:
metadataMap:
instanceId: ${spring.application.name}:${spring.application.instance_id:${random.value}}
client:
registerWithEureka: true
fetchRegistry: true
server:
waitTimeInMsWhenSyncEmpty: 0
serviceUrl:
defaultZone: http://localhost:8762/eureka/,http://localhost:8763/eureka/,http://localhost:8764/eureka/
each server is peering to each others, so registration info will be replicated among servsers.
But, when I checked the logs server, it always throw exception as beblow:
I also checked dashboard of eureka server and it shown as below:
I dont know why the registered replicas (http://localhost:8761/eureka/) did not change even I have configured
serviceUrl:
defaultZone: http://localhost:8762/eureka/,http://localhost:8763/eureka/,http://localhost:8764/eureka/
I have setup a micro services and register to the eureka cluster, but for somehow the service info only is registered on one of eureak server, I am expecting that info also replicated to others as well
is there anyone suffering from same issue, I have been trying searching a lot but could not find the right answer for my case.
High appreicate for any advice?
I have found the issues. I have put serviceUrl in wrong section (server).
it should be in the client part.
server:
port: 8763
eureka:
instance:
metadataMap:
instanceId: ${spring.application.name}:${spring.application.instance_id:${random.value}}
client:
registerWithEureka: false
fetchRegistry: false
serviceUrl:
defaultZone: http://localhost:8762/eureka/,http://localhost:8763/eureka/,http://localhost:876
server:
waitTimeInMsWhenSyncEmpty: 0

Springcloud: Zuul + Eureka not working

I spent all day trying to make Zuul + Eureka work together, but I hasn't been lucky with that. Zuul alone, without Eureka, works fine. I've tried a lot of different configurations like this one, which says that all I have to do is turn Zull in an Eureka client (by using #EnableDiscoveryClient at root application class).
My setup is very simple: it consists of a service:8080 (service 1), Zuul:9000 and Eureka:8761.
All the approaches I've tried using Eureka gave me the same error when I tried to access service 1 using Zull (http://localhost:9000/service1 in this case):
com.netflix.zuul.exception.ZuulException: Forwarding error
(...)
com.netflix.client.ClientException: Load balancer does not have available server for client: service1
Service 1 is working fine (I can access it directly from the browser address bar http://localhost:8080) and Eureka shows both Zull and Service 1 correctly registered:
The apps are configured like:
Service 1 (bootstrap.yml):
spring:
application:
name: service1
Service 1 (application.yml):
eureka:
instance:
leaseRenewalIntervalInSeconds: 1
leaseExpirationDurationInSeconds: 2
client:
serviceUrl:
defaultZone: http://127.0.0.1:8761/eureka/
Zuul:
ribbon:
eureka:
enabled: false
eureka:
instance:
leaseRenewalIntervalInSeconds: 1
leaseExpirationDurationInSeconds: 2
client:
serviceUrl:
defaultZone: http://127.0.0.1:8761/eureka/
healthcheck:
enabled: true
server:
port: 9000
Eureka:
server:
port: 8761
eureka:
client:
register-with-eureka: false
fetch-registry: false
logging:
level:
com:
netflix:
eureka: OFF
discovery: OFF
Zuul Annotations at root application class:
#EnableZuulProxy
#EnableDiscoveryClient
Am I missing any important point?

How to config multiple Eureka Servers from client in Spring Cloud

From the spring doc, I see we can have peer eureka server together, so for Eureka1, in application.yml, I can have:
spring:
profiles: peer1
eureka:
instance:
hostname: peer1
client:
serviceUrl:
defaultZone: http://peer2/eureka/
And in Eureka Server 2, I can have:
spring:
profiles: peer2
eureka:
instance:
hostname: peer2
client:
serviceUrl:
defaultZone: http://peer1/eureka/
Now these two eureka servers are aware each other, it is good.
BUT, now in configuring client, when they register again Eureka, how to do this?
In my client application, I have:
eureka:
instance:
hostname: ${host.instance.name:localhost}
nonSecurePort: ${host.instance.port:8080}
leaseRenewalIntervalInSeconds: 5 #default is 30, recommended to keep default
metadataMap:
instanceId: ${spring.application.name}:${spring.application.instance_id:${random.value}}
client:
serviceUrl:
defaultZone: http://(eurekaServerHost):8761/eureka/
server:
port: ${host.instance.port:8080}
So now my question is shall I use peer1 or peer2 as EurekaServerHost in the client application.yml?
Thanks
Use a comma separated list of peers in eureka.client.serviceUrl.defaultZone.
eureka.client.serviceUrl.defaultZone=http://<peer1host>:<peer1port>/eureka,http://<peer2host>:<peer2port>/eureka

Service instances register to both eureka services that are peers

We are running in a peer to peer Eureka configuration. See below for the configuration.
So when the services are started, they register to 10.202.10.95 (the primary) and we see them there with none showing on 10.202.10.97 (the secondary)
If we kill 10.202.10.95 (the primary) then we see them show up on 10.202.10.97 (the secondary eureka).
If we restart 10.202.10.95 (which was the primary), we see the services show up on 10.202.10.95 and also on 10.202.10.97. So services are visible on both eureka servers
If we restart 10.202.10.97 (the secondary), the services vanish and are just visible on the primary (10.202.10.95)
Case 3 is unexpected to me. Is this a case of improper peer to peer Eureka configuration?
The eureka configuration is as follows: (we point these 2 instances to each other in a peer to peer configuration)
spring:
profiles: api06-prod-int-discoveryserver # 10.202.10.95 eureka host
eureka:
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: http://10.202.10.97:8761/eureka/
---
spring:
profiles: api05-prod-int-discoveryserver # 10.202.10.97 eureka host
eureka:
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: http://10.202.10.95:8761/eureka/
Each service has its configuration to eureka set like this: (we point to both instances with x.x.x.95 being the primary)
eureka:
# these are settings for the client that gets services
client:
# enable these two settings if you want discovery to work
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: http://10.202.10.95:8761/eureka/,http://10.202.10.97:8761/eureka/
So, following some other posts and Spencer's response I checked and my configuration was incorrect. After I changed it, it responds correctly.
This is the Eureka settings. I run 2 eureka servers, one with profile peer1 and the other with profile peer2.
---
spring:
profiles: peer1 # not standalone
server:
port: 8761
eureka:
instance:
hostname: peer2
leaseRenewalIntervalInSeconds: 3
client:
serviceUrl:
defaultZone: http://localhost:8762/eureka/
---
spring:
profiles: peer2 # not standalone
server:
port: 8762
eureka:
instance:
hostname: peer1
leaseRenewalIntervalInSeconds: 3
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka/
server:
waitTimeInMsWhenSyncEmpty: 0
The Service that I have connecting is configured like this
eureka:
client:
registerWithEureka: true
fetchRegistry: true
serviceUrl:
defaultZone: http://localhost:8761/eureka/,http://localhost:8762/eureka/
instance:
statusPageUrlPath: /${info.app.name}/manage/info
homePageUrlPath: /${info.app.name}/manage
healthCheckUrlPath: /${info.app.name}/manage/health
preferIpAddress: true
After I run my service, I can see it connect to both discovery services and if I kill either then it is visible on the other discovery service.

Eureka peers not synchronized

I'm prototyping a set of Spring Cloud + Netflix OSS applications and have run into trouble with Eureka. In our setup, we have a Spring Cloud Config Server + Eureka Server, and then 2 modules that utilize that server component for bootstrapping and service discovery.
The problem I run into is that if I spin up 2 instances of the Eureka Server and try to pair them (based on the Two Peer Aware Eureka Servers in the docs) they don't synchronize with each other. See configs below and/or the code on GitHub.
Essentially, Peer1 starts up and looks fine. Peer2 will startup and look fine, with both peers showing each other in the services. However, if the "UserService" module spins up and registers itself with Peer1, Peer2 will never see it. If we then spin up the "Web" module pointing to Peer2, it can never resolve the UserService. They basically act in isolation.
I've tried several combinations of setting the serviceUrl both on the server and the instance of the Eureka servers but to no avail. Am I just configuring things wrong?
Peer 1 / default config:
server:
port: 8888
eureka:
dashboard:
path: /dashboard
instance:
hostname: peer1
leaseRenewalIntervalInSeconds: 3
client:
serviceUrl:
defaultZone: ${eureka.server.serviceUrl:http://localhost:${server.port}/eureka/}
server:
serviceUrl:
defaultZone: http://localhost:${server.port}/eureka/
peer2: http://peer2/eureka/
waitTimeInMsWhenSyncEmpty: 0
spring:
application:
name: demo-config-service
profiles:
active: native
# required for Spring Cloud Bus
rabbitmq:
host: ${DOCKER_IP:192.168.59.103}
port: 5672
username: guest
password: guest
virtualHost: /
cloud:
config:
server:
prefix: /configs
native:
searchLocations: /Users/dave/workspace/oss/distributed-spring/modules/config-server/src/main/resources/testConfigs
# git :
# uri: https://github.com/joshlong/microservices-lab-configuration
Peer 2 config:
server:
port: 8889
eureka:
dashboard:
path: /dashboard
instance:
hostname: peer2
leaseRenewalIntervalInSeconds: 3
client:
serviceUrl:
defaultZone: ${eureka.server.serviceUrl:http://localhost:${server.port}/eureka/}
server:
serviceUrl:
defaultZone: http://localhost:8888/eureka/
peer1: http://peer1/eureka/
waitTimeInMsWhenSyncEmpty: 0
spring:
application:
name: demo-config-service
profiles:
active: native
# required for Spring Cloud Bus
rabbitmq:
host: ${DOCKER_IP:192.168.59.103}
port: 5672
username: guest
password: guest
virtualHost: /
cloud:
config:
server:
prefix: /configs
native:
searchLocations: /Users/dave/workspace/oss/distributed-spring/modules/config-server/src/main/resources/testConfigs
# git :
# uri: https://github.com/joshlong/microservices-lab-configuration
There were a few problems. The defaultZone needs to be in the client section as noted in the docs. The defaultZone url needs the port.
/etc/hosts
127.0.0.1 peer1
127.0.0.1 peer2
Peer 1 Config (Partial)
eureka:
instance:
hostname: peer1
leaseRenewalIntervalInSeconds: 3
client:
serviceUrl:
defaultZone: http://peer2:8889/eureka/
Peer 2 Config (Partial)
eureka:
dashboard:
path: /dashboard
instance:
hostname: peer2
leaseRenewalIntervalInSeconds: 3
client:
serviceUrl:
defaultZone: http://peer1:8888/eureka/
server:
waitTimeInMsWhenSyncEmpty: 0
User service config (Partial) Config port was wrong.
spring:
application:
name: user-service
cloud:
config:
uri: http://localhost:8888/configs
You can see user-service replicated to both peer1 and peer2. I can post a PR to your code if you want.
Peer 1
Peer 2
#spencergibb's didn't mention why this hack-ish workaround is required. There is a gotcha with running more than one Eureka server on the same host. Netflix code (com.netflix.eureka.cluster.PeerEurekaNodes.isThisMyUrl) filters out the peer URLs that are on the same host. This may have been done to prevent the server registering as its own peer (I’m guessing here) but because they don’t check for the port, peer awareness doesn’t work unless the Eureka hostnames in the eureka.client.serviceUrl.defaultZone are different. The hacky workaround for this is to define unique hostnames and then map them to 127.0.0.1 in the /etc/hosts file (or its Windows equivalent).
I've created a blog post with the details of Eureka here, that fills in some missing detail from Spring doc or Netflix blog. It is the result of several days of debugging and digging through source code.