Eureka peers not synchronized - spring-cloud

Question

I'm prototyping a set of Spring Cloud + Netflix OSS applications and have run into trouble with Eureka. In our setup, we have a Spring Cloud Config Server + Eureka Server, and then 2 modules that utilize that server component for bootstrapping and service discovery.
The problem I run into is that if I spin up 2 instances of the Eureka Server and try to pair them (based on the Two Peer Aware Eureka Servers in the docs) they don't synchronize with each other. See configs below and/or the code on GitHub.
Essentially, Peer1 starts up and looks fine. Peer2 will startup and look fine, with both peers showing each other in the services. However, if the "UserService" module spins up and registers itself with Peer1, Peer2 will never see it. If we then spin up the "Web" module pointing to Peer2, it can never resolve the UserService. They basically act in isolation.
I've tried several combinations of setting the serviceUrl both on the server and the instance of the Eureka servers but to no avail. Am I just configuring things wrong?
Peer 1 / default config:
server:
port: 8888
eureka:
dashboard:
path: /dashboard
instance:
hostname: peer1
leaseRenewalIntervalInSeconds: 3
client:
serviceUrl:
defaultZone: ${eureka.server.serviceUrl:http://localhost:${server.port}/eureka/}
server:
serviceUrl:
defaultZone: http://localhost:${server.port}/eureka/
peer2: http://peer2/eureka/
waitTimeInMsWhenSyncEmpty: 0
spring:
application:
name: demo-config-service
profiles:
active: native
# required for Spring Cloud Bus
rabbitmq:
host: ${DOCKER_IP:192.168.59.103}
port: 5672
username: guest
password: guest
virtualHost: /
cloud:
config:
server:
prefix: /configs
native:
searchLocations: /Users/dave/workspace/oss/distributed-spring/modules/config-server/src/main/resources/testConfigs
# git :
# uri: https://github.com/joshlong/microservices-lab-configuration
Peer 2 config:
server:
port: 8889
eureka:
dashboard:
path: /dashboard
instance:
hostname: peer2
leaseRenewalIntervalInSeconds: 3
client:
serviceUrl:
defaultZone: ${eureka.server.serviceUrl:http://localhost:${server.port}/eureka/}
server:
serviceUrl:
defaultZone: http://localhost:8888/eureka/
peer1: http://peer1/eureka/
waitTimeInMsWhenSyncEmpty: 0
spring:
application:
name: demo-config-service
profiles:
active: native
# required for Spring Cloud Bus
rabbitmq:
host: ${DOCKER_IP:192.168.59.103}
port: 5672
username: guest
password: guest
virtualHost: /
cloud:
config:
server:
prefix: /configs
native:
searchLocations: /Users/dave/workspace/oss/distributed-spring/modules/config-server/src/main/resources/testConfigs
# git :
# uri: https://github.com/joshlong/microservices-lab-configuration

Answer 1

There were a few problems. The defaultZone needs to be in the client section as noted in the docs. The defaultZone url needs the port.
/etc/hosts
127.0.0.1 peer1
127.0.0.1 peer2
Peer 1 Config (Partial)
eureka:
instance:
hostname: peer1
leaseRenewalIntervalInSeconds: 3
client:
serviceUrl:
defaultZone: http://peer2:8889/eureka/
Peer 2 Config (Partial)
eureka:
dashboard:
path: /dashboard
instance:
hostname: peer2
leaseRenewalIntervalInSeconds: 3
client:
serviceUrl:
defaultZone: http://peer1:8888/eureka/
server:
waitTimeInMsWhenSyncEmpty: 0
User service config (Partial) Config port was wrong.
spring:
application:
name: user-service
cloud:
config:
uri: http://localhost:8888/configs
You can see user-service replicated to both peer1 and peer2. I can post a PR to your code if you want.
Peer 1
Peer 2

Answer 2

#spencergibb's didn't mention why this hack-ish workaround is required. There is a gotcha with running more than one Eureka server on the same host. Netflix code (com.netflix.eureka.cluster.PeerEurekaNodes.isThisMyUrl) filters out the peer URLs that are on the same host. This may have been done to prevent the server registering as its own peer (I’m guessing here) but because they don’t check for the port, peer awareness doesn’t work unless the Eureka hostnames in the eureka.client.serviceUrl.defaultZone are different. The hacky workaround for this is to define unique hostnames and then map them to 127.0.0.1 in the /etc/hosts file (or its Windows equivalent).
I've created a blog post with the details of Eureka here, that fills in some missing detail from Spring doc or Netflix blog. It is the result of several days of debugging and digging through source code.