Why eureka considers a service still 'UP' even if its last renewal time was two days ago - spring-cloud

Question

I have a service with 2 replicas running. But I found the service having 3 UP instances in eureka's dashboard.
Below the detail info of the service in eureka. The problematic instance is 8d0c39f3ed9b:ocr-server:8095, whose last renewal time was two days ago. And the status and health url can NOT be accessed. I'm wondering why eureka considers it's UP!
<application>
<name>OCR-SERVER</name>
<instance>
<instanceId>211a4634b45f:ocr-server:8095</instanceId>
<hostName>211a4634b45f</hostName>
<app>OCR-SERVER</app>
<ipAddr>10.0.0.38</ipAddr>
<status>UP</status>
<overriddenstatus>UNKNOWN</overriddenstatus>
<port enabled="true">8095</port>
<securePort enabled="false">443</securePort>
<countryId>1</countryId>
<dataCenterInfo class="com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo">
<name>MyOwn</name>
</dataCenterInfo>
<leaseInfo>
<renewalIntervalInSecs>30</renewalIntervalInSecs>
<durationInSecs>90</durationInSecs>
<registrationTimestamp>1505589414444</registrationTimestamp>
<lastRenewalTimestamp>1505833520772</lastRenewalTimestamp>
<evictionTimestamp>0</evictionTimestamp>
<serviceUpTimestamp>1505589414444</serviceUpTimestamp>
</leaseInfo>
<metadata class="java.util.Collections$EmptyMap"/>
<homePageUrl>http://211a4634b45f:8095/</homePageUrl>
<statusPageUrl>http://211a4634b45f:8096/manage/info</statusPageUrl>
<healthCheckUrl>http://211a4634b45f:8096/manage/health</healthCheckUrl>
<vipAddress>ocr-server</vipAddress>
<secureVipAddress>ocr-server</secureVipAddress>
<isCoordinatingDiscoveryServer>false</isCoordinatingDiscoveryServer>
<lastUpdatedTimestamp>1505589414444</lastUpdatedTimestamp>
<lastDirtyTimestamp>1505589412731</lastDirtyTimestamp>
<actionType>ADDED</actionType>
</instance>
<instance>
<instanceId>8d0c39f3ed9b:ocr-server:8095</instanceId>
<hostName>8d0c39f3ed9b</hostName>
<app>OCR-SERVER</app>
<ipAddr>10.0.0.38</ipAddr>
<status>UP</status>
<overriddenstatus>UNKNOWN</overriddenstatus>
<port enabled="true">8095</port>
<securePort enabled="false">443</securePort>
<countryId>1</countryId>
<dataCenterInfo class="com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo">
<name>MyOwn</name>
</dataCenterInfo>
<leaseInfo>
<renewalIntervalInSecs>30</renewalIntervalInSecs>
<durationInSecs>90</durationInSecs>
<registrationTimestamp>1505589356797</registrationTimestamp>
<lastRenewalTimestamp>1505589356797</lastRenewalTimestamp>
<evictionTimestamp>0</evictionTimestamp>
<serviceUpTimestamp>1505589356797</serviceUpTimestamp>
</leaseInfo>
<metadata class="java.util.Collections$EmptyMap"/>
<homePageUrl>http://8d0c39f3ed9b:8095/</homePageUrl>
<statusPageUrl>http://8d0c39f3ed9b:8096/manage/info</statusPageUrl>
<healthCheckUrl>http://8d0c39f3ed9b:8096/manage/health</healthCheckUrl>
<vipAddress>ocr-server</vipAddress>
<secureVipAddress>ocr-server</secureVipAddress>
<isCoordinatingDiscoveryServer>false</isCoordinatingDiscoveryServer>
<lastUpdatedTimestamp>1505589356797</lastUpdatedTimestamp>
<lastDirtyTimestamp>1505589356787</lastDirtyTimestamp>
<actionType>ADDED</actionType>
</instance>
<instance>
<instanceId>0c62739f5ca8:ocr-server:8095</instanceId>
<hostName>0c62739f5ca8</hostName>
<app>OCR-SERVER</app>
<ipAddr>10.0.0.34</ipAddr>
<status>UP</status>
<overriddenstatus>UNKNOWN</overriddenstatus>
<port enabled="true">8095</port>
<securePort enabled="false">443</securePort>
<countryId>1</countryId>
<dataCenterInfo class="com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo">
<name>MyOwn</name>
</dataCenterInfo>
<leaseInfo>
<renewalIntervalInSecs>30</renewalIntervalInSecs>
<durationInSecs>90</durationInSecs>
<registrationTimestamp>1505589403150</registrationTimestamp>
<lastRenewalTimestamp>1505833509700</lastRenewalTimestamp>
<evictionTimestamp>0</evictionTimestamp>
<serviceUpTimestamp>1505589403150</serviceUpTimestamp>
</leaseInfo>
<metadata class="java.util.Collections$EmptyMap"/>
<homePageUrl>http://0c62739f5ca8:8095/</homePageUrl>
<statusPageUrl>http://0c62739f5ca8:8096/manage/info</statusPageUrl>
<healthCheckUrl>http://0c62739f5ca8:8096/manage/health</healthCheckUrl>
<vipAddress>ocr-server</vipAddress>
<secureVipAddress>ocr-server</secureVipAddress>
<isCoordinatingDiscoveryServer>false</isCoordinatingDiscoveryServer>
<lastUpdatedTimestamp>1505589403150</lastUpdatedTimestamp>
<lastDirtyTimestamp>1505589400020</lastDirtyTimestamp>
<actionType>ADDED</actionType>
</instance>
</application>
The config of eureka client looks like below,
eureka:
instance:
statusPageUrlPath: /info
healthCheckUrlPath: /health
preferIpAddress: false
client:
fetchRegistry: true
registryFetchIntervalSeconds: 5
healthcheck:
enabled: true
serviceUrl:
defaultZone: http://eureka-peer1:8761/eureka,http://eureka-peer2:8761/eureka
The configuration of eureka server is,
eureka:
instance:
preferIpAddress: true
enableSelfPreservation: false
homePageUrl: http://${eureka.hostname}/
---
spring:
profiles: peer1
eureka:
instance:
hostname: eureka-peer1
client:
serviceUrl:
defaultZone: http://eureka-peer2:${server.port}/eureka/
---
spring:
profiles: peer2
eureka:
instance:
hostname: eureka-peer2
client:
serviceUrl:
defaultZone: http://eureka-peer1:${server.port}/eureka/
And both the eureka clients and servers are running in docker swarm.

Answer 1

Your server has probably entered the self preservation mode, that's why you see a stale instance there.
I see you have the enableSelfPreservation: false option however it is not in the right section, it should be eureka.server.enableSelfPreservation, your entry is under eureka.instance.enableSelfPreservation which is not right.