why was k8s service deleted but the cluster IP works still - kubernetes

Question

I can easily reporduce this and could not find an answer for this issue either in k8s doc or the community.
Simple reproduce steps:
create service and endpoint with below config
---
kind: Service
apiVersion: v1
metadata:
name: hostname
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 9376
---
kind: Endpoints
apiVersion: v1
metadata:
name: hostname
subsets:
- addresses:
- ip: 10.244.44.250
- ip: 10.244.154.235
ports:
- port: 9376
kubectl apply -f <filename> to apply the config
test the service and it works perfect. Assume the cluster IP is A
kubectl delete -f <filename> to delete the service and endpoint and kubectl apply -f <filename> again
we got another cluser IP B, which works perfect also
however, cluser IP A was not removed as expected. I can use A to access the service still.
Update the endpoint definition (add new endpoint IP or remove one) and apply, B sees the change while A uses old config still.
Is there someone can explain what happens there?
My k8s version is:
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:09:21Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}