Kubernetes service architecture - kubernetes

Question

Within the same kubernetes cluster,
Can I have multiple StatefulSets attached to one headless service or should each StatefulSet have it's own headless service? What are the pros and cons of doing this?
Can I mix standard and headless services in the same cluster? Specifically, I would like to use LoadBalancer service to load balance headless services. Can I define a service of type LoadBalancer and have headless services (ClusterIP = None) attached to it? If yes, how can I achieve this?
Here is my intended architecture:
Load Balancer Service
- Headless Service (Database-service)
- MySql
- BlazeGraph
- Headless Service (Web / Tomcat)
- Web Service (RESTful / GraphQL)
Any advice and insight is appreciated.
My setup
My service and the statefulsets attached to it have different labels.
database-service: app=database
mysqlset: app=mysql
My pods
khteh#khteh-T580:~ 2007 $ k get pods -l app=mysql -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
mysql-0 1/1 Running 1 18h 10.1.1.4 khteh-t580 <none>
khteh#khteh-T580:~ 2008 $ k get pods -l app=blazegraph -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
blazegraph-0 1/1 Running 1 18h 10.1.1.254 khteh-t580 <none>
khteh#khteh-T580:~ 2009 $ k describe service database-service
Name: database-service
Namespace: default
Labels: app=database
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"database"},"name":"database-service","namespace":"defaul...
Selector: app=database,tier=database
Type: ClusterIP
IP: None
Port: mysql 3306/TCP
TargetPort: 3306/TCP
Endpoints: <none>
Port: blazegraph 9999/TCP
TargetPort: 9999/TCP
Endpoints: <none>
Session Affinity: None
Events: <none>
Notice the service Endpoints is <none>. I am not sure this is the right setup.

Answer 1

Headless Service you should use in any case where you want to automatically discover all pods under the service as opposed to regular Service where you get ClusterIP instead. As an illustration from above mentioned example here is difference between DNS entries for Service (with ClusterIP) and Headless Service (without ClusterIP):
Standard service you will get the clusterIP value:
kubectl exec zookeeper-0 -- nslookup zookeeper
Server: 10.0.0.10
Address: 10.0.0.10#53
Name: zookeeper.default.svc.cluster.local
Address: 10.0.0.213
Headless service you will get IP of each pod
kubectl exec zookeeper-0 -- nslookup zookeeper
Server: 10.0.0.10
Address: 10.0.0.10#53
Name: zookeeper.default.svc.cluster.local
Address: 172.17.0.6
Name: zookeeper.default.svc.cluster.local
Address: 172.17.0.7
Name: zookeeper.default.svc.cluster.local
Address: 172.17.0.8
Now, If you connect two statefulset with single headless service, it will return the address of each pod in both the statefulset. There will be no way to differentiate the pods from two applications if you create two statefulset and one headless service for that. See the following article to understand why headless services are used
Headless service allow developer to reduce coupling from kubernetes system by allowing them to do discovery their own way. For such services, clusterIP is not allocated, kube-proxy doesn't handle these services and there is no load balancing and proxying done by platform for them. So, If you define clusterIP: None in your service there will be no load-balancing will be done from kubernetes end.
Hope this helps.
EDIT:
I did a little experiment to answer your queries, created two statefulsets of mysql database named mysql and mysql2, with 1 replica for each statefulset. They have their own PV, PVC but bound by only single headless service.
[root#ip-10-0-1-235 centos]# kubectl get pods -l app=mysql -o wide
NAME READY STATUS RESTARTS AGE IP NODE
mysql-0 1/1 Running 0 4m 192.168.13.21 ip-10-0-1-235.ec2.internal
mysql2-0 1/1 Running 0 3m 192.168.13.22 ip-10-0-1-235.ec2.internal
Now you can see the single headless service attached to both the pods
[root#ip-10-0-1-235 centos]# kubectl describe svc mysql
Name: mysql
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=mysql
Type: ClusterIP
IP: None
Port: <unset> 3306/TCP
TargetPort: 3306/TCP
Endpoints: 192.168.13.21:3306,192.168.13.22:3306
Session Affinity: None
Events: <none>
Now when you lookup the service from some other pod, it returns IP address of both the pods:
[root#rtp-worker-0 /]# nslookup mysql
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: mysql.default.svc.cluster.local
Address: 192.168.13.21
Name: mysql.default.svc.cluster.local
Address: 192.168.13.22
Now, it is impossible to identify which address(pod) is of which statefulset. Now I tried to identify the statefulset using its metadata name, but couldn't
[root#rtp-worker-0 /]# nslookup mysql2.mysql.default.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
** server can't find mysql2.mysql.default.svc.cluster.local: NXDOMAIN
Hope it clarifies.