What do the EndPoints configure in the ServiceManifest of an Service Fabric Service? - azure-service-fabric

Question

We have a Service Fabric Service project with multiple services: Actors, Stateful services and Stateless services combined into one ServiceManifest.
Two stateful services did not work: the constructors were called, the communicationlisteners (through remoting) were created, but the RunAsync method was not called.
After removing the endpoint listing from the ServiceManifest.xml the services started working again. But now we are left wondering why and how this works. Could someone explain?
To illustrate, the relevant section was
<Resources>
<Endpoints>
<Endpoint Name="WebServiceEndpoint" Type="Input" Protocol="http" Port="80" />
<Endpoint Name="StatelessServiceEndpoint1" Type="Input" Protocol="http" Port="10101" />
<Endpoint Name="ActorServiceEndpoint1" />
<Endpoint Name="ActorServiceReplicatorEndpoint1" />
<Endpoint Name="ActorServiceEndpoint2" />
<Endpoint Name="ActorServiceReplicatorEndpoint2" />
<Endpoint Name="ActorServiceEndpoint3" />
<Endpoint Name="ActorServiceReplicatorEndpoint3" />
<Endpoint Name="ActorServiceEndpoint4" />
<Endpoint Name="ActorServiceReplicatorEndpoint4" />
<Endpoint Name="StatefulServiceEndpoint1" Type="Input" Protocol="http" />
<Endpoint Name="StatefulServiceReplicatorEndpoint1" />
<Endpoint Name="StatefulServiceEndpoint2" Type="Input" Protocol="http" />
<Endpoint Name="StatefulServiceReplicatorEndpoint2" />
<Endpoint Name="StatelessServiceEndPoint2" Type="Input" Protocol="http" />
</Endpoints>
</Resources>
After changing it to this
<Resources>
<Endpoints>
<Endpoint Name="WebServiceEndpoint" Type="Input" Protocol="http" Port="80" />
<Endpoint Name="StatelessServiceEndpoint1" Protocol="http" />
<Endpoint Name="ActorServiceReplicatorEndpoint1" />
<Endpoint Name="ActorServiceReplicatorEndpoint2" />
<Endpoint Name="ActorServiceReplicatorEndpoint3" />
<Endpoint Name="ActorServiceReplicatorEndpoint4" />
<Endpoint Name="StatefulServiceReplicatorEndpoint1" />
<Endpoint Name="StatefulServiceReplicatorEndpoint2" />
</Endpoints>
</Resources>
everything worked. But why?
EDIT
The complete ServiceManifest is this:
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="Service" Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<StatefulServiceType ServiceTypeName="ActorService1Type" />
<StatefulServiceType ServiceTypeName="ActorService1Type" HasPersistedState="true" />
<StatefulServiceType ServiceTypeName="ActorService3Type" />
<StatefulServiceType ServiceTypeName="ActorService4Type" HasPersistedState="true" />
<StatefulServiceType ServiceTypeName="StatefulService1Type" HasPersistedState="true" />
<StatefulServiceType ServiceTypeName="StatefulService2Type" HasPersistedState="true" />
<StatelessServiceType ServiceTypeName="StatelessService1Type" />
<StatelessServiceType ServiceTypeName="StatelessService2Type" />
<StatelessServiceType ServiceTypeName="WebServiceType" />
</ServiceTypes>
<CodePackage Name="Code" Version="1.0.0">
<SetupEntryPoint>
<ExeHost>
<Program>Setup.exe</Program>
</ExeHost>
</SetupEntryPoint>
<EntryPoint>
<ExeHost>
<Program>Service.exe</Program>
</ExeHost>
</EntryPoint>
</CodePackage>
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<Endpoint Name="WebServiceEndpoint" Type="Input" Protocol="http" Port="80" />
<Endpoint Name="StatelessServiceEndpoint1" Protocol="http" />
<Endpoint Name="ActorServiceReplicatorEndpoint1" />
<Endpoint Name="ActorServiceReplicatorEndpoint2" />
<Endpoint Name="ActorServiceReplicatorEndpoint3" />
<Endpoint Name="ActorServiceReplicatorEndpoint4" />
<Endpoint Name="StatefulServiceReplicatorEndpoint1" />
<Endpoint Name="StatefulServiceReplicatorEndpoint2" />
</Endpoints>
</Resources>
</ServiceManifest>

Answer 1

Hard to know what happened in your initial reported case since there's no specific error or error message to work off of, but usually this is port conflicts when you end up sharing ports that you don't really want to or which can't be shared, or port exhaustion.
The endpoint resource in your service manifest is mainly for times when:
you want SF to help with allocating communication resources like ports for your services
you want SF to help configure those resources:
Allocating some port and consistently assigning it to some set of workloads
Punching a hole in the local firewall
Setting up a URLACL (relevant to http on windows through http.sys only)
Setting up and configuring certs to enable secure communication (same caveat)
In general you're free to ignore the endpoint resource if you don't need/want the help, since SF really is expecting the service code to do it's setup. In cases where you're not really using SF's programming models then the endpoint resource is more important since it's how you communicate to SF what your endpoints are.
The behavior you get really depends on the transport you're using, as well as the OS's dynamic port range and the Application port range that you've defined, as well as what the service code actually does.
https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-fabric-settings#section-name-fabricnode
Let's say you're setting up an http communication listener in your service like so, and walk through a few examples of what happens when you define and endpoint in your manifest or don't.
1) Let's say you put nothing in your service manifest about endpoints. This means that effectively you're specifying 0 as the port in code. In this case SF's not doing any allocation or management. The port is getting assigned by the OS from the OS dynamic port range. The port that actually gets assigned will be different for each service instance listener. This should work as a reasonable default choice in most scenarios.
2) Let's say you're specifying an endpoint in the manifest and not specifying any port at all, i.e.:
<Endpoint Name="HealthServiceEndpoint"/>
In this case, the port that is assigned will come from the SF application port range. It will be the same for any service instances hosted in the same process, but different across processes. (So it matters if you are using the Exclusive or Shared process hosting model) This also presumes that reusing the port is supported for your transport. Most transports don't (like http on via Kestrel in .NET, TCP under most cases), but there are some notable examples (http.sys based http transports on Windows like WebListener/HttpSys, tcp via net.tcp in WCF probably a few others).
3) Let's say you're specifying an endpoint in the service fabric manifest and explicitly specifying 0 for that port i.e.:
<Endpoint Name="HealthServiceEndpoint" Port="0" Protocol="http"/>
In this case the port that gets assigned will be from the OS dynamic port range, and it will end up the same/shared for any service instances hosted in the same process that use that endpoint. The port will be different across processes. (So again it matters if you are using the Exclusive or Shared process hosting model)
4) Naturally if the endpoint is specified and a specific port is specified, that port will be used for all service instances both within and across processes. This somewhat implicitly assumes that such sharing is going to work, which again depends on your transport and platform, or that you're never planning on running more than one instance of the service on this node.
Other trivia:
the "transport" element mainly determines whether SF registers your url with http.sys on windows or configures certificates to secure traffic (most of this can be done within your service code or a SetupEntryPoint).
as of this writing Type is ignored (this is a holdover from an older version of SF)
PathSuffix is used to create a default uri fragment that gets appended to the IP and port assigned by the platform. This is used in cases where there's code not using SF's listener APIs that sets up some listener on a different path like /api/value, like a the code inside a container might do.