"Failed to create endpoint [XXX] on network because of a duplicate name" on local 5-node Service Fabric cluster - azure-service-fabric

I have a local Service Fabric cluster of 5 nodes and I have a problem deploying my application on all nodes. When set to "1 Node", the cluster works fine. When set to "5 Nodes" it gives an error on all nodes but one. This is the error/warning message:
Error event: SourceId='System.Hosting', Property='CodePackageActivation:Code:EntryPoint:131919316927686034'.
There was an error during CodePackage activation.System.Fabric.FabricException (-2147017731)
Failed to start Container. ContainerName=sf-0-28e0002f-fd7d-412c-81b8-b78ca5339ce4_865991cc-9c36-493f-9b3d-95f6eba43851, ApplicationId=Proton.SFType_App0, ApplicationName=fabric:/Proton.SF.
DockerRequest returned StatusCode=InternalServerError with ResponseBody={"message":"failed to create endpoint sf-0-28e0002f-fd7d-412c-81b8-b78ca5339ce4_865991cc-9c36-493f-9b3d-95f6eba43851 on network nat: HNS failed with error : You were not connected because a duplicate name
The error message looks truncated. The application loads up fine, but on one node only. Am I missing something in 5-node configuration? The application we are deploying is a container which runs a .NET Core app. I've attached a screenshot of the error in Service Fabric Explorer.
Error Screenshot
ServiceManifest.xml
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="Proton.TestingPkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<!-- This is the name of your ServiceType.
The UseImplicitHost attribute indicates this is a guest service. -->
<StatelessServiceType ServiceTypeName="Proton.TestingType" UseImplicitHost="true" />
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
<ContainerHost>
<ImageName>proton.azurecr.io/protontesting:latest</ImageName>
</ContainerHost>
</EntryPoint>
<!-- Pass environment variables to your container: -->
<!--
<EnvironmentVariables>
<EnvironmentVariable Name="VariableName" Value="VariableValue"/>
</EnvironmentVariables>
-->
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<Endpoint Name="Proton.TestingTypeEndpoint" Port="8001" />
</Endpoints>
</Resources>
</ServiceManifest>
ApplicationManifest.xml
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest ApplicationTypeName="Proton.SFType"
ApplicationTypeVersion="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Parameters>
<Parameter Name="Proton.Testing_InstanceCount" DefaultValue="-1" />
</Parameters>
<!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion
should match the Name and Version attributes of the ServiceManifest element defined in the
ServiceManifest.xml file. -->
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="Proton.TestingPkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<ContainerHostPolicies CodePackageRef="Code">
<!-- See https://aka.ms/I7z0p9 for how to encrypt your repository password -->
<RepositoryCredentials AccountName="ProtonCluster" Password="XXX" PasswordEncrypted="false" />
<PortBinding ContainerPort="80" EndpointRef="Proton.TestingTypeEndpoint" />
</ContainerHostPolicies>
</Policies>
</ServiceManifestImport>
<DefaultServices>
<!-- The section below creates instances of service types, when an instance of this
application type is created. You can also create one or more instances of service type using the
ServiceFabric PowerShell module.
The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
<Service Name="Proton.Testing" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="Proton.TestingType" InstanceCount="[Proton.Testing_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
</ApplicationManifest>

Related

Getting 502 http status code on a Service Fabric stateless service deployed on lesser node than configured VM Scaleset nodes

We have deployed various stateless services on a 5 node cluster with -1 as instance count as Singleton partition scheme. Recently, we decided to deploy the few stateless services only on 3 nodes out of 5 by defining instance count as 3.
After deployment, the stateless services with -1 as instance count are working and responding with HttpStatus 200 Ok. however, a stateless service deployed with 3 instance node count are intermittently responding with HttpStatus 502 with following error (from fiddler):
The connection to 'someservername.centralus.cloudapp.azure.com' failed.
System.Security.SecurityException Failed to negotiate HTTPS connection with server.fiddler.network.https> HTTPS handshake to someservername.centralus.cloudapp.azure.com failed. System.IO.IOException Authentication failed because the remote party has closed the transport stream.
Below is the application manifest of deployed application for reference
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="MyService.ServiceFabricType" ApplicationTypeVersion="1.0.0.1.1" ManifestId="8747c387-a7fc-4b05-b189-b1c01958f066" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<Parameters>
<Parameter Name="My_Service_ASPNETCORE_ENVIRONMENT" DefaultValue="" />
<Parameter Name="My_Service_InstanceCount" DefaultValue="3" />
</Parameters>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="MyServicePkg" ServiceManifestVersion="1.0.0.1.1" />
<ConfigOverrides />
<EnvironmentOverrides CodePackageRef="code">
<EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[My_Service_ASPNETCORE_ENVIRONMENT]" />
</EnvironmentOverrides>
</ServiceManifestImport>
<DefaultServices>
<Service Name="MyService" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="MyServiceType" InstanceCount="[My_Service_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
</ApplicationManifest>
and service manifest :
<ServiceManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ManifestId="59ea463b-5e4c-44f5-8982-5658b35d6c89" Name="MyServicePkg" Version="1.0.0.1.1" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<ServiceTypes>
<StatelessServiceType ServiceTypeName="MyService" />
</ServiceTypes>
<CodePackage Name="Code" Version="1.0.0.1.1">
<EntryPoint>
<ExeHost>
<Program>MyService.exe</Program>
<WorkingFolder>CodePackage</WorkingFolder>
</ExeHost>
</EntryPoint>
<EnvironmentVariables>
<EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="" />
</EnvironmentVariables>
</CodePackage>
<ConfigPackage Name="Config" Version="1.0.0.1.1" />
<Resources>
<Endpoints>
<Endpoint Name="ServiceEndpoint" Protocol="https" Type="Input" Port="9226" />
</Endpoints>
</Resources>
</ServiceManifest>
Is it mandatory to deploy a stateless service all nodes in service fabric?
If no, how the above scenario can be configured?
Note - Currently Service Fabric is configured with Silver durability tier and with reverse proxy in disabled state. Also did not get any relevant solution from this azure documentation.

How to pass command line arguments to docker Grafana container in Service Fabric

I'm trying to run grafana as a container inside my service fabric cluster. The container is healthy if I deploy without commmand line arguments, but when I try to pass command line arguments as shown below in the service manifest it fails.
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="Grafana.Pkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<StatelessServiceType ServiceTypeName="Grafana.Type" UseImplicitHost="true" />
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ContainerHost>
<ImageName>myregistry.azurecr.io/grafana:6.3.2</ImageName>
<Commands>-e "GF_SERVER_ROOT_URL=https://serverurl.com/grafana"</Commands>
</ContainerHost>
</EntryPoint>
</CodePackage>
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<Endpoint Name="Grafana.TypeEndpoint" UriScheme="http" Protocol="http" Port="3000" />
</Endpoints>
</Resources>
</ServiceManifest>
The actual docker command for running grafana is shown below
specifically I need to pass
-e "GF_SERVER_ROOT_URL=http://grafana.server.name"
and
-e "GF_SECURITY_ADMIN_PASSWORD=secret"
What is the recommended way to pass command line arguments in the ServiceFabric manifest? Also is there a way to escape the double quotes that need to be passed in?
The -e arguments for the docker CLI, are environment variables to be set in the container. You can specify them in the manifest.
<CodePackage Name="Code" Version="1.0.0">
...
<EnvironmentVariables>
<EnvironmentVariable Name="GF_SERVER_ROOT_URL" Value="https://serverurl.com/grafana"/>
</EnvironmentVariables>
</CodePackage>

Service Fabric Explorer Health State Unknown

The node in my partition keeps switching between Health State = OK and Health State = unknown.
Sometimes the node disappears.
I have tried deleting the service, the app and unprovisioning the type, then redeploying, however I get the same problem.
It is a Service Fabric stateful service, and it's running fine locally, the issue I'm having is only in my dev environment.
I'm using 5 nodes.
ServiceManifest.xml:
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="Integration.Optical.ServicePkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<!-- This is the name of your ServiceType.
This name must match the string used in the RegisterServiceAsync call in Program.cs. -->
<StatefulServiceType ServiceTypeName="Integration.Optical.ServiceType" />
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>Integration.Optical.Service.exe</Program>
<WorkingFolder>CodePackage</WorkingFolder>
</ExeHost>
</EntryPoint>
<EnvironmentVariables>
<EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value=""/>
<EnvironmentVariable Name="KEYVAULT_ENDPOINT" Value=""/>
</EnvironmentVariables>
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<Endpoint Name="ServiceEndpoint" />
<!-- This endpoint is used by the replicator for replicating the state of your service.
This endpoint is configured through a ReplicatorSettings config section in the Settings.xml
file under the ConfigPackage. -->
<Endpoint Name="ReplicatorEndpoint" />
</Endpoints>
</Resources>
</ServiceManifest>
ApplicationManifest.xml:
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="Integration.OpticalType" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<Parameters>
<Parameter Name="Integration.Optical.Service_ASPNETCORE_ENVIRONMENT" DefaultValue="" />
<Parameter Name="Integration.Optical.Service_KEYVAULT_ENDPOINT" DefaultValue="" />
<Parameter Name="Integration.Optical.Service_MinReplicaSetSize" DefaultValue="3" />
<Parameter Name="Integration.Optical.Service_PartitionCount" DefaultValue="1" />
<Parameter Name="Integration.Optical.Service_TargetReplicaSetSize" DefaultValue="3" />
</Parameters>
<!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion
should match the Name and Version attributes of the ServiceManifest element defined in the
ServiceManifest.xml file. -->
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="Integration.Optical.ServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<EnvironmentOverrides CodePackageRef="code">
<EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[Integration.Optical.Service_ASPNETCORE_ENVIRONMENT]" />
<EnvironmentVariable Name="KEYVAULT_ENDPOINT" Value="[Integration.Optical.Service_KEYVAULT_ENDPOINT]" />
</EnvironmentOverrides>
</ServiceManifestImport>
<DefaultServices>
<!-- The section below creates instances of service types, when an instance of this
application type is created. You can also create one or more instances of service type using the
ServiceFabric PowerShell module.
The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
<Service Name="Integration.Optical.Service" ServicePackageActivationMode="ExclusiveProcess">
<StatefulService ServiceTypeName="Integration.Optical.ServiceType" TargetReplicaSetSize="[Integration.Optical.Service_TargetReplicaSetSize]" MinReplicaSetSize="[Integration.Optical.Service_MinReplicaSetSize]">
<UniformInt64Partition PartitionCount="[Integration.Optical.Service_PartitionCount]" LowKey="-9223372036854775808" HighKey="9223372036854775807" />
</StatefulService>
</Service>
</DefaultServices>
</ApplicationManifest>
ApplicationParameters/Cloud.xml:
<?xml version="1.0" encoding="utf-8"?>
<Application xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="fabric:/Integration.Optical" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<Parameters>
<Parameter Name="Integration.Optical.Service_ASPNETCORE_ENVIRONMENT" Value="" />
<Parameter Name="Integration.Optical.Service_KEYVAULT_ENDPOINT" Value="" />
<Parameter Name="Integration.Optical.Service_PartitionCount" Value="1" />
<Parameter Name="Integration.Optical.Service_MinReplicaSetSize" Value="1" />
<Parameter Name="Integration.Optical.Service_TargetReplicaSetSize" Value="1" />
</Parameters>
</Application>
Not sure what part of this fixed it. But this is what I did and it's now working:
In ServiceManifest.xml I added HasPersistedState = true:
<StatefulServiceType ServiceTypeName="Integration.Optical.ServiceType" HasPersistedState="true" />
I moved the app configuration code
ServiceRuntime.RegisterServiceAsync...
from Service.RunAsync() to Program.Main()

Service Fabric can't connect to guest executable on UDP (Unreal Engine Server)

I'm trying to deploy Unreal Server guest executable to Service Fabric.
I've been fallowing this blog post:
http://haishibai.blogspot.com/2017/03/setting-up-highly-available-minecraft.html
Except I'm not using containers and use Windows based service.
On local cluster everything works. I can connect from client to deployed server.
When trying to connect to remote server, I can't connect to it.
When looking at my service fabric managment page It seems like my server process is working (it have proper process id).
In my LoadBalancing service I have port forwarding on UDP/7777, though HealthProbes are on TCP (there is no option to select UDP, I don't know if that mnatter).
Here Is my ServiceManifest:
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="UnrealGuestPkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<!-- This is the name of your ServiceType.
The UseImplicitHost attribute indicates this is a guest executable service. -->
<StatelessServiceType ServiceTypeName="UnrealGuestType" UseImplicitHost="true" />
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<!-- The SetupEntryPoint is an optional element used to specify a
program to be executed before the service's code is launched. -->
<!--
<SetupEntryPoint>
<ExeHost>
<Program></Program>
</ExeHost>
</SetupEntryPoint>
-->
<EntryPoint>
<ExeHost>
<Program>ActionRPGGame\Binaries\Win64\ActionRPGGameServer.exe</Program>
<Arguments>-log</Arguments>
<WorkingFolder>CodeBase</WorkingFolder>
<!-- Uncomment to log console output (both stdout and stderr) to one of the
service's working directories. -->
<!-- <ConsoleRedirection FileRetentionCount="5" FileMaxSizeInKb="2048"/> -->
</ExeHost>
</EntryPoint>
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<Endpoint Name="UnrealGuestTypeEndpoint" Port="7777" Protocol="udp" />
</Endpoints>
</Resources>
</ServiceManifest>
1.You seem to be missing the service endpoint configuration.
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<Endpoint Name="ServiceEndpoint" Port="7777" />
</Endpoints>
</Resources>
UDP isn't supported for probes on the load balancer. So, expose a second service endpoint that uses tcp, just as a health endpoint.

How do I configure local cluster for addtional node types

I have a cluster configuration with two Node Types specified in the ServiceManifest.xml
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="MKopa.M2M.ConfigurationPkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<!-- This is the name of your ServiceType.
This name must match the string used in RegisterServiceType call in Program.cs. -->
<StatelessServiceType ServiceTypeName="ConfigurationType">
<PlacementConstraints>(NodeType == Internal)</PlacementConstraints>
</StatelessServiceType>
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>MKopa.M2M.Configuration.Service.exe</Program>
</ExeHost>
</EntryPoint>
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<Endpoint Name="ServiceEndpoint" />
<Endpoint Name="HttpEndpoint" Protocol="http" Port="8081"/>
</Endpoints>
</Resources>
</ServiceManifest>
My issue that this causes the deployment to the local cluster to fail as this NodeType doesn't exist in the Local Cluster.
I've seen mention of the cluster.xml file and I've found it but making changes to it doesn't seem to have any effect. I've tried a reset, start and stop but the reset overrides the changes.
Here's is hoping that the answer is not starting the services dynamically :-)
I don't know how it works while the cluster is running, but I was able to do it by re-installing the local cluster. These were my steps:
Go to C:\Program Files\Microsoft SDKs\Service Fabric\ClusterSetup\
Uninstall the existing cluster by calling .\CleanCluster.ps1
Create a backup of the file C:\Program Files\Microsoft SDKs\Service Fabric\ClusterSetup\NonSecure\ClusterManifestTemplate.xml
Now you can adjust this file and add placement properties to every node:
<NodeType ...>
<Endpoints>...</Endpoints>
<PlacementProperties>
<Property Name="NodeType" Value="Internal" />
</PlacementProperties>
</NodeType>
Re-create the cluster by calling .\DevClusterSetup.ps1