We have deployed various stateless services on a 5 node cluster with -1 as instance count as Singleton partition scheme. Recently, we decided to deploy the few stateless services only on 3 nodes out of 5 by defining instance count as 3.
After deployment, the stateless services with -1 as instance count are working and responding with HttpStatus 200 Ok. however, a stateless service deployed with 3 instance node count are intermittently responding with HttpStatus 502 with following error (from fiddler):
The connection to 'someservername.centralus.cloudapp.azure.com' failed.
System.Security.SecurityException Failed to negotiate HTTPS connection with server.fiddler.network.https> HTTPS handshake to someservername.centralus.cloudapp.azure.com failed. System.IO.IOException Authentication failed because the remote party has closed the transport stream.
Below is the application manifest of deployed application for reference
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="MyService.ServiceFabricType" ApplicationTypeVersion="1.0.0.1.1" ManifestId="8747c387-a7fc-4b05-b189-b1c01958f066" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<Parameters>
<Parameter Name="My_Service_ASPNETCORE_ENVIRONMENT" DefaultValue="" />
<Parameter Name="My_Service_InstanceCount" DefaultValue="3" />
</Parameters>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="MyServicePkg" ServiceManifestVersion="1.0.0.1.1" />
<ConfigOverrides />
<EnvironmentOverrides CodePackageRef="code">
<EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[My_Service_ASPNETCORE_ENVIRONMENT]" />
</EnvironmentOverrides>
</ServiceManifestImport>
<DefaultServices>
<Service Name="MyService" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="MyServiceType" InstanceCount="[My_Service_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
</ApplicationManifest>
and service manifest :
<ServiceManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ManifestId="59ea463b-5e4c-44f5-8982-5658b35d6c89" Name="MyServicePkg" Version="1.0.0.1.1" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<ServiceTypes>
<StatelessServiceType ServiceTypeName="MyService" />
</ServiceTypes>
<CodePackage Name="Code" Version="1.0.0.1.1">
<EntryPoint>
<ExeHost>
<Program>MyService.exe</Program>
<WorkingFolder>CodePackage</WorkingFolder>
</ExeHost>
</EntryPoint>
<EnvironmentVariables>
<EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="" />
</EnvironmentVariables>
</CodePackage>
<ConfigPackage Name="Config" Version="1.0.0.1.1" />
<Resources>
<Endpoints>
<Endpoint Name="ServiceEndpoint" Protocol="https" Type="Input" Port="9226" />
</Endpoints>
</Resources>
</ServiceManifest>
Is it mandatory to deploy a stateless service all nodes in service fabric?
If no, how the above scenario can be configured?
Note - Currently Service Fabric is configured with Silver durability tier and with reverse proxy in disabled state. Also did not get any relevant solution from this azure documentation.
Related
The node in my partition keeps switching between Health State = OK and Health State = unknown.
Sometimes the node disappears.
I have tried deleting the service, the app and unprovisioning the type, then redeploying, however I get the same problem.
It is a Service Fabric stateful service, and it's running fine locally, the issue I'm having is only in my dev environment.
I'm using 5 nodes.
ServiceManifest.xml:
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="Integration.Optical.ServicePkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<!-- This is the name of your ServiceType.
This name must match the string used in the RegisterServiceAsync call in Program.cs. -->
<StatefulServiceType ServiceTypeName="Integration.Optical.ServiceType" />
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>Integration.Optical.Service.exe</Program>
<WorkingFolder>CodePackage</WorkingFolder>
</ExeHost>
</EntryPoint>
<EnvironmentVariables>
<EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value=""/>
<EnvironmentVariable Name="KEYVAULT_ENDPOINT" Value=""/>
</EnvironmentVariables>
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<Endpoint Name="ServiceEndpoint" />
<!-- This endpoint is used by the replicator for replicating the state of your service.
This endpoint is configured through a ReplicatorSettings config section in the Settings.xml
file under the ConfigPackage. -->
<Endpoint Name="ReplicatorEndpoint" />
</Endpoints>
</Resources>
</ServiceManifest>
ApplicationManifest.xml:
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="Integration.OpticalType" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<Parameters>
<Parameter Name="Integration.Optical.Service_ASPNETCORE_ENVIRONMENT" DefaultValue="" />
<Parameter Name="Integration.Optical.Service_KEYVAULT_ENDPOINT" DefaultValue="" />
<Parameter Name="Integration.Optical.Service_MinReplicaSetSize" DefaultValue="3" />
<Parameter Name="Integration.Optical.Service_PartitionCount" DefaultValue="1" />
<Parameter Name="Integration.Optical.Service_TargetReplicaSetSize" DefaultValue="3" />
</Parameters>
<!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion
should match the Name and Version attributes of the ServiceManifest element defined in the
ServiceManifest.xml file. -->
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="Integration.Optical.ServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<EnvironmentOverrides CodePackageRef="code">
<EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[Integration.Optical.Service_ASPNETCORE_ENVIRONMENT]" />
<EnvironmentVariable Name="KEYVAULT_ENDPOINT" Value="[Integration.Optical.Service_KEYVAULT_ENDPOINT]" />
</EnvironmentOverrides>
</ServiceManifestImport>
<DefaultServices>
<!-- The section below creates instances of service types, when an instance of this
application type is created. You can also create one or more instances of service type using the
ServiceFabric PowerShell module.
The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
<Service Name="Integration.Optical.Service" ServicePackageActivationMode="ExclusiveProcess">
<StatefulService ServiceTypeName="Integration.Optical.ServiceType" TargetReplicaSetSize="[Integration.Optical.Service_TargetReplicaSetSize]" MinReplicaSetSize="[Integration.Optical.Service_MinReplicaSetSize]">
<UniformInt64Partition PartitionCount="[Integration.Optical.Service_PartitionCount]" LowKey="-9223372036854775808" HighKey="9223372036854775807" />
</StatefulService>
</Service>
</DefaultServices>
</ApplicationManifest>
ApplicationParameters/Cloud.xml:
<?xml version="1.0" encoding="utf-8"?>
<Application xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="fabric:/Integration.Optical" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<Parameters>
<Parameter Name="Integration.Optical.Service_ASPNETCORE_ENVIRONMENT" Value="" />
<Parameter Name="Integration.Optical.Service_KEYVAULT_ENDPOINT" Value="" />
<Parameter Name="Integration.Optical.Service_PartitionCount" Value="1" />
<Parameter Name="Integration.Optical.Service_MinReplicaSetSize" Value="1" />
<Parameter Name="Integration.Optical.Service_TargetReplicaSetSize" Value="1" />
</Parameters>
</Application>
Not sure what part of this fixed it. But this is what I did and it's now working:
In ServiceManifest.xml I added HasPersistedState = true:
<StatefulServiceType ServiceTypeName="Integration.Optical.ServiceType" HasPersistedState="true" />
I moved the app configuration code
ServiceRuntime.RegisterServiceAsync...
from Service.RunAsync() to Program.Main()
I have deployed trafeik in secured service fabric cluster , and i have made settings in servicemanifest of one of my application to use trafeik, eventhough trafeik is able to identify service but backend in trafeik is always red and my service is unreachable, if i use direct endpoint to my service it is reachable, need help in resolving same.
below is the toml of trafeik -
debug = true
logLevel = "INFO"
defaultEntryPoints = ["https"]
[entryPoints]
[entryPoints.https]
address = ":16080"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
certFile = "certs/servicefabric.crt"
keyFile = "certs/servicefabric.key"
[entryPoints.traefik]
address = ":16081"
[api]
entryPoint = "traefik"
dashboard = true
debug = true
[servicefabric]
clustermanagementurl = "https://localhost:19080"
apiversion = "3.0"
[serviceFabric.tls]
cert = "certs/servicefabric.crt"
key = "certs/servicefabric.key"
insecureskipverify = true
Service manifest of trafik service -
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="TraefikPkg" Version="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<!-- This is the name of your ServiceType.
The UseImplicitHost attribute indicates this is a guest executable service. -->
<StatelessServiceType ServiceTypeName="TraefikType" UseImplicitHost="true" />
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<!-- The SetupEntryPoint is an optional element used to specify a
program to be executed before the service's code is launched. -->
<EntryPoint>
<ExeHost>
<Program>traefik.exe</Program>
<Arguments>--configfile=traefik.toml</Arguments>
<WorkingFolder>CodePackage</WorkingFolder>
<!-- Uncomment to log console output (both stdout and stderr) to one of the
service's working directories. Do not use in production. -->
<!--<ConsoleRedirection FileRetentionCount="5" FileMaxSizeInKb="2048" />-->
</ExeHost>
</EntryPoint>
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<Endpoint Protocol="https" Name="TraefikTypeEndpoint" Type="Input" Port="16080" CertificateRef="ClusterThumbprint"/>
<Endpoint Protocol="https" Name="TraefikTypeAPIEndpoint" Type="Input" Port="16081" CertificateRef="ClusterThumbprint"/>
</Endpoints>
</Resources>
</ServiceManifest>
Application manifest of trafeik service -
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest ApplicationTypeName="TraefikType" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Parameters>
<Parameter Name="Traefik_InstanceCount" DefaultValue="-1" />
<Parameter Name="Stateless1_PlacementConstraints" DefaultValue="(NodeTypeName==fntensvcp)" />
</Parameters>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="TraefikPkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<RunAsPolicy CodePackageRef="Code" UserRef="AdminUser" EntryPointType="All" />
</Policies>
</ServiceManifestImport>
<DefaultServices>
<Service Name="Traefik" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="TraefikType" InstanceCount="[Traefik_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
</DefaultServices>
<Principals>
<Users>
<User Name="AdminUser">
<MemberOf>
<SystemGroup Name="Administrators" />
</MemberOf>
</User>
</Users>
</Principals>
<Certificates>
<EndpointCertificate X509FindValue="b22bb9d3e74da3339e5f6709798107443553ae48" Name="ClusterThumbprint" />
</Certificates>
</ApplicationManifest>
For my application below is service manifest -
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="CSM.IE.SMA.C360.RevenueServicePkg" Version="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ManifestId="25bba1ad-7380-4cbf-b9ed-9a90ee3dd086">
<ServiceTypes>
<!-- This is the name of your ServiceType.
This name must match the string used in RegisterServiceType call in Program.cs. -->
<StatelessServiceType ServiceTypeName="CSM.IE.SMA.C360.RevenueServiceType">
<Extensions>
<Extension Name="Traefik">
<Labels xmlns="http://schemas.microsoft.com/2015/03/fabact-no-schema">
<Label Key="traefik.frontend.rule.c360Revenue">PathPrefixStrip: /C360/RevenueService</Label>
<Label Key="traefik.enable">true</Label>
<Label Key="traefik.frontend.passHostHeader">true</Label>
</Labels>
</Extension>
</Extensions>
</StatelessServiceType>
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>CSM.IE.SMA.C360.RevenueService.exe</Program>
</ExeHost>
</EntryPoint>
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<!--<Endpoint Name="ServiceEndpoint" Protocol="http" Port="8082"/>-->
<Endpoint Protocol="https" Name="ServiceEndpoint" Type="Input" Port="19616" CertificateRef="ClusterThumbprint" />
</Endpoints>
</Resources>
</ServiceManifest>
for my application below is application manifest file -
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="IE_SmartAssistance" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric" ManifestId="1ff9d8a9-ecbc-4d55-931e-eb59eb229f15">
<Parameters>
<Parameter Name="MultiTPIDConsumptionService_InstanceCount" DefaultValue="2" />
<Parameter Name="SupportService_InstanceCount" DefaultValue="2" />
<Parameter Name="ConsumptionService_InstanceCount" DefaultValue="2" />
<Parameter Name="AgreementService_InstanceCount" DefaultValue="2" />
<Parameter Name="RevenueService_InstanceCount" DefaultValue="2" />
<Parameter Name="ProfileProxyService_InstanceCount" DefaultValue="2" />
<Parameter Name="Stateless1_PlacementConstraints" DefaultValue="(NodeTypeName==fntensvcp)" />
</Parameters>
<!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion
should match the Name and Version attributes of the ServiceManifest element defined in the
ServiceManifest.xml file. -->
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.MultiTPIDConsumptionServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.AgreementServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.RevenueServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.ProfileProxyServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.ConsumptionServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.SupportServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<DefaultServices>
<!-- The section below creates instances of service types, when an instance of this
application type is created. You can also create one or more instances of service type using the
ServiceFabric PowerShell module.
The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
<Service Name="CSM.IE.SMA.C360.MultiTPIDConsumptionService" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.MultiTPIDConsumptionServiceType" InstanceCount="[MultiTPIDConsumptionService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
<Service Name="IE_SmartAssistance_Customer360Service_AgreementService-Microservice_API">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.AgreementServiceType" InstanceCount="[AgreementService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
<Service Name="RevenueService-Microservice_API" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.RevenueServiceType" InstanceCount="[RevenueService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
<Service Name="CSM.IE.SMA.C360.ProfileProxyService" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.ProfileProxyServiceType" InstanceCount="[ProfileProxyService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
<Service Name="CSM.IE.SMA.C360.ConsumptionService" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.ConsumptionServiceType" InstanceCount="[ConsumptionService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
<Service Name="CSM.IE.SMA.C360.SupportService" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.SupportServiceType" InstanceCount="[SupportService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
</DefaultServices>
<Principals>
<Users>
<User Name="LocalAdmin" AccountType="NetworkService">
<MemberOf>
<SystemGroup Name="Administrators" />
</MemberOf>
</User>
</Users>
</Principals>
<Certificates>
<EndpointCertificate X509FindValue="b22bb9d3e74da3339e5f6709798107443553ae48" Name="ClusterThumbprint" />
</Certificates>
</ApplicationManifest>
I have a local Service Fabric cluster of 5 nodes and I have a problem deploying my application on all nodes. When set to "1 Node", the cluster works fine. When set to "5 Nodes" it gives an error on all nodes but one. This is the error/warning message:
Error event: SourceId='System.Hosting', Property='CodePackageActivation:Code:EntryPoint:131919316927686034'.
There was an error during CodePackage activation.System.Fabric.FabricException (-2147017731)
Failed to start Container. ContainerName=sf-0-28e0002f-fd7d-412c-81b8-b78ca5339ce4_865991cc-9c36-493f-9b3d-95f6eba43851, ApplicationId=Proton.SFType_App0, ApplicationName=fabric:/Proton.SF.
DockerRequest returned StatusCode=InternalServerError with ResponseBody={"message":"failed to create endpoint sf-0-28e0002f-fd7d-412c-81b8-b78ca5339ce4_865991cc-9c36-493f-9b3d-95f6eba43851 on network nat: HNS failed with error : You were not connected because a duplicate name
The error message looks truncated. The application loads up fine, but on one node only. Am I missing something in 5-node configuration? The application we are deploying is a container which runs a .NET Core app. I've attached a screenshot of the error in Service Fabric Explorer.
Error Screenshot
ServiceManifest.xml
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="Proton.TestingPkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<!-- This is the name of your ServiceType.
The UseImplicitHost attribute indicates this is a guest service. -->
<StatelessServiceType ServiceTypeName="Proton.TestingType" UseImplicitHost="true" />
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
<ContainerHost>
<ImageName>proton.azurecr.io/protontesting:latest</ImageName>
</ContainerHost>
</EntryPoint>
<!-- Pass environment variables to your container: -->
<!--
<EnvironmentVariables>
<EnvironmentVariable Name="VariableName" Value="VariableValue"/>
</EnvironmentVariables>
-->
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<Endpoint Name="Proton.TestingTypeEndpoint" Port="8001" />
</Endpoints>
</Resources>
</ServiceManifest>
ApplicationManifest.xml
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest ApplicationTypeName="Proton.SFType"
ApplicationTypeVersion="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Parameters>
<Parameter Name="Proton.Testing_InstanceCount" DefaultValue="-1" />
</Parameters>
<!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion
should match the Name and Version attributes of the ServiceManifest element defined in the
ServiceManifest.xml file. -->
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="Proton.TestingPkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<ContainerHostPolicies CodePackageRef="Code">
<!-- See https://aka.ms/I7z0p9 for how to encrypt your repository password -->
<RepositoryCredentials AccountName="ProtonCluster" Password="XXX" PasswordEncrypted="false" />
<PortBinding ContainerPort="80" EndpointRef="Proton.TestingTypeEndpoint" />
</ContainerHostPolicies>
</Policies>
</ServiceManifestImport>
<DefaultServices>
<!-- The section below creates instances of service types, when an instance of this
application type is created. You can also create one or more instances of service type using the
ServiceFabric PowerShell module.
The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
<Service Name="Proton.Testing" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="Proton.TestingType" InstanceCount="[Proton.Testing_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
</ApplicationManifest>
I created 2 Stateless Service Fabric services, that I need both exposed and be accessible from the web via https:
Engine, (Asp.net Core API) exposed via HTTP on port 1212 and HTTPS on port 8465
Website (Asp.net Core Web App) exposed via HTTPS on port 443
I'm for now LOCAL ONLY, using WebListener.
ServiceManifest.XML ENGINE
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="EnginePkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<StatelessServiceType ServiceTypeName="EngineType" />
</ServiceTypes>
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>Engine.exe</Program>
<WorkingFolder>CodePackage</WorkingFolder>
</ExeHost>
</EntryPoint>
</CodePackage>
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<Endpoint Protocol="http" Name="EngineEndpoint" Type="Input" Port="1212" />
<Endpoint Protocol="https" Name="EngineEndpointSecure" Type="Input" Port="8465" />
</Endpoints>
</Resources>
</ServiceManifest>
ServiceManifest.XML WEBSITE
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="WebsitePkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<StatelessServiceType ServiceTypeName="WebsiteType" />
</ServiceTypes>
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>Website.exe</Program>
<WorkingFolder>CodePackage</WorkingFolder>
</ExeHost>
</EntryPoint>
</CodePackage>
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<Endpoint Protocol="https" Name="WebsiteEndpoint" Type="Input" Port="443" />
</Endpoints>
</Resources>
</ServiceManifest>
ENGINE.CS
internal sealed class Engine : StatelessService
{
public Engine(StatelessServiceContext context)
: base(context)
{ }
/// <summary>
/// Optional override to create listeners (like tcp, http) for this service instance.
/// </summary>
/// <returns>The collection of listeners.</returns>
protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
return new ServiceInstanceListener[]
{
new ServiceInstanceListener(serviceContext =>
new WebListenerCommunicationListener(serviceContext, "EngineEndpoint", (url, listener) =>
{
ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting WebListener on {url}");
return new WebHostBuilder().UseWebListener()
.ConfigureServices(
services => services
.AddSingleton(serviceContext))
.UseContentRoot(Directory.GetCurrentDirectory())
.UseStartup<Startup>()
.UseApplicationInsights()
.UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
.UseUrls(url)
.Build();
}), "EngineEndpoint"),//Name is important for multiple endpoints
new ServiceInstanceListener(serviceContext =>
new WebListenerCommunicationListener(serviceContext, "EngineEndpointSecure", (url, listener) =>
{
ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting Secure WebListener on {url}");
return new WebHostBuilder().UseWebListener()
.ConfigureServices(
services => services
.AddSingleton(serviceContext))
.UseContentRoot(Directory.GetCurrentDirectory())
.UseStartup<Startup>()
.UseApplicationInsights()
.UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
.UseUrls(url)
.Build();
}), "EngineEndpointSecure")
};
}
}
WEBSITE.CS
internal sealed class Website : StatelessService
{
public Website(StatelessServiceContext context)
: base(context)
{ }
/// <summary>
/// Optional override to create listeners (like tcp, http) for this service instance.
/// </summary>
/// <returns>The collection of listeners.</returns>
protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
return new ServiceInstanceListener[]
{
new ServiceInstanceListener(serviceContext =>
new WebListenerCommunicationListener(serviceContext, "WebsiteEndpoint", (url, listener) =>
{
ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting WebListener on {url}");
return new WebHostBuilder().UseWebListener()
.ConfigureServices(
services => services
.AddSingleton(serviceContext))
.UseContentRoot(Directory.GetCurrentDirectory())
.UseStartup<Startup>()
.UseApplicationInsights()
.UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
.UseUrls(url)
.Build();
}), "WebsiteEndpoint"),
};
}
}
APPLICATIONMANIFEST.XML
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="ProjectSFType" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<Parameters>
<!--STATELESS-->
<Parameter Name="Engine_InstanceCount" DefaultValue="-1" />
<Parameter Name="Website_InstanceCount" DefaultValue="-1" />
</Parameters>
<ServiceManifestImport>
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="WebsiteEndpoint" CertificateRef="FabricFront" />
<EndpointBindingPolicy EndpointRef="EngineEndpointSecure" CertificateRef="FabricFront" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="EnginePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="WebsitePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
</ServiceManifestImport>
<DefaultServices>
<Service Name="Engine">
<StatelessService ServiceTypeName="EngineType" InstanceCount="[Engine_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
<Service Name="Website">
<StatelessService ServiceTypeName="WebsiteType" InstanceCount="[Website_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
<Certificates>
<EndpointCertificate X509FindValue="0000000000000" Name="FabricFront" />
</Certificates>
</ApplicationManifest>
What happens in Local Cluster when launched:
Engine HTTP on port 1212 always works
Website HTTPS endpoint on port 443 works ONLY if the Engine HTTPS endpoint is removed as endpoint and from Engine.cs
Engine HTTPS endpoint NEVER works even if Website is switched to HTTP and is so the only Secure endpoint in the solution, the browser show "ERR_CONNECTION RESET" error.
On ServiceFabric Explorer I see both active and Running, no errors in the Output.
I tried other ports with same results.
How can I make this work?
The error was that I was placing <Policies> in ApplicationManifest.xml on the top of the two <ServiceManifestImport>.
Each Service needs a different <Policy> inside its own<ServiceManifestImport> just below <ConfigOverrides />.
It´s a bit unclear with your question but if you are trying to run one on port 1601 and one on 443 and only 443 succeeds. Then it might be a privilegie problem? Different ports requires different privilegies.
On the other hand if you are trying to bind both on port 443 then it´s likely you get a conflict since they both use the same port and url. We had the same problem and we managed to get past it by doing the following:
Creating a HttpSetup Application that runs a powershell script that
Installs our certificate and registers it using netsh
&netsh http add sslcert hostnameport="${EndpointHost}:${EndpointPort}" certhash=$CertThumbprint certstorename=$CertStore appid=$AppId
Binds the certificates urls, using netsh:
&netsh http add urlacl url=$ReservationUrl"
Example urls
https://mydnsname.com/
https://mydnsname.com/api
In our main application we then bind our services to the full urls since with full urls there is no conflict. We pass the urls to the cluster via environment parameters.
NOTE: The reason we had to split up into two applications was because our main application deployed continously on each commit. And when netsh was run from multiple deployments at the same time it locked up and hanged on the nodes.
ApplicationManifest.xml
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="ApiPkg" ServiceManifestVersion="1.0.0" />
<EnvironmentOverrides CodePackageRef="Code">
<EnvironmentVariable Name="EndpointUri" Value="[Api_EndpointUri]" />
<EnvironmentVariable Name="CertThumbprint" Value="[Api_CertThumbprint]" />
</EnvironmentOverrides>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="UiPkg" ServiceManifestVersion="1.0.0" />
<EnvironmentOverrides CodePackageRef="Code">
<EnvironmentVariable Name="EndpointUri" Value="[App_EndpointUri]" />
<EnvironmentVariable Name="CertThumbprint" Value="[App_CertThumbprint]" />
</EnvironmentOverrides>
</ServiceManifestImport>
ServiceManifest.xml (for both packages)
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="UiPkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<StatelessServiceType ServiceTypeName="UiType" />
</ServiceTypes>
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>Ui.exe</Program>
<WorkingFolder>CodePackage</WorkingFolder>
</ExeHost>
</EntryPoint>
<EnvironmentVariables>
<EnvironmentVariable Name="EndpointUri" Value="" />
<EnvironmentVariable Name="CertThumbprint" Value="" />
</EnvironmentVariables>
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- To bind to a specific hostname use netsh from a SetupEntyPoint and change Protocol to tcp here to just open the firewall
-->
<Endpoint Name="ServiceEndpoint" Protocol="tcp" Port="443" />
</Endpoints>
</Resources>
</ServiceManifest>
Program.cs
var listeningAddress = $"{Environment.GetEnvironmentVariable("Api_EndpointUri")}:443/api/";
_webHost = new WebHostBuilder().UseWebListener()
.UseContentRoot(Directory.GetCurrentDirectory())
.UseStartup<Startup>()
.UseUrls(listeningAddress)
.Build();
We have a Service Fabric Service project with multiple services: Actors, Stateful services and Stateless services combined into one ServiceManifest.
Two stateful services did not work: the constructors were called, the communicationlisteners (through remoting) were created, but the RunAsync method was not called.
After removing the endpoint listing from the ServiceManifest.xml the services started working again. But now we are left wondering why and how this works. Could someone explain?
To illustrate, the relevant section was
<Resources>
<Endpoints>
<Endpoint Name="WebServiceEndpoint" Type="Input" Protocol="http" Port="80" />
<Endpoint Name="StatelessServiceEndpoint1" Type="Input" Protocol="http" Port="10101" />
<Endpoint Name="ActorServiceEndpoint1" />
<Endpoint Name="ActorServiceReplicatorEndpoint1" />
<Endpoint Name="ActorServiceEndpoint2" />
<Endpoint Name="ActorServiceReplicatorEndpoint2" />
<Endpoint Name="ActorServiceEndpoint3" />
<Endpoint Name="ActorServiceReplicatorEndpoint3" />
<Endpoint Name="ActorServiceEndpoint4" />
<Endpoint Name="ActorServiceReplicatorEndpoint4" />
<Endpoint Name="StatefulServiceEndpoint1" Type="Input" Protocol="http" />
<Endpoint Name="StatefulServiceReplicatorEndpoint1" />
<Endpoint Name="StatefulServiceEndpoint2" Type="Input" Protocol="http" />
<Endpoint Name="StatefulServiceReplicatorEndpoint2" />
<Endpoint Name="StatelessServiceEndPoint2" Type="Input" Protocol="http" />
</Endpoints>
</Resources>
After changing it to this
<Resources>
<Endpoints>
<Endpoint Name="WebServiceEndpoint" Type="Input" Protocol="http" Port="80" />
<Endpoint Name="StatelessServiceEndpoint1" Protocol="http" />
<Endpoint Name="ActorServiceReplicatorEndpoint1" />
<Endpoint Name="ActorServiceReplicatorEndpoint2" />
<Endpoint Name="ActorServiceReplicatorEndpoint3" />
<Endpoint Name="ActorServiceReplicatorEndpoint4" />
<Endpoint Name="StatefulServiceReplicatorEndpoint1" />
<Endpoint Name="StatefulServiceReplicatorEndpoint2" />
</Endpoints>
</Resources>
everything worked. But why?
EDIT
The complete ServiceManifest is this:
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="Service" Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<StatefulServiceType ServiceTypeName="ActorService1Type" />
<StatefulServiceType ServiceTypeName="ActorService1Type" HasPersistedState="true" />
<StatefulServiceType ServiceTypeName="ActorService3Type" />
<StatefulServiceType ServiceTypeName="ActorService4Type" HasPersistedState="true" />
<StatefulServiceType ServiceTypeName="StatefulService1Type" HasPersistedState="true" />
<StatefulServiceType ServiceTypeName="StatefulService2Type" HasPersistedState="true" />
<StatelessServiceType ServiceTypeName="StatelessService1Type" />
<StatelessServiceType ServiceTypeName="StatelessService2Type" />
<StatelessServiceType ServiceTypeName="WebServiceType" />
</ServiceTypes>
<CodePackage Name="Code" Version="1.0.0">
<SetupEntryPoint>
<ExeHost>
<Program>Setup.exe</Program>
</ExeHost>
</SetupEntryPoint>
<EntryPoint>
<ExeHost>
<Program>Service.exe</Program>
</ExeHost>
</EntryPoint>
</CodePackage>
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<Endpoint Name="WebServiceEndpoint" Type="Input" Protocol="http" Port="80" />
<Endpoint Name="StatelessServiceEndpoint1" Protocol="http" />
<Endpoint Name="ActorServiceReplicatorEndpoint1" />
<Endpoint Name="ActorServiceReplicatorEndpoint2" />
<Endpoint Name="ActorServiceReplicatorEndpoint3" />
<Endpoint Name="ActorServiceReplicatorEndpoint4" />
<Endpoint Name="StatefulServiceReplicatorEndpoint1" />
<Endpoint Name="StatefulServiceReplicatorEndpoint2" />
</Endpoints>
</Resources>
</ServiceManifest>
Hard to know what happened in your initial reported case since there's no specific error or error message to work off of, but usually this is port conflicts when you end up sharing ports that you don't really want to or which can't be shared, or port exhaustion.
The endpoint resource in your service manifest is mainly for times when:
you want SF to help with allocating communication resources like ports for your services
you want SF to help configure those resources:
Allocating some port and consistently assigning it to some set of workloads
Punching a hole in the local firewall
Setting up a URLACL (relevant to http on windows through http.sys only)
Setting up and configuring certs to enable secure communication (same caveat)
In general you're free to ignore the endpoint resource if you don't need/want the help, since SF really is expecting the service code to do it's setup. In cases where you're not really using SF's programming models then the endpoint resource is more important since it's how you communicate to SF what your endpoints are.
The behavior you get really depends on the transport you're using, as well as the OS's dynamic port range and the Application port range that you've defined, as well as what the service code actually does.
https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-fabric-settings#section-name-fabricnode
Let's say you're setting up an http communication listener in your service like so, and walk through a few examples of what happens when you define and endpoint in your manifest or don't.
1) Let's say you put nothing in your service manifest about endpoints. This means that effectively you're specifying 0 as the port in code. In this case SF's not doing any allocation or management. The port is getting assigned by the OS from the OS dynamic port range. The port that actually gets assigned will be different for each service instance listener. This should work as a reasonable default choice in most scenarios.
2) Let's say you're specifying an endpoint in the manifest and not specifying any port at all, i.e.:
<Endpoint Name="HealthServiceEndpoint"/>
In this case, the port that is assigned will come from the SF application port range. It will be the same for any service instances hosted in the same process, but different across processes. (So it matters if you are using the Exclusive or Shared process hosting model) This also presumes that reusing the port is supported for your transport. Most transports don't (like http on via Kestrel in .NET, TCP under most cases), but there are some notable examples (http.sys based http transports on Windows like WebListener/HttpSys, tcp via net.tcp in WCF probably a few others).
3) Let's say you're specifying an endpoint in the service fabric manifest and explicitly specifying 0 for that port i.e.:
<Endpoint Name="HealthServiceEndpoint" Port="0" Protocol="http"/>
In this case the port that gets assigned will be from the OS dynamic port range, and it will end up the same/shared for any service instances hosted in the same process that use that endpoint. The port will be different across processes. (So again it matters if you are using the Exclusive or Shared process hosting model)
4) Naturally if the endpoint is specified and a specific port is specified, that port will be used for all service instances both within and across processes. This somewhat implicitly assumes that such sharing is going to work, which again depends on your transport and platform, or that you're never planning on running more than one instance of the service on this node.
Other trivia:
the "transport" element mainly determines whether SF registers your url with http.sys on windows or configures certificates to secure traffic (most of this can be done within your service code or a SetupEntryPoint).
as of this writing Type is ignored (this is a holdover from an older version of SF)
PathSuffix is used to create a default uri fragment that gets appended to the IP and port assigned by the platform. This is used in cases where there's code not using SF's listener APIs that sets up some listener on a different path like /api/value, like a the code inside a container might do.