Trafeik , Service Fabric Backend , Stateless service, not reachable, always red in explorer - azure-service-fabric

Question

I have deployed trafeik in secured service fabric cluster , and i have made settings in servicemanifest of one of my application to use trafeik, eventhough trafeik is able to identify service but backend in trafeik is always red and my service is unreachable, if i use direct endpoint to my service it is reachable, need help in resolving same.
below is the toml of trafeik -
debug = true
logLevel = "INFO"
defaultEntryPoints = ["https"]
[entryPoints]
[entryPoints.https]
address = ":16080"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
certFile = "certs/servicefabric.crt"
keyFile = "certs/servicefabric.key"
[entryPoints.traefik]
address = ":16081"
[api]
entryPoint = "traefik"
dashboard = true
debug = true
[servicefabric]
clustermanagementurl = "https://localhost:19080"
apiversion = "3.0"
[serviceFabric.tls]
cert = "certs/servicefabric.crt"
key = "certs/servicefabric.key"
insecureskipverify = true
Service manifest of trafik service -
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="TraefikPkg" Version="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<!-- This is the name of your ServiceType.
The UseImplicitHost attribute indicates this is a guest executable service. -->
<StatelessServiceType ServiceTypeName="TraefikType" UseImplicitHost="true" />
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<!-- The SetupEntryPoint is an optional element used to specify a
program to be executed before the service's code is launched. -->
<EntryPoint>
<ExeHost>
<Program>traefik.exe</Program>
<Arguments>--configfile=traefik.toml</Arguments>
<WorkingFolder>CodePackage</WorkingFolder>
<!-- Uncomment to log console output (both stdout and stderr) to one of the
service's working directories. Do not use in production. -->
<!--<ConsoleRedirection FileRetentionCount="5" FileMaxSizeInKb="2048" />-->
</ExeHost>
</EntryPoint>
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<Endpoint Protocol="https" Name="TraefikTypeEndpoint" Type="Input" Port="16080" CertificateRef="ClusterThumbprint"/>
<Endpoint Protocol="https" Name="TraefikTypeAPIEndpoint" Type="Input" Port="16081" CertificateRef="ClusterThumbprint"/>
</Endpoints>
</Resources>
</ServiceManifest>
Application manifest of trafeik service -
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest ApplicationTypeName="TraefikType" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Parameters>
<Parameter Name="Traefik_InstanceCount" DefaultValue="-1" />
<Parameter Name="Stateless1_PlacementConstraints" DefaultValue="(NodeTypeName==fntensvcp)" />
</Parameters>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="TraefikPkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<RunAsPolicy CodePackageRef="Code" UserRef="AdminUser" EntryPointType="All" />
</Policies>
</ServiceManifestImport>
<DefaultServices>
<Service Name="Traefik" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="TraefikType" InstanceCount="[Traefik_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
</DefaultServices>
<Principals>
<Users>
<User Name="AdminUser">
<MemberOf>
<SystemGroup Name="Administrators" />
</MemberOf>
</User>
</Users>
</Principals>
<Certificates>
<EndpointCertificate X509FindValue="b22bb9d3e74da3339e5f6709798107443553ae48" Name="ClusterThumbprint" />
</Certificates>
</ApplicationManifest>
For my application below is service manifest -
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="CSM.IE.SMA.C360.RevenueServicePkg" Version="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ManifestId="25bba1ad-7380-4cbf-b9ed-9a90ee3dd086">
<ServiceTypes>
<!-- This is the name of your ServiceType.
This name must match the string used in RegisterServiceType call in Program.cs. -->
<StatelessServiceType ServiceTypeName="CSM.IE.SMA.C360.RevenueServiceType">
<Extensions>
<Extension Name="Traefik">
<Labels xmlns="http://schemas.microsoft.com/2015/03/fabact-no-schema">
<Label Key="traefik.frontend.rule.c360Revenue">PathPrefixStrip: /C360/RevenueService</Label>
<Label Key="traefik.enable">true</Label>
<Label Key="traefik.frontend.passHostHeader">true</Label>
</Labels>
</Extension>
</Extensions>
</StatelessServiceType>
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>CSM.IE.SMA.C360.RevenueService.exe</Program>
</ExeHost>
</EntryPoint>
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<!--<Endpoint Name="ServiceEndpoint" Protocol="http" Port="8082"/>-->
<Endpoint Protocol="https" Name="ServiceEndpoint" Type="Input" Port="19616" CertificateRef="ClusterThumbprint" />
</Endpoints>
</Resources>
</ServiceManifest>
for my application below is application manifest file -
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="IE_SmartAssistance" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric" ManifestId="1ff9d8a9-ecbc-4d55-931e-eb59eb229f15">
<Parameters>
<Parameter Name="MultiTPIDConsumptionService_InstanceCount" DefaultValue="2" />
<Parameter Name="SupportService_InstanceCount" DefaultValue="2" />
<Parameter Name="ConsumptionService_InstanceCount" DefaultValue="2" />
<Parameter Name="AgreementService_InstanceCount" DefaultValue="2" />
<Parameter Name="RevenueService_InstanceCount" DefaultValue="2" />
<Parameter Name="ProfileProxyService_InstanceCount" DefaultValue="2" />
<Parameter Name="Stateless1_PlacementConstraints" DefaultValue="(NodeTypeName==fntensvcp)" />
</Parameters>
<!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion
should match the Name and Version attributes of the ServiceManifest element defined in the
ServiceManifest.xml file. -->
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.MultiTPIDConsumptionServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.AgreementServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.RevenueServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.ProfileProxyServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.ConsumptionServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="CSM.IE.SMA.C360.SupportServicePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="ServiceEndpoint" CertificateRef="ClusterThumbprint" />
<RunAsPolicy CodePackageRef="Code" UserRef="LocalAdmin" />
</Policies>
</ServiceManifestImport>
<DefaultServices>
<!-- The section below creates instances of service types, when an instance of this
application type is created. You can also create one or more instances of service type using the
ServiceFabric PowerShell module.
The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
<Service Name="CSM.IE.SMA.C360.MultiTPIDConsumptionService" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.MultiTPIDConsumptionServiceType" InstanceCount="[MultiTPIDConsumptionService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
<Service Name="IE_SmartAssistance_Customer360Service_AgreementService-Microservice_API">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.AgreementServiceType" InstanceCount="[AgreementService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
<Service Name="RevenueService-Microservice_API" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.RevenueServiceType" InstanceCount="[RevenueService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
<Service Name="CSM.IE.SMA.C360.ProfileProxyService" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.ProfileProxyServiceType" InstanceCount="[ProfileProxyService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
<Service Name="CSM.IE.SMA.C360.ConsumptionService" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.ConsumptionServiceType" InstanceCount="[ConsumptionService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
<Service Name="CSM.IE.SMA.C360.SupportService" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="CSM.IE.SMA.C360.SupportServiceType" InstanceCount="[SupportService_InstanceCount]">
<SingletonPartition />
<PlacementConstraints>[Stateless1_PlacementConstraints]</PlacementConstraints>
</StatelessService>
</Service>
</DefaultServices>
<Principals>
<Users>
<User Name="LocalAdmin" AccountType="NetworkService">
<MemberOf>
<SystemGroup Name="Administrators" />
</MemberOf>
</User>
</Users>
</Principals>
<Certificates>
<EndpointCertificate X509FindValue="b22bb9d3e74da3339e5f6709798107443553ae48" Name="ClusterThumbprint" />
</Certificates>
</ApplicationManifest>