We have deployed various stateless services on a 5 node cluster with -1 as instance count as Singleton partition scheme. Recently, we decided to deploy the few stateless services only on 3 nodes out of 5 by defining instance count as 3.
After deployment, the stateless services with -1 as instance count are working and responding with HttpStatus 200 Ok. however, a stateless service deployed with 3 instance node count are intermittently responding with HttpStatus 502 with following error (from fiddler):
The connection to 'someservername.centralus.cloudapp.azure.com' failed.
System.Security.SecurityException Failed to negotiate HTTPS connection with server.fiddler.network.https> HTTPS handshake to someservername.centralus.cloudapp.azure.com failed. System.IO.IOException Authentication failed because the remote party has closed the transport stream.
Below is the application manifest of deployed application for reference
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="MyService.ServiceFabricType" ApplicationTypeVersion="1.0.0.1.1" ManifestId="8747c387-a7fc-4b05-b189-b1c01958f066" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<Parameters>
<Parameter Name="My_Service_ASPNETCORE_ENVIRONMENT" DefaultValue="" />
<Parameter Name="My_Service_InstanceCount" DefaultValue="3" />
</Parameters>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="MyServicePkg" ServiceManifestVersion="1.0.0.1.1" />
<ConfigOverrides />
<EnvironmentOverrides CodePackageRef="code">
<EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[My_Service_ASPNETCORE_ENVIRONMENT]" />
</EnvironmentOverrides>
</ServiceManifestImport>
<DefaultServices>
<Service Name="MyService" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="MyServiceType" InstanceCount="[My_Service_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
</ApplicationManifest>
and service manifest :
<ServiceManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ManifestId="59ea463b-5e4c-44f5-8982-5658b35d6c89" Name="MyServicePkg" Version="1.0.0.1.1" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<ServiceTypes>
<StatelessServiceType ServiceTypeName="MyService" />
</ServiceTypes>
<CodePackage Name="Code" Version="1.0.0.1.1">
<EntryPoint>
<ExeHost>
<Program>MyService.exe</Program>
<WorkingFolder>CodePackage</WorkingFolder>
</ExeHost>
</EntryPoint>
<EnvironmentVariables>
<EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="" />
</EnvironmentVariables>
</CodePackage>
<ConfigPackage Name="Config" Version="1.0.0.1.1" />
<Resources>
<Endpoints>
<Endpoint Name="ServiceEndpoint" Protocol="https" Type="Input" Port="9226" />
</Endpoints>
</Resources>
</ServiceManifest>
Is it mandatory to deploy a stateless service all nodes in service fabric?
If no, how the above scenario can be configured?
Note - Currently Service Fabric is configured with Silver durability tier and with reverse proxy in disabled state. Also did not get any relevant solution from this azure documentation.
I have a local Service Fabric cluster of 5 nodes and I have a problem deploying my application on all nodes. When set to "1 Node", the cluster works fine. When set to "5 Nodes" it gives an error on all nodes but one. This is the error/warning message:
Error event: SourceId='System.Hosting', Property='CodePackageActivation:Code:EntryPoint:131919316927686034'.
There was an error during CodePackage activation.System.Fabric.FabricException (-2147017731)
Failed to start Container. ContainerName=sf-0-28e0002f-fd7d-412c-81b8-b78ca5339ce4_865991cc-9c36-493f-9b3d-95f6eba43851, ApplicationId=Proton.SFType_App0, ApplicationName=fabric:/Proton.SF.
DockerRequest returned StatusCode=InternalServerError with ResponseBody={"message":"failed to create endpoint sf-0-28e0002f-fd7d-412c-81b8-b78ca5339ce4_865991cc-9c36-493f-9b3d-95f6eba43851 on network nat: HNS failed with error : You were not connected because a duplicate name
The error message looks truncated. The application loads up fine, but on one node only. Am I missing something in 5-node configuration? The application we are deploying is a container which runs a .NET Core app. I've attached a screenshot of the error in Service Fabric Explorer.
Error Screenshot
ServiceManifest.xml
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="Proton.TestingPkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<!-- This is the name of your ServiceType.
The UseImplicitHost attribute indicates this is a guest service. -->
<StatelessServiceType ServiceTypeName="Proton.TestingType" UseImplicitHost="true" />
</ServiceTypes>
<!-- Code package is your service executable. -->
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<!-- Follow this link for more information about deploying Windows containers to Service Fabric: https://aka.ms/sfguestcontainers -->
<ContainerHost>
<ImageName>proton.azurecr.io/protontesting:latest</ImageName>
</ContainerHost>
</EntryPoint>
<!-- Pass environment variables to your container: -->
<!--
<EnvironmentVariables>
<EnvironmentVariable Name="VariableName" Value="VariableValue"/>
</EnvironmentVariables>
-->
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- This endpoint is used by the communication listener to obtain the port on which to
listen. Please note that if your service is partitioned, this port is shared with
replicas of different partitions that are placed in your code. -->
<Endpoint Name="Proton.TestingTypeEndpoint" Port="8001" />
</Endpoints>
</Resources>
</ServiceManifest>
ApplicationManifest.xml
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest ApplicationTypeName="Proton.SFType"
ApplicationTypeVersion="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Parameters>
<Parameter Name="Proton.Testing_InstanceCount" DefaultValue="-1" />
</Parameters>
<!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion
should match the Name and Version attributes of the ServiceManifest element defined in the
ServiceManifest.xml file. -->
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="Proton.TestingPkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
<Policies>
<ContainerHostPolicies CodePackageRef="Code">
<!-- See https://aka.ms/I7z0p9 for how to encrypt your repository password -->
<RepositoryCredentials AccountName="ProtonCluster" Password="XXX" PasswordEncrypted="false" />
<PortBinding ContainerPort="80" EndpointRef="Proton.TestingTypeEndpoint" />
</ContainerHostPolicies>
</Policies>
</ServiceManifestImport>
<DefaultServices>
<!-- The section below creates instances of service types, when an instance of this
application type is created. You can also create one or more instances of service type using the
ServiceFabric PowerShell module.
The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
<Service Name="Proton.Testing" ServicePackageActivationMode="ExclusiveProcess">
<StatelessService ServiceTypeName="Proton.TestingType" InstanceCount="[Proton.Testing_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
</ApplicationManifest>
I am running a Log4J2 TCPSocketServer on an edge node in a cluster. All the data nodes send log events to the TCPSocketServer on the edge node and also log locally in the data node using the log4j2.xml configuration file as shown below. The Application Name is stored as a System property and is accessible in the data node or client's log4j2.xml configuration using ${sys:ABC.appname}. How can I send the same appname to the edge node where TCPSocketServer is running using the log4j2.xml. I would be using the same Application Name in the log4j2-server.xml to log events into separate log files just like I am doing locally on data node.
Sample snippet from data node or Client - log4j2.xml :
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" name="MyApp" >
<Appenders>
<Socket name="socket" host="localhost" port="12345" >
<SerializedLayout />
</Socket>
<File name="MyFile" fileName="/var/log/${sys:ABC.appname}.log" >
<PatternLayout>
<Pattern>%d{ISO8601} %p %c{1.} [%t] %m%n</Pattern>
</PatternLayout>
</File>
</Appenders>
<Loggers>
<Root level="debug">
<AppenderRef ref="socket"/>
<AppenderRef ref="MyFile"/>
</Root>
</Loggers>
</Configuration>
Sample snippet from edge node or Server - log4j2-server.xml
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<File name="MyFile" fileName="/var/log/data/${hostName}-<**This is where I would like to see the appname from data node**>.log" >
<PatternLayout>
<Pattern>%d{ISO8601} %p %c{1.} [%t] %m%n</Pattern>
</PatternLayout>
</File>
<Async name="AsyncFile">
<AppenderRef ref="MyFile" />
</Async>
</Appenders>
<Loggers>
<Root level="WARN">
<AppenderRef ref="AsyncFile"/>
</Root>
</Loggers>
</Configuration>
I used ThreadContext to resolve this issue. Its pretty easy to add Thread Context into the codebase and later use Routing to segregate the log events based on the ThreadContext. I followed the example on this link to do the same https://logging.apache.org/log4j/2.x/faq.html#separate_log_files
I created 2 Stateless Service Fabric services, that I need both exposed and be accessible from the web via https:
Engine, (Asp.net Core API) exposed via HTTP on port 1212 and HTTPS on port 8465
Website (Asp.net Core Web App) exposed via HTTPS on port 443
I'm for now LOCAL ONLY, using WebListener.
ServiceManifest.XML ENGINE
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="EnginePkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<StatelessServiceType ServiceTypeName="EngineType" />
</ServiceTypes>
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>Engine.exe</Program>
<WorkingFolder>CodePackage</WorkingFolder>
</ExeHost>
</EntryPoint>
</CodePackage>
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<Endpoint Protocol="http" Name="EngineEndpoint" Type="Input" Port="1212" />
<Endpoint Protocol="https" Name="EngineEndpointSecure" Type="Input" Port="8465" />
</Endpoints>
</Resources>
</ServiceManifest>
ServiceManifest.XML WEBSITE
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="WebsitePkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<StatelessServiceType ServiceTypeName="WebsiteType" />
</ServiceTypes>
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>Website.exe</Program>
<WorkingFolder>CodePackage</WorkingFolder>
</ExeHost>
</EntryPoint>
</CodePackage>
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<Endpoint Protocol="https" Name="WebsiteEndpoint" Type="Input" Port="443" />
</Endpoints>
</Resources>
</ServiceManifest>
ENGINE.CS
internal sealed class Engine : StatelessService
{
public Engine(StatelessServiceContext context)
: base(context)
{ }
/// <summary>
/// Optional override to create listeners (like tcp, http) for this service instance.
/// </summary>
/// <returns>The collection of listeners.</returns>
protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
return new ServiceInstanceListener[]
{
new ServiceInstanceListener(serviceContext =>
new WebListenerCommunicationListener(serviceContext, "EngineEndpoint", (url, listener) =>
{
ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting WebListener on {url}");
return new WebHostBuilder().UseWebListener()
.ConfigureServices(
services => services
.AddSingleton(serviceContext))
.UseContentRoot(Directory.GetCurrentDirectory())
.UseStartup<Startup>()
.UseApplicationInsights()
.UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
.UseUrls(url)
.Build();
}), "EngineEndpoint"),//Name is important for multiple endpoints
new ServiceInstanceListener(serviceContext =>
new WebListenerCommunicationListener(serviceContext, "EngineEndpointSecure", (url, listener) =>
{
ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting Secure WebListener on {url}");
return new WebHostBuilder().UseWebListener()
.ConfigureServices(
services => services
.AddSingleton(serviceContext))
.UseContentRoot(Directory.GetCurrentDirectory())
.UseStartup<Startup>()
.UseApplicationInsights()
.UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
.UseUrls(url)
.Build();
}), "EngineEndpointSecure")
};
}
}
WEBSITE.CS
internal sealed class Website : StatelessService
{
public Website(StatelessServiceContext context)
: base(context)
{ }
/// <summary>
/// Optional override to create listeners (like tcp, http) for this service instance.
/// </summary>
/// <returns>The collection of listeners.</returns>
protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
return new ServiceInstanceListener[]
{
new ServiceInstanceListener(serviceContext =>
new WebListenerCommunicationListener(serviceContext, "WebsiteEndpoint", (url, listener) =>
{
ServiceEventSource.Current.ServiceMessage(serviceContext, $"Starting WebListener on {url}");
return new WebHostBuilder().UseWebListener()
.ConfigureServices(
services => services
.AddSingleton(serviceContext))
.UseContentRoot(Directory.GetCurrentDirectory())
.UseStartup<Startup>()
.UseApplicationInsights()
.UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.None)
.UseUrls(url)
.Build();
}), "WebsiteEndpoint"),
};
}
}
APPLICATIONMANIFEST.XML
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="ProjectSFType" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<Parameters>
<!--STATELESS-->
<Parameter Name="Engine_InstanceCount" DefaultValue="-1" />
<Parameter Name="Website_InstanceCount" DefaultValue="-1" />
</Parameters>
<ServiceManifestImport>
<ConfigOverrides />
<Policies>
<EndpointBindingPolicy EndpointRef="WebsiteEndpoint" CertificateRef="FabricFront" />
<EndpointBindingPolicy EndpointRef="EngineEndpointSecure" CertificateRef="FabricFront" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="EnginePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="WebsitePkg" ServiceManifestVersion="1.0.0" />
<ConfigOverrides />
</ServiceManifestImport>
<DefaultServices>
<Service Name="Engine">
<StatelessService ServiceTypeName="EngineType" InstanceCount="[Engine_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
<Service Name="Website">
<StatelessService ServiceTypeName="WebsiteType" InstanceCount="[Website_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
<Certificates>
<EndpointCertificate X509FindValue="0000000000000" Name="FabricFront" />
</Certificates>
</ApplicationManifest>
What happens in Local Cluster when launched:
Engine HTTP on port 1212 always works
Website HTTPS endpoint on port 443 works ONLY if the Engine HTTPS endpoint is removed as endpoint and from Engine.cs
Engine HTTPS endpoint NEVER works even if Website is switched to HTTP and is so the only Secure endpoint in the solution, the browser show "ERR_CONNECTION RESET" error.
On ServiceFabric Explorer I see both active and Running, no errors in the Output.
I tried other ports with same results.
How can I make this work?
The error was that I was placing <Policies> in ApplicationManifest.xml on the top of the two <ServiceManifestImport>.
Each Service needs a different <Policy> inside its own<ServiceManifestImport> just below <ConfigOverrides />.
It´s a bit unclear with your question but if you are trying to run one on port 1601 and one on 443 and only 443 succeeds. Then it might be a privilegie problem? Different ports requires different privilegies.
On the other hand if you are trying to bind both on port 443 then it´s likely you get a conflict since they both use the same port and url. We had the same problem and we managed to get past it by doing the following:
Creating a HttpSetup Application that runs a powershell script that
Installs our certificate and registers it using netsh
&netsh http add sslcert hostnameport="${EndpointHost}:${EndpointPort}" certhash=$CertThumbprint certstorename=$CertStore appid=$AppId
Binds the certificates urls, using netsh:
&netsh http add urlacl url=$ReservationUrl"
Example urls
https://mydnsname.com/
https://mydnsname.com/api
In our main application we then bind our services to the full urls since with full urls there is no conflict. We pass the urls to the cluster via environment parameters.
NOTE: The reason we had to split up into two applications was because our main application deployed continously on each commit. And when netsh was run from multiple deployments at the same time it locked up and hanged on the nodes.
ApplicationManifest.xml
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="ApiPkg" ServiceManifestVersion="1.0.0" />
<EnvironmentOverrides CodePackageRef="Code">
<EnvironmentVariable Name="EndpointUri" Value="[Api_EndpointUri]" />
<EnvironmentVariable Name="CertThumbprint" Value="[Api_CertThumbprint]" />
</EnvironmentOverrides>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="UiPkg" ServiceManifestVersion="1.0.0" />
<EnvironmentOverrides CodePackageRef="Code">
<EnvironmentVariable Name="EndpointUri" Value="[App_EndpointUri]" />
<EnvironmentVariable Name="CertThumbprint" Value="[App_CertThumbprint]" />
</EnvironmentOverrides>
</ServiceManifestImport>
ServiceManifest.xml (for both packages)
<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="UiPkg"
Version="1.0.0"
xmlns="http://schemas.microsoft.com/2011/01/fabric"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<ServiceTypes>
<StatelessServiceType ServiceTypeName="UiType" />
</ServiceTypes>
<CodePackage Name="Code" Version="1.0.0">
<EntryPoint>
<ExeHost>
<Program>Ui.exe</Program>
<WorkingFolder>CodePackage</WorkingFolder>
</ExeHost>
</EntryPoint>
<EnvironmentVariables>
<EnvironmentVariable Name="EndpointUri" Value="" />
<EnvironmentVariable Name="CertThumbprint" Value="" />
</EnvironmentVariables>
</CodePackage>
<!-- Config package is the contents of the Config directoy under PackageRoot that contains an
independently-updateable and versioned set of custom configuration settings for your service. -->
<ConfigPackage Name="Config" Version="1.0.0" />
<Resources>
<Endpoints>
<!-- To bind to a specific hostname use netsh from a SetupEntyPoint and change Protocol to tcp here to just open the firewall
-->
<Endpoint Name="ServiceEndpoint" Protocol="tcp" Port="443" />
</Endpoints>
</Resources>
</ServiceManifest>
Program.cs
var listeningAddress = $"{Environment.GetEnvironmentVariable("Api_EndpointUri")}:443/api/";
_webHost = new WebHostBuilder().UseWebListener()
.UseContentRoot(Directory.GetCurrentDirectory())
.UseStartup<Startup>()
.UseUrls(listeningAddress)
.Build();
I'm able to start the OrientDb studio using and embedded (plocal) server. However, I have to grant permissions to RuntimePermission.createClassLoader. I'd rather not grant that permission and I may not need a class loader; hot redeployment may not fit my needs and the server will be embedded (not independent from my application).
Is it possible to load OrientDb Studio without a class loader?
Here is my code to start the server:
OServer server;
System.setProperty("ORIENTDB_HOME", getGraphDatabaseDirectoryPath());
server = OServerMain.create();
server.startup(orientServerConfigFile.getInputStream());
server.activate();
orientServerConfigFile points to my confile. Here is my config file:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<orient-server>
<network>
<protocols>
<protocol implementation="com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary" name="binary"/>
<protocol name="http" implementation="com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpDb"/>
</protocols>
<listeners>
<listener protocol="binary" socket="default" port-range="2424-2430" ip-address="0.0.0.0"/>
<listener protocol="http" port-range="2480-2485" ip-address="0.0.0.0">
<commands>
<command implementation="com.orientechnologies.orient.server.network.protocol.http.command.get.OServerCommandGetStaticContent" pattern="GET|www GET|studio/ GET| GET|*.htm GET|*.html GET|*.xml GET|*.jpeg GET|*.jpg GET|*.png GET|*.gif GET|*.js GET|*.css GET|*.swf GET|*.ico GET|*.txt GET|*.otf GET|*.pjs GET|*.svg">
<parameters>
<entry value="Cache-Control: no-cache, no-store, max-age=0, must-revalidate\r\nPragma: no-cache" name="http.cache:*.htm *.html"/>
<entry value="Cache-Control: max-age=120" name="http.cache:default"/>
</parameters>
</command>
</commands>
</listener>
</listeners>
</network>
<users>
<user resources="*" password="root" name="root"/>
<user resources="connect,server.listDatabases,server.dblist" password="guest" name="guest"/>
</users>
<properties>
<entry value="1" name="db.pool.min"/>
<entry value="50" name="db.pool.max"/>
<entry value="true" name="profiler.enabled"/>
</properties>