I'm starting with Service Fabric. I have created a very simple console application that runs the following code:
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Hello world!");
File.AppendAllText("c:\\temp\\hello.txt", "Hello world!" + DateTime.Now.ToString() + "\r\n");
Console.ReadLine();
}
}
Then I create a guest executable project with Visual Studio and point it to the exe application. It gets installed in Service Fabric, I can see that the file is created, but then service fabric throws an error:
Error event: SourceId='System.FM', Property='State'.
Partition is below target replica or instance count.
fabric:/Test3/Test3Service -1 1 5ef5a0eb-5621-4821-95cb-4c1920ab7f0c
(Showing 0 out of 0 replicas. Total available replicas: 0.)
Is this approach correct? Can I have exe applications hosted in Service Fabric or do I need to implement/inherit from something?
EDIT
When the application is deployed it enters in a Warning state, showing the following messages:
Soon afterwards it transitions to an error state:
Yes you can host a simple Console Application in Service Fabric as a Guest Executable, that should not be a problem.
The issue you are seeing is likely because the application is trying to write to a file in c:/temp where your Guest Exe by default doesnt have permissions. Try removing that part of your sample code, or change it to write to hello.txt and it will end up in the same folder your Guest Exe is running in.
You should consider file storage on a Service Fabric node as temporary however and not rely on storing data there as your service could be moved between nodes by Service Fabric as part of it's cluster maintenance.
See this answer for some more details on file system access in SF https://stackoverflow.com/a/37966158/1062217
Related
I created the GrpcGreeter and GrpcGreeterClient projects in Visual Studio 2019 from the following page:
[https://learn.microsoft.com/en-us/aspnet/core/tutorials/grpc/grpc-start?view=aspnetcore-5.0&tabs=visual-studio][1]
The only change I made to these examples was that in order for the GrpcGreeter app to run as a Windows service, I added ".UseWindowsService()" to IHostBuilder CreateHostBuilder. I published both to local folders while in VS, and selected Self Contained for the Deployment Mode.
Server and client work fine using https://localhost:5001 when run from the either the VS environment or when running the published GrpcGreeter.exe and GrpcGreeterClient.exe directly.
I then used "Sc create" to successfully create a Windows service with GrpcGreeter.exe. Then on the Services window I started the service.
The problem is that when run as a Windows service the GrpcGreeter.exe does not listen on port 5001, as shown with netstat -anb (it does listen to port 5354, apparently). And of course when I then run GrpcGreeterClient.exe it does not connect. When GrpcGreeter.exe is run not as a Windows service netstat shows that it is listening to 5001, and GrpcGreeterClient.exe talks to it just fine.
A look at Event Viewer shows 3 errors happening immediately whenever I start the service on the Services window. I'm abbreviating them below.
1st:
Faulting application name: GrpcGreeter.exe, version: 1.0.0.0, time stamp: 0x5f6b3846
Faulting module name: ntdll.dll, version: 10.0.19041.546, time stamp: 0xd49544eb
Exception code: 0xc0000374
Fault offset: 0x000e6763
...
2nd:
Fault bucket , type 0
Event Name: FaultTolerantHeap
Response: Not available
Cab Id: 0
Problem signature:
P1: GrpcGreeter.exe
...
3rd:
Fault bucket 2242750238749681031, type 1
Event Name: APPCRASH
Response: Not available
Cab Id: 0
Problem signature:
P1: GrpcGreeter.exe
...
Please help. Thank you.
this is a very old post but I too came across with this issue when deploying a windows service with gRPC. Not sure will it solve your problem or not but my issue was that when you deploy into the windows service, it needs to have a certificate configured. It was stated in this documentation here under the "Set HTTPS certificates by using configuration" part
So I have created a self signed certificate using openssl where you can refer here too, then just add the .pfx file into the kestrel configuration as shown by the Microsoft documentation, build it and publish it as a windows service. After that, just proceed with the normal service creation procedure using
sc create
// and then
sc start
The windows service should now be running with the gRPC server without any issue (For my case at least). One thing to note is that because this is a self signed certificate which is not exactly trustable, when the frontend attempts to communicate with the server, it will have an error about the cert. You just need to trust it and it will be fine.
On browser, just go to the link that is hosting the gRPC, for example https://localhost:5001, click advanced and trust it.
In my case, I was using electron + angular so I just need to add this code snippet that I have gotten from here. Now my frontend can communicate with the gRPC server in the windows service normally.
// ignore self signed certificate in dev mode
if (process.env.NODE_ENV === 'development') {
// SSL/TSL: this is the self signed certificate support
app.on('certificate-error', (event, webContents, url, error, certificate, callback) => {
// On certificate error we disable default behaviour (stop loading the page)
// and we then say "it is all fine - true" to the callback
event.preventDefault();
callback(true);
});
}
After sleeping (in lite account type) node-red, created by node-red starter kit, crashes while starting. It is possible to login in editor for a few seconds and then it crashes with error code "an instance of the app crashed: APP/PROC/WEB: Exite with status 1 (out of memory)". Dashboard (node-red-dashboard) was installed before sleeping and worked correctly.
I tried to restart Node-RED, Stop and Start.
I solved this problem. The problem may be due to the memory overflow in the container Garden. Taking into account that the content is stored in the cache, the application cannot start after the restart process, it issues an Exit status 1 (out of memory) error.
The cache is updated only by pushing the application into the cloud.
An option that was checked for application recovery:
View the name of the database for NodeRED (which stores all information about the Node-RED) in Cloudant, for example, "nodered."
Install to PC Cloud Foundry Command Line Interface - CLI https://docs.cloudfoundry.org/cf-cli/install-go-cli.html
Download from github and unarchive the application's code bluemix-starter https://github.com/knolleary/node-red-bluemix-starter (clone or download -> download zip)
In the downloaded folder add a record to a manifest file (manifest.yml) in the env section, in which set the database name (for example, nodered) in Cloudant to environment variable NODE_RED_STORAGE_DB_NAME. Four spaces must be made before NODE_RED_STORAGE_DB_NAME. It is better to make changes using the Notepad ++ editor.
---
applications:
- memory: 256M
env:
OPTIMIZE_MEMORY: true
NODE_RED_STORAGE_DB_NAME: nodered
command: node index.js --settings ./bluemix-settings.js –v
Save the file after changing.
Run the command line (cmd) and then:
a. go to a folder with a downloaded project, such as Windows
cd c:/node-red-bluemix-starter
b. specify the api endpoint where the application is located, in our case:
cf api https://api.eu-gb.bluemix.net
c. send a registration command in the cloud
cf login
d. specify the mail and password (password is entered without explicit character display)
e. pushing the project by specifying the name of your instance Node-RED, for example NameApp
cf push NameApp
I am new to service Fabric and trying to integrate my windows service application into service fabric. For logging information, we are planning to use Application Insights. But the events are not logged if i send it through my SF application. At the same time, through a normal console/windows application, I can able to log the message to applicationinsights and can be viewed from there.
Then I tried to create a VM in azure environment, and create SF application there and send the log information to AI and its worked successfully. I copied the same codebase into my local machine and run it, its not working. I am not sure whether its related to any firewall or proxy settings. Can anyone help on this?
I have used the nuget package to install Microsoft.ApplicationInsights dll in my machine. The version that I used is 2.2.0. And I am using .Net framework 4.6.1
You could look at EventFlow to help you capture Service Fabric ETW Events from your SF services and send them to Application Insights.
It's easy enough to setup, simply add Microsoft.Diagnostics.EventFlow.ServiceFabric NuGet to your Service Fabric service project and then setup a pipline
public static void Main(string[] args)
{
try
{
using (var diagnosticsPipeline = ServiceFabricDiagnosticPipelineFactory.CreatePipeline("MyApplication-MyService-DiagnosticsPipeline"))
{
ServiceRuntime.RegisterServiceAsync("MyServiceType", ctx => new MyService(ctx)).Wait();
ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(MyService).Name);
Thread.Sleep(Timeout.Infinite);
}
}
catch (Exception e)
{
ServiceEventSource.Current.ServiceHostInitializationFailed(e.ToString());
throw;
}
}
In your eventflow.config you can then setup Application Insights as an output:
{
"inputs": [
{
"type": "EventSource",
"sources": [
{ "providerName": "Your-Service-EventSource" }
]
},
],
"filters": [
{
"type": "drop",
"include": "Level == Verbose"
}
],
"outputs": [
// Please update the instrumentationKey.
{
"type": "ApplicationInsights",
"instrumentationKey": "00000000-0000-0000-0000-000000000000"
}
],
"schemaVersion": "2016-08-11",
"extensions": []
}
An alternative to the EventFlow approach suggested by yoape would be Azure Diagnostics (WAD).
Setup WAD in SF VMSS
When you're running an Azure Service Fabric cluster, it's a good idea
to collect the logs from all the nodes in a central location. Having
the logs in a central location helps you analyze and troubleshoot
issues in your cluster, or issues in the applications and services
running in that cluster. One way to upload and collect logs is to use
the Windows Azure Diagnostics (WAD) extension, which uploads logs to
Azure Storage, and also has the option to send logs to Azure
Application Insights or Event Hubs. You can also use an external
process to read the events from storage and place them in an analysis
platform product, such as OMS Log Analytics or another log-parsing
solution.
Setup AI upload in WAD
Cloud services, Virtual Machines, Virtual Machine Scale Sets and
Service Fabric all use the Azure Diagnostics extension to collect
data. Azure diagnostics sends data to Azure Storage tables. However,
you can also pipe all or a subset of the data to other locations using
Azure Diagnostics extension 1.5 or later. This article describes how
to send data from the Azure Diagnostics extension to Application
Insights.
The nice thing about it is that it's completely managed by Azure, and you don't need to change anything in your project.
You can adapt the watchdog Service from Microsoft Samples. The watchdog Service is a generic standalone Service Fabric Stateful Service that will log data into Application Insights.
Add the watchdog app and watchdog service into your solution.
Add your Azure App ID in the WatchDogService -PackageRoot/Config/ServiceManifest.xml
In the service that you need monitored, in the Run Async command, add the following lines (Example is in the Test Stateless Service provided in the link below)
Code:
protected override async Task RunAsync(CancellationToken cancellationToken)
{
// Register the health check and metrics with the watchdog.
bool healthRegistered = await this.RegisterHealthCheckAsync(cancellationToken);
bool metricsRegistered = await this.RegisterMetricsAsync(cancellationToken);
while (true)
{
// Report some fake metrics to Service Fabric.
this.ReportFakeMetrics(cancellationToken);
await Task.Delay(TimeSpan.FromSeconds(30), cancellationToken);
// Check that registration was successful. Could also query the watchdog for additional safety.
if (false == healthRegistered)
{
healthRegistered = await this.RegisterHealthCheckAsync(cancellationToken);
}
if (false == metricsRegistered)
{
metricsRegistered = await this.RegisterMetricsAsync(cancellationToken);
}
}
}
Copy the RegisterHealthCheckAsync, RegisterMetricsAsync and ReportFakeMetrics methods, as is, into your service.cs file.
That should be it! It uses Azure Storage optionally. I did not have to implement that to get the watchdog up and running.
Here is the link : https://github.com/Azure-Samples/service-fabric-watchdog-service
For sending application/service telemetry to Application Insights, I strongly recommend you have a look at App Insights Service Fabric. It works great for:
Sending error and exception info
Populating the application map with all your services and their dependencies (including database)
Reporting on app performance metrics, as well as,
Tracing service call dependencies end-to-end,
Integrating with native as well as non-native SF applications
One thing however that the above won't solve is providing overall cluster health information - e.g. when/how often nodes go up/down, how much CPU/Memory and disk IO is consumed on individual nodes.
When running in Azure, the above should be fairly simple and I recommend you start here.
Doing this on-premise is not quite as simple. For this you could try MS EventFlow, or some of the other solutions already mentioned above.
Personally, I ended up creating a simple/custom windows service that use standard App Insights nuget packages to report the following info:
Cluster and Node ETW events from the Service Fabric Operational ETW channel
Performance counters (configurable via app insights config file)
Using 5.1.163 version of service fabric run time.
Created a service fabric application with one stateless web api (i.e. using owin communication listener).
Modified the generated code so that listening endpoint to contain partition id/instance id/new_guid (just as is the case for stateful services). This should allow me to create another app instance so that I can have multi-tenancy at application level.
By default, Local.xml file is set to 1 instance for this service.
Deployed it to local machine by F5. Verified that it is deployed to only one instance.
Verified that service is working fine.
Navigated to local service fabric explorer and clicked on the Cluster/Application/AppType node. Clicked on 'Create app instance'.
It successfully created 2nd app instance.
However in this new instance, the service is deployed to all 5 nodes.
I was expecting it deploy the service instance only one node. Is this a bug? But only in this version of service fabric?
When you deploy a Service Fabric application using Visual Studio (or from PowerShell) you use the Deploy-FabricApplication.ps1 that is generated for your application and found in /scripts under your SF project. This script does two things (mainly):
Create/update the application type
Create a new/upgrade existing instance of the application type
The second part there is similar to what you do in the SF Explorer, except this one also considers the publisher profile file you supply. The PS-script actually reads your publisher profile xml files and extracts any parameters in there to a hashset (a dictionary) and passes that as an argument in step 2.
You can create an instance of an SF application type using the PS cmdlets (alternatively you can use FabricClient). The following command does this: New-ServiceFabricApplication. Here you have the chance to supply your own application parameters, including instance count for services in your new application instance (if you have a dynamic parameter for that in your application manifest).
So, when you use the SF explorer to create a new application instance you cannot control how that instance is created, it is always using the default parameter values as specified directly in ApplicationManifest.xml, not values you have specified in your publisher profiles (local1, local5, cloud, etc.).
To controll the creation, run New-ServiceFabricApplication with yor parameters as a hashset.
I tried removing application from service fabric using service fabric explorer.
I deleted my application using Delete Application action.
Then When I tried Unprovision application type I got error saying,
Error: Application Type of version 1.0.0 could not be unprovisioned as
it still contains active applications.
I could see that even after deleting the application , the actor service inside the application is still active in some of the nodes. Am attaching a screenshot of my service fabric explorer.
Any help regarding completely removing the applications?
This can happen if services in your application don't play nice by not shutting down when requested by the platform, for example by ignoring the cancellation token in RunAsync.
Here's a quick PowerShell script that will go through an application and force remove all replicas of all stateful services:
Connect-ServiceFabricCluster -ConnectionEndpoint localhost:19000
$nodes = Get-ServiceFabricNode
foreach($node in $nodes)
{
$replicas = Get-ServiceFabricDeployedReplica -NodeName $node.NodeName -ApplicationName "fabric:/MyApp"
foreach ($replica in $replicas)
{
Remove-ServiceFabricReplica -ForceRemove -NodeName $node.NodeName -PartitionId $replica.Partitionid -ReplicaOrInstanceId $replica.ReplicaOrInstanceId
}
}
Go to http://localhost:19080/-->Expand Cluster-->Applications-->Service
Click on the 3 dots beside service and select option "Delete service"
Similarly delete the application and then unprovision the type