I have a bunch of backend service in Azure Service Fabric, I want to log any uncaught exceptions to App Insights, along with all my other logs. Is there any way in an Azure Service Farbic app to catch all uncaught exceptions and log them before re-throwing them?
You're using .net so you have access to the standard AppDomain way of handling all uncaught exceptions. Use this event.
Add the following lines into your Program.cs with logging code in there
AppDomain.CurrentDomain.UnhandledException += (sender,e)
=> {
//log exception
};
For sending application/service telemetry to Application Insights, I strongly recommend you have a look at App Insights Service Fabric. It works great for:
Sending error and exception info
Populating the application map with all your services and their dependencies (including database)
Reporting on app performance metrics, as well as,
Tracing service call dependencies end-to-end,
Integrating with native as well as non-native SF applications
If you're also interesting in monitoring the overall health of your cluster (e.g. CPU/Memory and when nodes go up/down), have a look at EventFlow or this github project
Related
I have created a sample play framework api which has one endpoint.
http://play-demo-broker.cfapps.io/say?number=20
Which just return me number that have passed.
I am able successfully deploy the service. Next want this service to Act like service broker
For same want to register this as by using below command
cf create-service-broker play-demo-broker admin admin http://play-demo-broker.cfapps.io --space-scoped
This command it giving me below error -
The service broker rejected the request. Status Code: 404 Not Found
Not sure what is causing this issue as there not much information available for Play Framework Service broker Setup.
The play framework is implemented above the akka packages. Akka rejects paths that are not implemented.
If I an not mistaken, cf create-service-broker command access the / endpoint. If you implemented only say?number=20 endpoint, then be default all other paths, such as the empty path, are rejected by Akka.
In order to open that endpoint you need to add it into the routes.
For example you can add:
GET / controllers.ControllerName.GetEmptyPath
And implement the GetEmptyPath method in ControllerName
We have a setup of WSO2 API Management in a distributed pattern (pattern-3) in Kubernetes. We are using a PostgreSQL DB which is running outside the Kubernetes cluster for all the databases.
I have published some APIs in the publisher and am able to invoke them from the store.
I had to make a change in api-manager.xml for the API Publisher and API Store configmap files and recreated the pod. When the pods were available, I observed that the APIs that I had published and working earlier are not visible anymore.
I tried to add the same APIs again and it is complaining that the APIs by that name already exists.
Following is the log from the plubisher pod:
[2019-05-16 08:19:38,266] ERROR - APIProviderHostObject Error occurred while adding the document. PizzaShack API Documentation already exists for API PizzaShackAPI-1.0.0
[2019-05-16 08:19:38,273] ERROR - docs:jag org.wso2.carbon.apimgt.api.APIManagementException: Error occurred while adding the document. PizzaShack API Documentation already exists for API PizzaShackAPI-1.0.0
While creating the API again on the Publisher, following error is displayed: "Duplicate API Name"
It clearly seems to be some synchronization issue. How can this issue be fixed?
I had shared the instance of Carbon DB across the components. This was causingthe issue. Using separate instance for each component in disbuted mode solved it
I am new to service Fabric and trying to integrate my windows service application into service fabric. For logging information, we are planning to use Application Insights. But the events are not logged if i send it through my SF application. At the same time, through a normal console/windows application, I can able to log the message to applicationinsights and can be viewed from there.
Then I tried to create a VM in azure environment, and create SF application there and send the log information to AI and its worked successfully. I copied the same codebase into my local machine and run it, its not working. I am not sure whether its related to any firewall or proxy settings. Can anyone help on this?
I have used the nuget package to install Microsoft.ApplicationInsights dll in my machine. The version that I used is 2.2.0. And I am using .Net framework 4.6.1
You could look at EventFlow to help you capture Service Fabric ETW Events from your SF services and send them to Application Insights.
It's easy enough to setup, simply add Microsoft.Diagnostics.EventFlow.ServiceFabric NuGet to your Service Fabric service project and then setup a pipline
public static void Main(string[] args)
{
try
{
using (var diagnosticsPipeline = ServiceFabricDiagnosticPipelineFactory.CreatePipeline("MyApplication-MyService-DiagnosticsPipeline"))
{
ServiceRuntime.RegisterServiceAsync("MyServiceType", ctx => new MyService(ctx)).Wait();
ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(MyService).Name);
Thread.Sleep(Timeout.Infinite);
}
}
catch (Exception e)
{
ServiceEventSource.Current.ServiceHostInitializationFailed(e.ToString());
throw;
}
}
In your eventflow.config you can then setup Application Insights as an output:
{
"inputs": [
{
"type": "EventSource",
"sources": [
{ "providerName": "Your-Service-EventSource" }
]
},
],
"filters": [
{
"type": "drop",
"include": "Level == Verbose"
}
],
"outputs": [
// Please update the instrumentationKey.
{
"type": "ApplicationInsights",
"instrumentationKey": "00000000-0000-0000-0000-000000000000"
}
],
"schemaVersion": "2016-08-11",
"extensions": []
}
An alternative to the EventFlow approach suggested by yoape would be Azure Diagnostics (WAD).
Setup WAD in SF VMSS
When you're running an Azure Service Fabric cluster, it's a good idea
to collect the logs from all the nodes in a central location. Having
the logs in a central location helps you analyze and troubleshoot
issues in your cluster, or issues in the applications and services
running in that cluster. One way to upload and collect logs is to use
the Windows Azure Diagnostics (WAD) extension, which uploads logs to
Azure Storage, and also has the option to send logs to Azure
Application Insights or Event Hubs. You can also use an external
process to read the events from storage and place them in an analysis
platform product, such as OMS Log Analytics or another log-parsing
solution.
Setup AI upload in WAD
Cloud services, Virtual Machines, Virtual Machine Scale Sets and
Service Fabric all use the Azure Diagnostics extension to collect
data. Azure diagnostics sends data to Azure Storage tables. However,
you can also pipe all or a subset of the data to other locations using
Azure Diagnostics extension 1.5 or later. This article describes how
to send data from the Azure Diagnostics extension to Application
Insights.
The nice thing about it is that it's completely managed by Azure, and you don't need to change anything in your project.
You can adapt the watchdog Service from Microsoft Samples. The watchdog Service is a generic standalone Service Fabric Stateful Service that will log data into Application Insights.
Add the watchdog app and watchdog service into your solution.
Add your Azure App ID in the WatchDogService -PackageRoot/Config/ServiceManifest.xml
In the service that you need monitored, in the Run Async command, add the following lines (Example is in the Test Stateless Service provided in the link below)
Code:
protected override async Task RunAsync(CancellationToken cancellationToken)
{
// Register the health check and metrics with the watchdog.
bool healthRegistered = await this.RegisterHealthCheckAsync(cancellationToken);
bool metricsRegistered = await this.RegisterMetricsAsync(cancellationToken);
while (true)
{
// Report some fake metrics to Service Fabric.
this.ReportFakeMetrics(cancellationToken);
await Task.Delay(TimeSpan.FromSeconds(30), cancellationToken);
// Check that registration was successful. Could also query the watchdog for additional safety.
if (false == healthRegistered)
{
healthRegistered = await this.RegisterHealthCheckAsync(cancellationToken);
}
if (false == metricsRegistered)
{
metricsRegistered = await this.RegisterMetricsAsync(cancellationToken);
}
}
}
Copy the RegisterHealthCheckAsync, RegisterMetricsAsync and ReportFakeMetrics methods, as is, into your service.cs file.
That should be it! It uses Azure Storage optionally. I did not have to implement that to get the watchdog up and running.
Here is the link : https://github.com/Azure-Samples/service-fabric-watchdog-service
For sending application/service telemetry to Application Insights, I strongly recommend you have a look at App Insights Service Fabric. It works great for:
Sending error and exception info
Populating the application map with all your services and their dependencies (including database)
Reporting on app performance metrics, as well as,
Tracing service call dependencies end-to-end,
Integrating with native as well as non-native SF applications
One thing however that the above won't solve is providing overall cluster health information - e.g. when/how often nodes go up/down, how much CPU/Memory and disk IO is consumed on individual nodes.
When running in Azure, the above should be fairly simple and I recommend you start here.
Doing this on-premise is not quite as simple. For this you could try MS EventFlow, or some of the other solutions already mentioned above.
Personally, I ended up creating a simple/custom windows service that use standard App Insights nuget packages to report the following info:
Cluster and Node ETW events from the Service Fabric Operational ETW channel
Performance counters (configurable via app insights config file)
hi I am trying several ways to configure app url, but no luck, i am trying 1.started all services like app management service,managed meta data services,Microsoft SharePoint Foundation Subscription Settings Service Application,2.and giving full controls permissions
Check that those service application are running by browsing them like this :
http://[HOST]:[PORTNUMBER]/[PoolAppId]/SubscriptionSettings.svc for Subscription Settings Service Application
http://[HOST]:[PORTNUMBER]/[PoolAppId]/AppMng.svc for App Management Service Application
because sometimes the Central Administration tells you that the services are running, but you find error logs in the event viewer saying something like the message below, to inform that your server needs more ressources to activate all services :
Error source : System.ServiceModel 4.0.0.0
WebHost failed to process a request ....
Exception : System.ServiceModel.ServiceActivationException: the service '/c720557bd2fa437c94391bd06bdb4d44/XXXXXX.svc' can't be activated due to an exception in the compilation phase.
If your server hasn't enough RAM memory, you can edit the web.config file of the desired service by modifying the balise and including this one :
<serviceHostingEnvironment minFreeMemoryPercentageToActivateService="1"/>
hope this will help you !
When I try to unbind a specific SendGrid service from my Java Liberty application running on Bluemix, I get the following error message:
Service broker error: {"description"=>"Received exception with class 'javax.ws.rs.client.ResponseProcessingException' and message 'Problem with reading the data, class com.appdirect.backend.integration.custom.sendgrid.output.SendGridResult, ContentType: application/xml;charset=utf-8.' while calling integration endpoint 'null'."}
I get the same message using the Bluemix gui and cf cli.
Go here and open a support ticket. This isn't something you can address on your end. Thanks!
It looks like an error with the broker and SendGrid. Can you try deleting your app as well then try to delete the service?
I can recreate this problem; you're not doing anything wrong. It's an operational problem with the Sendgrid service broker that they need to fix.
Yes there are some issues deleting specific service instances like that of mysql community service instance. Raise a support ticket at https://support.ng.bluemix.net/gethelp/ , the support team would help with deleting of the service instances from your org/space. Ensure you provide the details of your service, space and org while opening the service ticket.