I have 3-node on-prem cluster. Now i want to collect and analyze reverse proxy logs (and other system service fabric logs). I google and found this article and it says
Refer to Collect reverse proxy events to enable collecting events from
these channels in local and Azure Service Fabric clusters.
But that link describes how to enable, configure and collect reverse proxy logs for clusters in Azure. And I don't understand how to do it on-prem.
Please, help!
Service Fabric Events are just ETW events, you have the option to use the Builtin mechanism to collect and forward these events to a Monitoring Application like Windows Azure Diagnostics, or you can build your own.
If you decide to follow the approach in the documents, it will work on azure or On premises, the only caveat is that On-Premises it will send the logs to Azure, but it will work the same way.
On Premises, another way is build you own collector using EventFlow, you can configure EventFlow to collect the ReverseProxy ETW events and then forward these to ELK or any other monitoring platform.
Related
I have some limitations with the rights required by Flink native deployment.
The prerequisites say
KubeConfig, which has access to list, create, delete pods and **services**, configurable
Specifically, my issue is I cannot have a service account with the rights to create/remove services. create/remove pods is not an issue. but services by policy only can be created within an internal tool.
could it be any workaround for this?
Flink creates two service in native Kubernetes integration.
Internal service, which is used for internal communication between JobManager and TaskManager. It is only created when the HA is not enabled. Since the HA service will be used for the leader retrieval when HA enabled.
Rest service, which is used for accessing the webUI or rest endpoint. If you have other ways to expose the rest endpoint, or you are using the application mode, then it is also optional. However, it is always be created currently. I think you need to change some codes to work around.
We are trying to figure out which monitoring options will be suitable for our environment.
We have two clusters in GCP and we installed Istio (with Helm) in both of them. We are also using Workload Identity and Stackdriver Monitoring.
Now, we would like to create dashboards (or charts) for kubernetes objects (such as, deployments, containers, cronjobs, services, etc.) and want to set alerts on them. So can anyone suggest free monitoring options to achieve these all? We don't want to go with any third party paid software.
Thank you in advance.
If you are using GCP GKE then default stack driver logging & monitoring is best option.
It's free if you are using GCP service and using stack driver monitoring you can monitoring and creat respective dashboards as per need.
For alerts, you can use the GCP Uptime check option available in monitoring itself which sends the email. For call alerts, you may have to use some custom or third applications.
You can read more at : https://cloud.google.com/monitoring/docs
Uptime checks : https://cloud.google.com/monitoring/uptime-checks
We have several services running in an ASF cluster. Is it possible to take pause/stop/restart a service in an ASF cluster across all nodes programmatically?
At the moment you cannot stop a 'service' in Service Fabric. You can only remove it. But you can start/stop, enable/disable nodes within the cluster.
There is a enhancement request in Azure feedback forums to have the service start/stop feature. Vote for it and wait for it to be implemented.
https://feedback.azure.com/forums/293901-service-fabric/suggestions/13714473-ability-to-stop-disable-services-without-removing
I found this from several months back on Application Insights and Service Fabric and I'm wondering if there is any new information.
I would really like to get CPU, Memory, Storage and other metrics out of service fabric and the reliable actors. Having it presented in a user friendly HUD like app insights provides would be awesome!
Thanks!
On the azure portal, you can now create a resource called 'Service Fabric Analytics' to create a nice dashboard for your cluster. Configure as cluster like described here. It's OMS based, not Appinsights though.
The Service Fabric Solution helps identify and troubleshoot issues
across your Service Fabric cluster, by providing visibility into how
your Service Fabric virtual machines are performing and how your
applications and micro-services are running. Available features
include: • Get insight into your Service Fabric framework • Get
insight into the performance of your Service Fabric applications and
micro-services • View events from your applications and micro-services
Data collected: Service Fabric Reliable Service Events, Service Fabric
Actor Events, Service Fabric Operational Events, Event tracing for
Windows events and Windows event logs. Requirements: This solution
will only work if you have set up Azure Diagnostics on your Service
Fabric VMs, and have configured OMS to collect data for your WAD
tables.
In service fabric with Eventflow https://github.com/Azure/diagnostics-eventflow you had a option to send the diagnostics data to multiple data stores like WAD Tables, OMS, elastic search and application insights.
Have a look at it. It is really straight forward and easily integrate with ETW events and will serve your purpose.
I have a bit of trouble finding differences about Azure and on-premise Service Fabric versions. I did read somewhere that on-premise version does not support auto-scaling, but this is easy to understand.
However, does on-premise version offer any type of operational capabilities such as resource managers, visual management of cluster, etc.?
The core Service Fabric platform is simply a runtime that gets installed on a set of virtual or physical machines. Once you tell those machines how to find each other, they form a cluster and provide a set of management capabilities that includes the Service Fabric Explorer UI, a REST API, and a TCP endpoint for PowerShell. All of that is common whether you're running on Azure, on-premises, or in another public cloud.
What's different in those environments is everything that lives outside of the machines that form the cluster. That includes:
Autoscaling
While Service Fabric can easily handle new machines being added and removed from the cluster, it has no knowledge of how that process actually works, so some external agent needs to handle it. In Azure, that's a virtual machine scale set.
Failure domain/Upgrade domain management
Good management of failure and upgrade domains is critical to ensuring availability and data reliability in Service Fabric. In Azure, clusters are automatically spread across FDs/UDs and maintenance is coordinated to avoid impact to your clusters. In other environments, this is your responsibility.
Cluster setup and management
In Azure, a Service Fabric cluster is a 1st class resource that can be created and managed through the Azure Resource Manager and the Azure portal. Outside of Azure, you must do that management using the cluster configuration JSON template.
Incidentally, just so there's no confusion since there are overloaded terms... you can't currently use the Azure Resource Manager (ARM) with Service Fabric outside of the Azure environment. However, Service Fabric's cluster resource manager is part of the core runtime and is available everywhere.
Diagnostics pipeline
By default, Service Fabric logging (on Windows) is done via ETW. However, without any component to pick up those events from the individual machines in the cluster and ship them somewhere for easy aggregation and inspection, the logs aren't very useful. In Azure, that process is handled by the Windows Azure Diagnostics (WAD) agent, whereas in other environments you are responsible for setting up that pipeline.
You don't get to use the resource manager on premises. You can access the Service Fabric Explorer at port 19080.
https://azure.microsoft.com/en-us/documentation/articles/service-fabric-deploy-anywhere/
https://azure.microsoft.com/en-us/documentation/articles/service-fabric-visualizing-your-cluster/
Powershell management & deployment will also work.