I've got a single node service fabric instance hosted in Azure, just for testing purposes. When I try and upgrade the service fabric version to 7.0 from 6.5 I get the message:
SingleNodeClusterUpdateNotAllowed
Is there anything I can do to allow this?
The short answer is no.
The reason for this is that in order to upgrade service fabric has to takes down a node, updates and restarts it. This is repeated for all nodes until the update is complete. In a single node cluster this would mean taking the cluster offline completely. This is not allowed by the service fabric rules (at the very least one node must be available).
A single node 'cluster' therefore cannot update the platform or applications running on it.
The only way you can update a single node cluster is to delete and reinstall it. The same goes for applications (delete the application type before deploying an updated version). Depending on where you have the software deployed (development box, a server, azure) I would recommend scripting as much as possible. This will allow you to easily delete and redeploy. I am using a combination of an Azure template (arm), DevOps pipeline and script to initialise and load some default data into the application.
Related
We have a 5-node Azure Service Fabric Cluster as our main Production microservices hub. Up until now, for testing purposes, we've just been pushing out separate versions of our applications (the production application with ".Test" appended to the name) to that production SFC.
We're looking for a better approach, namely a separate test Service Fabric Cluster. But the issue comes down to costs. The smallest SFC you can create in Azure is 3 nodes. Further, you can't shutdown a SFC when it's not being used, which we would also need to do to save on costs.
So now I'm looking at just spinning up a plain Windows VM in Azure and installing the local Service Fabric Cluster app (which allows just one-node setup). Is it possible to do this and be able to communicate with the cluster from outside the VM?
What you are trying to accomplish is setup a standalone cluster. The steps to do it is documented in this docs.
Yes, you can access the cluster from outside the VM, In simple terms enable access to the network and open the firewall ports.
Technically both deployments(Guide and DevCluster) are very similar, the main difference is that you have better control on the templates following the standalone guide, using the development setup you don't have much options and all the process is automated.
PS: I would highly recommend you have a UAT\Staging cluster with the
exact same specs as the production version, the approach you used
could be a good idea for staging environment. Having different
environments increase the risk of issues, mainly related to
configuration and concurrency.
Requirement - New Relic monitoring for an application running in pods as part of a kubernetes cluster.
I have installed Kube-state-metrics on my cluster and able to see kubernetes dashboard using newrelic insights.
Also, need to configure the Application monitoring for the same. Following https://blog.newrelic.com/2017/11/27/monitoring-application-performance-in-kubernetes/ for the same.
Have some questions for the same -
Can this be achieved using kube-state-metrics ?
Do I need to have separate yaml file for each pod containing license key?
Do I need to make changes in my application also or adding the information in spec will work?
Do I need to install Java agent in every pod? If yes, will it eat resources?
Somehow, Installation of application monitoring is becoming complex. Please explain the exact requirement of installation
You didn't mention your stack, you should follow instructions on their site for your language. Typically you just pull in their agent library and configure credentials to get started. You should not have a reason to tell your pods apart, so the agent credentials should be the same for all pods
Installing agents at infrastructure will let you have infrastructure data. So you'll get alerts if you're running out of memory/space/cpu and such. Infrastructure agent cannot possibly know about application data. If you want application performance data (apm) you need to install the agent at the application level too and you'll get data such as http request rates, error rates and response times if it's a webserver. You can also annotate current transaction with data which is all application specific. They have a bunch of client agents, see if there's one for your stack. For example all you need for a nodejs service is require('newrelic') at the top of your app and configuration
I've an on-premise, secure, development cluster that I wish to upgrade. The current version is 5.7.198.9494. I've followed the steps listed here.
At the time of writing, the latest version of SF is 6.2.283.9494. However, running Get-ServiceFabricRuntimeUpgradeVersion -BaseVersion 5.7.198.9494 shows that I first must update to 6.0.232.9494, before upgrade to 6.2.283.9494.
I run the following in Powershell, and the upgrade does start:
Copy-ServiceFabricClusterPackage -Code -CodePackagePath .\MicrosoftAzureServiceFabric.6.0.232.9494.cab -ImageStoreConnectionString "fabric:ImageStore"
Register-ServiceFabricClusterPackage -Code -CodePackagePath MicrosoftAzureServiceFabric.6.0.232.9494.cab
Start-ServiceFabricClusterUpgrade -Code -CodePackageVersion 6.0.232.9494 -Monitored -FailureAction Rollback
However, after a few minutes the following happens:
Powershell IDE crashes
The Service Fabric Cluster becomes unreachable
Service Fabric Local Cluster Manager disappears from the task bar
Event Viewer will log the events, see below.
Quite some time later, the vm will reboot. Service Fabric Local Cluster Manager will only give options to Setup or Restart the local cluster.
Event viewer has logs in the under Applications and Services Logs -> Microsoft-Service Fabric -> Operational. Most are information about opening, closing, and aborting one of the upgrade domains. There are some warnings about a vm failing to open an upgrade domain stating error: Lease Failed.
This behavior happens consistently, and I've not yet been able to update the cluster. My guess is that we are not able to upgrade a development cluster, but I've not found an article that states that.
Am I doing something incorrectly here, or is it impossible to upgrade a development cluster?
I will assume you have a development cluster with a single node or multiple nodes in a single VM.
As described in the first section of the documentation from the same link your provided:
service-fabric-cluster-upgrade-windows-server
You can upgrade your cluster to the new version only if you're using a
production-style node configuration, where each Service Fabric node is
allocated on a separate physical or virtual machine. If you have a
development cluster, where more than one Service Fabric node is on a
single physical or virtual machine, you must re-create the cluster
with the new version.
I am searching for chef features that, does below jobs for deployment.
Configuration:
1) Configuration of deployment node machines in specific environment
2) Configure no of service instances to be alive in environment at all time
Deployment:
Now, Just doing above configuration. When I trigger deployment of N services.
It will randomly pick up nodes from deployment environments and will start total N services.
Multiple Services:
If I have 2 nodes and I want to bring up 4 services, it should bring up 2 services on each node.
Service Failure recovery:
If any machine goes down or any service goes down in any node.
It will bring up a new service in any of environment node.
I answered this over on stackexchange and then saw this posted here so answering it here too.
You should take a look at BOSH. Its the tool that is used by CloudFoundry, its services and a distro of Kubernetes called Kubo for installation, management and "Day 2" operations.
It's basically a declarative, cloud-agnostic orchestration tool that features rolling updates, canary deployments, scaling, monitoring and self healing. It can monitor processes on VMs (i.e. services) as well as the VM themselves and will make sure that the deployment is running as you specified it to in the deployment manifest.
In order to do all of this (especially the monitoring and self-healing bits) it has a client-server architecture which is deployed with a cut-down version of BOSH itself called bosh bootloader or bbl for short. You use this to deploy the BOSH director. You talk to the bosh director by installing the bosh-cli (brew install bosh-cli on a mac).
For you deployment you would first need to create what's called a BOSH release and this can, admittedly, be a little daunting if you are not familiar with BOSH but as CF, its services and Kubo are all open source there are tons of references out there. There are also lots of pre-backed releases and stemcells (OSes).
After creating your release you upload it to your bosh director and bosh deploy. To upgrade it you upload the next version of your release and bosh deploy. To patch a security vulnerability you upload the latest blessed stemcell from bosh.io and bosh deploy. I am sure you get the picture.
If you want to go the next level then there is a good getting started guide here.
Chef does not do multi-node orchestration.
I have a bit of trouble finding differences about Azure and on-premise Service Fabric versions. I did read somewhere that on-premise version does not support auto-scaling, but this is easy to understand.
However, does on-premise version offer any type of operational capabilities such as resource managers, visual management of cluster, etc.?
The core Service Fabric platform is simply a runtime that gets installed on a set of virtual or physical machines. Once you tell those machines how to find each other, they form a cluster and provide a set of management capabilities that includes the Service Fabric Explorer UI, a REST API, and a TCP endpoint for PowerShell. All of that is common whether you're running on Azure, on-premises, or in another public cloud.
What's different in those environments is everything that lives outside of the machines that form the cluster. That includes:
Autoscaling
While Service Fabric can easily handle new machines being added and removed from the cluster, it has no knowledge of how that process actually works, so some external agent needs to handle it. In Azure, that's a virtual machine scale set.
Failure domain/Upgrade domain management
Good management of failure and upgrade domains is critical to ensuring availability and data reliability in Service Fabric. In Azure, clusters are automatically spread across FDs/UDs and maintenance is coordinated to avoid impact to your clusters. In other environments, this is your responsibility.
Cluster setup and management
In Azure, a Service Fabric cluster is a 1st class resource that can be created and managed through the Azure Resource Manager and the Azure portal. Outside of Azure, you must do that management using the cluster configuration JSON template.
Incidentally, just so there's no confusion since there are overloaded terms... you can't currently use the Azure Resource Manager (ARM) with Service Fabric outside of the Azure environment. However, Service Fabric's cluster resource manager is part of the core runtime and is available everywhere.
Diagnostics pipeline
By default, Service Fabric logging (on Windows) is done via ETW. However, without any component to pick up those events from the individual machines in the cluster and ship them somewhere for easy aggregation and inspection, the logs aren't very useful. In Azure, that process is handled by the Windows Azure Diagnostics (WAD) agent, whereas in other environments you are responsible for setting up that pipeline.
You don't get to use the resource manager on premises. You can access the Service Fabric Explorer at port 19080.
https://azure.microsoft.com/en-us/documentation/articles/service-fabric-deploy-anywhere/
https://azure.microsoft.com/en-us/documentation/articles/service-fabric-visualizing-your-cluster/
Powershell management & deployment will also work.