I have a stateless ASP.NET Core (RC1) service running in my Azure Service Fabric cluster. It has the following manifest:
<ServiceManifest Name="MyServicePkg" Version="1.0.2" ...>
<ServiceTypes>
<StatelessServiceType ServiceTypeName="MyServiceType" />
</ServiceTypes>
...
</ServiceManifest>
My cluster is configured with placement properties. I have 5 servers with "nodeType=Backend" and 3 servers with "nodeType=Frontend".
I would like to upgrade my Service and specify that it may only be placed on "Backend" nodes. This is my updated manifest:
<ServiceManifest Name="MyServicePkg" Version="1.0.3" ...>
<ServiceTypes>
<StatelessServiceType ServiceTypeName="MyServiceType">
<PlacementConstraints>(nodeType==Backend)</PlacementConstraints>
</StatelessServiceType>
</ServiceTypes>
...
</ServiceManifest>
However, if I now execute the upgrade, I get the following error:
Start-ServiceFabricApplicationUpgrade : Default service descriptions
must not be modified as part of upgrade. Modified default service:
fabric:/MyApp/MyService
Why isn't it possible to change the constraints with an upgrade?
Would I have to delete and re-create the service? This would seem extremely problematic to me because it would result in downtime and data loss for stateful services.
So the issue here is actually with the DefaultService part of the ApplicationManifest. When services are created as part of the DefaultService, there are things you can't change about it afterwards. You might be able to change it through the ServiceFabric explorer, but I'm not sure.
One recommendation would be to keep the DefaultServices empty in the ApplicationManifest, and instead create your services manually. With manual I mean either through powershell, code or the ServiceFabric Explorer.
That gives you more flexibility about changing parts of the service afterwards. When it's done that way, you I know you have the possibility to change things like placement constraints after the service is running.
To create Services with PowerShell you can use the New-ServiceFabricService command.
To create it from code, you can use FabricClient to do it. A sample of that can be found here: Azure Service Fabric Multi-Tenancy
There's actually a fairly easy way to do this without having to write a bunch of code to manually define the application on the fabric cluster.
While you can declare the placement constraints in the service manifest, you can also declare them in the application manifest. Anything declared in the application manifest will override what's in the service manifest. And with the setting in the application manifest, you can then use parameters to alter the values based on the parameter file you want to a specific deployment.
I've just written up a blog post that discusses this approach in greater detail. I hope you find it useful. :)
Related
I'm creating a Deployment Group in CodeDeploy with a CloudFormation template.
The Deployment Group is successfully created and the application is deployed perfectly fine.
The CF resource that I defined (Type: AWS::CodeDeploy::DeploymentGroup) has the "Deployment" property set. The thing is that I would like to configure automatic rollbacks for this deployment, but as per CF documentation for "AutoRollbackConfiguration" property: "Information about the automatic rollback configuration that is associated with the deployment group. If you specify this property, don't specify the Deployment property."
So my understanding is that if I specify "Deployment", I cannot set "AutoRollbackConfiguration"... Then how are you supposed to configure any rollback for the deployment? I don't see any other resource property that relates to rollbacks.
Should I create a second DeploymentGroup resource and bind it to the same instances that the original Deployment Group has? I'm not sure this is possible or makes sense but I ran out of options.
Thanks,
Nicolas
First i like to describe why you cannot specify both, deployment and rollback configuration:
Whenever you specify a deployment directly for the group, you already state which revision you like to deploy. This conflicts with the idea of CloudFormation of having resources managed by it without having a drift in the actual configuration of those resources.
I would recommend the following:
Use CloudFormation to deploy the 'underlying' infrastructure (the deployment group, application, roles, instances, etc.)
Create a CodePipline within this infrastructure template, which then includes a CodeDeploy deployment action (https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference-CodeDeploy.html, https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codepipeline-pipeline-stages-actions-actiontypeid.html)
The pipeline can triggered whenever you have a new version inside you revision location
This approach clearly separates the underlying stuff, which is not changing dynamically and the actual application deployment, done using a proper pipeline.
Additionally in this way you can specify how you like to deploy (green/blue, canary) and how/when rollbacks should be handled. The status of your deployment also to be seen inside CodePipeline.
I didn't mention it but what you are suggesting about CodePipeline is exactly what I did.
In fact, I have one CloudFormation template that creates all the infrastructure and includes the DeploymentGroup. With this, the application is deployed for the first time to my EC2 instances.
Then I have another CF template for CI/CD purposes with a CodeDeploy stage/action that references the previous DeploymentGroup. Whenever I push some code to my repository, the Pipeline is triggered, code is built and new version successfully deployed to the instances.
However, I don't see how/where in any of the CF templates to handle/configure the rollback for the DeploymentGroup as you were saying. I think I get the idea of your explanation about the conflict CF might have in case of having a drift, but my impression is that in case of errors during the CF stack creation, CF rollback should just remove the DeploymentGroup you're trying to create. In other words, for me there's no CodeDeploy deployment rollback involved in that scenario, just removing the resource (DeploymentGroup) CF was trying to create.
One thing that really impresses me is that you can enable/disable automatic rollbacks for the DeploymentGroup through the AWS Console. Just edit and go to Advanced Configuration for the DeploymentGroup and you have a checkbox. I tried it and triggered the Pipeline again and worked perfectly. I made a faulty change to make the deployment fail in purpose, and then CodeDeploy automatically reverted back to the previous version of my application... completely expected behavior. Doesn't make much sense that this simple boolean/flag option is not available through CF.
Hope this makes sense and helps clarifying my current situation. Any extra help would be highly appreciated.
Thanks again
Working with a stateful service in ServiceFabric version 5.1.163.9590, I am attempting to deploy a demo application with three WebApi services that manage their own state.
Two of the three services start and create their partitions without errors, but the last spews events a series of warnings and errors, the error detail has this intriguing message:
Microsoft.ServiceFabric.Replicator.LoggingReplicator : GetCopyState The parameter copyContext is null. This might be caused by deployment bug that 'hasPersistedState' attribute is false.
I can't locate any external references to this error message.
Is there a way to correct this from the application and service deployment side, or from the cluster management side?
The error indicates you have a stateful service with persisted state, but didn't tell Service Fabric about that when you deployed the service.
There's a flag that needs to be set to indicate to Service Fabric that a stateful service has persisted state (as opposed to state that is "volatile," meaning in-memory only).
In your ServiceManifest.xml, make sure you have this flag set on the service type:
<ServiceTypes>
<StatefulServiceType ServiceTypeName="MyServiceType" HasPersistedState="true" />
</ServiceTypes>
Then if you're deploying through PowerShell, make sure you set this flag when you create an instance of the service:
PS > New-ServiceFabricService -Stateful -HasPersistedState -ServiceTypeName "MyServiceType" ...
trying to setup that specific services get deploy to specific node types I am getting this error using Visual Studio publish dialog (that breaks calling new-servicefabricapplication PS command)
I am using the service manifest to define the placementConstraints like this:
<StatelessServiceType ServiceTypeName="VisualObjects2.WebServiceType" >
<PlacementConstraints>(nodeType==node2)</PlacementConstraints>
</StatelessServiceType>
How can i define this placement constraints on the nodes?
In the Azure portal, go to your SF Cluster, select node types and for each one you can add a key-value list of placement constraints. There I put the key-value: nodetype = node2. After this, the deployment was done only in the nodes with this attribute
I'm trying to upgrade a Service Fabric application with a mix of stateful and stateless actors. I did some refactoring and so removed some actors I didn't need any more. Now, when I try to upgrade the application, I get the following error:
Services must be explicitly deleted before removing their Service Types.
After thinking about it a little bit, I think I understand the trouble that could come from removed services and upgrades, but then what's the correct way to do this?
You need to remove the service instances before you can upgrade to a version that doesn't contain the removed service package. Either:
In SF Explorer, navigate to the service and click Actions > Delete Service
In PowerShell:
Connect-ServiceFabricCluster
Remove-ServiceFabricService -ServiceName fabric:/MyApp/MyService
DO BE CAREFUL - If you're deleting a stateful service you'll lose all its data. Always be sure to have a periodic backup of production data.
Is it possible to configure startup order when starting up the services.
A Service1 has to be running before Service2 can be started.
Clarification:
I'm didn't mean micro services when I mentioned Service, I meant stateless services like REST API (Service1) and WebSocket (Service2).
So when then solution is deployed the WebSocket service (Service2) must be up and running before the REST API (Service1)?
Of course you can, because you control when services are created. It's not immediately obvious if you've only ever deployed applications through Visual Studio, because Visual Studio sets you up with Default Services. This is what you see in ApplicationManifest.xml when you create an application through Visual Studio:
<DefaultServices>
<Service Name="Stateless1">
<StatelessService ServiceTypeName="Stateless1Type" InstanceCount="[Stateless1_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
<Service Name="Stateful1">
<StatefulService ServiceTypeName="Stateful1Type" TargetReplicaSetSize="[Stateful1_TargetReplicaSetSize]" MinReplicaSetSize="[Stateful1_MinReplicaSetSize]">
<UniformInt64Partition PartitionCount="[Stateful1_PartitionCount]" LowKey="-9223372036854775808" HighKey="9223372036854775807" />
</StatefulService>
</Service>
</DefaultServices>
This is a nice convenience when you know you always want certain services created a certain way each time you create an application instance. You can define them declaratively here and Service Fabric will create them whenever you create an instance of the application.
But it has some drawbacks. Most notably, in your case, is that you have no control over the order in which the services are created.
It also hides some of the concepts around application and service types and application and service instances, which again can be convenient until you want to do something more advanced, like in your case.
When you "deploy" an application, there are actually several steps:
Create the application package
Copy the package up to the cluster
Register the application type and version
Create an instance of the registered application type and version
Create instances of each registered service type in that application
With Default Services, you skip step 5 because Service Fabric does it for you. Without Default Services though, you get to create your service instances yourself, so you can determine what order to do it in. You can do other things like check if a service is ready before creating the next one. All of these actions are available in Service Fabric's C# SDK and PowerShell cmdlets. Here's a quick PowerShell example:
Copy-ServiceFabricApplicationPackage -ApplicationPackagePath C:\temp\MyApp -ImageStoreConnectionString fabric:ImageStore -ApplicationPackagePathInImageStore MyApp
Register-ServiceFabricApplicationType MyApp
New-ServiceFabricApplication -ApplicationName fabric:/MyAppInstance -ApplicationTypeName MyApp -ApplicationTypeVersion 1.0
New-ServiceFabricService -ApplicationName fabric:/MyAppInstance -InstanceCount 1 -PartitionSchemeSingleton -ServiceName fabric:/MyAppInstance/MyStatelessService -ServiceTypeName MyStatelessService -Stateless
New-ServiceFabricService -ApplicationName fabric:/MyAppInstance -MinReplicaSetSize 2 -PartitionSchemeSingleton -ServiceName fabric:/MyAppInstance/MyStatefulService -ServiceTypeName MyStatefulServiceType -Stateful -TargetReplicaSetSize 3
Of course, this just applies to creating the service instances. When it comes time to upgrading your services, the "upgrade unit" is actually the application, so you can't pick the order in which services within an application get upgraded, at least not in one single upgrade. You can, however, choose which services get upgraded during an application upgrade, so if you have the same ordering dependency, you can accomplish that by doing two separate application upgrades.
So you get some level of control. But, it's really best if your services are resilient to missing dependent services, because there will likely be times when a service is unavailable for one reason or another.
Edit:
I showed you a lot of PowerShell but I should mention the C# APIs are also just as powerful. You have the full set of management tools at your disposal in C#. For example, you can have a service that creates and manages other services. In your case, if Service A depends on Service B, then you can have Service A create an instance of Service B before Service A itself does any work, and throughout Service A's life it can keep an eye on Service B. Here's an example of a service that creates other applications and services: https://github.com/Azure-Samples/service-fabric-dotnet-management-party-cluster/blob/master/src/PartyCluster.ApplicationDeployService/FabricClientApplicationOperator.cs
In service fabric world it's called Service Affinity (https://azure.microsoft.com/en-us/documentation/articles/service-fabric-cluster-resource-manager-advanced-placement-rules-affinity/)
Anyway, try to avoid these situations in microservices world.
Good luck.