Lets say I'm deploying an Azure VM by scripting which requires multiple resources interdependent on one another.
Lets say a NIC fails,
Does the deployment script still go through to the deployment of the VM? So that I have a VM with no NIC
Does it fail rolling back the entire script?
No ARM templates are not executed within a transaction.
It's possible that you have deployed resources without deploying the whole ARM template. In your case it's not possible to have a VM without a NIC (But you test the deployment of the ARM template and make it work at the end.)
It does not roll back.
I think now they have added "--rollback-on-error" flag to rollback on last successful deployment if the deployment fails. Also you can specify the name of the deployment to which you need to roll back refer this: https://learn.microsoft.com/en-us/cli/azure/group/deployment?view=azure-cli-latest#commands.
Related
My kubernetes application is made of several flavors of nodes, a couple of “schedulers” which send tasks to quite a few more “worker” nodes. In order for this app to work correctly all the nodes must be of exactly the same code version.
The deployment is performed using a standard ReplicaSet and when my CICD kicks in it just does a simple rolling update. This causes a problem though since during the rolling update, nodes of different code versions co-exist for a few seconds, so a few tasks during this time get wrong results.
Ideally what I would want is that deploying a new version would create a completely new application that only communicates with itself and has time to warm its cache, then on a flick of a switch this new app would become active and start to get new client requests. The old app would remain active for a few more seconds and then shut down.
I’m using Istio sidecar for mesh communication.
Is there a standard way to do this? How is such a requirement usually handled?
I also had such a situation. Kubernetes alone cannot satisfy your requirement, I was also not able to find any tool that allows to coordinate multiple deployments together (although Flagger looks promising).
So the only way I found was by using CI/CD: Jenkins in my case. I don't have the code, but the idea is the following:
Deploy all application deployments using single Helm chart. Every Helm release name and corresponding Kubernetes labels must be based off of some sequential number, e.g. Jenkins $BUILD_NUMBER. Helm release can be named like example-app-${BUILD_NUMBER} and all deployments must have label version: $BUILD_NUMBER . Important part here is that your Services should not be a part of your Helm chart because they will be handled by Jenkins.
Start your build with detecting the current version of the app (using bash script or you can store it in ConfigMap).
Start helm install example-app-{$BUILD_NUMBER} with --atomic flag set. Atomic flag will make sure that the release is properly removed on failure. And don't delete previous version of the app yet.
Wait for Helm to complete and in case of success run kubectl set selector service/example-app version=$BUILD_NUMBER. That will instantly switch Kubernetes Service from one version to another. If you have multiple services you can issue multiple set selector commands (each command executes immediately).
Delete previous Helm release and optionally update ConfigMap with new app version.
Depending on your app you may want to run tests on non user facing Services as a part of step 4 (after Helm release succeeds).
Another good idea is to have preStop hooks on your worker pods so that they can finish their jobs before being deleted.
You should consider Blue/Green Deployment strategy
I'm creating a Deployment Group in CodeDeploy with a CloudFormation template.
The Deployment Group is successfully created and the application is deployed perfectly fine.
The CF resource that I defined (Type: AWS::CodeDeploy::DeploymentGroup) has the "Deployment" property set. The thing is that I would like to configure automatic rollbacks for this deployment, but as per CF documentation for "AutoRollbackConfiguration" property: "Information about the automatic rollback configuration that is associated with the deployment group. If you specify this property, don't specify the Deployment property."
So my understanding is that if I specify "Deployment", I cannot set "AutoRollbackConfiguration"... Then how are you supposed to configure any rollback for the deployment? I don't see any other resource property that relates to rollbacks.
Should I create a second DeploymentGroup resource and bind it to the same instances that the original Deployment Group has? I'm not sure this is possible or makes sense but I ran out of options.
Thanks,
Nicolas
First i like to describe why you cannot specify both, deployment and rollback configuration:
Whenever you specify a deployment directly for the group, you already state which revision you like to deploy. This conflicts with the idea of CloudFormation of having resources managed by it without having a drift in the actual configuration of those resources.
I would recommend the following:
Use CloudFormation to deploy the 'underlying' infrastructure (the deployment group, application, roles, instances, etc.)
Create a CodePipline within this infrastructure template, which then includes a CodeDeploy deployment action (https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference-CodeDeploy.html, https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codepipeline-pipeline-stages-actions-actiontypeid.html)
The pipeline can triggered whenever you have a new version inside you revision location
This approach clearly separates the underlying stuff, which is not changing dynamically and the actual application deployment, done using a proper pipeline.
Additionally in this way you can specify how you like to deploy (green/blue, canary) and how/when rollbacks should be handled. The status of your deployment also to be seen inside CodePipeline.
I didn't mention it but what you are suggesting about CodePipeline is exactly what I did.
In fact, I have one CloudFormation template that creates all the infrastructure and includes the DeploymentGroup. With this, the application is deployed for the first time to my EC2 instances.
Then I have another CF template for CI/CD purposes with a CodeDeploy stage/action that references the previous DeploymentGroup. Whenever I push some code to my repository, the Pipeline is triggered, code is built and new version successfully deployed to the instances.
However, I don't see how/where in any of the CF templates to handle/configure the rollback for the DeploymentGroup as you were saying. I think I get the idea of your explanation about the conflict CF might have in case of having a drift, but my impression is that in case of errors during the CF stack creation, CF rollback should just remove the DeploymentGroup you're trying to create. In other words, for me there's no CodeDeploy deployment rollback involved in that scenario, just removing the resource (DeploymentGroup) CF was trying to create.
One thing that really impresses me is that you can enable/disable automatic rollbacks for the DeploymentGroup through the AWS Console. Just edit and go to Advanced Configuration for the DeploymentGroup and you have a checkbox. I tried it and triggered the Pipeline again and worked perfectly. I made a faulty change to make the deployment fail in purpose, and then CodeDeploy automatically reverted back to the previous version of my application... completely expected behavior. Doesn't make much sense that this simple boolean/flag option is not available through CF.
Hope this makes sense and helps clarifying my current situation. Any extra help would be highly appreciated.
Thanks again
My project is Windows Service application which could be installed in several roles (the difference is in service name, exe path and some setting in app.config). Each role could be scaled horizontally by instances count. And all these {roles x replica counts} should be deployed over a set of servers in specific proportions for effective performance and utilization.
As an example:
ServerA
ServiceAlfa.1
ServiceAlfa.2
ServiceBravo
ServiceDelta
ServerB
ServiceBravo
ServiceCharlie
ServiceDelta.1
ServiceDelta.2
ServiceDelta.3
How can I achieve this with Azure DevOps (Dev17.M153.5) instruments?
I know brand new yaml pipeline introduces some conception of Environments and VM. It's just not available in latest stable version yet. But it's like a replacement for Deployment Groups early used for deployment to multiple machines, which I can use. I have already installed deployment agents and registered it. But I still cannot figure it out how better configure my complex mapping of instances to servers in release pipeline.
I can create a one job stage per role and link them with corresponding variable groups like
StageAlfa
ServerA:2
StageBravo
ServerA:1
ServerB:1
StageCharlie
ServerB:1
StageDelta
ServerA:1
ServerB:3
So I should check and compare the server name in my script
Or I can do the opposite: create a stage per machine and link it with variable group describing count of specific role replicas on current server. So in every stage I could select specific machine from deployment group by tag.
Looks like the second approach is simpler but they both are felt so awkward!
P.S. Windows Services on Machines not a containers in Kubernetes due to specific Windows software dependencies.
Your approaches are correct. You may consider migrating to Azure DevOps Service or upgrade to Azure DevOps Server 2020, which supports Envorinments and VM:
https://learn.microsoft.com/en-us/azure/devops/server/release-notes/azuredevops2020?view=azure-devops#continuous-deployment-in-yaml
Azure Devops has a standard task for deploying apps to ServiceFabric. The task is named Service Fabric Application Deployment and is documented here. Among other settings, it contains an optional boolean isUpgrade setting (default value 'true'). I tried to set it explicitly to true and false, but I did not find any difference in the behavior of the task. In both cases, the deployment was successful, all previously deployed packages were still provisioned, and Azure Pipelines logs were the same. The time of the deployment was the same, too.
My question is what the setting affects? Maybe, somebody has used it in his CI pipelines.
There are 2 types of deployment in Service Fabric. This isUpgrade flag controls which type op upgrade you are executing.
Regular
Basically this removes the old application and deploys the new version. So if you have Statefull services, this will remove all state. You will have downtime when you do a regular upgrade.
Upgrade
An upgrade will do a lot of things, It will keep the state, it will do health checking, make sure the services are available. Does a rollback when the healthcheck fails, ...
If your application or services didn't change, nothing changes in your cluster.
Typically an upgrade will take more time (This is highly dependent on your health check rules). See the application upgrade flowchart
More info about the 2 types
If you look at the code of the task. You see that it will only take effect if overridePublishProfileSettings is true. Otherwise the PulishProfile.xml is used.
We would like to try building a release pipeline for our product in VSTS - however, our product requires a separate instance of the application per customer (there are some legacy in the picture here :)). What we THINK we want, is a process like this:
For each customer:
Update DB schema
Configure a container, with customer-specific configuration etc.
Publish the container into Azure Container Registry
Deploy the container in Azure Container Service (OR on-prem if the customer runs on-prem)
The configuration can be multiple things: Extensions of the API in the application (new DLLs basically), connection strings, ...
I figure we can do this fairly easily using a custom PowerShell script, but I would like to not write anything custom (at least for the "looping" issue) if I don't have to. We could also create separate environments in VSTS for each customer, but that seems quite unmaintainable with well over 100 customers.
Some additional details:
- There's a separate DB per customer
- There's two separate web applications per customer
So what's the best practice here? Any advice? Thanks! :-)
You could think of doing it in two ways.
1 - By creating one environment for each customer. So you could have the exact same tasks for each environment, or have the flexibility to change steps in a particular environment.
This approach would give you also the ability to use a flow pipeline, because your build will be released only after is passes your internal QA and other processes.
To do it easily, you could also create task groups to reuse then in each environment.
2 - The other way is to create create separate releases for each customer or group of customers. This will also give you the same flexibility, you can use your builds, but you have to add some extra steps to make sure you are using the right build, since you can choose any build when you create a release, which you can do mannualy.
Updated
A third option could be to create on environment for all customers and then have the one deployment agent installed for every customer, using all of them on the same deployment group. Then have one file with all your variables per customer, with the file named with the agent name, and a powershell script that uses the agent name variable to find what file to run. This powershell script would then run all your individual configurations.
In that case, I suspect that you would end up doing almost all your deployment in powershell, which could be more time consuming for you to maintain. You also have to keep in mind that in this particular scenario you would update all your customers the same time, because all agents would be on the same deployment group.