I have a project that uses Entity Framework 6.x, ASP .NET WebApi 5.x. Data Access is in a secondary project inside the solution. I want to use VSTS (aka Visual Studio Online) to build and release it as a website to environments for integration/dev, qa, stage, and production. There are some great videos on Channel 9 that deal with the generic high-level description. (for example https://channel9.msdn.com/Series/DevOps-Release-Management and https://channel9.msdn.com/Series/DevOps-Fundamentals/Infrastructure-as-Code) There are all kinds of articles and videos about how to do migrations from inside Visual Studio including generating scripts.
Searching around the web I don't even find any older resources or concrete examples of continuous deployment with code first migrations. There must be examples and best practices for methods other than auto-migrations or SQL scripts.
I have configured a Web Deployment Package publish profile. I use it via the PublishProfile msbuild.exe directive. The package is added to the artifacts and then deployed by the Azure Web App Deployment task in each Release environment. However once this package is built, I don't know of a way of changing the connection string in the build package for each time it is released to an environment.
There is probably something I am overlooking, but how should environment specific migrations be done with via VSTS Release?
For Code First Migration, you can "Write App_Start code to run Migrations" or "Write Web.config transforms to configure the MigrateDatabaseToLatestVersion initializer to run", refer to this article for details: http://blogs.msdn.com/b/webdev/archive/2014/04/09/ef-code-first-migrations-deployment-to-an-azure-cloud-service.aspx
For the connection string transformation with profile, you need to add a web.config file for the publish profile and then enter the connection string in this web.config. Refer to this link for details: http://awaitwisdom.com/publish-profile-config-transform/
I hate to answer my own questions here but, ultimately my research took me to the conclusion I am posting at length here. In sort, Web.config and Parameters.xml require some custom scripting that will require you to maintain your own deployment automation. These routes will still require you to additionally create your resource groups or manage them manually.
To avoid these complications and cobbling tools and scripts together, the whole operation can be achieved with two JSON files. These JSON deployment templates allow you to create or update your resource group when your deployment runs. They also allow you to automate setting appsettings and connectionstrings that overwrite your Web.config values in the same manner as you can through the Azure Portal.
the steps: (1) Add the two JSON files to the project setting the name of you connection string on line 88 (2) Add a Azure Resource Group Deployment task to the Release environment. (3) Set Template (WebSite.json) and Template Parameters (WebSite.parameters.json) paths in the task. (4) Set Override Template parameters to -hostingPlanName "myHostingPlan" -webSiteName "myWebsiteName" -connectionString "the-actual-connection-string" (5) make sure you are using the same website name in your Azure App Deployment task.
This does depend on having your code first migrations run via App_Start or something similar. I took the first part of #Eddie's suggestion since App_Start is easy to deal with and doesn't seem to run too often.
As a bonus you can add environment variables for any of this configuration so you can clone the environment and then just change the variables. This ultimately makes your application or api connection string a Release variable.
Related
I am trying to create a release pipeline in Azure DevOps. We already have a functioning build pipeline that works well, it is able to package the build with VSBuild and publish it as an artifact. Then in the release pipeline I am using an IIS Deployment job (which includes IIS Manage and IIS Deploy tasks) and it gets that artifact to deploy.
The problem is that we already have a publish profile (.pubxml) that should take care of pretty much everything the IIS Deployment is doing (at least as far I as I understand it). So to me it seems I have two options that don't require me to refactor the project configuration itself.
I can try to mimic the settings on the IIS Deployment job to match our .pubxml as closely as possible and manually applying any changes that aren't doable through the task settings. Obviously this is not ideal as that would require us to update both when ever we make changes and it introduces a large chance of the pipeline breaking down over time.
I can scrap the idea of using IIS Deployment and just use a VSBuild task that uses arguments /p:DeployOnBuild=true /p:PublishProfile=Staging. This doesn't seem like best practices because it means my release pipeline isn't passing a build package to deploy, it is just creating a new one at each stage.
So is there a better option that would allow me to utilize the package I created with VSBuild and the .pubxml configuration together in a deploy? If that isn't possible then are either of my options the "correct" way to handle my situation or am I just missing another method of deployment I could use?
Thank you for any help or insight you can provide. Please let me know if there is any more information I can give that would be useful.
You can try using publish settings file (*.publishsettings) for your IIS deployment.
A publish settings file (.publishsettings) is different than a publishing profile (.pubxml) created in Visual Studio. A publish settings file is created by IIS or Azure App Service, or it can be manually created, and then it can be imported into Visual Studio.
To view more details, you can see:
Publish an application to IIS by importing publish settings in Visual Studio
Deploy your app to a folder, IIS, Azure, or another destination
So unfortunately there doesn't seem to be a way I can achieve everything I wanted in this. The publish profiles are required for when we build the project so without making changes to how we configure those I need to build the project whenever I want to deploy. Ultimately I went with option #2. I essentially just copied most of the build tasks used in the testing pipeline and placed those in the release pipeline with a few modified commands to actually deploy the build once finished. It all seems to work just fine but still doesn't feel like best practices. If I am missing something please let me know and I will make updates as appropriate.
My question is about maintainability of vNext/Octopus processes vs XAML based processes. Or rather about the impossibility to maintain them sanely leading me to believe we are doing something terribly wrong.
Given:
Microsoft pushes to phase out its TFS XAML builds in favour of the vNext builds
Octopus Deploy is a popular deployment automation framework
We have many XAML based builds, but starting to port to vNext
The deployments are automated with Octopus Deploy
Concretely, we have three kinds of builds going on in QA:
Old XAML based compilation builds producing artifacts to be deployed
Ultimately just builds the code, zips it and places in a well-known location
New vNext compilation builds producing artifacts to be deployed
Same as above
Deployment builds
XAML based build definition per deployment environment. This is the source of truth for the particular deployment, containing all the configuration URLs, connection strings, certificate thumbprints, etc.. The build definition has over 100 build parameters. Each time a new environment is setup we clone an existing XAML build definition and change the parameters.
This build unpacks the build artifact, generates all the web/app config files based on the configuration parameters and kicks off Octopus Deploy with a lot of parameters using octo.exe and waiting for the end
Octopus Deploy process
Creates 3 packages from the build artifact previously unpacked by the XAML build to match three areas of deployment - web farm, background job engine cluster and the database
Delivers the relevant packages to the relevant tentacles.
The tentacles unpack and setup their respective packages
So, if we have 50 deployment environments, then we have 50 XAML deployment builds, each capturing the context of the respective environment. But the XAML deployment build delegates the deployment job to Octopus and here we are forced to have 50 Octopus projects - one per deployment.
Why is it so? We examined the option of having just one Octopus Project, but what would be the Release versions of such project? In order for us to be able to navigate amongst the gazillion releases, the release version must include:
The build version of the deployed code, e.g. 55.0.18709.3
The name of the deployment environment, e.g. atwfm
Using the example above this gives us 55.0.18709.3-atwfm, but sometimes we want to deploy the same build artifact in the same deployment environment twice. But the only Octopus project would already have the release 55.0.18709.3-atwfm, so how to deploy 55.0.18709.3 in atwfm again, without deleting the already existing release?
We could not find a workaround and so, we have Octopus project per deployment.
THIS IS ABSOLUTELY CRAZY because Octopus projects are a pain to update. Suppose we need to add a step - go do it in 50 projects. There are great advises on the Internet to use automation to edit multiple projects. Not ideal at all.
vNext, BTW, has the same problem. If I am to port the existing XAML builds to vNext I will end up with 50 vNext deployment builds. If I decide to add a step, I need to do it in all the 50 builds!!!
Note, that XAML builds do not have this problem (they have many others, though), because their the process is separate from the parameters. I can modify the workflow once and all the XAML builds are now updated with the new process change.
My question is - how do people work with vNext and Octopus, because our process drives me crazy. There must be a better way.
EDIT 1
I would like to clarify. We sometimes want to deploy the same build artifacts twice. We are not recompiling them and reusing the same version. No. We already have the build artifacts handy with the build version baked inside the artifact. We may want to deploy it the second time into the same environment because, for example, some databases in that environment have been misconfigured and now this is fixed and we need to redeploy. This does not mean we can rerun the already existing Octopus release, because the fix may involve tweaking the deployment parameters of the respective XAML deployment build definition. Hence we may be forced to restart the XAML deployment build, which never compiles code.
EDIT 2
First of all, why do we drive the deployment from TFS XAML builds rather than from Octopus? Historic reasons. We did not have Octopus at first. The deployment was done by our ad hoc code. When we introduced Octopus we decided to keep the XAML deploymenet builds for two reasons:
To save the cost of migrating all the XAML deployment builds with all the gazillion deployment parameters to Octopus. Maybe it was a wrong decision, maybe we could have automated the migration.
Because TFS has better facility to display test results. The deployment may end with deployment smoke tests and their results has to be published somewhere. We do not see how Octopus can help us publish the results, TFS can.
Why would one redeploy? For example, one of the deployment parameters is certificate thumbprint. When the certificate is renewed, this parameter must be changed (we do have automation for updating XAML build parameters). But often we discover that it was already deployed with wrong thumbprint. So, we fix the deployment and redeploy. Or, we discover some strange behavior of the deployed application and wish to redeploy with some extra tracing/debugging features.
There is a lot to unpack here, but I'll give it a go.
TL;DR It's the way you version the releases that's causing you all the pain. Change that and everything else will fall in to place
Lets start at the end and work backwards.
Octopus Deploy has a concept of Environments. This means that you can Deploy the same project to multiple environments and use Octopus's scoping mechanism to manage environment specific configuration.
So using your example.
Creates 3 packages from the build artifact previously unpacked by the
XAML build to match three areas of deployment - web farm, background
job engine cluster and the database
I set up an Environment in Octopus for each of your 50 Environments. (I'll use 3 environments in the example to keep it simple, but the principles apply no matter how many environments you have)
In my Dev Environment I have a single server so I create an environment called "Dev" and add the tentacle for that specific server. Then I tag the tentacle with the deployment type "Web", "Job", "Database"
I then set up a test environment which has 3 servers so I create the Environment and add the 3 servers. I then tag each tentacle with the deployment type "Web", "Job", "Database"
Finally I set up the Production environment. This has 5 web servers, 1 job server and 1 database server. I add all 7 tentacles to the environment, and tag them appropriately.
Now I only need 1 project to deploy to all 3 environments. In my project I have 3 steps.
Step 1 Deploy Web Site
Step 2 Deploy Jobs
Step 3 Deploy database
I can tag each of these steps to say what kind of tentacle I want to deploy to. Now when I run the deployment the link between the tags on the step, and the tags on the tentacle mean Octopus knows where to deploy the code.
Variables: Your variables can be scoped to an environment. So for example if your dev environment database connection string is dev.database.net/Instance and your test environment database connection string is test.Database.net/Instance then you can scope these in the variables section of the project. If your DNS is consitant with your environment names you could even use some of the built in variables to make adding environments more easy. i.e. ${Octopus.Environment.Name}.Database.net/Instance
Releases and version numbers: So here is where I think your problem lies. Adding the environment name to the release and trying to create multiple releases with the same version is basically causing you all of the pain.
Using the example above this gives us 55.0.18709.3-atwfm, but
sometimes we want to deploy the same build artifact in the same
deployment environment twice. But the only Octopus project would
already have the release 55.0.18709.3-atwfm, so how to deploy
55.0.18709.3 in atwfm again, without deleting the already existing release?
There are a couple of things here. In Octopus you can easily deploy again from the UI, however it sounds like you're rebuilding the artifact and trying to create a new release with the same version number. Don't do this! Each new build should have a distinct and unique build number / release number.
The principle I follow is "build once deploy many"
When you create a release it requires a version number, this release then flows through the environments. So I build my code and it gets a versions number 55.0.18709.3 then I deploy it to Dev. When the deployment has been verified I then want to "Promote" the release to test I can do this from within Octopus or I can get TFS to do this.
So I promote 55.0.18709.3 to test and then on to prod. If I need to know which release is in which environment, Octopus tells me this via the dashboard or API.
Finally I can "Orchestrate" the flow of releases through my environments using Build v.next.
So my end to end process looks something like.
Build vNext Build
Compile
Run Unit Tests
Package output
Publish package
build vNext Release
Call Octopus to create the release passing in the version number
Optionally deploy the release to the first environment on your way to live
I now have everything I need in Octopus to deploy to ANY environment with a single project and my environment specific configuration.
I can either "Deploy" the release to a specific environment or "Promote" the release from one environment to another. This can be done easily from within the Octopus UI
Or I can create a "Promote" using the Octopus plugin in TFS and use that to orchestrate the promotion of code through the environments.
Octopus Terminology.
Create release - This pulls together the Artifacts and Release number in Octopus to create an Immutable thing which will be deployed to one of more environments.
Deploy release - The act of pushing your code to a specific environment.
Promote release - Once the code has been deployed in to a single environment, it can them be promoted in to other environments
If you have a specific sequence of environments then you can use the "Lifecycles" feature of Octopus to enforce that workflow. but that's a topic for another day!
EDIT1 Response
I don't think the edit changes my answer, you can re-deploy the same release many times as you like. what you cannot do is create a new release with the same version number. You might want to decouple these steps could you add some more detail about what changes in the XAML build? You can change variables in a release, you can update them in octopus and then redeploy
EDIT 2 Response
That makes things clearer. I think you need to take the hit and migrate the parameters to Octopus. It's variable management is much better than XAML builds and although build vNext is comparable to Octopus it makes more sense to have the config in Octopus. As XAML builds are on their way out, it makes sense to move this stuff now. Whilst it might be a lot of work, at the end you'll have a much smother workflow and you can really take advantage of the power of Octopus.
The Test results point. I agree this is better suited to build vNext, so at this point you will be using build vNext as your Orchestrator and Octopus Deploy as your release management tool.
The process would look something like
Build vNext
Compile code.
Run Unit tests
Run Octopack
Publish packages
Call Octopus and Create release
Call Octopus to Deploy to "Dev"
Run Smoke tests
Run Integration Tests
Call "Selenium" to run Run UI tests
Call Octopus to Promote release to "Test"
Run Smoke tests
Run Integration Tests
Call "Selenium" to run Run UI tests
Call Octopus to Promote release to "Production" (Perhaps with a manual innervation)
Run Smoke tests
Run Integration Tests
Call "Selenium" to run Run UI tests
I have an ASP .Net MVC application with 4 different publishing profiles: dev, test, demo and prod.
These publising profiles are build using the same two steps: NuGet restore followed by an MSBuild. Then, they are deployed to lots of different servers: a few dev servers (one server dev per team), one test server, one demo server and several production servers.
msbuild /p:Configuration="$(Configuration)"
/p:PlatformTarget="any cpu"
/p:DeployOnBuild="True"
/p:DeployTarget="MsDeployPublish"
/p:MSDeployServiceURL="$(MSDeployServiceURL)"
/p:DeployIISAppPath="portal"
/p:CreatePackageOnPublish="False"
/p:MsDeployPublishMethod="WMSVC"
/p:AllowUntrustedCertificate="True"
/p:UserName="Deploy_User_For_TFS"
/p:Password="P#ssw0rd"
/p:AutoParameterizationWebConfigConnectionStrings=False
/p:ExcludeFilesFromDeployment="Cache"
Currently I have 4 TFS Build configurations (one for every publishing profile) and a file where I have all the possible values for MSDeployServiceURL parameter.
There are two issues with this approach:
When we had to add a new parameter AutoParameterizationWebConfigConnectionStrings we had to change it in 4 places instead of one.
We have to have a shared file as the source for the parameters. it is not easy to understand where which value should be copied and people often make mistakes.
So I have two questions:
Is there any way to have a one universal template where I can specify only my parameters Configuration and list of possible MSDeployServiceURLs and have everything else stay the same? Having such a template should fix problem #1.
Is there a way to define a drop-down like variable, where the value could not be typed in by the user, but should be selected from a pre-defined list of values?
Is there any way to have a one universal template where I can specify
only my parameters Configuration and list of possible
MSDeployServiceURLs and have everything else stay the same? Having
such a template should fix problem #1.
You could simplify such that the release configuration creates a templated publish profile. Then using TFS's release management, you could update the publish profile with the appropriate values. Based on your description, it seems like you are trying to combine both the compilation and the release.
For example, in TFS you could have one build (for example, MyApp-Release) that builds the code in the release configuration. As part of that process, it passes in placeholders for things like the deploy URL. For example, /p:MSDeployServiceURL="$(MSDeployServiceURL)" would be /p:MSDeployServiceURL="__MSDeployServiceURL__".
In the TFS release, you'd have a step that the replaces tokens (if you need one, you can use Colin's ALM Corner Custom Build Tasks) in the publish profile. The replace token task would then update the __MSDeployServiceURL__ with the value from an release environment variable with the same name (minus the underscores). So your release would have a dev, test, demo, and prod environment and for each environment, there would be a variable named MSDeployServerURL in each with a different value and a replace tokens step.
I'm creating a test environment in Azure. I want to have an accurate script of what of the configuration so it's easy to replicate for other test, pre-prod and prod environments later on. The environment has an existing subscription, and I want the entire hierarchy of resources from Resource Groups through to Web Apps to be created by script.
I'm currently rolling my own script in PowerShell utilising AzureRm. This is working well, but I can't help feel I'm reinventing the wheel. What is the existing method for creating an entire Azure environment by script?
Yes, that way is called Azure Resource Manager Templates. Quote:
With Resource Manager, you can create a template (in JSON format) that defines the infrastructure and configuration of your Azure solution. By using a template, you can repeatedly deploy your solution throughout its lifecycle and have confidence your resources are deployed in a consistent state. When you create a solution from the portal, the solution automatically includes a deployment template. You do not have to create your template from scratch because you can start with the template for your solution and customize it to meet your specific needs. You can retrieve a template for an existing resource group by either exporting the current state of the resource group, or viewing the template used for a particular deployment. Viewing the exported template is a helpful way to learn about the template syntax.
Reference: https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#template-deployment
Edit: you can use powershell, azure cli, azure cli2, azure sdk to deploy those templates (or simply Azure portal, search for "Template Deployment")
I'm using Visual Studio Team Services and I'm trying to set up Release Management to allow automated deployments for our Azure Web App to multiple environments. I would like the same source to be deployed to each environment, but with modified configuration settings.
I was hoping that I could create a single Build for my application, and then modify the configuration at deployment time for each environment. I'm aware that this can be done for appSettings and connectionStrings (either through Tokenization, or even managing those settings via the Azure portal), but I'd like to be able to make more general changes to the web.config file. For example, I want to be able to:
Update 'simple' settings such as appSettings/connectionStrings
Update multiple attributes on elements (like httpErrors)
Insert or rewrite sections of the config file itself (for example to add IIS rewrite rules, or to remove unwanted HTTP handlers for production)
Currently we achieve this by using config file transformations and separate publish profiles for each environment (manual deployment). I can't see a way to re-use this if I want a single release pipeline.
I'm hoping someone can help point me in the right direction. I'm also happy to accept alternative solutions - ultimately I just want to be able to deploy the same source (e.g. from the same commit in source control) to multiple environments, with different configuration, while keeping some kind of flow from dev, to test, to eventually production.
You can use Tokenization Task to update the files base on the environment variables.
More similar extensions: Replace Tokens and Colin's ALM Corner Build & Release Tools.