Which tool / technology: System management for databases and dependent services - powershell

A follow-up on this system management question:
Since I probably will not get much feedback on serverfault I'll give it a try here.
My main concern is to reflect the dependencies between the databases, services ans tasks/jobs I'll have to manage.
Besides considering Powershell, I even thought about using MSBuild because it would allow for modeling dependencies and reuse configuration targets.
In other words: What technology should I use to develop a flexible solution that will allow me to stop service A, B and C on machine D in the right order and disable task E on machine F when taking down database X?
... and I'd like to reuse the part for stopping service B and C when taking down database Y.

If you are familiar with PowerShell and want to work with dependencies, try psake. What it looks like:
psake script.ps1:-------
properties {
$dbServer = 'sqlexpress'
...
}
task default -depend StopServer1, StopServer2
task StopS1 -depend MakeS1Backup, StopSqlServer1 {
...
}
task MakeS1Backup {
... make backup
}
task StopSqlServer1 {
stop-service ...
}
# and anything similar to StopServer2
Then you may call it like this (there are more options):
Invoke-Psake script.ps1
#or
Invoke-Psake script.ps1 -task StopS1 #calls only StopS1 task and all other scripts it depends on
#or
Invoke-Psake script.ps1 -task MakeS1Backup #only backups S1, it doesn't depend on anything else
What it does - it stops server 1 (task StopS1) and before that it processes all the tasks that StopS1 depends on. So before stopping S1 backup of S1 is made and Sql server 1 is stopped and so on.
I like it much better than msbuild configuration, which is very verbose and ugly (although very powerfull).

Related

How can I hide skipped tasks output in Ansible

I have Ansible role, for example
---
- name: Deploy app1
include: deploy-app1.yml
when: 'deploy_project == "{{app1}}"'
- name: Deploy app2
include: deploy-app2.yml
when: 'deploy_project == "{{app2}}"'
But I deploy only one app in one role call. When I deploy several apps, I call role several times. But every time there is a lot of skipped tasks output (from tasks which do not pass condition), which I do not want to see. How can I avoid it?
I'm assuming you don't want to see the skipped tasks in the output while running Ansible.
Set this to false in the ansible.cfg file.
display_skipped_hosts = false
Note. It will still output the name of the task although it will not display "skipped" anymore.
UPDATE: by the way you need to make sure ansible.cfg is in the current working directory.
Taken from the ansible.cfg file.
ansible will read ANSIBLE_CONFIG,
ansible.cfg in the current working directory, .ansible.cfg in
the home directory or /etc/ansible/ansible.cfg, whichever it
finds first.
So ensure you are setting display_skipped_hosts = false in the right ansible.cfg file.
Let me know how you go
Since ansible 2.4, a callback plugin name full_skip was added to suppress the skipping of task names and skipping keyword in the ansible output. You can try the below ansible configuration:
[defaults]
stdout_callback = full_skip
Ansible allows you to control its output by using custom callbacks.
In this case you can simply use the skippy callback which will not output anything on a skipped task.
That said, skippy is now deprecated and will be removed in ansible v2.11.
If you don't mind losing colours you can elide the skipped tasks by piping the output through sed:
ansible-playbook whatever.yml | sed -nr '/^TASK/{h;n;/^skipping:/{n;b};H;x};p'
If you are using roles, you can use when to cancel the include in main.yml
# roles/myrole/tasks/main.yml
- include: somefile.yml
when: somevar is defined
# roles/myrole/tasks/somefile.yml
- name: this task will only run (and be seen in the output) if somevar is defined
debug:
msg: "Hello World"

Why can't I debug my puppet code?

I'm learning Puppet and the biggest frustration I have with the entire paradigm is the try/run/fix development process I'm using to build functional Puppet code. My background is in Java and I'm naturally use to debugging my code to find errors instead of just running the program to see where it bombs making development much faster but I can't seem to find a way to do this using Puppet and Eclipse. I know writing a debugger for Puppet would require some creativity given its nature but I think this is something the community could really benefit from.
I've written debuggers and know the Eclipse SDK but unfortunately it does not map cleanly to the Puppet architecture which is a bit awkward in the sense its runtime stack and execution flow does not happen in natural order as well as the fact the runtime requires a target machine to apply changes on.
I'm curious if the community has done any development work on trying to create some kind of debugger where code can be stepped. To write this I think it would make sense to extend Eclipse with a new Puppet debug configuration type where you specify a target sandbox host to test your code as well as a puppet project in your workspace you want to debug (leveraging existing Gepetto tooling). Then when you start a new Puppet debugging session Eclipse could connect to the remote host, execute puppet apply with some additional debug arguments and somehow provide feedback from the runtime about what line of code is currently being executed.
This still might be awkward but would allow puppet developers to quickly see things like oh duh.. I can't create this directory because the parent path does not exist, wait... why is this if statement not going here like I planned, oh I see here that Puppet is not very clear on single or double quotes or now I see why this fails because this class was not executed first etc. etc.
Instead all we get is a big ugly output on the agent console that yes can give us insight on errors but does not cleanly map exceptions to our code that in my view shows an underlying pain and weakness of Puppet, can you at least give me a stack trace and line number so I know where to look? Nope sorry.
Don't get me wrong, I love how Puppet can make me look very productive throughout the work week when all I'm doing is running Puppet apply on new machines which my manager has not yet figured out but I think for Puppet to really be useful this lack of debugging support is something that needs to be addressed.
Does anyone else feel this pain? - Duncan Krebs
It would be impossible to "step through" puppet code, unless you want to debug against the ruby codebase itself. It's not just that the order of "execution" is unclear, its that the manifest themselves are never executed at a single time. They are actually evaluated in multiple phases throughout execution.
There are ways to simplify finding problems though. The biggest one is writing unit tests using rspec-puppet. It lets you essentially test the compilation phase of puppet, helping you catch errors like circular dependencies, incorrect conditional logic, etc.
There is a new tool called the puppet-debugger which allows you to set breakpoints in your puppet code in order to step through it. So this is no longer "impossible" as it has been available for about 8 months.
You will first need to install the puppet-debugger gem
https://github.com/nwops/puppet-debugger
Then install the debug module, include it in your fixtures or just ensure it is in your module path.
https://forge.puppet.com/nwops/debug
Then just set a breakpoint in your code using debug::break() function.
Ruby Version: 2.0.0
Puppet Version: 4.9.4
Puppet Debugger Version: 0.6.0
Created by: NWOps
Type "exit", "functions", "vars", "krt", "whereami", "facts", "resources", "classes",
"play", "classification", "types", "datatypes", "reset", or "help" for more information.
>> $vars = ['one', 'two', 'three']
=> [
[0] "one",
[1] "two",
[2] "three"
]
>> $vars.each | String $var | {
debug::break()
notify{$var:}
}
From file: puppet_debugger_input20170417-97123-qjwbaj.pp
1: $vars.each | String $var | {
=> 2: debug::break()
3: notify{$var:}
4: }
1:>> $var
=> "one"
2:>> exit
From file: puppet_debugger_input20170417-97123-qjwbaj.pp
1: $vars.each | String $var | {
=> 2: debug::break()
3: notify{$var:}
4: }
1:>> $var
=> "two"
2:>> exit
From file: puppet_debugger_input20170417-97123-qjwbaj.pp
1: $vars.each | String $var | {
=> 2: debug::break()
3: notify{$var:}
4: }
1:>> $var
=> "three"

OctopusDeploy - Every website in the deploy has a different AppPool and Website name; how to deal; no other differences

I'm trying to setup a deploy process that targets 16 web sites each hosting an instance of the same application.
Websites and AppPools are named as such:
appServer1:
app10.site.com
app11.site.com
app12.site.com
app13.site.com
appServer2:
app20.site.com
app21.site.com
app22.site.com
app23.site.com
etc.
etc.
...with each website having a correspondingly named AppPool.
I am desperately trying to determine how to use a single Deploy NuGet Package step to target all of these websites/app pools using variables and a combination of powershell scripts if possible.
I'd like to have a single step where I can variable substitute the website and app pool names. As this is the only difference. I basically need the equivalent of being able to loop the nuget package step passing it a list of website and app pool names. I cannot simply use variables because I can only resolve to the machine level with variable scoping.
Create list of all Website and AppPool names, iterate them passing each value to a Step for execution. ForEach processing step for lack of better words.
I do have the ability to rename the AppPools if need be for a more consistent pattern, but I cannot change the website names
Any ideas would be greatly appreciated.
http://help.octopusdeploy.com/discussions/questions/3481-every-website-in-the-deploy-has-a-different-apppool-and-website-name-how-to-deal-no-other-differences
There's a lot to your question, but I'm going to take a stab at explaining our approach, in hopes of jogging your creative juices.
tl;dr
simply put, use your own powershell scripts to install the web-application. In there you can set the app pool name on a per website basis
For starters, we do do a separate deployment step for each project. The scripts we use will allow you to do all deployments from a single deploy.ps1 (including unique appPool names), but we find that it really helps keep each deployment nice and lean, and easy to manage. Each project get's it's own nupkg and therein contains the predeploy.ps1, deploy.ps1, and postdeploy.ps1 as well as a folder of build/deploy scripts that we've open sourcesd, and a folder of environment config xml files.
A sample of an environment config would be this. The name is simply [envName].xml
<!-- environments\Production.xml -->
<environmentSettings>
<webSites>
<app>
<physicalPathRoot>c:\inetpub</physicalPathRoot>
<physicalFolderPrefix>appname</physicalFolderPrefix>
<siteProtcol>https</siteProtcol>
<siteName>appname.tld</siteName>
<siteHost>appname.tld</siteHost>
<portNumber>443</portNumber>
<appPath>/</appPath>
<appPool>
<name>appname.tld</name>
<!-- valid identityTypes are: [LocalSystem, LocalService, NetworkService, SpecificUser, ApplicationPoolIdentity] -->
<identityType>NetworkService</identityType>
<!-- Set this value to the User the Service will run under in the format DOMAIN\username -->
<!-- If Running as 'NetworkService' then 'NT AUTHORITY\Network Service' is used -->
<userName>NT AUTHORITY\Network Service</userName>
<!-- Leave blank unless using SpecificUser -->
<password></password>
<maxWorkerProcesses>5</maxWorkerProcesses>
</appPool>
</app>
</webSites>
<serverDatabase>
<name>database_name</name>
<connectionString>REPLACED BY OCTOPUS</connectionString>
<providerName>System.Data.SqlClient</providerName>
</serverDatabase>
</environmentSettings>
You can see in the corresponding Get-EnvironmentSettings.ps1 where we load up the config, and then update it with any Octopus variables. This is the trickiest part, because we use dot-Notation to update the paths (case sensitive).
Our octopus variables really only contain information that is secret, as everything else lives in [environment].xml
| Name | Value | Scope
--------------------------------------------------------------------------
| webSites.app.appPool.password | supersecret | Production
So now a typical deployment script simply imports the modules, grab environmentSettings, update config, and install the web app.
# Top of the script, get Octopus environment and version
param(
[string] $version = $OctopusPackageVersion,
[string] $environment = $OctopusEnvironmentName
)
# Make sure a failed deployment actually fails
$ErrorActionPreference = "Stop"
# Import the modules
$currentDir = Split-Path $script:MyInvocation.MyCommand.Path
$moduleDir = "$currentDir\modules"
Import-Module BuildDeployModules
# Grab the environment settings
$environmentSettings = Get-EnvironmentSettings $environment "//environmentSettings"
$databaseSettings = $environmentSettings.serverDatabase
$websiteSettings = $environmentSettings.webSites.app
# update the config
Update-XmlConfigValues $currentDir\website\Web.config "//appSettings/add[#key='databaseName']" $($databaseSettings.name) "value"
Update-XmlConfigValues $currentDir\website\Web.config "//connectionStrings/add[#name='databaseConnection']" $($databaseSettings.connectionString) "connectionString"
Update-XmlConfigValues $currentDir\website\Web.config "//connectionStrings/add[#name='databaseConnection']" $($databaseSettings.providerName) "providerName"
# Install the web application
Install-WebApplication $environment $websiteSettings $version "anonymousAuthentication"
In doing all of this, the web application is installed into IIS with a specific application pool, and appropriate config transforms without relying on any unknowns.
Our nupkg structure looks something like this
appname.1.2.3.4.nupkg
environments
dev.xml
staging.xml
qual.xml
production.xml
modules
[all of our build modules]
website
[all of our website files]
This is super repeatable, easy to maintain, and easy to edit config. Hope it helps

Can I dynamically create psake tasks?

Is it possible to dynamically create psake tasks based on configuration (much as rake does)? For example, given a hashtable of environments:
environments {
$dev = 'server1',
$uat= 'server2'
}
I'd like tasks deploy.db.dev, deploy.db.uat, deploy.app.dev, deploy.app.uat etc. where the appropriate server name is provided to the task template?
Thanks

Waiting for a new deployment to fully initialize before swap the staging/production slot (swap VIP)?

I use the following code to swap my newly deployed application from the staging slot into the production slot (swap VIP):
Get-HostedService -serviceName $serviceName -subscriptionId $subcription -certificate $certificate | Get-Deployment -slot staging | Move-Deployment |Get-OperationStatus –WaitToComplete
I thought that the -WaitToComplete flag would make sure all VMs have fully initialize before doing the swap however it doesn't and it performs the swap at which time the newly deployed application in the production slot is still initializing and is unavailable for about 5/10min while it initializes fully.
What is the best way to make sure that the application is fully initialized before doing the Swap VIP operation?
This PowerShell snippet will wait until every instance is ready (building on the answer #astaykov gave).
It queries the state of the running instances in the staging slot, and only if all are showing as 'ready' will it leave the loop.
$hostedService = "YOUR_SERVICE_NAME"
do {
# query the status of the running instances
$list = (Get-AzureRole -ServiceName $hostedService `
-Slot Staging `
-InstanceDetails).InstanceStatus
# total number of instances
$total = $list.Length
# count the number of ready instances
$ready = ($list | Where-Object { $_ -eq "ReadyRole" }).Length
Write-Host "$ready out of $total are ready"
$notReady = ($ready -ne $total)
If ($notReady) {
Start-Sleep -s 10
}
}
while ($notReady)
I am guessing that what you might actually be seeing is the delay that it takes for the DNS entries to be propagated and become available.
What you should find is that once the status is reported as Ready you may not be able to access your site using the staging URL "http://.cloudapp.net" you will find that it might not come up... but if you look on the Management Portal you will see at the bottom of the Properties a value for 'VIP' - if you use that IP address "http://xxx.xxx.xxx.xxx you should be able to get to your site.
When you do a SWAP you will find similar behavior. It will take some time for the DNS updates to propagate, but you will likely see that you can still access the site with either the IP address or the staging address (if it has become available).
Finally, 1 question... based on your question it sounds like you might be deploying to staging as part of your build then immediately promoting to a production deployment... is this correct, and if so why not just deploy to the production deployment? (I'm not suggesting that deploying directly into production is a best practice... but if that is your workflow I see no benefit to the temporary deployment to staging)
Hope this helps!
I am not very familiar with PowerShell, but from my experience with shells at all you are pipelining commands. Each set before a pipe charachter (|) represents a single command which would pass its result to the next command in the pipe (command after the pipe character). And because you are executing these commands before the depolyment is fully complete, that's why you get the newly deployed app swapped to the production slot.
First thing to note here is that you have "-WaitToComplete" argument just for the last command, which is actually Get-OperationStatus.
Other thing that I see is that this powershell commands will just do the vip swap. What about deployment?
From what you descriped it appears that your build server is auto deploying to staging, and you have post-build event that executes the swap script. What Mike Erickson suggests here would make sense, if your flow is like that - immediately swap after depoloy to staging. Why would you deploy to staging, if you are going to make a swap without checking application health first? However I would not recommend direct depolyment to the server (delete + deploy), but a service upgrade. Because when we do service upgrade, our deployment keeps its public IP address. If we delete + deploy we get a new public IP address. And the public IP address for a hosted service is already guaranteed to not be changed untill deployment is deleted.
Finally, you shall expand your PowerShell script a bit. First include a routine which will check (and wait untill) the staging slot to be "ready", and then perform the swap. As I said, I'm not much into powershell, but I'm sure this is feasible.
Just my 2 cents.
UPDATE
After revisiting this guide, I now understand something. You are waiting for operation to complete, but this is the VIP-SWAP operation which you are waiting to complete. If your stating deployment is not yet ready, you have to wait for it to become ready. And also like Mike mentioned, there might be DNS delay, which is noted at the end of the guide:
Note:
If you visit the production site shortly after its promotion, the DNS
name might not be ready. If you encounter a DNS error (404), wait a
few minutes and try again. Keep in mind that Windows Azure creates DNS
name entries dynamically and that the changes might take few minutes
to propagate.
UPDATE 2
Well, you will have to query for all the roles and all of their instances and wait for all of them to be ready. Technical you could conduct the VIP swap with at least one instance per role being ready, but I think that would complicate the script even more.
Here's a minor tweak to Richard Astbury's example above that will retry a limited number of times. All credit to him for original sample code, so I'd vote for him as the answer most to the point. Simply posting this variation here as an alternative for people to copy/paste as needed:
$hostedService = "YOUR_SERVICE_NAME"
# Wait roughly 10 minutes, plus time required for Azure methods
$remainingTries = 6 * 10
do {
$ready=0
$total=0
$remainingTries--
# query the status of the running instances
$list = (Get-AzureRole -ServiceName $hostedService -Slot Staging -InstanceDetails).InstanceStatus
# count the number of ready instances
$list | foreach-object { IF ($_ -eq "ReadyRole") { $ready++ } }
# count the number in total
$list | foreach-object { $total++ }
"$ready out of $total are ready"
if (($ready -ne $total) -and ($remainingTries -gt 0)) {
# Not all ready, so sleep for 10 seconds before trying again
Start-Sleep -s 10
}
else {
if ($ready -ne $total) {
throw "Timed out while waiting for service to be ready: $hostedService"
}
break;
}
}
while ($true)