Install Windows HotFix using Terraform on AWS - powershell

Question

I have a very simple PowerShell script that uploads a generated test file to an AWS S3 bucket from a Windows 2008 R2 Datacenter server (clean AWS instance). If I run the script remotely on the server using Terraform (remote-exec provisioner), the script fails on the S3 upload with a StackOverflowException. When I run the script directly on the server, it runs fine and uploads the file.
I've experimented with different sizes for the file and 14.5MB seems to be about the maximum that works before the StackOverflowException occurs. Just about any size works fine when I RDP into the server and run the script directly. I've tested 200MB and it works fine.
Any idea why this is happening or what I can do to fix it? The actual file I need to upload is 50MB.
Here are the essential parts to recreate the problem. terraform.tf file:
resource "aws_instance" "windows" {
count = "1"
ami = "ami-e935fc94" #base win 2008 R2 datacenter
instance_type = "t2.micro"
connection {
type = "winrm"
user = "<username>"
password = "<password>"
timeout = "30m"
}
provisioner "file" {
source = "windows/upload.ps1"
destination = "C:\\scripts\\upload.ps1"
}
provisioner "remote-exec" {
inline = [
"powershell.exe -File C:\\scripts\\upload.ps1"
]
}
}
The PowerShell script is very simple. upload.ps1:
$f = new-object System.IO.FileStream C:\Temp\test.dat, Create, ReadWrite
$f.SetLength(40MB) # change this to 14.5MB and it works!
$f.Close()
Write-S3Object -BucketName "mybucket" -Folder "C:\Temp" -KeyPrefix "20180322" -SearchPattern "*.dat"
The error that I receive when launching the script from Terraform (remote-exec provisioner):
aws_instance.windows (remote-exec): Process is terminated due to StackOverflowException.
Running upload.ps1 from RDP on the server itself works fine, including larger files (tested up to 200MB).
Here is the version information:
Microsoft Windows Server 2008 R2 Datacenter
Powershell Version: 3.0
AWS Tools for Windows PowerShell, Version 3.3.245.0
Amazon Web Services SDK for .NET, Core Runtime Version 3.3.21.15

Answer 1

This problem results from a Windows bug. This is all fine and good for a standard Windows server -- you can patch and move on. But, things are more tricky with AWS automation using Terraform.
The ideal solution would allow 1) use of the base AMI, 2) apply the hotfix to itself, and 3) then run the WinRM remote-exec, all from Terraform. Another solution would be to create an AMI with the hotfix installed and have Terraform generate instances using that AMI. However, then you're stuck maintaining AMIs.
Normally, I grab the Microsoft-provided base AMI using a filter:
data "aws_ami" "windows2008" {
most_recent = true
filter {
name = "virtualization-type"
values = ["hvm"]
}
filter {
name = "name"
values = ["Windows_Server-2008-R2_SP1-English-64Bit-Base*",]
}
owners = ["801119661308", "amazon"]
}
Then I use that AMI to create the AWS instance:
resource "aws_instance" "windows" {
count = "1"
ami = "${data.aws_ami.windows2008.id}"
...
}
But, the base AMI doesn't have the hotfix installed allowing you to avoid this WinRM/Windows bug. This is were it gets tricky.
You can use a userdata script to perform a multi-phase setup. In the first boot of the instance (Phase 1), we'll block the instance so that the remote-exec doesn't come in before we're ready. Then, we'll download and install the hotfix and we'll reboot (thanks to Niklas Akerlund, Micky Balladelli and Techibee). On the second boot (in method described here), we'll unblock the instance (enable WinRM) so that the remote-exec can connect.
Here's my userdata/PowerShell script:
$StateFile = "C:\Temp\userdata_state.txt"
If(-Not (Test-Path -Path $StateFile))
{
# PHASE 1
# Close the instance to WinRM connections until instance is ready (probably already closed, but just in case)
Start-Process -FilePath "winrm" -ArgumentList "set winrm/config/service/auth #{Basic=`"false`"}" -Wait
# Set the admin password for WinRM connections
$Admin = [adsi]("WinNT://./Administrator, user")
$Admin.psbase.invoke("SetPassword", "${tfi_rm_pass}")
# Create state file so after reboot it will know
New-Item -Path $StateFile -ItemType "file" -Force
# Make it so that userdata will run again after reboot
$EC2SettingsFile="C:\Program Files\Amazon\Ec2ConfigService\Settings\Config.xml"
$Xml = [xml](Get-Content $EC2SettingsFile)
$XmlElement = $Xml.get_DocumentElement()
$XmlElementToModify = $XmlElement.Plugins
Foreach ($Element in $XmlElementToModify.Plugin)
{
If ($Element.name -eq "Ec2HandleUserData")
{
$Element.State="Enabled"
}
}
$Xml.Save($EC2SettingsFile)
# Download and install hotfix
# Download self-extractor
$DownloadUrl = "https://hotfixv4.trafficmanager.net/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix467402/7600/free/463984_intl_x64_zip.exe"
$HotfixDir = "C:\hotfix"
$HotfixFile = "$HotfixDir\KB2842230.exe"
mkdir $HotfixDir
(New-Object System.Net.WebClient).DownloadFile($DownloadUrl, $HotfixFile)
# Extract self-extractor
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Compression.ZipFile]::ExtractToDirectory($HotfixFile, $HotfixDir)
# Install - NOTE: wusa returns immediately, before install completes, so you must check process to see when it finishes
Get-Item "$HotfixDir\*.msu" | Foreach { wusa ""$_.FullName /quiet /norestart"" ; While (#(Get-Process wusa -ErrorAction SilentlyContinue).Count -ne 0) { Start-Sleep 3 } }
# Reboot
Restart-Computer
}
Else
{
# PHASE 2
# Open WinRM for remote-exec
Start-Process -FilePath "winrm" -ArgumentList "quickconfig -q"
Start-Process -FilePath "winrm" -ArgumentList "set winrm/config/service #{AllowUnencrypted=`"true`"}" -Wait
Start-Process -FilePath "winrm" -ArgumentList "set winrm/config/service/auth #{Basic=`"true`"}" -Wait
Start-Process -FilePath "winrm" -ArgumentList "set winrm/config #{MaxTimeoutms=`"1900000`"}"
}