Intermittent errors with automation of virtual machines using Powershell - powershell

Question

I have an intermittent problem with Powershell when automating some tests on a virtual machine.
The scenario and set up is as follows:
Server running HyperV
One virtual machine with multiple snapshots
Powershell script that restores a given snapshot, copies files over, runs a test and retrieves log files
Batch file that calls the Powershell script multiple times with different parameters
The batch file parameters specify things like which snapshot to use, which test to run, etc.
The problem is as follows:
I can run the batch and some of the tests will fail to copy files / fail to create a scheduled task / fail to retrieve log files / etc. It varies which if any (or all) sections fail. Some of the tests will work completely. If I re-run the same batch file, again some tests may fail and others will work; there is no consistency in terms of which fail and which run. Sometimes I have two adjacent tests that use the same snapshot, 1 will work and 1 won’t (see errors below).
To restore the snapshots I am using the “PowerShell Management Library for Hyper-V” from: (http://pshyperv.codeplex.com/releases)
Below is some of the code:
Powershell (minus a few functions / variable declarations / reading xml config file / reading and validating command line inputs / and other non-relevant sections):
Function ApplySnapshot
{
LogAction "Starting apply snapshot"
LogAction $("Restoring snapshot {0}" -f $ss)
#Stop a running VM, restore snapshot, start it up and connect to it
$vmstate = get-vmstate $vmname
$vmstate = $vmstate.EnabledState
if ($vmstate -ne "Stopped")
{
stop-vm $vmname -force
Start-Sleep -Second 15
}
get-vmsnapshot $vmname | where {$_.ElementName -eq $ss} | Restore-VMSnapshot -force
start-vm $vmname -force
Start-Sleep -Second 20
LogAction $("Snapshot {0} restored" -f $ss)
LogAction "End apply snapshot"
}
Function CopyFiles
{
LogAction "Start copy installation files"
$from = "\\server\folderx"
$to = "\\" + $hostname + "\C$\test"
Enter-PSSession -ComputerName $hostname -Credential $cred
Copy-Item $from $to -Recurse
LogAction "End copy installation files"
}
Function CreateSchedule ($hn, $tn, $tr, $sd, $st, $un, $pw)
{
LogAction "Starting create schedule"
Invoke-Command -ComputerName $hn -ScriptBlock {
param($hn, $tn, $tr, $sd, $st, $un, $pw)
Write-Host $("Host name: [{0}]" -f $hn);
$cmd = $("schtasks.exe /create /S ""{0}"" /tn ""{1}"" /tr ""{2}"" /sc once /sd {3} /st {4} /ru ""{5}"" /rp ""{6}"" /rl highest /V1" -f $hn, $tn, $tr, $sd, $st, $un, $pw);
Invoke-Expression $cmd;
} -ArgumentList #($hn, $tn, $tr, $sd, $st, $un, $pw)
LogAction "End create schedule"
}
...setting variables etc...
ApplySnapshot
CopyFiles
CreateSchedule -hn $hostname -tn $taskname -tr $taskrun -sd $setdate -st $settime -un $username -pw $password
Batch file:
PowerShell -Command "& C:\Auto.ps1" <...params...>
PowerShell -Command "& C:\Auto.ps1" <...params...>
PowerShell -Command "& C:\Auto.ps1" <...params...>
PowerShell -Command "& C:\Auto.ps1" <...params...>
pause
Example output:
C:\Auto>PowerShell -Command "& C:\Auto.ps1" <...params...>
WARNING: The job to Change state of VM TestVM to Stopped is still
running in the background.
You can check its progress with Test-wmiJob or Test-wmiJob -statusOnly using
the following job id:
\\Server\root\virtualization:Msvm_ConcreteJob.InstanceID="A207CEBA-F582-4A42-
BCDE-3312C7FB6DCC"
JobStarted
WARNING: The job to Change state of VM TestVM to Running is still
running in the background.
You can check its progress with Test-wmiJob or Test-wmiJob -statusOnly using
the following job id:
\\Server\root\virtualization:Msvm_ConcreteJob.InstanceID="42C31CEF-00E2-40A7-
AF70-578B0B91B05D"
JobStarted
Enter-PSSession : Connecting to remote server failed with the following error m
essage : The WinRM client cannot complete the operation within the time specifi
ed. Check if the machine name is valid and is reachable over the network and fi
rewall exception for Windows Remote Management service is enabled. For more inf
ormation, see the about_Remote_Troubleshooting Help topic.
At C:\Auto.ps1:192 char:18
+ Enter-PSSession <<<< -ComputerName $hostname -Credential $cred
+ CategoryInfo : InvalidArgument: (TestVM:String) [Enter-PSS
ession], PSRemotingTransportException
+ FullyQualifiedErrorId : CreateRemoteRunspaceFailed
[TestVM] Connecting to remote server failed with the following error messa
ge : The WinRM client cannot complete the operation within the time specified.
Check if the machine name is valid and is reachable over the network and firewa
ll exception for Windows Remote Management service is enabled. For more informa
tion, see the about_Remote_Troubleshooting Help topic.
+ CategoryInfo : OpenError: (:) [], PSRemotingTransportException
+ FullyQualifiedErrorId : PSSessionStateBroken
So, in this example, the snapshot has been successfully applied (despite the warnings). The “Enter-PSSession” error appears after the files have been copied to the virtual machine.
As a test, I tried this on a different server (also running HyperV etc etc), and I found that I still get the initial error (after the file copying stage) but I do not get the error creating the scheduled task.
All my efforts to search for information on the “Connecting to remote server failed with the following error message : The WinRM client cannot complete the operation within the time specified.” Error seem to say “make sure the machine is set up for remote use”; well I know it is because sometimes it works and if I run just an “Enter-PSSession” command by itself, I can connect.
The server(s) and virtual machine(s) are on the same domain.
I know there’s a lot to take in here, but I would really appreciate some help in how to troubleshoot / fix this problem.
Thank you

Answer 1

Maybe the targets are not always up when the connection attempts are being made.