I am writing a Powershell script to copy unencrypted EBS Snapshots in AWS to Encrypted Snapshots. In AWS the max number of concurrent copies is currently 20 at one time, but I have 1400 snapshots to copy. I wrote a script in Powershell using a For Each loop to loop through the snapshot IDs stored in a Text file, and it works as expected until it gets to 20 snapshots being copied. Then it will throw the following error and fail:
An error occurred (ResourceLimitExceeded) when calling the CopySnapshot operation: Too many snapshot copies in progress. The limit is 20 for this destination region.
I have tried to use a While Do statement, but I believe I am missing some items on here. The script is listed below. Essentially I am trying to have it if the script gets to 20 concurrent copies, it will retry on the one snapshot until a free spot opens up and then move on to the next. Ideally I would like to just have this run in the background for a day or so. See the current script below:
function Get-TimeStamp {
return "[{0:MM/dd/yy} {0:HH:mm:ss}]" -f (Get-Date)
}
$kmsID = "blah"
$region = "us-east-1"
$stoploop = $false
[int]$Retrycount = "0"
Foreach($line in get-content C:\snaps4.txt) {
do {
$desc = aws ec2 describe-snapshots --snapshot-ids $line | ConvertFrom-Json
$description = $desc.Snapshots.Description
Write-Output "$description"
$snap = aws ec2 copy-snapshot --description "[Copied $line from us-east-1] $description" --source-region $region --source-snapshot-id $line --encrypted --kms-key-id $kmsID | ConvertFrom-Json
$newsnap = $snap.SnapshotId
Write-Output "$(Get-TimeStamp) Created copy of $line $description with NEW SnapshotID $newsnap" >> C:\log.txt
$stoploop = $true
}
While ($Stoploop -eq $false)
}
Please let me know if you have any questions, and I appreciate any help in advance.
Thanks!
You can put the copy command inside a try/catch block.
Something like this:
try {
copy command;
mark as complete
}
catch{
mark as failed
}
one approach is to make the content file a csv with file name and complete columns. Use import-csv to read it; iterate the imported list when $_.complete -ne "Y" and set complete to "Y" when it succeeds. Export the file at the end.
Re-run as needed
Related
I have a script below, the purpose of the script is as follows.
Pass a file to the script containing a list of SQL instances.
Pass an array of files to the script.
Run each and every script on all the SQL instances defined in the file from step 1
param(
[string[]] $file_list,
[string] $in_sql_server_instances_file_path
)
if (!(test-path $in_sql_server_instances_file_path)) {
write-output "The file does not exist"
return 1
}
$file_server_list = get-content -path $in_sql_server_instances_file_path
workflow execute-parallel {
param(
[string []]
$sql_instances,
[string []]
$files_to_execute
)
foreach -parallel -throttlelimit 8 ($sql_instance in $sql_instances) {
inlineScript {
$wf_sql_instance = $using:sql_instance
$wf_files_to_execute = $using:files_to_execute
foreach ($file_to_execute in $wf_files_to_execute) {
Invoke-Sqlcmd -ServerInstance "$wf_sql_instance" -Database "tempdb" -InputFile $file_to_execute
}
}
}
}
execute-parallel -sql_instances $file_server_list -files_to_execute $file_list
To execute the script.
.\execute.ps1 -file_list 'C:\Scripts\a.sql','C:\Scripts\b.sql' -in_sql_server_instances_file_path 'C:\Scripts\sql_file.txt'
sql_file.txt contains a list of SQL instances.
I am trying to achieve the following.
If a file fails, stop execution on the server but move onto the next server in the loop.
Handle error control
Deploy the files in the order specified in the -file_list array parameter.
How can I be certain that a.sql will always run before b.sql?
PowerShell standard hash tables are weird when it comes to order. But not arrays.
I created the following function which I wanted to use for a very simple CTI solution I have to use at work. This CTI process is writing all received calls to a text logile.
This function starts a new powershell Job and checks if the .log has been saved during the last 2 seconds and gets the last 4 lines of the log (receiving calls always creates 4 new lines).
During the job update I'm using regex to find the line with the phonenumber and time and append this to a richtextbox in a form.
In theory this works exactly as I want it to work. If I manually add new lines and save the file, it's always showing the timecode and phone number.
In the field however, this doesn't work as the CTI process is opening the file and doesn't save the it unless the process is shutting down.
I know that I can use get-content -wait to display new lines. I already tested this in the console and it's displaying new lines as soon as the .log is updated from the CTI process. What I don't know is how to rewrite the function to work with that, displaying only new lines and not all the old stuff when first running the script. I need to keep it in the job for a responsive form. Another thing is, that the computer running the form, doesn't have that much power. I don't know if get-content -wait could cause high memory usage after several hours. Maybe there are also some alternative solutions for a case like that available?
function start-CTIlogMonitoring
{
Param ($CTIlogPath)
Write-Debug "startCTI monitor"
Add-JobTracker -Name "CTILogger" -ArgumentList $CTIlogPath `
-JobScript {
#--------------------------------------------------
#TODO: Set a script block
#Important: Do not access form controls from this script block.
Param ($CTIlogPath) #Pass any arguments using the ArgumentList parameter
while ($true)
{
$diff = ((Get-ChildItem $CTIlogPath).LastWriteTime - (get-date)).totalseconds
Write-Debug "diff $diff"
if ($diff -gt -2)
{
Write-Debug "cti log DIFF detected"
Get-Content -Path "$CTIlogPath" -Tail 4
Start-Sleep -Seconds 1
}
}
#--------------------------------------------------
}`
-CompletedScript { Param ($Job) }`
-UpdateScript {
Param ($Job)
$results = Receive-Job -Job $Job | Out-String # Out-String required to get new lines in RTB
#get the stuff from results and make it more appearing to read for humans
if ($results -match '(Ein, E, (\d+))')
{
Write-debug "Incoming Call:"
$time = ([regex]'[0-9]{2}:[0-9]{2}:[0-9]{2}').Match($results)
$phoneNumber = ([regex]'Ein, E, (\d+)').Split($results)[1]
Write-Debug "$time ----> $phoneNumber"
if ($richtextboxCTIlogs.lines.count -eq 0)
{
$richtextboxCTIlogs.AppendText("$time ----> $phoneNumber")
}
else
{
$richtextboxCTIlogs.AppendText("`n$time ----> $phoneNumber")
}
$richtextboxCTIlogs.SelectionStart = $richtextboxCTIlogs.TextLength;
$richtextboxCTIlogs.ScrollToCaret()
}
<#else
{
Write-Debug "found nothin"
}#>
}
}
I have a text file to process. Text file has some configuration data and some networking commands. I want to run all those network commands and redirect output in some log file.
At starting of text file,there are some configuration information like File-name and file location. This can be used for naming log file and location of log file. These line starts with some special characters like '<#:'. just to know that rest of the line is config data about file not the command to execute.
Now, before i want start executing networking commands (starts with some special characters like '<:'), first i want to read all configuration information about file i.e. file name, location, overwrite flag etc. Then i can run all commands and dump output into log file.
I used get-content iterator to loop over entire text file.
Question: Is there any way to start looping over file from a specific line again?
So that i can process config information first (loop till i first encounter command to execute, remember this line number), create log file and then keep running commands and redirect output to log file (loop from last remembered line number).
Config File looks like:
<#Result_File_Name:dump1.txt
<#Result_File_Location:C:\powershell
<:ping www.google.com
<:ipconfig
<:traceroute www.google.com
<:netsh interface ip show config
My powerhsell script looks like:
$content = Get-Content C:\powershell\config.txt
foreach ($line in $content)
{
if($line.StartsWith("<#Result_File_Name:")) #every time i am doing this, even for command line
{
$result_file_arr = $line.split(":")
$result_file_name = $result_file_arr[1]
Write-Host $result_file_name
}
#if($line.StartsWith("<#Result_File_Location:"))#every time i am doing this, even for command line
#{
# $result_file_arr = $line.split(":")
# $result_file_name = $result_file_arr[1]
#}
if( $conf_read_over =1)
{
break;
}
if ($line.StartsWith("<:")) #In this if block, i need to run all commands
{
$items = $line.split("<:")
#$items[0]
#invoke-expression $items[2] > $result_file_name
invoke-expression $items[2] > $result_file_name
}
}
If all the config information starts with <# just process those out first separately. Once that is done you can assume the rest are commands?
# Collect config lines and process
$config = $content | Where-Object{$_.StartsWith('<#')} | ForEach-Object{
$_.Trim("<#") -replace "\\","\\" -replace "^(.*?):(.*)" , '$1 = $2'
} | ConvertFrom-StringData
# Process all the lines that are command lines.
$content | Where-Object{!$_.StartsWith('<#') -and ![string]::IsNullOrEmpty($_)} | ForEach-Object{
Invoke-Expression $_.trimstart("<:")
}
I went a little over board with the config section. What I did was convert it into a hashtable. Now you will have your config options, as they were in file, accessible as an object.
$config
Name Value
---- -----
Result_File_Name dump1.txt
Result_File_Location C:\powershell
Small reconfiguration of your code, with some parts missing, would look like the following. You will most likely need to tweak this to your own needs.
# Collect config lines and process
$config = ($content | Where-Object{$_.StartsWith('<#')} | ForEach-Object{
$_.Trim("<#") -replace "\\","\\" -replace "^(.*?):(.*)" , '$1 = $2'
} | Out-String) | ConvertFrom-StringData
# Process all the lines that are command lines.
$content | Where-Object{!$_.StartsWith('<#') -and ![string]::IsNullOrEmpty($_)} | ForEach-Object{
Invoke-Expression $_.trimstart("<:") | Add-Content -Path $config.Result_File_Name
}
As per your comment you are still curious about your restart loop logic which was part of your original question. I will add this as a separate answer to that. I would still prefer my other approach.
# Use a flag to determine if we have already restarted. Assume False
$restarted = $false
$restartIndexPoint = 4
$restartIndex = 2
for($contentIndex = 0; $contentIndex -lt $content.Length; $contentIndex++){
Write-Host ("Line#{0} : {1}" -f $contentIndex, $content[$contentIndex])
# Check to see if we are on the $restartIndexPoint for the first time
if(!$restarted -and $contentIndex -eq $restartIndexPoint){
# Set the flag so this does not get repeated.
$restarted = $true
# Reset the index to repeat some steps over again.
$contentIndex = $restartIndex
}
}
Remember that array indexing is 0 based when you are setting your numbers. Line 20 is element 19 in the string array for example.
Inside the loop we run a check. If it passes we change the current index to something earlier. The write-host will just print the lines so you can see the "restart" portion. We need a flag to be set so that we are not running a infinite loop.
I wasn't sure how to describe this problem in the title so here goes.
I call a function from a script in another script. In that function i have a while loop that basically keeps looping through a set of ip's and looks up their hostname. when the while loop times out or we have all the host names.
it returns the hostnames.
My problem is that the return value contains every single Write-Host i'm doing in that function.
i know it's because Write-Host puts stuff on the pipeline and the return just returns whatever it has.
How do i go about fixing this?
The entire script i run get's logged in a log file which is why i want to have some verbose logging.
| out-null on write-host fixes the issue but it doesn't print the write-host values in the script.
in main.psm1 i have a function like so:
$nodes = #("ip1", "ip2", "ip3", "ip4")
$nodesnames = DoStuff -nodes $nodes
then in functions.psm1 i have functions like:
Function DoStuff
{
param($nodes)
$timeout = 300
$timetaken = 0
$sleepseconds = 5
$nodenames = #("$env:COMPUTERNAME")
while(($nodenames.count -lt $nodes.count) -and ($timetaken -lt $timeout))
{
try
{
Write-Host "Stuff"
foreach($node in $nodes)
{
$nodename = SuperawesomeFunction $node
Write-Host "$nodename"
if($nodenames -notcontains $nodename)
{
$nodenames += #($nodename)
}
}
}
catch
{
Write-Host "DoStuff Failed because $_"
}
Start-Sleep $sleepseconds
$timetaken += $sleepseconds
}
return $nodenames
}
Function SuperawesomeFunction
{
param($node)
$nodename = [System.Net.Dns]::GetHostEntry("$node")
return $nodename
}
Thanks.
So the answer is, your function is working like it is by design. In PowerShell a function will return output in general to the pipeline, unless specifically directed otherwise.
You used Echo before, which is an alias of Write-Output, and output is passed down the pipe as I mentioned before. As such it would be collected along with the returned $nodenames array.
Replacing Echo with Write-Host changes everything because Write-Host specifically tells PowerShell to send the information to the host application (usually the PowerShell Console or PowerShell ISE).
How do you avoid this? You could add a parameter specifying a path for a logfile, and have your function update the logfile directly, and only output the relevant data.
Or you can make an object with a pair of properties that gets passed back down the pipe which has the DNS results in one property, and the errors in another.
You could use Write-Error in the function, and set it up as an advanced function to support -errorvariable and capture the errors in a separate variable. To be honest, I'm not sure how to do that, I've never done it, but I'm 90% sure that it can be done.
I'm new at writing in powershell but this is what I'm trying to accomplish.
I want to compare the dates of the two excel files to determine if one is newer than the other.
I want to convert a file from csv to xls on a computer that doesn't have excel. Only if the statement above is true, the initial xls file was copied already.
I want to copy the newly converted xls file to another location
If the file is already open it will fail to copy so I want to send out an email alert on success or failure of this operation.
Here is the script that I'm having issues with. The error is "Expressions are only allowed as the first element of a pipeline." I know it's to do with the email operation but I'm at a loss as to how to write this out manually with all those variables included. There are probably more errors but I'm not seeing them now. Thanks for any help, I appreciate it!
$CSV = "C:filename.csv"
$LocalXLS = "C:\filename.xls"
$RemoteXLS = "D:\filename.xls"
$LocalDate = (Get-Item $LocalXLS).LASTWRITETIME
$RemoteDate = (Get-Item $RemoteXLS).LASTWRITETIME
$convert = "D:\CSV Converter\csvcnv.exe"
if ($LocalDate -eq $RemoteDate) {break}
else {
& $convert $CSV $LocalXLS
$FromAddress = "email#address.com"
$ToAddress = "email#address.com"
$MessageSubject = "vague subject"
$SendingServer = "mail.mail.com"
$SMTPMessage = New-Object System.Net.Mail.MailMessage $FromAddress, $ToAddress, $MessageSubject, $MessageBody
$SMTPClient = New-Object System.Net.Mail.SMTPClient $SendingServer
$SendEmailSuccess = $MessageBody = "The copy completed successfully!" | New-Object System.Net.Mail.SMTPClient mail.mail.com $SMTPMessage
$RenamedXLS = {$_.BaseName+(Get-Date -f yyyy-MM-dd)+$_.Extension}
Rename-Item -path $RemoteXLS -newname $RenamedXLS -force -erroraction silentlycontinue
If (!$error)
{ $SendEmailSuccess | copy-item $LocalXLS -destination $RemoteXLS -force }
Else
{$MessageBody = "The copy failed, please make sure the file is closed." | $SMTPClient.Send($SMTPMessage)}
}
You get this error when you are trying to execute an independent block of code from within a pipeline chain.
Just as a different example, imagine this code using jQuery:
$("div").not(".main").console.log(this)
Each dot (.) will chain the array into the next function. In the above function this breaks with console because it's not meant to have any values piped in. If we want to break from our chaining to execute some code (perhaps on objects in the chain - we can do so with each like this:
$("div").not(".main").each(function() {console.log(this)})
The solution is powershell is identical. If you want to run a script against each item in your chain individually, you can use ForEach-Object or it's alias (%).
Imagine you have the following function in Powershell:
$settings | ?{$_.Key -eq 'Environment' } | $_.Value = "Prod"
The last line cannot be executed because it is a script, but we can fix that with ForEach like this:
$settings | ?{$_.Key -eq 'Environment' } | %{ $_.Value = "Prod" }
This error basically happens when you use an expression on the receiving side of the pipeline when it cannot receive the objects from the pipeline.
You would get the error if you do something like this:
$a="test" | $a
or even this:
"test" | $a
I don't know why are trying to pipe everywhere. I would recommend you to learn basics about Powershell pipelining. You are approaching it wrong. Also, I think you can refer to the link below to see how to send mail, should be straight forward without the complications that you have added with the pipes : http://www.searchmarked.com/windows/how-to-send-an-email-using-a-windows-powershell-script.php