Commands behaving differently in script vs shell - powershell

I've got two powershell functions in a file. The first one (Get-ImapAttachments) connects using the NetCmdlets IMAP cmdlets, downloads all *.csv attachments from any messages in the INBOX, and then moves the messages to another folder to keep the INBOX clean. I got this function from the NetCmdlets site, and tweaked it slightly for my needs. On it's own, it works the way I need it to.
I've written a second function (Import-D2LData) to process the CSV file. It's a badly formatted CSV file, complete with a report title on the first line, and the column headings are repeated every 23 lines or so. Since Import-Csv isn't quite as flexible as I need, I use a combination of get-content and select-object -skip 1 to start off the import, and basically build a long string to pass to ConvertFrom-Csv. If I run that function manually, it currently works the way I need it to, also.
The problem arises when I run it in the .ps1 file I've got them in. The first function fires off just fine, and the second one seems to. The ConvertFrom-Csv fails and I end up with a bunch of empty lines of output. The real strangeness is, I can use the . command to source the .ps1 file, and run the last 7 lines of the script (starting with the call to Get-ImapAttachments), and I get the expected results.
I'm sure I'm doing something pretty braindead, but after staring at the code for a couple of hours, I'm just not seeing it at this point...
# Needed for the IMAP Stuff
Add-PSSnapin -Name NetCmdlets -ErrorAction SilentlyContinue
function Get-ImapAttachments {
param(
[string] $server,
[string] $user,
[string] $password,
[string] $downloadDir="c:\temp\attachments",
[string] $folder = "INBOX"
)
$destFolder = "$folder.Processed"
$imapConnection = Connect-Imap -server $server -user $user -password $password
$m_messages = Get-Imap -connection $imapConnection -folder $folder
foreach($msg in $m_messages)
{
if($msg.ContentType.StartsWith("multipart")) #A MIME message with multiple parts
{
for($i=0;$i -lt $msg.PartCount;$i++)
{
if($msg.PartFileName[$i] -like "*.csv" ) #There is a CSV attachment
{
$localFile = $downloadDir + "\" + $msg.PartFileName[$i]
if( Test-Path -Path $localFile ) {
Remove-Item $localFile -Force
}
Get-Imap -connection $imapConnection -folder $folder -view $msg.Id -localfile $localFile -part $msg.PartId[$i]
Move-Imap -Connection $imapConnection -folder $folder -Message $msg.Id -Destination $destFolder
}
}
}
}
Disconnect-IMAP -Connection $imapConnection
}
function Import-D2LData {
param(
[string] $CsvFile
)
Write-Host -ForegroundColor Cyan $CsvFile
$file = get-content $CsvFile | Select-Object -Skip 1
$head = $file[0].Split(',')
$data = "$head`n"
foreach ( $line in $file ) {
if( $line -ne $file[0] ) {
$data += "$line`n"
}
}
$csv = $data | ConvertFrom-Csv -Header $head
$csv
}
Get-ImapAttachments -server imap.contoso.com -user sa_mailbox -password 'password'
$csvFiles = Get-ChildItem -Path "c:\temp\attachments" -Filter "*.csv"
foreach( $file in $csvFiles ) {
Import-D2LData -CsvFile $file.FullName
}

Well, it wasn't pretty, but I ended up breaking the functions into two different scripts. When that wasn't enough, I ran the first script in it's own PowerShell instance, which seemed to isolate the runspaces enough to allow both functions to operate properly:
$BasePath = 'Z:\scripts\Import-Data'
powershell.exe -NoProfile -command "& '$BasePath\fetch-attachments.ps1'"
& "$BasePath\process-attachments.ps1"
Not exactly an elegant solution, but it got me past the issues I was having...

Related

How to parse through folders and files using PowerShell?

I am trying to construct a script that moves through specific folders and the log files in it, and filters the error codes. After that it passes them into a new file.
I'm not really sure how to do that with for loops so I'll leave my code bellow.
If someone could tell me what I'm doing wrong, that would be greatly appreciated.
$file_name = Read-Host -Prompt 'Name of the new file: '
$path = 'C:\Users\user\Power\log_script\logs'
Add-Type -AssemblyName System.IO.Compression.FileSystem
function Unzip
{
param([string]$zipfile, [string]$outpath)
[System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath)
}
if ([System.IO.File]::Exists($path)) {
Remove-Item $path
Unzip 'C:\Users\user\Power\log_script\logs.zip' 'C:\Users\user\Power\log_script'
} else {
Unzip 'C:\Users\user\Power\log_script\logs.zip' 'C:\Users\user\Power\log_script'
}
$folder = Get-ChildItem -Path 'C:\Users\user\Power\log_script\logs\LogFiles'
$files = foreach($logfolder in $folder) {
$content = foreach($line in $files) {
if ($line -match '([ ][4-5][0-5][0-9][ ])') {
echo $line
}
}
}
$content | Out-File $file_name -Force -Encoding ascii
Inside the LogFiles folder are three more folders each containing log files.
Thanks
Expanding on a comment above about recursing the folder structure, and then actually retrieving the content of the files, you could try something line this:
$allFiles = Get-ChildItem -Path 'C:\Users\user\Power\log_script\logs\LogFiles' -Recurse
# iterate the files
$allFiles | ForEach-Object {
# iterate the content of each file, line by line
Get-Content $_ | ForEach-Object {
if ($_ -match '([ ][4-5][0-5][0-9][ ])') {
echo $_
}
}
}
It looks like your inner loop is of a collection ($files) that doesn't yet exist. You assign $files to the output of a ForEach(...) loop then try to nest another loop of $files inside it. Of course at this point $files isn't available to be looped.
Regardless, the issue is you are never reading the content of your log files. Even if you managed to loop through the output of Get-ChildItem, you need to look at each line to perform the match.
Obviously I cannot completely test this, but I see a few issues and have rewritten as below:
$file_name = Read-Host -Prompt 'Name of the new file'
$path = 'C:\Users\user\Power\log_script\logs'
$Pattern = '([ ][4-5][0-5][0-9][ ])'
if ( [System.IO.File]::Exists( $path ) ) { Remove-Item $path }
Expand-Archive 'C:\Users\user\Power\log_script\logs.zip' 'C:\Users\user\Power\log_script'
Select-String -Path 'C:\Users\user\Power\log_script\logs\LogFiles\*' -Pattern $Pattern |
Select-Object -ExpandProperty line |
Out-File $file_name -Force -Encoding ascii
Note: Select-String cannot recurse on its own.
I'm not sure you need to write your own UnZip function. PowerShell has the Expand-Archive cmdlet which can at least match the functionality thus far:
Expand-Archive -Path <SourceZipPath> -DestinationPath <DestinationFolder>
Note: The -Force parameter allows it to over write the destination files if they are already present. which may be a substitute for testing if the file exists and deleting if it does.
If you are going to test for the file that section of code can be simplified as:
if ( [System.IO.File]::Exists( $path ) ) { Remove-Item $path }
Unzip 'C:\Users\user\Power\log_script\logs.zip' 'C:\Users\user\Power\log_script'
This is because you were going to run the UnZip command regardless...
Note: You could also use Test-Path for this.
Also there are enumerable ways to get the matching lines, here are a couple of extra samples:
Get-ChildItem -Path 'C:\Users\user\Power\log_script\logs\LogFiles' |
ForEach-Object{
( Get-Content $_.FullName ) -match $Pattern
# Using match in this way will echo the lines that matched from each run of
# Get-Content. If nothing matched nothing will output on that iteration.
} |
Out-File $file_name -Force -Encoding ascii
This approach will read the entire file into an array before running the match on it. For large files it may pose a memory issue, however it enabled the clever use of -match.
OR:
Get-ChildItem -Path 'C:\Users\user\Power\log_script\logs\LogFiles' |
Get-Content |
ForEach-Object{ If( $_ -match $Pattern ) { $_ } } |
Out-File $file_name -Force -Encoding ascii
Note: You don't need the alias echo or its real cmdlet Write-Output
UPDATE: After fuzzing around a bit and trying different things I finally got it to work.
I'll include the code below just for demonstration purposes.
Thanks everyone
$start = Get-Date
"`n$start`n"
$file_name = Read-Host -Prompt 'Name of the new file: '
Out-File $file_name -Force -Encoding ascii
Expand-Archive -Path 'C:\Users\User\Power\log_script\logs.zip' -Force
$i = 1
$folders = Get-ChildItem -Path 'C:\Users\User\Power\log_script\logs\logs\LogFiles' -Name -Recurse -Include *.log
foreach($item in $folders) {
$files = 'C:\Users\User\Power\log_script\logs\logs\LogFiles\' + $item
foreach($file in $files){
$content = Get-Content $file
Write-Progress -Activity "Filtering..." -Status "File $i of $($folders.Count)" -PercentComplete (($i / $folders.Count) * 100)
$i++
$output = foreach($line in $content) {
if ($line -match '([ ][4-5][0-5][0-9][ ])') {
Add-Content -Path $file_name -Value $line
}
}
}
}
$end = Get-Date
$time = [int]($end - $start).TotalSeconds
Write-Output ("Runtime: " + $time + " Seconds" -join ' ')

Get-childitem is not working with foreach loop

Trying to delete files from multiple path,
So have created a csv file like path,days,filter
importing this file in shell and looping over each object to delete contents, but getchilditem is failing again and again,
Unable to understand reason behind that,
Below is the code what m trying to achieve
Start-Transcript -Path "D:\delete.log"
$pathlist= Import-csv -LiteralPath "D:\diskpath.csv"
$count = 0
foreach($p in $pathlist){
Write-host $p.path " | " $p.days -ForegroundColor DarkCyan
$path = $p.path
$days = $p.days
$filter = $p.filter
Get-ChildItem -Path $path -Filter $filter | where-object{$_.LastWriteTime -lt [datetime]::Now.AddDays(-$days)}|Remove-Item -Force -Verbose -Recurse -ErrorAction SilentlyContinue -Confirm $false
}
Stop-Transcript
without for loop, script executes properly, but with for loop it fails
Please let know if any further information needed on this query,
will like provide the same,
have already google and read multiple questions here at SO, but unable to find reason behind failure,
#T-Me and #Theo
Thanks for highlighting error, haven't looked type error in script my bad, whereas while manual typing in PowerShell was writing correctly, but in script made mistake, now its working-
Start-Transcript -Path "D:\delete.log"
$pathlist= Import-csv -LiteralPath "D:\diskpath.csv"
$count = 0
foreach($p in $pathlist){
Write-host $p.path " | " $p.days -ForegroundColor DarkCyan
$path = $p.path
$days = $p.days
$filter = $p.filter
Get-ChildItem -Path $path -Filter $filter | where-object{$_.LastWriteTime -lt [datetime]::Now.AddDays(-$days)}|Remove-Item -Force -Verbose -Recurse
}

Issues with Copy-item script

I have a script that needs to copy a list of files to ceraitn directories and locations on target servers.
I was able to understand that I need to creat an csv file as follows:
I need to understand from you how to make the files from source location to their adjacent file in target location. Any ideas?
My code looks like this:
# customize log file
$date = (Get-Date -Format d).ToString() | foreach {$_ -replace "/", "_"}
$time = (Get-Date)
$scriptDir = "D:\Scripts\ServerBuildToolkitT1\SingleFileUpfate\"
$logDir = "D:\Scripts\ServerBuildToolkitT1\Logs\SingleFileUpfate\"
$logFileName = "SingleFileUpfate $date.log"
$sources = #()
$destinsyions = #()
function CSV {
Import-Csv D:\Scripts_PS\SD.csv | ForEach-Object {
$sources += $_."Source Location"
$destinations += $_."Destination Location"
}
}
# this file contains the list of destination server that you want copy
# file/folder to
$computers = Get-Content "D:\Scripts_PS\ServerList.txt"
function main
{
foreach ($computer in $computers) {
foreach ($destination in $destinations) {
Write-Output "$([DateTime]::Now) Copying files update to $computer now" |
Out-File -FilePath "$logDir\$logFileName" -Append
Copy-Item $sources -Destination "\\$computer\$destination" -Force -Recurse -Verbose:$false
}
}
}
csv
main
Write-Output "$([DateTime]::Now) the operation SingleFileUpdate completed successefully" |
Out-File -FilePath "$logDir\$logFileName" -Append
i have updtaed the Script (as seen above) and now i am getting the following ERROR
"WARNING: One or more headers were not specified. Default names starting with "H" have been used in place of any missing headers."

PowerShell run script simultaneously

I created a PowerShell script to remove all files and folders older than X days. This works perfectly fine and the logging is also ok. Because PowerShell is a bit slow, it can take some time to delete these files and folders when big quantities are to be treated.
My questions: How can I have this script ran on multiple directories ($Target) at the same time?
Ideally, we would like to have this in a scheduled task on Win 2008 R2 server and have an input file (txt, csv) to paste some new target locations in.
Thank you for your help/advise.
The script
#================= VARIABLES ==================================================
$Target = \\share\dir1"
$OlderThanDays = "10"
$Logfile = "$Target\Auto_Clean.log"
#================= BODY =======================================================
# Set start time
$StartTime = (Get-Date).ToShortDateString()+", "+(Get-Date).ToLongTimeString()
Write-Output "`nDeleting folders that are older than $OlderThanDays days:`n" | Tee-Object $LogFile -Append
Get-ChildItem -Directory -Path $Target |
Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-$OlderThanDays) } | ForEach {
$Folder = $_.FullName
Remove-Item $Folder -Recurse -Force -ErrorAction SilentlyContinue
$Timestamp = (Get-Date).ToShortDateString()+" | "+(Get-Date).ToLongTimeString()
# If folder can't be removed
if (Test-Path $Folder)
{ "$Timestamp | FAILLED: $Folder (IN USE)" }
else
{ "$Timestamp | REMOVED: $Folder" }
} | Tee-Object $LogFile -Append # Output folder names to console & logfile at the same time
# Set end time & calculate runtime
$EndTime = (Get-Date).ToShortDateString()+", "+(Get-Date).ToLongTimeString()
$TimeTaken = New-TimeSpan -Start $StartTime -End $EndTime
# Write footer to log
Write-Output ($Footer = #"
Start Time : $StartTime
End Time : $EndTime
Total runtime : $TimeTaken
$("-"*79)
"#)
# Create logfile
Out-File -FilePath $LogFile -Append -InputObject $Footer
# Clean up variables at end of script
$Target=$StartTime=$EndTime=$OlderThanDays = $null
One way to achieve this would be to write an "outer" script that passes the directory-to-be-cleaned, into the "inner" script, as a parameter.
For your "outer" script, have something like this:
$DirectoryList = Get-Content -Path $PSScriptRoot\DirList;
foreach ($Directory in $DirectoryList) {
Start-Process -FilePath powershell.exe -ArgumentList ('"{0}\InnerScript.ps1" -Path "{1}"' -f $PSScriptRoot, $Directory);
}
Note: Using Start-Process kicks off a new process that is, by default, asynchronous. If you use the -Wait parameter, then the process will run synchronously. Since you want things to run more quickly and asynchronously, omitting the -Wait parameter should achieve the desired results.
Invoke-Command
Alternatively, you could use Invoke-Command to kick off a PowerShell script, using the parameters: -File, -ArgumentList, -ThrottleLimit, and -AsJob. The Invoke-Command command relies on PowerShell Remoting, so that must enabled, at least on the local machine.
Add a parameter block to the top of your "inner" script (the one you posted above), like so:
param (
[Parameter(Mandatory = $true)]
[string] $Path
)
That way, your "outer" script can pass in the directory path, using the -Path parameter for the "inner" script.

Powershell Foreach-Object Foreach loop read string from several files and write to one

I have trouble to use several commands in one Foreach or Foreach-Object loop
My situation is -
I have many text files, about 100.
So they are read Get-ChildItem $FilePath -Include *.txt
Every file's structure is same only key information is different.
Example
User: Somerandomname
Computer: Somerandomcomputer
With -Replace command I remove "User:" and "Computer:" so $User = Somerandomname and $computer = "Somerandomcomputer.
In each circle $user and $Computer with -Append should be written to one file. And then next file should be read.
foreach-object { $file = $_.fullname;
should be used, but I can not figure out the right syntax for it. Could someone help me with it?
Assuming you've defined $FilePath, $user, and/or $computer elsewhere, try something like this.
$files = Get-ChildItem $FilePath\*.txt
foreach ($file in $files)
{
(Get-Content $file) |
Foreach-Object { $content = $_ -replace "User:", "User: $user" ; $content -replace "Computer:", "Computer: $computer" } |
Set-Content $file
}
You can use ; to delimit additional commands in within the Foreach-Object, for example if you wanted to have separate commands for your user and computer name. If you don't enclose the Get-Content cmdlet with parenthesis you will get an error because that process will still have $file open when Set-Content tries to use it.
Also note that with Powershell, strings in double quotes will evaluate variables, so you can put $user in the string to do something like "User: $user" if you so desired.
Try this:
gci $FilePath -Include *.txt | % {
gc $_.FullName | ? { $_ -match '^(?:User|Computer): (.*)' } | % { $matches[1] }
} | Out-File 'C:\path\to\output.txt'
If User and Computer are on separate lines, you need to read the lines two at a time. The ReadCount parameter of Get-Content allows you to do that.
Get-ChildItem $FilePath -Include *.txt `
| Get-Content -ReadCount 2 `
| %{ $user = $_[0] -replace '^User: ', ''; $computer = $_[1] -replace '^Computer: ', ''; "$user $computer" } `
| Out-File outputfile.txt
This makes the assumption that every file contains only lines of the exact form
User: someuser
Computer: somecomputer
User: someotheruser
Computer: someothercomputer
...
If this is not the case, you will need to provide whatever is the exact file format.