This script that identifies duplicate files triggers a virus scanner. How can we slow it down?
Get-ChildItem -Recurse -File `
| Group-Object -Property Length `
| ?{ $_.Count -gt 1 } `
| %{ $_.Group } `
| Get-FileHash `
| Group-Object -Property Hash `
| ?{ $_.Count -gt 1 } `
| %{ $_.Group }
| %{ $_.path -replace "$([regex]::escape($(pwd)))",'' }
Is there a way to put like a 2 second pause in between files so it takes a long time to complete?
TIA
Edits for comments:
Don't want to say the antivirus software but it's very advanced.
I got the backticks from stack overflow, so garbage in garbage out :) [seriously thanks for the tip]
It works flawlessly on network shares with about 100 files on it.
The speed of your script isn't the problem with an A/V scanner. My guess is possibly the use of [regex]::replace(pattern, text) and Get-FileHash could be something your A/V flags on during heuristic analysis. Without knowing the A/V software, it's impossible to know if others have experienced and resolved the same problem you are having.
The real correct answer is to open a ticket with your A/V vendor on it flagging false positives. Signing your scripts is also known to help scripts with A/V some. Some A/Vs allow whitelisting by checksum, which you could use to approve your scripts if your vendor doesn't have any alternatives. Using the checksum of a signed script is even safer, as you can guarantee the code came from your organization at the time the checksum is calculated.
You can also configure A/V software to whitelist a directory, and you can effectively work around the issue by running scripts out of that directory while you sort the issue with your vendor. However, whitelisting by path should not be your permanent solution. Figure out why your scripts are getting flagged with the vendor, then follow their recommendations.
That said, to answer your original question "Is there a way to put like a 2 second pause in between files....?", yes. Start-Sleep will achieve what you want (but I have serious doubts it would affect your A/V results). The last block can be one line but is made multiline for readability (the semicolon ; is required if on one line):
Note: I've also replaced the backticks with better multi-line formatting. You can end a line with | operator and continue the code on the next line in a single expression. This also works for other operators as well.
This change has also fixed an issue in your original sample where you forgot the penultimate backtick. Backticks are easy to forget, and can be hard to look for. This is one reason why their use is not recommended for multi-line expressions.
Get-ChildItem -Recurse -File |
Group-Object -Property Length |
?{ $_.Count -gt 1 } |
%{ $_.Group } |
Get-FileHash |
Group-Object -Property Hash |
?{ $_.Count -gt 1 } |
%{ $_.Group } |
%{
$_.path -replace "$([regex]::escape($(pwd)))",'';
Start-Sleep 2
}
Related
I am novice in powershell and using it very rarely for some little things.
I am using this one liner in order to extract emails
recursive
(Get-ChildItem -Include *.txt -Recurse | Get-Content | Select-String -Pattern "(?:[a-zA-Z0-9_\-\.]+)#(?:[a-zA-Z0-9_\-\.]+)\.(?:[a-zA-Z]{2,5})").Matches | Select-Object -ExpandProperty Value -Unique
In order to access Matches property I've added parentheses. Later I come to that way:
Get-ChildItem -Include *.txt -Recurse | Get-Content | Select-String -Pattern "(?:[a-zA-Z0-9_\-\.]+)#(?:[a-zA-Z0-9_\-\.]+)\.(?:[a-zA-Z]{2,5})" | Select-Object -ExpandProperty Matches -Unique | Select-Object -ExpandProperty Value
I want to to ask what parentheses do exactly in the first version.
Say you have an $output via some function (gci in your case) and you are interested in the field $output.Matches.
If you run $output | select Matches (example 1), you run a
Foreach-Object statement against every object in your array. This
pipeline will use some RAM (very limited, indeed) that are used in a
serial calculation, so every object of $output is processed one after
the other.
If you run $output.Matches (example 2), you select a field from an
array. This will use a lot of RAM at once, but the field will be
processed as one big object instead of many little objects.
As it comes to performance. As always, note that PowerShell is not the way to go if you need high performance. It was never designed to be a fast programming language.
When you're using small objects (like gci $env:userprofile\Desktop), the performance hit will be small. When using large objects or using a lot of nested pipes, the performance hit will be large.
I've just tested it with a gci Z:\ -recurse when Z:\ is a network drive. Performance is dropped with a factor of 20 in this specific case. (Use Measure-Command to test this.)
I am trying to determine if some environment variables are set (for postgres environment). They usually start with PG. (E.g. PGUSER, PGPASSWORD, etc). The following command does output it. (Provided I set it previously).
gci env:* | sort name | more
To eliminate the scrolling I tried the following:
gci env:* | sort name | select-string "PG"
This doesn't return anything. What am I doing wrong here?
Edit: the alternative I have for now:
gci env:* | sort name | % { $var = $_.Name + ":" + $_.Value; Write-Output $var } | select-string "PG"
There must be a better alternative.
You're using the wrong mindset. Don't try to work with PowerShell like everything is a string. That's Unix-like thinking, and it's going to work as well as driving nails with a screwdiver. You need to switch to object-oriented thinking because in PowerShell you're working with objects 99% of the time.
Generally, you would just do this for something as simple as what you're looking for:
Get-ChildItem Env:PG* | Sort-Object -Property Name
If the globbing that Get-ChildItem supports doesn't work, you would want to use Where-Object with the -like operator which is similar globbing to what Get-ChildItem can do:
Get-ChildItem Env:* | Where-Object Name -like 'PG*' | Sort-Object -Property Name
If you need to search values, you can do it like this:
Get-ChildItem Env:* | Where-Object Value -like 'PG*' | Sort-Object -Property Name
And if you want to do both, you'd use the full synax of Where-Object:
Get-ChildItem Env:* | Where-Object { $_.Name -like 'PG*' -or $_.Value -like 'PG*' } | Sort-Object -Property Name
Or you can use the -match operator, which lets you specify a .Net regular expression:
Get-ChildItem Env:* | Where-Object Name -match '^PG' | Sort-Object -Property Name
Or if you know exactly what you're looking for:
$Vars = 'PGUSER', 'PGPASSWORD'
Get-ChildItem Env:* | Where-Object Name -in $Vars | Sort-Object -Property Name
Remembering, of course, that PowerShell is usually case-insensitive. You can specify -clike, -cmatch, -cin, etc. if you want case-sensitive operators.
Alternately, you can use the $env: automatic variable namespace.
if ($null -eq $env:PGUSER) { 'Not set' }
See also Get-Help about_Environment_Variables.
Beware that setting environment variables permanently is not exactly self-evident. It's described briefly in the above link, but the bottom line is that you have to call [System.Environment]::SetEnvironmentVariable(), which you can find documented here. In Windows land, environment variables are basically legacy features with the exception of Windows OS level variables (like PATH) so they're no longer supported like you might expect.
Your approach to how this command should work and your instinct that there has to be a better alternative is exactly correct. This is quite a frustrating issue in my mind and I also asked a variation on this question a few days back.
Select-String only handles strings and what you are passing to it in the above is not a string, so it returns nothing. Obviously, you might think that since Select-String requires a string, that it would implicitly change it into a string, but no. So the next thing to consider is to change it to a string, but that creates even more confusion.
gci env:* | sort name | out-string | select-string "Pro"
So now you just get everything returned. What's happening here is that out-string returns all lines as a single string, so if there is any hit for "Pro" you get everything returned.
What you need to do is to use out-string -stream which splits the string up by linebreaks so that you get a string per line, and only then do you get rational output.
gci env:* | sort name | out-string -stream | select-string "Pro"
More on this here: Using PowerShell sls (Select-String) vs grep vs findstr. The github request linked to in there is trying to change the functionality so that select-string will implicitly have an out-string -stream in the background so that your original command will work.
Often we need strings to output results and there is nothing wrong with wanting to manipulate strings (in fact, it depends what you need of course - if you need to do further object manipulations, keep it as an object for that, but if you just need the string output, you should not have to jump through hoops to get that!). If you use a string-manipulation tool like select-string then PowerShell should at least convert the incoming information to a string to provide meaningful output. Compare with findstr: if you pipe the above to findstr.exe, exactly that will happen and it will implicitly convert with | out-string -stream for findstr (and all other external / non-PowerShell programs) and so gci env:* | findstr "Pro" (on a PowerShell console!) gives you rational output. select-string is a string-manipulation tool so I find the idea that people are not thinking right about it for expecting a string-manipulation tool to manipulate the incoming information as a string to be unfair on users. PowerShell is an incredibly versatile language but I think this is a common area of confusion. Hopefully, future versions of select-string will operate in the above fashion as a result of the change request on GitHub, but in the meantime, just use | out-string -stream and it will all work as expected, including for other string manipulations which you can then deal with easily:
(gci env:* | sort name | out-string -stream) -replace "Pro", "XXX" | select-String "XXX"
to keep this short: Your approach doesn't work in PowerShell. All you need to do is
# Short Version
gci env: | ? Name -match "PG" | sort Name
# Long Version
Get-ChildItem -Path env: |
Where-Object -FilterScript { $_.Name -match "PG" } |
Sort-Object -Property Name
Select-String works fine with string content piped one by one instead of a big stream.
Cheers
I am attempting to output full directory path and lastaccesstime in one line.
Needed --
R:\Directory1\Directory2\Directory3, March 10, 1015
What I am getting --
R:\Directory1\Directory2\Directory3
March 10, 1015
Here is my code, It isn't that complicated, but it is beyond me.
Get-ChildItem -Path "R:\" -Directory | foreach-object -process{$_.FullName, $_.LastAccessTime} | Where{ $_.LastAccessTime -lt [datetime]::Today.AddYears(-2) } | Out-File c:\temp\test.csv
I have used foreach-object in the past in order to ensure I do not truncate the excessively long directory names and paths, but never used it when pulling two properties. I would like the information to be on all one line, but haven't been successful. Thanks in advance for the assist.
I recommend filtering (Where-Object) before selecting the properties you want. Also I think you want to replace ForEach-Object with Select-Object, and lastly I think you want Export-Csv rather than Out-File. Example:
Get-ChildItem -Path "R:\" -Directory |
Where-Object { $_.LastAccessTime -lt [DateTime]::Today.AddYears(-2) } |
Select-Object FullName,LastAccessTime |
Export-Csv C:\temp\test.csv -NoTypeInformation
We can get your output on one line pretty easily, but to make it easy to read we may have to split your script out to multiple lines. I'd recommend saving the script below as a ".ps1" which would allow you to right click and select "run with powershell" to make it easier in the future. This script could be modified to play around with more inputs and variables in order to make it more modular and work in more situations, but for now we'll work with the constants you provided.
$dirs = Get-ChildItem -Path "R:\" -Directory
We'll keep the first line you made, since that is solid and there's nothing to change.
$arr = $dirs | Select-Object {$_.FullName, $_.LastAccessTime} | Where-Object{ $_.LastAccessTime -lt [datetime]::Today.AddYears(-2) }
For the second line, we'll use "Select-Object" instead. In my opinion, it's a lot easier to create an array this way. We'll want to deal with the answers as an array since it'll be easiest to post the key,value pairs next to each other this way. I've expanded your "Where" to "Where-Object" since it's best practice to use the full cmdlet name instead of the alias.
Lastly, we'll want to convert our "$arr" object to csv before putting in the temp out-file.
ConvertTo-CSV $arr | Out-File "C:\Temp\test.csv"
Putting it all together, your final script will look like this:
$dirs = Get-ChildItem -Path "C:\git" -Directory
$arr = $dirs | Select-Object {$_.FullName, $_.LastAccessTime} | Where{ $_.LastAccessTime -lt [datetime]::Today.AddYears(-2) }
ConvertTo-CSV $arr | Out-File "C:\Temp\test.csv"
Again, you can take this further by creating a function, binding it to a cmdlet, and creating parameters for your path, output file, and all that fun stuff.
Let me know if this helps!
I have been tasked with reporting all of the ACL's on each folder in our Shared drive structure. Added to that, I need to do a look up on the membership of each unique group that gets returned.
Im using the NTFSSecurity module in conjunction with the get-childitem2 cmdlet to get past the 260 character path length limit. The path(s) I am traversing are many hundreds of folders deep and long since pass the 260 character limit.
I have been banging on this for a couple of weeks. My first challenge was crafting my script to do my task all at once, but now im thinking thats my problem... The issue at hand is resources, specifically memory exhaustion. Once the script gets into one of the deep folders, it consumes all RAM and starts swapping to disk, and I eventually run out of disk space.
Here is the script:
$csvfile = 'C:\users\user1\Documents\acl cleanup\dept2_Dir_List.csv'
foreach ($record in Import-Csv $csvFile)
{
$Groups = get-childitem2 -directory -path $record.FullName -recurse | Get-ntfsaccess | where -property accounttype -eq -value group
$groups2 = $Groups | where -property account -notmatch -value '^builtin|^NT AUTHORITY\\|^Creator|^AD\\Domain'
$groups3 = $groups2 | select account -Unique
$GroupMembers = ForEach ($Group in $Groups3) {
(Get-ADGroup $Group.account.sid | get-adgroupmember | select Name, #{N="GroupName";e={$Group.Account}}
)}
$groups2 | select FullName,Account,AccessControlType,AccessRights,IsInherited | export-csv "C:\Users\user1\Documents\acl cleanup\Dept2\$($record.name).csv"
$GroupMembers | export-csv "C:\Users\user1\Documents\acl cleanup\Dept2\$($record.name)_GroupMembers.csv"
}
NOTE: The dir list it reads in is the top level folders created from a get-childitem2 -directory | export-csv filename.csv
During the run, it appears to not be flushing memory properly. This is just a guess from observation. At the end of each run through the code, the variables should be getting over-written, I thought, but memory doesn't go down, so it looked to me that since memory didn't go back down, that it wasn't properly releasing it? Like I said, a guess... I have been reading about runspaces but I am confused about how to implement that with this script. Is that the right direction for this?
Thanks in advance for any assistance...!
Funny you should post about this as I just finished a modified version of the script that I think works much better. A friend turned me on to 'Function Filters' that seem to work well here. Ill test it on the big directories tomorrow to see how much better the memory management is but so far it looks great.
#Define the function ‘filter’ here and call it ‘GetAcl’. Process is the keyword that tells the function to deal with each item in the pipeline one at a time
Function GetAcl {
PROCESS {
Get-NTFSAccess $_ | where -property accounttype -eq -value group | where -property account -notmatch -value '^builtin|^NT AUTHORITY\\|^Creator|^AD\\Domain'
}
}
#Import the directory top level paths
$Paths = import-csv 'C:\users\rknapp2\Documents\acl cleanup\dept2_Dir_List.csv'
#Process each line from the importcsv one at a time and run GetChilditem against it.
#Notice the second part – I ‘|’ pipe the results of the GetChildItem to the function that because of the type of function it is, handles each item one at a time
#When done, pass results to Exportcsv and send it to a file name based on the path name. This puts each dir into its own file.
ForEach ($Path in $paths) {
(Get-ChildItem2 -path $path.FullName -Recurse -directory) | getacl | export-csv "C:\Users\rknapp2\Documents\acl cleanup\TestFilter\$($path.name).csv" }
I am trying to compose a script/one liner, which will find files which have been modified over 10 hours ago in a specific folder and if there are no files I need it to print some value or string.
Get-ChildItem -Path C:\blaa\*.* | where {$_.Lastwritetime -lt (date).addhours(-10)}) | Format-table Name,LastWriteTime -HideTableHeaders"
With that one liner I am getting the wanted result when there are files with
modify time over 10 hours, but I also need it to print value/string if there are
no results, so that I can monitor it properly.
The reason for this is to utilize the script/one liner for monitoring purposes.
Those cmdlet Get-ChildItem and where clause you have a would return null if nothing was found. You would have to account for that separately. I would also caution the use of Format-Table for output unless you are just using it for screen reading. If you wanted a "one-liner" you would could this. All PowerShell code can be a one liner if you want it to be.
$results = Get-ChildItem -Path C:\blaa\*.* | where {$_.Lastwritetime -lt (date).addhours(-10)} | Select Name,LastWriteTime; if($results){$results}else{"No files found matching criteria"}
You have an added bracket in your code, that might be a copy artifact, I had to remove. Coded properly would look like this
$results = Get-ChildItem -Path "C:\blaa\*.*" |
Where-Object {$_.Lastwritetime -lt (date).addhours(-10)} |
Select Name,LastWriteTime
if($results){
$results
}else{
"No files found matching criteria"
}