>$search="<table id="
$linenumber= Get-Content ".\145039.html" | select-string $search | Select-Object LineNumber
$search="</table>"
$linenumber2= Get-Content ".\145039.html" | select-string $search | Select-Object LineNumber
#$linenumber2
# the list of line numbers to fetch
$linesToFetch = $linenumber[2]..$linenumber2[2]
$currentLine = 1
$result = switch -File ".\145039.html" {
default { if ($linesToFetch -contains $currentLine++) { $_ }}
}
# write to file and also display on screen by using -PassThru
$result | Set-Content -Path ".\excerpt.html" -PassThru
Cannot convert the "#{LineNumber=6189}" value of type "Selected.Microsoft.PowerShell.Commands.MatchInfo" to type "System.Int32".
At line:10 char:1
+ $linesToFetch = $linenumber[2]..$linenumber2[2]
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [], RuntimeException
+ FullyQualifiedErrorId : ConvertToFinalInvalidCastException
$linenumber and $linenumber2 return values like below but I just need to get the number not the column header.
LineNumber
----------
6015
Also, the final version needs to loop through all the html files in a directory not just one static file.
Sorry, there is probably a better way to do this but not sure how.
Thanks in advance!
Did a lot of googling but could not find the right solution.
Updated code:
$search1="disconnect-status"
$linenumber1= Get-Content ".\145039.html" | select-string
$search1
| Select-Object -ExpandProperty LineNumber
$search2="</table>"
$linenumber2= Get-Content ".\145039.html" | select-string
$search2 | Select-Object -ExpandProperty LineNumber
# the list of line numbers to fetch
$linesToFetch = $linenumber1[3]..$linenumber2[1]
$currentLine = 1
$result = switch -File ".\145039.html" {
default { if ($linesToFetch -contains $currentLine++) { $_ }}
}
# write to file and also display on screen by using -PassThru
$result | Set-Content -Path ".\excerpt.html" -PassThru
_____________________________________________________________
Thank you # mklement0
This now works for one file at a time now I need it go select text from all the HTML files in the directory.
Your immediate problem is that you need to change Select-Object LineNumber (which, due to positional parameter binding, is equivalent to Select-Object -Property LineNumber) to Select-Object -ExpandProperty LineNumber.
That is, you must use Select-Object's -ExpandProperty parameter in order to only get the values of the input objects' .LineNumber properties - see this post.
That said, your approach can be optimized in a number of ways, allowing you to make do with only a switch statement:
$output = $false; $openTagCount = 0
$result =
switch -File .\145039.html {
'<table id=' {
if (++$openTagCount -eq 3) { $output = $true } # 3rd block found
continue
}
'</table>' {
if ($output) { break } # end of 3rd block -> exit
continue
}
default {
if ($output) { $_ } # inside 3rd block -> output line
}
}
Note: This extracts the lines inside the third <table> element that has an id attribute, as implied by your original solution attempt; the solution you later edited into the question works differently.
Taking a step back:
It looks like your input is HTML, so you're usually better off using HTML parsing to handle your input:
In Windows PowerShell you may be able to use Invoke-WebRequest relying on the Internet Explorer engine if present (it isn't anymore by default in recent Windows versions).
In recent versions of Windows and in PowerShell (Core) (v6+), you'll either need New-Object -Com HTMLFile - see this answer - or a third-party solution such as such as the PowerHTML module that wraps the HTML Agility Pack - see this answer.
Related
First, my PS knowledge is very basic, so know that up front.
I'm working on a basic script to search EventIDs in archived .evtx files and kick out "reports". The Where-Object queries are in .txt files stored in .\AuditEvents\ folder. I'm trying to do a ForEach on the .txt files and pass each query to Get-WinEvent.
Here's an example of how the queries appear in the .txt files:
{($_.ID -eq "11")}
The script is:
$ae = Get-ChildItem .\AuditEvents\
ForEach ($f in $ae) {
$qs = Get-Content -Path .\AuditEvents\$f
Get-WinEvent -Path .\AuditReview\*.evtx -MaxEvents 500 | Select-Object TimeCreated, ID, LogName, MachineName, ProviderName, LevelDisplayName, Message | Where-Object $qs | Out-GridView -Title $f.Name
}
This is the error:
Where-Object : Cannot bind argument to parameter 'FilterScript' because it is null.
At C:\Users\######\Desktop\PSAuditReduction\PSAuditReduction.ps1:6 char:177
+ ... e, ProviderName, LevelDisplayName, Message | Where-Object $qs | Out-G ...
+ ~~~
+ CategoryInfo : InvalidData: (:) [Where-Object], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.WhereObjectCommand
Your symptom implies that $qs is $null, which in turn implies that file .\AuditEvents\$f is empty.
However, even if it had content, you couldn't pass the resulting string as-is to the (positionally implied) -FilterScript parameter of Where-Object requires a script block ({ ... }).
You must create a script block from the string explicitly, using [scriptblock]::Create().
A simplified example:
# Simulated input using a literal string instead of file input via Get-Content
$qs = '{ 0 -eq $_ % 2 }' # Sample filter: return $true for even numbers.
# Remove the enclosing { and }, as they are NOT part of the code itself
# (they are only needed to define script-block *literals* in source code).
# NOTE: If you control the query files, you can simplify them
# by omitting { and } to begin with, which makes this
# -replace operation unnecessary.
$qs = $qs.Trim() -replace '^\{(.+)\}$', '$1'
# Construct a script block from the string and pass it to Where-Object
1..4 | Where-Object ([scriptblock]::Create($qs)) # -> 2, 4
Note:
Your code assumes that each .\AuditEvents\$f file contains just one line, and that that line contains valid PowerShell source code suitable for use a Where-Object filter.
Generally, be sure to only load strings that you'll execute as code from sources you trust.
Taking a step back:
As Abraham Zinala points out, a much faster way to filter event-log entries is by using Get-WinEvent's -FilterHashtable parameter.
This allows you to save hastable literals in your query files, which you can read directly into a hashtable with Import-PowerShellDataFile:
# Create a file with a sample filter.
'#{Path=".\AuditEvents\.*evtx";ID=11}' > sample.txt
# Read the file into a hashtable...
$hash = Import-PowerShellDataFile sample.txt
# ... and pass it to Get-WinEvent
Get-WinEvent -MaxEvents 500 -FilterHashtable $hash | ...
This question already has an answer here:
Not all properties displayed
(1 answer)
Closed 1 year ago.
This is a follow-up question from PowerShell | EVTX | Compare Message with Array (Like)
I changed the tactic slightly, now I am collecting all the services installed,
$7045 = Get-WinEvent -FilterHashtable #{ Path="1system.evtx"; Id = 7045 } | select
#{N=’Timestamp’; E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
Id,
#{N=’Machine Name’; E={$_.MachineName}},
#{N=’Service Name’; E={$_.Properties[0].Value}},#{N=’Image Path’;E=$_.Properties[1].Value}},
#{N=’RunAsUser’; E={$_.Properties[4].Value}},#{N=’Installed By’; E={$_.UserId}}
Now I match each object for any suspicious traits and if found, I add a column 'Suspicious' with the value 'Yes'. This is because I want to leave the decision upto the analyst and pretty sure the bad guys might use something we've not seen before.
foreach ($Evt in $7045)
{
if ($Evt.'Image Path' -match $sus)
{
$Evt | Add-Member -MemberType NoteProperty -Name 'Suspicious' -Value 'Yes'
}
}
Now, I'm unable to get PowerShell to display all columns unless I specifically Select them
$7045 | Format-Table
Same goes for CSV Export. The first two don't include the Suspicious Column but the third one does but that's because I'm explicitly asking it to.
$7045 | select * | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Select-Object Timestamp, Id, 'Machine Name', 'Service Name', 'Image Path', 'RunAsUser', 'Installed By', Suspicious | Export-Csv -Path test.csv -NoTypeInformation
I read the Export-CSV documentation on MS. Searched StackOverFlow for some tips, I think it has something to do with PS checking the first Row and then compares if the property exists for the second row and so on.
Thank you
The issue you're experiencing is partially because of how objects are displayed to the console, the first object's Properties determines the displayed Properties (Columns) to the console.
The bigger problem though, is that Export-Csv will not export those properties that do not match with first object's properties unless they're explicitly added to the remaining objects or the objects are reconstructed, for this one easy way is to use Select-Object as you have pointed out in the question.
Given the following example:
$test = #(
[pscustomobject]#{
A = 'ValA'
}
[pscustomobject]#{
A = 'ValA'
B = 'ValB'
}
[pscustomobject]#{
C = 'ValC'
D = 'ValD'
E = 'ValE'
}
)
Format-Table will not display the properties B to E:
$test | Format-Table
A
-
ValA
ValA
Format-List can display the objects properly, this is because each property with it's corresponding value has it's own console line in the display:
PS /> $test | Format-List
A : ValA
A : ValA
B : ValB
C : ValC
D : ValD
E : ValE
Export-Csv and ConvertTo-Csv will also miss properties B to E:
$test | ConvertTo-Csv
"A"
"ValA"
"ValA"
You have different options as a workaround for this, you could either add the Suspicious property to all objects and for those events that are not suspicious you could add $null as Value.
Another workaround is to use Select-Object explicitly calling the Suspicious property (this works because you know the property is there and you know it's Name).
If you did not know how many properties your objects had, a dynamic way to solve this would be to discover their properties using the PSObject intrinsic member.
using namespace System.Collections.Generic
function ConvertTo-NormalizedObject {
[CmdletBinding()]
param(
[Parameter(ValueFromPipeline, Mandatory)]
[object[]] $InputObject
)
begin {
$list = [List[object]]::new()
$props = [HashSet[string]]::new([StringComparer]::InvariantCultureIgnoreCase)
}
process {
foreach($object in $InputObject) {
$list.Add($object)
foreach($property in $object.PSObject.Properties) {
$null = $props.Add($property.Name)
}
}
}
end {
$list | Select-Object ([object[]] $props)
}
}
Usage:
# From Pipeline
$test | ConvertTo-NormalizedObject | Format-Table
# From Positional / Named parameter binding
ConvertTo-NormalizedObject $test | Format-Table
Lastly, a pretty easy way of doing it thanks to Select-Object -Unique:
$prop = $test.ForEach{ $_.PSObject.Properties.Name } | Select-Object -Unique
$test | Select-Object $prop
Using $test for this example, the result would become:
A B C D E
- - - - -
ValA
ValA ValB
ValC ValD ValE
Continuing from my previous answer, you can add a column Suspicious straight away if you take out the Where-Object filter and simply add another calculated property to the Select-Object cmdlet:
# create a regex for the suspicious executables:
$sus = '(powershell|cmd|psexesvc)\.exe'
# alternatively you can join the array items like this:
# $sus = ('powershell.exe','cmd.exe','psexesvc.exe' | ForEach-Object {[regex]::Escape($_)}) -join '|'
$7045 = Get-WinEvent -FilterHashtable #{ LogName = 'System';Id = 7045 } |
Select-Object Id,
#{N='Timestamp';E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
#{N='Machine Name';E={$_.MachineName}},
#{N='Service Name'; E={$_.Properties[0].Value}},
#{N='Image Path'; E={$_.Properties[1].Value}},
#{N='RunAsUser'; E={$_.Properties[4].Value}},
#{N='Installed By'; E={$_.UserId}},
#{N='Suspicious'; E={
if ($_.Properties[1].Value -match $sus) { 'Yes' } else {'No'}
}}
$7045 | Export-Csv -Path 'X:\Services.csv' -UseCulture -NoTypeInformation
Because you have many columns, this will not fit the console width anymore if you do $7045 | Format-Table, but the CSV file will hold all columns you wanted.
I added switch -UseCulture to the Export-Csv cmdlet, which makes sure you can simply double-click the csv file so it opens correctly in your Excel.
As sidenote: Please do not use those curly so-called 'smart-quotes' in code as they may lead to unforeseen errors. Straighten these ’ thingies and use normal double or single quotes (" and ')
I'm trying to write a script that counts all comments in multiple files, including both single line (//) and multi-line (/* */) comments and prints out the total. So, the following file would return 4
// Foo
var text = "hello world";
/*
Bar
*/
alert(text);
There's a requirement to include specific file types and exclude certain file types and folders, which I already have working in my code.
My current code is:
( gci -include *.cs,*.aspx,*.js,*.css,*.master,*.html -exclude *.designer.cs,jquery* -recurse `
| ? { $_.FullName -inotmatch '\\obj' } `
| ? { $_.FullName -inotmatch '\\packages' } `
| ? { $_.FullName -inotmatch '\\release' } `
| ? { $_.FullName -inotmatch '\\debug' } `
| ? { $_.FullName -inotmatch '\\plugin-.*' } `
| select-string "^\s*//" `
).Count
How do I change this to get multi-line comments as well?
UPDATE: My final solution (slightly more robust than what I was asking for) is as follows:
$CodeFiles = Get-ChildItem -include *.cs,*.aspx,*.js,*.css,*.master,*.html -exclude *.designer.cs,jquery* -recurse |
Where-Object { $_.FullName -notmatch '\\(obj|packages|release|debug|plugin-.*)\\' }
$TotalFiles = $CodeFiles.Count
$IndividualResults = #()
$CommentLines = ($CodeFiles | ForEach-Object{
#Get the comments via regex
$Comments = ([regex]::matches(
[IO.File]::ReadAllText($_.FullName),
'(?sm)^[ \t]*(//[^\n]*|/[*].*?[*]/)'
).Value -split '\r?\n') | Where-Object { $_.length -gt 0 }
#Get the total lines
$Total = ($_ | select-string .).Count
#Add to the results table
$IndividualResults += #{
File = $_.FullName | Resolve-Path -Relative;
Comments = $Comments.Count;
Code = ($Total - $Comments.Count)
Total = $Total
}
Write-Output $Comments
}).Count
$TotalLines = ($CodeFiles | select-string .).Count
$TotalResults = New-Object PSObject -Property #{
Files = $TotalFiles
Code = $TotalLines - $CommentLines
Comments = $CommentLines
Total = $TotalLines
}
Write-Output (Get-Location)
Write-Output $IndividualResults | % { new-object PSObject -Property $_} | Format-Table File,Code,Comments,Total
Write-Output $TotalResults | Format-Table Files,Code,Comments,Total
To be clear: Using string matching / regular expressions is not a fully robust way to detect comments in JavaScript / C# code, because there can be false positives (e.g., var s = "/* hi */";); for robust parsing you'd need a language parser.
If that is not a concern, and it is sufficient to detect comments (that start) on their own line, optionally preceded by whitespace, here's a concise solution (PSv3+):
(Get-ChildItem -include *.cs,*.aspx,*.js,*.css,*.master,*.html -exclude *.designer.cs,jquery* -recurse |
Where-Object { $_.FullName -notmatch '\\(obj|packages|release|debug|plugin-.*)' } |
ForEach-Object {
[regex]::matches(
[IO.File]::ReadAllText($_.FullName),
'(?sm)^[ \t]*(//[^\n]*|/[*].*?[*]/)'
).Value -split '\r?\n'
}
).Count
With the sample input, the ForEach-Object command yields 4.
Remove the ^[ \t]* part to match comments starting anywhere on a line.
The solution reads each input file as a single string with [IO.File]::ReadAllText() and then uses the [regex]::Matches() method to extract all (potentially line-spanning) comments.
Note: You could use Get-Content -Raw instead to read the file as a single string, but that is much slower, especially when processing multiple files.
The regex uses in-line options s and m ((?sm)) to respectively make . match newlines too and to make anchors ^ and $ match line-individually.
^[ \t]* matches any mix of spaces and tabs, if any, at the start of a line.
//[^\n]*$ matches a string that starts with // through the end of the line.
/[*].*?[*]/ matches a block comment across multiple lines; note the lazy quantifier, *?, which ensures that very next instance of the closing */ delimiter is matched.
The matched comments (.Value) are then split into individual lines (-split '\r?\n'), which are output.
The resulting lines across all files are then counted (.Count)
As for what you tried:
The fundamental problem with your approach is that Select-String with file-info object input (such as provided by Get-ChildItem) invariably processes the input files line by line.
While this could be remedied by calling Select-String inside a ForEach-Object script block in which you pass each file's content as a single string to Select-String, direct use of the underlying regex .NET types, as shown above, is more efficient.
An IMO better approach is to count net code lines by removing single/multi line comments.
For a start a script that handles single files and returns for your above sample.cs the result 5
((Get-Content sample.cs -raw) -replace "(?sm)^\s*\/\/.*?$" `
-replace "(?sm)\/\*.*?\*\/.*`n" | Measure-Object -Line).Lines
EDIT: without removing empty lines, build the difference from total lines
## Q:\Test\2018\10\31\SO_53092258.ps1
$Data = Get-ChildItem *.cs | ForEach-Object {
$Content = Get-Content $_.FullName -Raw
$TotalLines = (Measure-Object -Input $Content -Line).Lines
$CodeLines = ($Content -replace "(?sm)^\s*\/\/.*?$" `
-replace "(?sm)\/\*.*?\*\/.*`n" | Measure-Object -Line).Lines
$Comments = $TotalLines - $CodeLines
[PSCustomObject]#{
File = $_.FullName
Lines = $TotalLines
Comments= $Comments
}
}
$Data
"="*40
"TotalLines={0} TotalCommentLines={1}" -f (
$Data | Measure-Object -Property Lines,Comments -Sum).Sum
Sample output:
> Q:\Test\2018\10\31\SO_53092258.ps1
File Lines Comments
---- ----- --------
Q:\Test\2018\10\31\example.cs 10 5
Q:\Test\2018\10\31\sample.cs 9 4
============================================
TotalLines=19 TotalCommentLines=9
So I am a complete beginner at Powershell but need to write a script that will take a file, compare it against another file, and tell me what strings are different in the first compared to the second. I have had a go at this but I am struggling with the outputs as my script will currently only tell me on which line things are different, but it also seems to count lines that are empty too.
To give some context for what I am trying to achieve, I would like to have a static file of known good Windows processes ($Authorized) and I want my script to pull a list of current running processes, filter by the process name column so to just pull the process name strings, then match anything over 1 character, sort the file by unique values and then compare it against $Authorized, plus finally either outputting the different process strings found in $Processes (to the ISE Output Pane) or just to output the different process names to a file.
I have spent today attempting the following in Powershell ISE and also Googling around to try and find solutions. I heard 'fc' is a better choice instead of Compare-Object but I could not get that to work. I have thus far managed to get it to work but the final part where it compares the two files it seems to compare line by line, for which would always give me false positives as the line position of the process names in the file supplied would change, furthermore I only want to see the changed process names, and not the line numbers which it is reporting ("The process at line 34 is an outlier" is what currently gets outputted).
I hope this makes sense, and any help on this would be very much appreciated.
Get-Process | Format-Table -Wrap -Autosize -Property ProcessName | Outfile c:\users\me\Desktop\Processes.txt
$Processes = 'c:\Users\me\Desktop\Processes.txt'
$Output_file = 'c:\Users\me\Desktop\Extracted.txt'
$Sorted = 'c:\Users\me\Desktop\Sorted.txt'
$Authorized = 'c:\Users\me\Desktop\Authorized.txt'
$regex = '.{1,}'
select-string -Path $Processes -Pattern $regex |% { $_.Matches } |% { $_.Value } > $Output_file
Get-Content $Output_file | Sort-Object -Unique > $Sorted
$dif = Compare-Object -ReferenceObject $(Get-Content $Sorted) -DifferenceObject $(get-content $Authorized) -IncludeEqual
$lineNumber = 1
foreach ($difference in $dif)
{
if ($difference.SideIndicator -ne "==")
{
Write-Output "The Process at Line $linenumber is an Outlier"
}
$lineNumber ++
}
Remove-Item c:\Users\me\Desktop\Processes.txt
Remove-Item c:\Users\me\Desktop\Extracted.txt
Write-Output "The Results are Stored in $Sorted"
From the length and complexity of your script, I feel like I'm missing something, but your description seems clear
Running process names:
$ProcessNames = #(Get-Process | Select-Object -ExpandProperty Name)
.. which aren't blank: $ProcessNames = $ProcessNames | Where-Object {$_ -ne ''}
List of authorised names from a file:
$AuthorizedNames = Get-Content 'c:\Users\me\Desktop\Authorized.txt'
Compare:
$UnAuthorizedNames = $ProcessNames | Where-Object { $_ -notin $AuthorizedNames }
optional output to file:
$UnAuthorizedNames | Set-Content out.txt
or in the shell:
#(gps).Name -ne '' |? { $_ -notin (gc authorized.txt) } | sc out.txt
1 2 3 4 5 6 7 8
1. #() forces something to be an array, even if it only returns one thing
2. gps is a default alias of Get-Process
3. using .Property on an array takes that property value from every item in the array
4. using an operator on an array filters the array by whether the items pass the test
5. ? is an alias of Where-Object
6. -notin tests if one item is not in a collection
7. gc is an alias of Get-Content
8. sc is an alias of Set-Content
You should use Set-Content instead of Out-File and > because it handles character encoding nicely, and they don't. And because Get-Content/Set-Content sounds like a memorable matched pair, and Get-Content/Out-File doesn't.
I am writing a script that after each iteration through a loop (array of selected services) it will gather the 4 values for each service that are: server name, service name, service state, and service start name
So for each iteration, I would like to output the 4 mentioned values to an external file (txt, svc, or html) such that each value will be arranged in its own column. Currently I use tab `t to arrange the values in each column but it doesn't work quite well because some service name is a lot longer or a lot shorter so it screws up the column alignment. What other approach do you suggest so all columns are aligned properly
Below is a snippet of my script on how I currently format the output to a txt file
ForEach($service in services)
$startname = $service.startname
$state = $service.state
$servicename = $service.name
write-output "$server `t $servicename `t $state `t $startname is current" | out-file -append $ScriptDirectory
If you just want to dump the results to text in a nicely-formatted way (i.e. you don't have requirements for making this CSV, or tab-delimited, or anything else besides "easy for a person to read"), then just use Format-Table -AutoSize.
AutoSize does exactly what you want - it inspects the length of all properties you are outputting, then dynamically adjusts the column width so that as much as possible is shown.
You don't explain where $server comes from, I will assume that is defined somewhere else...
$services `
| Format-Table -AutoSize #{N='Server';E={$server}},StartName,State,Name `
| Out-String `
| Out-File results.txt
Instead of using several variables, use a Powershell object to store your output. Something like this:
ForEach($service in $services) {
New-Object PSObject -Property #{
StartName = $service.startname
State = $service.state
ServiceName = $service.name
}
} | Out-File $ScriptDirectory
You may need to add a Select-Object in the chain to ensure the columns are in the correct order that you want for your final output.
If you want to keep the variables, You could try the following String formatting to space out the variable in the string evenly. In the example below the spacing is 20 characters between each value:
ForEach($service in services){
$startname = $service.startname
$state = $service.state
$servicename = $service.name
"{0,-20} | {1,-20} | {2,-20} | {3,-20}" -f $server,$servicename,$state,$startname `
| Out-File -append $ScriptDirectory
}
It's a little unclear what you're looking for as some of the properties of the object Get-Service returns don't exist as written and the code seems incomplete. Taking a guess at your intent though:
$servers = #("server1","server2");
$services = get-service -computername $servers;
$svcCollection = #();
ForEach($service in $services) {
$svccollection+=New-Object PSObject -Property #{
Servername = $service.MachineName;
StartName = $service.servicename;
State = $service.Status;
ServiceName = $service.DisplayName;
}
}
# Various output formats
$svccollection|ConvertTo-Html|Out-File -path Services.html; # Create a full HTML file
$svcCollection|Export-Csv -NoTypeInformation -Path Services.csv; # Create a "traditional" CSV file
$svcCollection|Export-Csv -Delimiter "`t" -Path Services-tab.csv; # Create a tab-delimited CSV file
$svcCollection|ConvertTo-Xml|Out-File -path Services.xml; # Create an XML file
$svcCollection|ConvertTo-Json|Out-File -path Services.js; # Create a JSON object (v3 only)