Running Powershell command Out-File in Azure Pipeline cuts lines - powershell

As part of a CI pipeline, I need to generate a list of all the files in the repository, including certain properties of them, and to Out-File it into a file. The command I use:
Get-ChildItem $Path -File -Recurse | Select-Object -Property LastWriteTime, #
{
label = "Size(KB)"
expr = { [string]::Format("{0:0.00}", $_.Length/1KB) }
}, FullName, <some_other_property> | Out-File $OutputFile
My problem is, that running this script from the command line gives the desired result.
However, running this during a Azure Pipeline build does 2 bad things:
cuts the lines in the FullName column when they are long:
LastWriteTime Size(KB) Name
------------ ------- ----
<some_date> <some size> ASomeWhatLong...
Doesn't display the rest of the properties such as <some_other_property>
If I turn FullName into Name it all goes OK, but I really need the FullName property.
Because I'm working in a Air-gapped environment I can't copy all the outputs and everything.
I've tried using the -Width flag for Out-File with no success.

I believe what's happening under the hood, ps uses the ToString() method of the object created which outputs it as the Format-Table cmdlet does. You get truncated properties because of the Window's size. To look at it you could use:
(Get-Host).ui.RawUI.WindowSize
Probably this is too small.
What I would suggest is the following:
Pipe the object into Format-Table
Get-ChildItem $Path -File -Recurse | Select-Object -Property LastWriteTime, #
{
label = "Size(KB)"
expr = { [string]::Format("{0:0.00}", $_.Length/1KB) }
}, FullName, <some_other_property> | Format-Table | Out-String | Out-File $OutputFile
This probably won't work as it is, but you could play with the Format-Table's properties like: -Wrap. By default it will allocate enough space for the first properties, and the last one it would try to 'fit' it, which might look as:
LastWriteTime Size(KB) Name
------------ ------- ----
<some_date> <some size> ASomeWhatLong
foobarfoobarfo
foobarfoobarfo
To solve this, you can use the -Property argument, which needs to be as:
$propertWidth = [int]((Get-Host).ui.RawUI.WindowSize.Width / 3)
$property = #(
#{ Expression = 'LastWriteTime'; Width = $propertWidth; },
#{ Expression = 'Size(KB)'; Width = $propertWidth; },
#{ Expression = 'FullName'; Width = $propertWidth; }
)
... | Format-Table -Property $property -Wrap | ...
If you don't mind having a JSON into your file, you could use:
Get-ChildItem $Path -File -Recurse | Select-Object -Property LastWriteTime, #
{
label = "Size(KB)"
expr = { [string]::Format("{0:0.00}", $_.Length/1KB) }
}, FullName, <some_other_property> | ConvertTo-Json | Out-File $OutputFile
But take into account that ConvertTo-Json has a default Depth of 2. This will also truncate your objects if you have nested objects. But as far as property lengths, it will do fine.

Related

How to use PS calculated properties to collate data into a cell?

I'm stumped on this one. I have a PS custom object with strings only and I want to build a report where I'm outputting strings of data into a new pipeline output object.
$myObjectTable |
Select-Object #{
n = "OldData";
e = {
$_ | Select-Object name, *_old | Format-List | Out-String
}
},
#{
n = "NewData";
e = {
$_ | Select-Object name, *_new | Format-List | Out-String
}
}
Running this produces blank output.
I tried running the code above with just the $_ object in the expressions, but I only got ... as the output. Wrapping the expressions in parenthesis did not change the output.
The ... as property value means that the value is either a multi-line string or it just didn't fit in a tabular format. See Using Format commands to change output view more details.
You can fix those empty lines added by Out-String using Trim. Then if you want to properly display this object having multi-line property values, Format-Table -Wrap will be needed.
Here is a little example:
[pscustomobject]#{
Name = 'foo'
Something_Old = 123
Something_New = 456
} | ForEach-Object {
[pscustomobject]#{
OldData = $_ | Format-List name, *_old | Out-String | ForEach-Object Trim
NewData = $_ | Format-List name, *_new | Out-String | ForEach-Object Trim
}
} | Format-Table -Wrap
Resulting object would become:
OldData NewData
------- -------
Name : foo Name : foo
Something_Old : 123 Something_New : 456

PowerShell Export-CSV - Missing Columns [duplicate]

This question already has an answer here:
Not all properties displayed
(1 answer)
Closed 1 year ago.
This is a follow-up question from PowerShell | EVTX | Compare Message with Array (Like)
I changed the tactic slightly, now I am collecting all the services installed,
$7045 = Get-WinEvent -FilterHashtable #{ Path="1system.evtx"; Id = 7045 } | select
#{N=’Timestamp’; E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
Id,
#{N=’Machine Name’; E={$_.MachineName}},
#{N=’Service Name’; E={$_.Properties[0].Value}},#{N=’Image Path’;E=$_.Properties[1].Value}},
#{N=’RunAsUser’; E={$_.Properties[4].Value}},#{N=’Installed By’; E={$_.UserId}}
Now I match each object for any suspicious traits and if found, I add a column 'Suspicious' with the value 'Yes'. This is because I want to leave the decision upto the analyst and pretty sure the bad guys might use something we've not seen before.
foreach ($Evt in $7045)
{
if ($Evt.'Image Path' -match $sus)
{
$Evt | Add-Member -MemberType NoteProperty -Name 'Suspicious' -Value 'Yes'
}
}
Now, I'm unable to get PowerShell to display all columns unless I specifically Select them
$7045 | Format-Table
Same goes for CSV Export. The first two don't include the Suspicious Column but the third one does but that's because I'm explicitly asking it to.
$7045 | select * | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Select-Object Timestamp, Id, 'Machine Name', 'Service Name', 'Image Path', 'RunAsUser', 'Installed By', Suspicious | Export-Csv -Path test.csv -NoTypeInformation
I read the Export-CSV documentation on MS. Searched StackOverFlow for some tips, I think it has something to do with PS checking the first Row and then compares if the property exists for the second row and so on.
Thank you
The issue you're experiencing is partially because of how objects are displayed to the console, the first object's Properties determines the displayed Properties (Columns) to the console.
The bigger problem though, is that Export-Csv will not export those properties that do not match with first object's properties unless they're explicitly added to the remaining objects or the objects are reconstructed, for this one easy way is to use Select-Object as you have pointed out in the question.
Given the following example:
$test = #(
[pscustomobject]#{
A = 'ValA'
}
[pscustomobject]#{
A = 'ValA'
B = 'ValB'
}
[pscustomobject]#{
C = 'ValC'
D = 'ValD'
E = 'ValE'
}
)
Format-Table will not display the properties B to E:
$test | Format-Table
A
-
ValA
ValA
Format-List can display the objects properly, this is because each property with it's corresponding value has it's own console line in the display:
PS /> $test | Format-List
A : ValA
A : ValA
B : ValB
C : ValC
D : ValD
E : ValE
Export-Csv and ConvertTo-Csv will also miss properties B to E:
$test | ConvertTo-Csv
"A"
"ValA"
"ValA"
You have different options as a workaround for this, you could either add the Suspicious property to all objects and for those events that are not suspicious you could add $null as Value.
Another workaround is to use Select-Object explicitly calling the Suspicious property (this works because you know the property is there and you know it's Name).
If you did not know how many properties your objects had, a dynamic way to solve this would be to discover their properties using the PSObject intrinsic member.
using namespace System.Collections.Generic
function ConvertTo-NormalizedObject {
[CmdletBinding()]
param(
[Parameter(ValueFromPipeline, Mandatory)]
[object[]] $InputObject
)
begin {
$list = [List[object]]::new()
$props = [HashSet[string]]::new([StringComparer]::InvariantCultureIgnoreCase)
}
process {
foreach($object in $InputObject) {
$list.Add($object)
foreach($property in $object.PSObject.Properties) {
$null = $props.Add($property.Name)
}
}
}
end {
$list | Select-Object ([object[]] $props)
}
}
Usage:
# From Pipeline
$test | ConvertTo-NormalizedObject | Format-Table
# From Positional / Named parameter binding
ConvertTo-NormalizedObject $test | Format-Table
Lastly, a pretty easy way of doing it thanks to Select-Object -Unique:
$prop = $test.ForEach{ $_.PSObject.Properties.Name } | Select-Object -Unique
$test | Select-Object $prop
Using $test for this example, the result would become:
A B C D E
- - - - -
ValA
ValA ValB
ValC ValD ValE
Continuing from my previous answer, you can add a column Suspicious straight away if you take out the Where-Object filter and simply add another calculated property to the Select-Object cmdlet:
# create a regex for the suspicious executables:
$sus = '(powershell|cmd|psexesvc)\.exe'
# alternatively you can join the array items like this:
# $sus = ('powershell.exe','cmd.exe','psexesvc.exe' | ForEach-Object {[regex]::Escape($_)}) -join '|'
$7045 = Get-WinEvent -FilterHashtable #{ LogName = 'System';Id = 7045 } |
Select-Object Id,
#{N='Timestamp';E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
#{N='Machine Name';E={$_.MachineName}},
#{N='Service Name'; E={$_.Properties[0].Value}},
#{N='Image Path'; E={$_.Properties[1].Value}},
#{N='RunAsUser'; E={$_.Properties[4].Value}},
#{N='Installed By'; E={$_.UserId}},
#{N='Suspicious'; E={
if ($_.Properties[1].Value -match $sus) { 'Yes' } else {'No'}
}}
$7045 | Export-Csv -Path 'X:\Services.csv' -UseCulture -NoTypeInformation
Because you have many columns, this will not fit the console width anymore if you do $7045 | Format-Table, but the CSV file will hold all columns you wanted.
I added switch -UseCulture to the Export-Csv cmdlet, which makes sure you can simply double-click the csv file so it opens correctly in your Excel.
As sidenote: Please do not use those curly so-called 'smart-quotes' in code as they may lead to unforeseen errors. Straighten these ’ thingies and use normal double or single quotes (" and ')

Getting a specific process id from a powershell output

Hi I'm new to powershell scripting and I would like to retrieve a specific process id based on the file's hash. However I can only get either a table with the hash value or a table with the id,process name and path
$ps = Get-Process | Select-Object -Property Id,ProcessName,Path
$hashlist = Get-FileHash(Get-Process|Select-Object -ExpandProperty Path) -Algorithm MD5
Is it possible for me to merge the two tables together so that I can get a view of Id,ProcessName and Hash using the path to link them together?
EDIT: totally different approach due to new information from comment
I don't think it is so easy to identify malware with a file MD5 hash.
Modern AntiVirusSoftware uses heuristics to overcome the problem of mean malware which includes random data and also obfuscates it's origin.
## Q:\Test\2018\11\11\SO_53247430.ps1
# random hex string replace with your malware signature
$MalwareMD5Hash = 'D52C11B7E076FCE593288439ABA0F6D4'
Get-Process | Where-Object Path | Select-Object ID,Path | Group-Object Path | ForEach-Object {
if ($MalwareMD5Hash -eq (Get-FileHash $_.Name -Alg MD5).Hash){
##iterate group to kill all processes matching
ForEach ($PID in $_.Group.ID){
Stop-Process -ID $PID -Force -WhatIF
}
}
$_.Name | Remove-Item -Force -WhatIf # to delete the physical file.
}
As I suggested in my comment:
$HashList = [ordered]#{}
Get-Process |Where-Object Path | Select-Object Path |Sort-Object Path -Unique | ForEach-Object {
$HashList[$_.Path]=(Get-FileHash $_.Path -Alg MD5).Hash
## or the reverse, the hash as key and the path as value
# $HashList[(Get-FileHash $_.Path -Alg MD5).Hash]=$_.Path
}
$Hashlist | Format-List
Shorted sample output
Name : C:\Program Files\Mozilla Firefox\firefox.exe
Value : BFE829AB5A4B729EE4565700FC8853DA
Name : C:\WINDOWS\Explorer.EXE
Value : E4A81EDDFF8B844D85C8B45354E4144E
Name : C:\WINDOWS\system32\conhost.exe
Value : EA777DEEA782E8B4D7C7C33BBF8A4496
Name : C:\WINDOWS\system32\DllHost.exe
Value : 2528137C6745C4EADD87817A1909677E
> $hashlist['C:\WINDOWS\Explorer.EXE']
E4A81EDDFF8B844D85C8B45354E4144E
Or with the reversed list
> $hashlist['E4A81EDDFF8B844D85C8B45354E4144E']
C:\WINDOWS\Explorer.EXE

Edit an object non-destructively in PowerShell

I am trying to format the resulting object without destroying it. But all my efforts and research has failed me. Any tips are welcome.
My code looks like this:
Set-Location 'C:\Temp'
$Files = Get-ChildItem -File | Select-Object FullName, Length
And what I get, is this:
FullName Length
-------- ------
C:\Temp\CleanupScript.txt 10600
C:\Temp\Columns.csv 4214
C:\Temp\Content.html 271034
C:\Temp\Content.txt 271034
C:\Temp\DirSizes.csv 78
What I want is this:
FullName Length
-------- ------
Temp\CleanupScript.txt 10600
Temp\Columns.csv 4214
Temp\Content.html 271034
Temp\Content.txt 271034
Temp\DirSizes.csv 78
When I tried this:
$Files = Get-ChildItem -File | Select-Object FullName, Length | % { $_.FullName.Remove(0, 3) }
I got the right result, but I lost the Length column.
PS C:\Temp> $Files
Temp\CleanupScript.txt
Temp\Columns.csv
Temp\Content.html
Temp\Content.txt
Temp\DirSizes.csv
Please help.
Big thanks
Patrik
The easiest way to do this is to construct the property you want in the Select command, such as:
$Files = Get-ChildItem -File | Select #{l='FullName';e={$_.FullName.Substring(3)}},Length
The format for this is a hashtable with two entries. The keys are lable (or name), and expression. You can shorten them to l (or n), and e. The label entry defines the name of the property you are constructing, and the expression defines the value.
If you want to retain all of the original methods and properties of the objects you should add a property to them rather than using calculated properties. You can do that with Add-Member as such:
$Files = GCI -File | %{Add-Member -inputobject $_ -notepropertyname 'ShortPath' -notepropertyvalue $_.FullName.Substring(3) -PassThru}
Then you can use that property by name like $Files | FT ShortPath,Length -Auto, while still retaining the ability to use the file's methods like Copy() and what not.
I would recommend using a calculated property and Split-Path -NoQualifier; e.g.:
Get-ChildItem -File | Select-Object `
#{Name = "NameNoQualifier"; Expression = {Split-Path $_.FullName -NoQualifier}},
Length
For help on calculated properties, see the help for Select-Object.
(Aside: To correct your terminology a bit, this is not modifying objects non-destructively but rather outputting new objects containing the properties you want formatted how you want them.)

Compare-Object Output Format

I have two CSV files I need to compare. Both of which may (or may not) contain an additional delimited field (delimited via a "|" character), like so:
(new.csv)
Title,Key,Value
Jason,Son,Hair=Red|Eyes=Blue
James,Son,Hair=Brown|Eyes=Green
Ron,Father,Hair=Black
Susan,Mother,Hair=Black|Eyes=Brown|Dress=Green
(old.csv)
Title,Key,Value
Jason,Son,Hair=Purple|Eyes=Blue
James,Son,Hair=Brown|Eyes=Green
Ron,Father,Hair=Purple
Susan,Mother,Hair=Black|Eyes=Brown|Dress=Blue
My problem comes in when I attempt to compare the two files...
$fileNew = "new.csv"
$fileOld = "old.csv"
$fileDiffOutputFile = "diff.txt"
$csvNewLog = (Import-CSV ($fileNew))
$csvOldLog = (Import-CSV ($fileOld))
$varDifferences = Compare-Object $csvOldLog $csvNewLog -property Title,Value
$varDifferences | Group-Object -Property Title | % {New-Object -TypeName PSObject -Property #{ NewValue=($_.group[0].Value); Title=$_.name; OldValue=($_.group[1].Value) } } | Out-File $fileDiffOutputFile -Append
Resulting in this output:
(diff.txt)
OldValue Title NewValue
-------- ----- --------
Hair=Purple|Eyes=Blue Jason Hair=Red|Eyes=Blue
Hair=Purple Ron Hair=Black
Hair=Black|Eyes=Brown|D... Susan Hair=Black|Eyes=Brown|...
Some of the values are inevitably going to extend out past the max length of the column, as it does with Susan above.
So, my question could have a couple of solutions that I can think of:
Is there an easier way to isolate the values so that I only pull out the changed values, and not the entire string of delimited values?
If not, is it possible to get the format to show the entire string (including the unchanged values part of the delimited string) instead?
If you include a format-table -wrap in your last line, like so?
$fileNew = "new.csv"
$fileOld = "old.csv"
$fileDiffOutputFile = "diff.txt"
$csvNewLog = (Import-CSV ($fileNew))
$csvOldLog = (Import-CSV ($fileOld))
$varDifferences = Compare-Object $csvOldLog $csvNewLog -property Title,Value
$varDifferences | Group-Object -Property Title | % {New-Object -TypeName PSObject -Property #{ NewValue=($_.group[0].Value); Title=$_.name; OldValue=($_.group[1].Value) } } | Format-Table -wrap | Out-File $fileDiffOutputFile -Append