The result of Get-Process -Name explorer | select Handles,WS,CPU,Id,ProcessName | ft -HideTableHeaders returns the following output:
2623 255336448 178.125 10080 explorer
In order to ingest this data into a third party system, I need to pipe delimit the result as such:
2623|255336448|178.125|10080|explorer
What is the best way to achieve this?
How about:
(Get-Process explorer |
Select-Object Handles,Ws,CPU,ID,ProcessName |
ConvertTo-Csv -Delimiter '|' -NoTypeInformation |
Select-Object -Skip 1) -replace '"',''
Only use ft (Format-Table) for easy viewing in the PowerShell console (it's not good for sending data to other applications, because then you would have to undo the formatting - so don't format in the first place).
To offer a more concise (and slightly faster) alternative to Bill Stewart's helpful answer:
Get-Process -Name explorer | ForEach-Object {
$(foreach ($prop in 'Handles', 'WS', 'CPU', 'Id', 'ProcessName') { $_.$prop }) -join '|'
}
foreach ($prop in 'Handles', 'WS', 'CPU', 'Id', 'ProcessName') { $_.$prop } outputs all the properties of interest for each process object ($_, the input object at hand provided by the ForEach-Object cmdlet).
$(...) collects them as an [object[]] array, which ...
... enables that array's use as the LHS of the -join operator in order to join the array's elements with | as the separator to form a single string.
Overall, you get a single string per input object, as desired.
Related
This question already has an answer here:
Not all properties displayed
(1 answer)
Closed 1 year ago.
This is a follow-up question from PowerShell | EVTX | Compare Message with Array (Like)
I changed the tactic slightly, now I am collecting all the services installed,
$7045 = Get-WinEvent -FilterHashtable #{ Path="1system.evtx"; Id = 7045 } | select
#{N=’Timestamp’; E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
Id,
#{N=’Machine Name’; E={$_.MachineName}},
#{N=’Service Name’; E={$_.Properties[0].Value}},#{N=’Image Path’;E=$_.Properties[1].Value}},
#{N=’RunAsUser’; E={$_.Properties[4].Value}},#{N=’Installed By’; E={$_.UserId}}
Now I match each object for any suspicious traits and if found, I add a column 'Suspicious' with the value 'Yes'. This is because I want to leave the decision upto the analyst and pretty sure the bad guys might use something we've not seen before.
foreach ($Evt in $7045)
{
if ($Evt.'Image Path' -match $sus)
{
$Evt | Add-Member -MemberType NoteProperty -Name 'Suspicious' -Value 'Yes'
}
}
Now, I'm unable to get PowerShell to display all columns unless I specifically Select them
$7045 | Format-Table
Same goes for CSV Export. The first two don't include the Suspicious Column but the third one does but that's because I'm explicitly asking it to.
$7045 | select * | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Select-Object Timestamp, Id, 'Machine Name', 'Service Name', 'Image Path', 'RunAsUser', 'Installed By', Suspicious | Export-Csv -Path test.csv -NoTypeInformation
I read the Export-CSV documentation on MS. Searched StackOverFlow for some tips, I think it has something to do with PS checking the first Row and then compares if the property exists for the second row and so on.
Thank you
The issue you're experiencing is partially because of how objects are displayed to the console, the first object's Properties determines the displayed Properties (Columns) to the console.
The bigger problem though, is that Export-Csv will not export those properties that do not match with first object's properties unless they're explicitly added to the remaining objects or the objects are reconstructed, for this one easy way is to use Select-Object as you have pointed out in the question.
Given the following example:
$test = #(
[pscustomobject]#{
A = 'ValA'
}
[pscustomobject]#{
A = 'ValA'
B = 'ValB'
}
[pscustomobject]#{
C = 'ValC'
D = 'ValD'
E = 'ValE'
}
)
Format-Table will not display the properties B to E:
$test | Format-Table
A
-
ValA
ValA
Format-List can display the objects properly, this is because each property with it's corresponding value has it's own console line in the display:
PS /> $test | Format-List
A : ValA
A : ValA
B : ValB
C : ValC
D : ValD
E : ValE
Export-Csv and ConvertTo-Csv will also miss properties B to E:
$test | ConvertTo-Csv
"A"
"ValA"
"ValA"
You have different options as a workaround for this, you could either add the Suspicious property to all objects and for those events that are not suspicious you could add $null as Value.
Another workaround is to use Select-Object explicitly calling the Suspicious property (this works because you know the property is there and you know it's Name).
If you did not know how many properties your objects had, a dynamic way to solve this would be to discover their properties using the PSObject intrinsic member.
using namespace System.Collections.Generic
function ConvertTo-NormalizedObject {
[CmdletBinding()]
param(
[Parameter(ValueFromPipeline, Mandatory)]
[object[]] $InputObject
)
begin {
$list = [List[object]]::new()
$props = [HashSet[string]]::new([StringComparer]::InvariantCultureIgnoreCase)
}
process {
foreach($object in $InputObject) {
$list.Add($object)
foreach($property in $object.PSObject.Properties) {
$null = $props.Add($property.Name)
}
}
}
end {
$list | Select-Object ([object[]] $props)
}
}
Usage:
# From Pipeline
$test | ConvertTo-NormalizedObject | Format-Table
# From Positional / Named parameter binding
ConvertTo-NormalizedObject $test | Format-Table
Lastly, a pretty easy way of doing it thanks to Select-Object -Unique:
$prop = $test.ForEach{ $_.PSObject.Properties.Name } | Select-Object -Unique
$test | Select-Object $prop
Using $test for this example, the result would become:
A B C D E
- - - - -
ValA
ValA ValB
ValC ValD ValE
Continuing from my previous answer, you can add a column Suspicious straight away if you take out the Where-Object filter and simply add another calculated property to the Select-Object cmdlet:
# create a regex for the suspicious executables:
$sus = '(powershell|cmd|psexesvc)\.exe'
# alternatively you can join the array items like this:
# $sus = ('powershell.exe','cmd.exe','psexesvc.exe' | ForEach-Object {[regex]::Escape($_)}) -join '|'
$7045 = Get-WinEvent -FilterHashtable #{ LogName = 'System';Id = 7045 } |
Select-Object Id,
#{N='Timestamp';E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
#{N='Machine Name';E={$_.MachineName}},
#{N='Service Name'; E={$_.Properties[0].Value}},
#{N='Image Path'; E={$_.Properties[1].Value}},
#{N='RunAsUser'; E={$_.Properties[4].Value}},
#{N='Installed By'; E={$_.UserId}},
#{N='Suspicious'; E={
if ($_.Properties[1].Value -match $sus) { 'Yes' } else {'No'}
}}
$7045 | Export-Csv -Path 'X:\Services.csv' -UseCulture -NoTypeInformation
Because you have many columns, this will not fit the console width anymore if you do $7045 | Format-Table, but the CSV file will hold all columns you wanted.
I added switch -UseCulture to the Export-Csv cmdlet, which makes sure you can simply double-click the csv file so it opens correctly in your Excel.
As sidenote: Please do not use those curly so-called 'smart-quotes' in code as they may lead to unforeseen errors. Straighten these ’ thingies and use normal double or single quotes (" and ')
In our company there are many users and many applications with restricted access and database with evidence of those accessess. I don´t have access to that database, but what I do have is automatically generated (once a day) csv file with all accessess of all my users. I want them to have a chance to check their access situation so i am writing a simple powershell script for this purpose.
CSV:
user;database1_dat;database2_dat;database3_dat
john;0;0;1
peter;1;0;1
I can do:
import-csv foo.csv | where {$_.user -eq $user}
But this will show me original ugly headres (with "_dat" suffix). Can I delete last four characters from every header which ends with "_dat", when i can´t predict how many headers will be there tomorrow?
I am aware of calculated property like:
Select-Object #{ expression={$_.database1_dat}; label='database1' }
but i have to know all column names for that, as far as I know.
Am I convicted to "overingeneer" it by separate function and build whole "calculated property expression" from scratch dynamically or is there a simple way i am missing?
Thanks :-)
Assuming that file foo.csv fits into memory as a whole, the following solution performs well:
If you need a memory-throttled - but invariably much slower - solution, see Santiago Squarzon's helpful answer or the alternative approach in the bottom section.
$headerRow, $dataRows = (Get-Content -Raw foo.csv) -split '\r?\n', 2
# You can pipe the result to `where {$_.user -eq $user}`
ConvertFrom-Csv ($headerRow -replace '_dat(?=;|$)'), $dataRows -Delimiter ';'
Get-Content -Raw reads the entire file into memory, which is much faster than reading it line by line (the default).
-split '\r?\n', 2 splits the resulting multi-line string into two: the header line and all remaining lines.
Regex \r?\n matches a newline (both a CRLF (\r\n) and a LF-only newline (\n))
, 2 limits the number of tokens to return to 2, meaning that splitting stops once the 1st token (the header row) has been found, and the remainder of the input string (comprising all data rows) is returned as-is as the last token.
Note the $null as the first target variable in the multi-assignment, which is used to discard the empty token that results from the separator regex matching at the very start of the string.
$headerRow -replace '_dat(?=;|$)'
-replace '_dat(?=;|$)' uses a regex to remove any _dat column-name suffixes (followed by a ; or the end of the string); if substring _dat only ever occurs as a name suffix (not also inside names), you can simplify to -replace '_dat'
ConvertFrom-Csv directly accepts arrays of strings, so the cleaned-up header row and the string with all data rows can be passed as-is.
Alternative solution: algorithmic renaming of an object's properties:
Note: This solution is slow, but may be an option if you only extract a few objects from the CSV file.
As you note in the question, use of Select-Object with calculated properties is not an option in your case, because you neither know the column names nor their number in advance.
However, you can use a ForEach-Object command in which you use .psobject.Properties, an intrinsic member, for reflection on the input objects:
Import-Csv -Delimiter ';' foo.csv | where { $_.user -eq $user } | ForEach-Object {
# Initialize an aux. ordered hashtable to store the renamed
# property name-value pairs.
$renamedProperties = [ordered] #{}
# Process all properties of the input object and
# add them with cleaned-up names to the hashtable.
foreach ($prop in $_.psobject.Properties) {
$renamedProperties[($prop.Name -replace '_dat(?=.|$)')] = $prop.Value
}
# Convert the aux. hashtable to a custom object and output it.
[pscustomobject] $renamedProperties
}
You can do something like this:
$textInfo = (Get-Culture).TextInfo
$headers = (Get-Content .\test.csv | Select-Object -First 1).Split(';') |
ForEach-Object {
$textInfo.ToTitleCase($_) -replace '_dat'
}
$user = 'peter'
Get-Content .\test.csv | Select-Object -Skip 1 |
ConvertFrom-Csv -Delimiter ';' -Header $headers |
Where-Object User -EQ $user
User Database1 Database2 Database3
---- --------- --------- ---------
peter 1 0 1
Not super efficient but does the trick.
I've created the following small script to remove 2++ strings from a CSV.
Each row is a log of a given person and a answer they give.
The CSV has X columns.
The column named FIRST identifies the person.
What I need to do is when I delete a row matching the answer, I also need to delete the person from the whole CSV if it had one of the two strings.
What I've made so far, removes the row of people having the answers but the person is still left in the overall CSV with other answers. I want to remove the person fully if the questions have been answered.
Can somebody help me out with making the addition or changes to make this happen?
INPUT File
FIRST,LAST,ADDR,ADDR2,GENDER,HOME,WORK
1,N/A,N/A,N/A,N/A,BAF,N/A
10005,JAS,AA,N/A,,ZAV,N/A
10007,JADE,BB,N/A,OMA,N/A,N/A
10007,JADE,N/A,RAV,N/A,N/A,N/A
10011,KIAH,N/A,N/A,BALI,BB,N/A
SCRIPT
$CSVfile = "C:\Temp\Test\Test.csv"
$CSVfile_filtered = "C:\Temp\Test\Test.csv"
$regex001 = "AA"
$regex002 = "BB"
$filterArray = #($regex001,$regex002)
Get-Content $CSVfile | Select-String -pattern $filterArray -notmatch | Set-Content $CSVfile_filtered
The file should then remove 10005, 10011 and both lines of 10007. But my version only removes one of the 10007 since it only matches one of the two patterns.
Using more of PowerShell's built-in cmdlets can make this a little easier to manage.
# Assuming searching only properties ADDR and ADDR2
$filter = 'AA','BB'
# Grouping by First and Last values to easily remove duplicates
# -match uses regex so | is needed for an OR of multiple items
Import-Csv Test.csv | Group-Object First,Last |
Where {!($_.Group.ADDR,$_.Group.ADDR2 -match ($filter -join '|'))} |
Foreach-Object Group |
Export-Csv output.csv -NoType
You would think strictly using text manipulation would be simpler, but it adds other scenarios to consider:
You will need to track users that have duplicate entries and potentially back track to remove them (if not grouping). This could require reading the file contents twice.
Your header row could match the string you want to filter so you will need to add it to the output if filtering removes it.
Keeping the scenarios above in mind, you can still use a grouping concept:
$filter = 'AA','BB'
$file = Get-Content Test.csv
# $file[0] is the header row
# -split string uses regex and splits at the second comma
# -split results' [0] element is First,Last values
$file[0],($file |
Select-Object -Skip 1 |
Group-Object {($_ -split '(?<=^[^,]*,[^,]*),')[0]} |
where {!($_.Group -match ($filter -join '|'))} |
Foreach-Object Group) | Set-Content output.csv
If I got it right you could do something like this:
$SearchPattern = 'AA', 'BB'
$INPUTCSV = #'
FIRST,LAST,ADDR,ADDR2,GENDER,HOME,WORK
1,N/A,N/A,N/A,N/A,BAF,N/A
10005,JAS,AA,N/A,,ZAV,N/A
10007,JADE,BB,N/A,OMA,N/A,N/A
10007,JADE,N/A,RAV,N/A,N/A,N/A
10011,KIAH,N/A,N/A,BALI,BB,N/A
'# | ConvertFrom-Csv
$ActualSearchPattern =
$INPUTCSV |
Where-Object {
$_.LAST -in $SearchPattern -or
$_.ADDR -in $SearchPattern -or
$_.ADDR2 -in $SearchPattern -or
$_.GENDER -in $SearchPattern -or
$_.HOME -in $SearchPattern -or
$_.Work -in $SearchPattern
} |
Select-Object -ExpandProperty FIRST
$INPUTCSV |
Where-Object -Property FIRST -NotIn -Value $ActualSearchPattern |
Format-Table -AutoSize
There might be more sophisticated or more elegant ways but I cannot think about one at the moment. ;-)
There is a nice PowerShell module you can use to manipulate the content of a csv or xlsx file: ImportExcel
This give you a lot of options to manipulate the sheets, columns etc.
How to find the count of unique numbers in a CSV file? When I use the following command in PowerShell ISE
1,2,3,4,2 | Sort-Object | Get-Unique
I can get the unique numbers but I'm not able to get this to work with CSV files. If for example I use
$A = Import-Csv C:\test.csv | Sort-Object | Get-Unique
$A.Count
it returns 0. I would like to count unique numbers for all the files in a given folder.
My data looks similar to this:
Col1,Col2,Col3,Col4
5,,7,4
0,,9,
3,,5,4
And the result should be 6 unique values (preferably written inside the same CSV file).
Or would it be easier to do it with Notepad++? So far I have found examples only on how to count the unique rows.
You can try the following (PSv3+):
PS> (Import-CSV C:\test.csv |
ForEach-Object { $_.psobject.properties.value -ne '' } |
Sort-Object -Unique).Count
6
The key is to extract all property (column) values from each input object (CSV row), which is what $_.psobject.properties.value does;
-ne '' filters out empty values.
Note that, given that Sort-Object has a -Unique switch, you don't need Get-Unique (you need Get-Unique only if your input already is sorted).
That said, if your CSV file is structured as simply as yours, you can speed up processing by reading it as a text file (PSv2+):
PS> (Get-Content C:\test.csv | Select-Object -Skip 1 |
ForEach-Object { $_ -split ',' -ne '' } |
Sort-Object -Unique).Count
6
Get-Content reads the CSV file as a line of strings.
Select-Object -Skip 1 skips the header line.
$_ -split ',' -ne '' splits each line into values by commas and weeds out empty values.
As for what you tried:
Import-CSV C:\test.csv | Sort-Object | Get-Unique:
Fundamentally, Sort-Object emits the input objects as a whole (just in sorted order), it doesn't extract property values, yet that is what you need.
Because no -Property argument is passed to Sort-Object to base the sorting on, it compares the custom objects that Import-Csv emits as a whole, by their .ToString() values, which happen to be empty[1]
, so they all compare the same, and in effect no sorting happens.
Similarly, Get-Unique also determines uniqueness by .ToString() here, so that, again, all objects are considered the same and only the very first one is output.
[1] This may be surprising, given that using a custom object in an expandable string does yield a value: compare $obj = [pscustomobject] #{ foo ='bar' }; $obj.ToString(); '---'; "$obj". This inconsistency is discussed in this GitHub issue.
I have a big file which content somme blocks of text data like this.
[SERVER1]
LIBELLE=DATA, SOMME DATA
VARIABLES=A,B,C,D,E
PHYSICAL NAME=E:\SOMME\PATH\FILE.INI
ARTICLE SIZE=50
MACHINE=SOME SERVER
PARAMETER =
OPTION =
[SERVER2]
LIBELLE=DATA2, SOMME DATA2
VARIABLES=A,B,C,D,E
PHYSICAL NAME=Z:\SOMME\PATH\FILE2.INI
ARTICLE SIZE=150
MACHINE=SOME SERVER XY
PARAMETER =
OPTION 1 = VOID
OPTION 2 =
OPTION 3 =
OPTION 4 =
OPTION 5 =
What i would like to do is retrieving every block [SERVERX] and put it into a CSV file in this format (horizontally)
ColumnA__ | ColumnB (LIBELLE)___ | ColumnC (VARIABLES) | ColumnD ect...
[SERVER1] | DATA1, SOMME DATA1 | A,B,C,D,E___________ | Ect...
[SERVER2] | DATA2, SOMME DATA2 | A,B,C,D,E___________ | Ect...
I've tried this, the output work as i want but it need to be automated and exported to scv, which doesn't work for me.
$mydata = Get-Content my_file.txt
write-Host $mydata[0] $mydata[1] $mydata[2] $mydata[3] $mydata[4] $mydata[5] | Export-Csv -Path $rep\results.csv -Force -UseCulture -NoTypeInformation
Tried also somthing with select-string but i dont know if this is the right way to do my job..
select-String -path $my_file -Pattern '\[*\]', 'IDENTIFIANTS=','LIBELLE=','VARIABLES=' | Select-Object -Property LineNumber, Line | Export-Csv -Path $rep\results.csv -Force -UseCulture -NoTypeInformation
Thanks for your advices.
So, I would read the whole file in as a multi-line string using the -Raw switch for Get-Content. Then split the file up based on the [ character to denote records. The get the properties from the ConvertFrom-StringData cmdlet (have to prepend "SERVER=" to each record), and make an object from it. Then we find out what all properties any given record can have, make sure to add them all to the first record if it doesn't have it (this is done because when you export to CSV it bases the columns off of the first entries' property list). Then you can export a CSV.
$Data = (Get-Content my_file.txt -Raw) -split "(\[[^[]+)" | ?{![string]::IsNullOrWhiteSpace($_)}
$Records = $Data -replace '\\','\\'|%{$Record="SERVER="+$_.trim()|ConvertFrom-StringData;New-Object PSObject -Prop $Record}
$Props = $Records|%{$_.psobject.properties.name}|select -Unique
$Props | Where{$_ -notin $Records[0].PSObject.Properties.Name}|%{Add-Member -InputObject $Records[0] -NotepropertyName $_ -NotepropertyValue $Null}
$Records|Export-CSV .\my_file.csv -notype
Edit: For those of you out there running PowerShell 2.0 (3 versions out of date at this point in time), you can't use the -Raw parameter. Here's the alternative:
$Data = (Get-Content my_file.txt) -Join "`r`n" -split "(\[[^[]+)" | ?{![string]::IsNullOrWhiteSpace($_)}
Alternative: Thanks #Matt for the suggestion, it is always good to have a different point of view on these things. As Matt suggested, you can use Out-String to combine the array of strings that Get-Content generates, and end up with a single multi-line string. Here's the usage!
$Data = (GC my_file.txt | Out-String) -split "(\[[^[]+)" | ?{![string]::IsNullOrWhiteSpace($_)}