I am having some difficulty getting an Export-Csv to work. I am creating an array like this...
[pscustomobject] #{
Servername = $_.Servername
Name = $_.Servername
Blk = ""
Blk2 = ""
Method = "RDP"
Port = "3389"
}
The issue I have is when I try to export that to a CSV I get garbage that looks like this...
"9e210fe47d09416682b841769c78b8a3",,,,,
I have read a ton of articles addressing this issue, but I just don't understand how to get the data right.
For testing, I built a CSV file w/ the servernames, and read it in, and the following works in PS4:
$serverList = import-csv "datafile.csv"
$AllObjects = #()
$serverList | ForEach-Object {
$AllObjects += [pscustomobject]#{
Servername = $_.Servername
Name = $_.Servername
Blk = ""
Blk2 = ""
Method = "RDP"
Port = "3389"
}
}
$AllObjects | Export-Csv -Path "outfile.csv" -NoTypeInformation
This happens when you try to pipe out from any of the Format-* commands.
The Format-List, Format-Table and Format-Wide cmdlets are special in PowerShell, in that they're meant to consume the pipeline, and transform it for display in the console. So, you can't pipe from FL, FT or FW into Export-csv. As Don Jones says "Format On the Right".
Don't believe me? Observe, as I run Get-Process, send it through Format-Table and then convert to Csv.
gps | ft | ConvertTo-Csv
#TYPE Microsoft.PowerShell.Commands.Internal.Format.FormatStartData
"ClassId2e4f51ef21dd47e99d3c952918aff9cd","pageHeaderEntry","pageFooterEntry","autosizeInfo","shapeInfo","groupingEntry"
"033ecb2bc07a4d43b5ef94ed5a35d280",,,,"Microsoft.PowerShell.Commands.Internal.Format.TableHeaderInfo",
"9e210fe47d09416682b841769c78b8a3",,,,,
"27c87ef9bbda4f709f6b4002fa4af63c",,,,,
"27c87ef9bbda4f709f6b4002fa4af63c",,,,,
"27c87ef9bbda4f709f6b4002fa4af63c",,,,,
It's even the same string! Why do we get this string? I really wish I knew, but I think it has something to do with the way the Format-* commands convert objects into text instructions for display in the console.
If you REALLY love the way your Format-Table looks, send it to Out-File, which is the only way to redirect the output of a Format-* command.
Another message is to use Tee-Object to dump a copy of your pipe to a variable, and then send that out to Export.
Get-Process | Tee-Object -Variable ExportMe | Format-Table
$exportMe | export-Csv .\Export.csv
I call this the 'have your cake and eat it too approach'.
Use Select-Object to prevent the bad CSV export.
Example:
Get-Services | Select-Object * | export-csv -Path C:\log.csv
Related
This question already has an answer here:
Not all properties displayed
(1 answer)
Closed 1 year ago.
This is a follow-up question from PowerShell | EVTX | Compare Message with Array (Like)
I changed the tactic slightly, now I am collecting all the services installed,
$7045 = Get-WinEvent -FilterHashtable #{ Path="1system.evtx"; Id = 7045 } | select
#{N=’Timestamp’; E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
Id,
#{N=’Machine Name’; E={$_.MachineName}},
#{N=’Service Name’; E={$_.Properties[0].Value}},#{N=’Image Path’;E=$_.Properties[1].Value}},
#{N=’RunAsUser’; E={$_.Properties[4].Value}},#{N=’Installed By’; E={$_.UserId}}
Now I match each object for any suspicious traits and if found, I add a column 'Suspicious' with the value 'Yes'. This is because I want to leave the decision upto the analyst and pretty sure the bad guys might use something we've not seen before.
foreach ($Evt in $7045)
{
if ($Evt.'Image Path' -match $sus)
{
$Evt | Add-Member -MemberType NoteProperty -Name 'Suspicious' -Value 'Yes'
}
}
Now, I'm unable to get PowerShell to display all columns unless I specifically Select them
$7045 | Format-Table
Same goes for CSV Export. The first two don't include the Suspicious Column but the third one does but that's because I'm explicitly asking it to.
$7045 | select * | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Select-Object Timestamp, Id, 'Machine Name', 'Service Name', 'Image Path', 'RunAsUser', 'Installed By', Suspicious | Export-Csv -Path test.csv -NoTypeInformation
I read the Export-CSV documentation on MS. Searched StackOverFlow for some tips, I think it has something to do with PS checking the first Row and then compares if the property exists for the second row and so on.
Thank you
The issue you're experiencing is partially because of how objects are displayed to the console, the first object's Properties determines the displayed Properties (Columns) to the console.
The bigger problem though, is that Export-Csv will not export those properties that do not match with first object's properties unless they're explicitly added to the remaining objects or the objects are reconstructed, for this one easy way is to use Select-Object as you have pointed out in the question.
Given the following example:
$test = #(
[pscustomobject]#{
A = 'ValA'
}
[pscustomobject]#{
A = 'ValA'
B = 'ValB'
}
[pscustomobject]#{
C = 'ValC'
D = 'ValD'
E = 'ValE'
}
)
Format-Table will not display the properties B to E:
$test | Format-Table
A
-
ValA
ValA
Format-List can display the objects properly, this is because each property with it's corresponding value has it's own console line in the display:
PS /> $test | Format-List
A : ValA
A : ValA
B : ValB
C : ValC
D : ValD
E : ValE
Export-Csv and ConvertTo-Csv will also miss properties B to E:
$test | ConvertTo-Csv
"A"
"ValA"
"ValA"
You have different options as a workaround for this, you could either add the Suspicious property to all objects and for those events that are not suspicious you could add $null as Value.
Another workaround is to use Select-Object explicitly calling the Suspicious property (this works because you know the property is there and you know it's Name).
If you did not know how many properties your objects had, a dynamic way to solve this would be to discover their properties using the PSObject intrinsic member.
using namespace System.Collections.Generic
function ConvertTo-NormalizedObject {
[CmdletBinding()]
param(
[Parameter(ValueFromPipeline, Mandatory)]
[object[]] $InputObject
)
begin {
$list = [List[object]]::new()
$props = [HashSet[string]]::new([StringComparer]::InvariantCultureIgnoreCase)
}
process {
foreach($object in $InputObject) {
$list.Add($object)
foreach($property in $object.PSObject.Properties) {
$null = $props.Add($property.Name)
}
}
}
end {
$list | Select-Object ([object[]] $props)
}
}
Usage:
# From Pipeline
$test | ConvertTo-NormalizedObject | Format-Table
# From Positional / Named parameter binding
ConvertTo-NormalizedObject $test | Format-Table
Lastly, a pretty easy way of doing it thanks to Select-Object -Unique:
$prop = $test.ForEach{ $_.PSObject.Properties.Name } | Select-Object -Unique
$test | Select-Object $prop
Using $test for this example, the result would become:
A B C D E
- - - - -
ValA
ValA ValB
ValC ValD ValE
Continuing from my previous answer, you can add a column Suspicious straight away if you take out the Where-Object filter and simply add another calculated property to the Select-Object cmdlet:
# create a regex for the suspicious executables:
$sus = '(powershell|cmd|psexesvc)\.exe'
# alternatively you can join the array items like this:
# $sus = ('powershell.exe','cmd.exe','psexesvc.exe' | ForEach-Object {[regex]::Escape($_)}) -join '|'
$7045 = Get-WinEvent -FilterHashtable #{ LogName = 'System';Id = 7045 } |
Select-Object Id,
#{N='Timestamp';E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
#{N='Machine Name';E={$_.MachineName}},
#{N='Service Name'; E={$_.Properties[0].Value}},
#{N='Image Path'; E={$_.Properties[1].Value}},
#{N='RunAsUser'; E={$_.Properties[4].Value}},
#{N='Installed By'; E={$_.UserId}},
#{N='Suspicious'; E={
if ($_.Properties[1].Value -match $sus) { 'Yes' } else {'No'}
}}
$7045 | Export-Csv -Path 'X:\Services.csv' -UseCulture -NoTypeInformation
Because you have many columns, this will not fit the console width anymore if you do $7045 | Format-Table, but the CSV file will hold all columns you wanted.
I added switch -UseCulture to the Export-Csv cmdlet, which makes sure you can simply double-click the csv file so it opens correctly in your Excel.
As sidenote: Please do not use those curly so-called 'smart-quotes' in code as they may lead to unforeseen errors. Straighten these ’ thingies and use normal double or single quotes (" and ')
I am trying to use below code with date range as Where-Object filter but that is slowing down my output speed as well as unable to export to csv.
$Prids = get-content -Path C:\Temp\sqltest.txt
foreach ($prid in $prids){
$filterDate = [datetime]::Today.AddDays(-22)
Get-CdPac2000Problems -PId $Prid | Where-Object {$_.ClosedDate.Date -ge $filterDate} |ft PID,ClosedDate,ClosedByELID,ResponsibleGroup,ReferredDate -autosize
}
How can I change Where-Object to parameter that looks something like -closedate $variable like I did with -PID? The biggest struggle for me is to creating a datetime variable.
First of, you should move the $filterDateout of the loop, second you should NEVER use Format-Table, and ABSOLUTELY NEVER in a loop.
Try this:
$filterDate = [datetime]::Today.AddDays(-22)
$Prids = get-content -Path C:\Temp\sqltest.txt
$Result = foreach ($prid in $prids){
Get-CdPac2000Problems -PId $Prid | Where-Object {(Get-Date $_.ClosedDate.Date) -ge $filterDate}
}
$Result | Format-Table PID,ClosedDate,ClosedByELID,ResponsibleGroup,ReferredDate -autosize
One of the most beautyfull things in PowerShell is, that you can interfere directly with all objects. All methods and properties are preserved, when passing the objects to a variable or to the next command in a pipe. Format-Table converts objects to a table - something human readable and something stripped from methods and life.
Having a simple issue that's only affecting export-csv output, out-gridview and results to the console are fine. Looking to capture the top 5 processes by "handles" on a set of servers.
Code is as follows:
$Servers = "Server1", "Server2", "Server3"
$OutArray = #()
ForEach ($Item in $Servers)
$Top5 = Get-Process -Computer $Item | Sort Handles -descending |Select -First 5
$OutArray += New-Object PSObject -property # {
Server = $Item
Top5 = $Top5
} #OutArray
} #ForEach
$OutArray | Export-csv Test.csv
The results of which come out looking fine via console as follows
Server Top5
------ ----
SERVER1 {#{ProcessName=svchost.exe; PercentCpuLoad=13.79}, #{ProcessName=services.exe; PercentCpuLoad=11.4}, #{ProcessName=WmiPrvSE.exe; PercentCpuLoad=10.03}, #{ProcessName=irfilcol.exe; PercentCpuLoad=9.79}...}
...However, in the csv they show as follows:
Server Top5
Server1 System.Object[]
Server2 System.Object[]
Server3 System.Object[]
I'm thinking it's because the $Top5 variable is an variable with multiple properties (5 each) for one server. How would do I correct the code so that export-csv shows the actual values?
any help appreciated!
I would like the csv results to look like the following that's shown in GRIDVIEW
Using the suggestion from BenH to review the post from Powershell legend Boe Prox, I now have the following working:
$Top5 = Get-Process -Computer $Item | Sort Handles -descending |Select -expand Handles | |Select -First 5
$new = [pscustomobject]#{ Top5 = (#($Top5) -join ',')
}
Just about got this working now:
i'd like to add more piece of formatting, where the Top5Processes have the actual CPU % used in (brackets) right now, I've got the following for output
Top2Proc Top2CPU
services.exe,BESClient.exe 32.76,16.6
However, it would be nicer output-wise, if i could combine the above two values into one, so it looks like this:
Top2Proc
Services(32.76), BesClient.exe(16.6)
Any idea how that would be done?
Use Select-Object to turn your process objects into strings before piping them to Export-Csv:
$OutArray |Select-Object Server,#{Expression={$_.Top5.Name -join ';'}} |Export-Csv test.csv
If you want that table to appear in your csv file then you would need to format the string Top5 property as such. Using Out-String will do just that
Sends objects to the host as a series of strings.
So a simple change should get you what you want.
$Top5 = Get-Process -Computer $Item |
Sort Handles -descending |
Select -First 5 |
Out-String
It will look a little ugly when not displayed with a mono-space font much like you see in Out-GridView. Also consider using .Trim() to remove the leading and trailing whitespace on your $top5.
There are other ways to tackle this. You could use the above in conjunction with Format-Table / Format-List depending what you want. In general if you want the output to be saved as it is displayed in host Out-String is something to test with.
I would have tried to add one row for each process with a the first column being the computer name. That way you would have better structured output that can be sorted or queried as needed.
ComputerName ProcessName Handles
------------ ----------- -------
Computer1 avp 54639
Computer1 OUTLOOK 7708
Computer1 RDTabs 6108
Computer1 svchost 3160
Computer1 chrome 2530
Keep in mind that you can use other methods to export this data while keeping the objects entact. Really depends the data recipeint but remeber there are other cmdlets like Export-CLIMXL and ConvertTo-JSON | Set-Content.
I have a big file which content somme blocks of text data like this.
[SERVER1]
LIBELLE=DATA, SOMME DATA
VARIABLES=A,B,C,D,E
PHYSICAL NAME=E:\SOMME\PATH\FILE.INI
ARTICLE SIZE=50
MACHINE=SOME SERVER
PARAMETER =
OPTION =
[SERVER2]
LIBELLE=DATA2, SOMME DATA2
VARIABLES=A,B,C,D,E
PHYSICAL NAME=Z:\SOMME\PATH\FILE2.INI
ARTICLE SIZE=150
MACHINE=SOME SERVER XY
PARAMETER =
OPTION 1 = VOID
OPTION 2 =
OPTION 3 =
OPTION 4 =
OPTION 5 =
What i would like to do is retrieving every block [SERVERX] and put it into a CSV file in this format (horizontally)
ColumnA__ | ColumnB (LIBELLE)___ | ColumnC (VARIABLES) | ColumnD ect...
[SERVER1] | DATA1, SOMME DATA1 | A,B,C,D,E___________ | Ect...
[SERVER2] | DATA2, SOMME DATA2 | A,B,C,D,E___________ | Ect...
I've tried this, the output work as i want but it need to be automated and exported to scv, which doesn't work for me.
$mydata = Get-Content my_file.txt
write-Host $mydata[0] $mydata[1] $mydata[2] $mydata[3] $mydata[4] $mydata[5] | Export-Csv -Path $rep\results.csv -Force -UseCulture -NoTypeInformation
Tried also somthing with select-string but i dont know if this is the right way to do my job..
select-String -path $my_file -Pattern '\[*\]', 'IDENTIFIANTS=','LIBELLE=','VARIABLES=' | Select-Object -Property LineNumber, Line | Export-Csv -Path $rep\results.csv -Force -UseCulture -NoTypeInformation
Thanks for your advices.
So, I would read the whole file in as a multi-line string using the -Raw switch for Get-Content. Then split the file up based on the [ character to denote records. The get the properties from the ConvertFrom-StringData cmdlet (have to prepend "SERVER=" to each record), and make an object from it. Then we find out what all properties any given record can have, make sure to add them all to the first record if it doesn't have it (this is done because when you export to CSV it bases the columns off of the first entries' property list). Then you can export a CSV.
$Data = (Get-Content my_file.txt -Raw) -split "(\[[^[]+)" | ?{![string]::IsNullOrWhiteSpace($_)}
$Records = $Data -replace '\\','\\'|%{$Record="SERVER="+$_.trim()|ConvertFrom-StringData;New-Object PSObject -Prop $Record}
$Props = $Records|%{$_.psobject.properties.name}|select -Unique
$Props | Where{$_ -notin $Records[0].PSObject.Properties.Name}|%{Add-Member -InputObject $Records[0] -NotepropertyName $_ -NotepropertyValue $Null}
$Records|Export-CSV .\my_file.csv -notype
Edit: For those of you out there running PowerShell 2.0 (3 versions out of date at this point in time), you can't use the -Raw parameter. Here's the alternative:
$Data = (Get-Content my_file.txt) -Join "`r`n" -split "(\[[^[]+)" | ?{![string]::IsNullOrWhiteSpace($_)}
Alternative: Thanks #Matt for the suggestion, it is always good to have a different point of view on these things. As Matt suggested, you can use Out-String to combine the array of strings that Get-Content generates, and end up with a single multi-line string. Here's the usage!
$Data = (GC my_file.txt | Out-String) -split "(\[[^[]+)" | ?{![string]::IsNullOrWhiteSpace($_)}
The below command will output the correct information to the screen, however when exporting to a csv file it looks nothing like what is shown on the screen. I have been able to export with other cmdlets, Get-DistributionGroup seems to corrupt the data when using with export.csv.
Outputting to screen works great!
Get-DistributionGroup | Where {$_.emailaddresses -like "*#adomain.com*"} | FT Name,Alias,EmailAddresses
Exporting to csv doesnt work correctly
Get-DistributionGroup | Where {$_.emailaddresses -like "*#adomain.com*"} | FT Name,Alias,EmailAddresses | Export-csv C:/thisis.csv
Instead of Format-Table (alias: ft) you should use Select-Object. Export-Csv expects objects as input, not formating instructions.
Format-Table, by definition, will convert objects to something that looks well in your output, but it's one way trip: you loose original objects as a part of the process.
Select-Object on the other hand can create objects with subset of properties, so that cmdlets like Export-Csv can still use data from original source.
EDIT
Thing I missed originally: you try to export a property that is a collection (EmailAddresses). That won't work unless you 'flatten' the collection first:
Get-DistributionGroup | Where {$_.emailaddresses -like "*#adomain.com*"} | select Name,Alias, #{
Name = 'EmailAddresses'
Expression = { $_.EmailAddresses -join '|' }
}
Thank you the answer has been resolved.
BartekB's answer
Get-DistributionGroup | Where {$.emailaddresses -like "*#adomain.com*"} | select Name,Alias, #{
Name = 'EmailAddresses'
Expression = { $.EmailAddresses -join '|' }
}