I am having the script to sort the excel data to descending order format and I need to save the data to another location. But while executing the below script error is showing.
Import-Csv -Path D:\Excel\details.csv | Sort-Object AMCore Content Version | Export-Csv D:\sb.csv
The error comes from an unquoted argument, Sort-Object is binding AMCore positionally as argument for the -Property parameter and has no clue on what to do with Content as the next argument.
... | Sort-Object 'AMCore Content Version' | ...
The above will be syntactically correct, however that's not enough to do a proper sorting on these Values, it would require an expression script block which would convert each value to either decimal or double to do proper sorting:
Import-Csv -Path D:\Excel\details.csv |
Sort-Object { [double] $_.'AMCore Content Version' } |
Export-Csv D:\sb.csv -NoTypeInformation
You can find an example below for testing:
$details = #'
AMCore Content Version
2.5
1.5
1
0.5
'# | ConvertFrom-Csv
$details | Sort-Object { [double] $_.'AMCore Content Version' }
Related
This question already has an answer here:
Not all properties displayed
(1 answer)
Closed 1 year ago.
This is a follow-up question from PowerShell | EVTX | Compare Message with Array (Like)
I changed the tactic slightly, now I am collecting all the services installed,
$7045 = Get-WinEvent -FilterHashtable #{ Path="1system.evtx"; Id = 7045 } | select
#{N=’Timestamp’; E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
Id,
#{N=’Machine Name’; E={$_.MachineName}},
#{N=’Service Name’; E={$_.Properties[0].Value}},#{N=’Image Path’;E=$_.Properties[1].Value}},
#{N=’RunAsUser’; E={$_.Properties[4].Value}},#{N=’Installed By’; E={$_.UserId}}
Now I match each object for any suspicious traits and if found, I add a column 'Suspicious' with the value 'Yes'. This is because I want to leave the decision upto the analyst and pretty sure the bad guys might use something we've not seen before.
foreach ($Evt in $7045)
{
if ($Evt.'Image Path' -match $sus)
{
$Evt | Add-Member -MemberType NoteProperty -Name 'Suspicious' -Value 'Yes'
}
}
Now, I'm unable to get PowerShell to display all columns unless I specifically Select them
$7045 | Format-Table
Same goes for CSV Export. The first two don't include the Suspicious Column but the third one does but that's because I'm explicitly asking it to.
$7045 | select * | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Select-Object Timestamp, Id, 'Machine Name', 'Service Name', 'Image Path', 'RunAsUser', 'Installed By', Suspicious | Export-Csv -Path test.csv -NoTypeInformation
I read the Export-CSV documentation on MS. Searched StackOverFlow for some tips, I think it has something to do with PS checking the first Row and then compares if the property exists for the second row and so on.
Thank you
The issue you're experiencing is partially because of how objects are displayed to the console, the first object's Properties determines the displayed Properties (Columns) to the console.
The bigger problem though, is that Export-Csv will not export those properties that do not match with first object's properties unless they're explicitly added to the remaining objects or the objects are reconstructed, for this one easy way is to use Select-Object as you have pointed out in the question.
Given the following example:
$test = #(
[pscustomobject]#{
A = 'ValA'
}
[pscustomobject]#{
A = 'ValA'
B = 'ValB'
}
[pscustomobject]#{
C = 'ValC'
D = 'ValD'
E = 'ValE'
}
)
Format-Table will not display the properties B to E:
$test | Format-Table
A
-
ValA
ValA
Format-List can display the objects properly, this is because each property with it's corresponding value has it's own console line in the display:
PS /> $test | Format-List
A : ValA
A : ValA
B : ValB
C : ValC
D : ValD
E : ValE
Export-Csv and ConvertTo-Csv will also miss properties B to E:
$test | ConvertTo-Csv
"A"
"ValA"
"ValA"
You have different options as a workaround for this, you could either add the Suspicious property to all objects and for those events that are not suspicious you could add $null as Value.
Another workaround is to use Select-Object explicitly calling the Suspicious property (this works because you know the property is there and you know it's Name).
If you did not know how many properties your objects had, a dynamic way to solve this would be to discover their properties using the PSObject intrinsic member.
using namespace System.Collections.Generic
function ConvertTo-NormalizedObject {
[CmdletBinding()]
param(
[Parameter(ValueFromPipeline, Mandatory)]
[object[]] $InputObject
)
begin {
$list = [List[object]]::new()
$props = [HashSet[string]]::new([StringComparer]::InvariantCultureIgnoreCase)
}
process {
foreach($object in $InputObject) {
$list.Add($object)
foreach($property in $object.PSObject.Properties) {
$null = $props.Add($property.Name)
}
}
}
end {
$list | Select-Object ([object[]] $props)
}
}
Usage:
# From Pipeline
$test | ConvertTo-NormalizedObject | Format-Table
# From Positional / Named parameter binding
ConvertTo-NormalizedObject $test | Format-Table
Lastly, a pretty easy way of doing it thanks to Select-Object -Unique:
$prop = $test.ForEach{ $_.PSObject.Properties.Name } | Select-Object -Unique
$test | Select-Object $prop
Using $test for this example, the result would become:
A B C D E
- - - - -
ValA
ValA ValB
ValC ValD ValE
Continuing from my previous answer, you can add a column Suspicious straight away if you take out the Where-Object filter and simply add another calculated property to the Select-Object cmdlet:
# create a regex for the suspicious executables:
$sus = '(powershell|cmd|psexesvc)\.exe'
# alternatively you can join the array items like this:
# $sus = ('powershell.exe','cmd.exe','psexesvc.exe' | ForEach-Object {[regex]::Escape($_)}) -join '|'
$7045 = Get-WinEvent -FilterHashtable #{ LogName = 'System';Id = 7045 } |
Select-Object Id,
#{N='Timestamp';E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
#{N='Machine Name';E={$_.MachineName}},
#{N='Service Name'; E={$_.Properties[0].Value}},
#{N='Image Path'; E={$_.Properties[1].Value}},
#{N='RunAsUser'; E={$_.Properties[4].Value}},
#{N='Installed By'; E={$_.UserId}},
#{N='Suspicious'; E={
if ($_.Properties[1].Value -match $sus) { 'Yes' } else {'No'}
}}
$7045 | Export-Csv -Path 'X:\Services.csv' -UseCulture -NoTypeInformation
Because you have many columns, this will not fit the console width anymore if you do $7045 | Format-Table, but the CSV file will hold all columns you wanted.
I added switch -UseCulture to the Export-Csv cmdlet, which makes sure you can simply double-click the csv file so it opens correctly in your Excel.
As sidenote: Please do not use those curly so-called 'smart-quotes' in code as they may lead to unforeseen errors. Straighten these ’ thingies and use normal double or single quotes (" and ')
The following code gives me the correct output to console. But I would need it in a csv file:
$array = #{}
$files = Get-ChildItem "C:\Temp\Logs\*"
foreach($file in $files){
foreach($row in (Get-Content $file | select -Last 2)){
if($row -like "Total peak job memory used:*"){
$sp_memory = $row.Split(" ")[5]
$array.Add(($file.BaseName),([double]$sp_memory))
break
}
}
}
$array.GetEnumerator() | sort Value -Descending |Format-Table -AutoSize
current output (console):
required output (csv):
In order to increase performance I would like to avoid the array and write output directly to csv (no append).
Thanks in advance!
Change your last line to this -
$array.GetEnumerator() | sort Value -Descending | select #{l='FileName'; e={$_.Name}}, #{l='Memory (MB)'; e={$_.Value }} | Export-Csv -path $env:USERPROFILE\Desktop\Output.csv -NoTypeInformation
This will give you a csv file named Output.csv on your desktop.
I am using Calculated properties to change the column headers to FileName and Memory (MB) and piping the output of $array to Export-Csv cmdlet.
Just to let you know, your variable $array is of type Hashtable which won't store duplicate keys. If you need to store duplicate key/value pairs, you can use arrays. Just suggesting! :)
I filtered by date this file data1.csv
2017.11.1,09:55,1.1,1.2,1.3,1.4,1
2017.11.2,09:55,1.5,1.6,1.7,1.8,2
I don't get a header with -NoTypeInformation:
$CutOff = (Get-Date).AddDays(-2)
$filePath = "data1.csv"
$Data = Import-Csv $filePath -Header Date,Time,A,B,C,D,E
$Data2 = $Data | Where-Object {$_.Date -as [datetime] -gt $Cutoff} | convertto-csv -NoTypeInformation -Delimiter "," | % {$_ -replace '"',''}
But when rewriting with Out-File
$Data2 | Out-File "data2.csv" -Encoding utf8 -Force
I get header back as data2.csv contains:
Date,Time,A,B,C,D,E
2017.11.2,09:55,1.5,1.6,1.7,1.8,2
Why do I have Date,Time,A,B,C,D,E ?
-NoTypeInformation is not about the header but the data type of the rows in the file. Remove it to see what shows up. From Microsoft
Omits the type information header from the output. By default, the string in the output contains #TYPE followed by the fully-qualified name of the object type.
Emphasis mine.
CSVs need headers. That is why it is making one. If you don't want to see the header in the output use Select-Object -Skip 1 to remove it.
$Data |
Where-Object {$_.Date -as [datetime] -gt $Cutoff} |
ConvertTo-CSV -NoTypeInformation -Delimiter "," |
Select-Object -Skip 1 |
% {$_ -replace '"'}
I would not pipe Out-File to itself. You could pipe to Set-Content here just as well.
I am guessing this whole process is to keep the source file in the same state just with some lines filtered out based on date. You could skip most of this just by parsing the date out in each line.
$threshold = (Get-Date).AddDays(-2)
$filePath = "c:\temp\bagel.txt"
(Get-Content $filePath) | Where-Object{
$date,$null=$_.Split(",",2)
[datetime]$date -gt $threshold
} | Set-Content $filePath
Now you don't have to worry about PowerShell CSV object structure or output since we act on the raw data of the file itself.
That will take each line of the input file and filter it out if the parsed date does not match the threshold. Change encoding on the input output cmdlets as you see necessary. What $date,$null=$_.Split(",",2) is doing is splitting the line
on the comma into 2 parts. First of which becomes $date and since this is just a filtering condition we dump the rest of the line into $null.
Properly-formed CSV files must have column headers. Your use of -NoTypeInformation in generating the CSV does not affect column headers; instead, it affects whether the PowerShell object type information is included. If you Export-CSV without -NoTypeInformation, the first line of your CSV file will have a line that looks like #TYPE System.PSCustomObject, which you don't want if you're going to open the CSV in a spreadsheet program.
If you subsequently Import-CSV, the headers (Date, Time, A, B, C) are used to create the fields of a PSObject, so that you can refer to them using the standard dot notation (e.g., $CSV[$line].Date).
The ability to specify -Header on Import-CSV is essentially a "hack" to allow the cmdlet to handle files that are comma-separated, but which did not include column headers.
I want the greatest value (mailboxSize) at the top of the file. I have a cvs as inport.
When I do the following sort cmd:
Import-Csv import.csv| Sort-Object MailboxSize,DisplayName -Descending | Export-Csv SORT.csv
I get the following result:
"DisplayName","MailboxSize"
"persone6","9941"
"persone3","8484"
"persone1","7008"
"persone4","4322"
"persone5","3106"
"persone7","27536"
"persone10","24253"
"persone8","1961"
"persone9","17076"
"persone11","17012"
"persone2","15351"
"persone12","11795"
"persone14","1156"
"persone13","1008"
But I want this as a result!
"persone7","27536"
"persone10","24253"
"persone9","17076"
"persone11","17012"
"persone2","15351"
"persone12","11795"
"persone6","9941"
"persone3","8484"
"persone1","7008"
"persone4","4322"
"persone5","3106"
"persone14","1156"
"persone13","1008"
When importing a CSV-file, all properties are made string-type. You have to cast the MailboxSize to an int before you can sort it properly. Try:
Import-Csv import.csv |
Sort-Object {[int]$_.MailboxSize}, DisplayName -Descending |
Export-Csv SORT.csv
You should also use the -NoTypeInformation switch in Export-CSV to avoid the #TYPE ..... line (first line in an exported CSV-file).
Sample:
$data = #"
"DisplayName","MailboxSize"
"persone6","9941"
"persone3","8484"
"persone1","7008"
"persone4","4322"
"persone5","3106"
"persone7","27536"
"persone10","24253"
"persone8","1961"
"persone9","17076"
"persone11","17012"
"persone2","15351"
"persone12","11795"
"persone14","1156"
"persone13","1008"
"# | ConvertFrom-Csv
$data |
Sort-Object {[int]$_.MailboxSize}, DisplayName -Descending |
Export-Csv SORT.csv -NoTypeInformation
SORT.csv
"DisplayName","MailboxSize"
"persone7","27536"
"persone10","24253"
"persone9","17076"
"persone11","17012"
"persone2","15351"
"persone12","11795"
"persone6","9941"
"persone3","8484"
"persone1","7008"
"persone4","4322"
"persone5","3106"
"persone8","1961"
"persone14","1156"
"persone13","1008"
I'm guessing the usernames are fake, but be aware that the same issue goes for DisplayName if your usernames actually was personeXX where XX is an int. Like:
persone7 27536
persone20 27536
persone13 27536
To sort them probably, you'd have to create a scriptblock for Sort-Object or create your own function to split the value and sort them correctly.
I have the following code:
$b = 1
Import-Csv c:\Awsaf\powershell\Beforenew.csv | select name, CustomAttribute1, CustomAttribute2, #{n='Counter';e={$b}} | Export-Csv -NoTypeInformation c:\Awsaf\PowerShell\afternew.csv
I want to increment the value of $b by 1. I have tried $b++, $b+=, for loop, do-while, nothing seems to be working. How can I do it?
I have also tried the following additional code, but I cannot figure out how to increase the value of $b.
$b = 0
Import-CSV c:\Awsaf\powershell\afternew.csv -Delimiter ',' | `
ForEach-Object { $_.Counter = "$b"; return $_ } | `
Export-CSV c:\awsaf\powershell\afterX.csv -Delimiter ',' -NoTypeInformation
Use explicit script scope for the variable 'b'. After increment (++) do not forget to output the value.
Here is the code that works fine for me (it is a slightly changed version of yours):
# Prepare some data for the test
Get-ChildItem | Select-Object Name | Export-Csv test1.csv -NoTypeInformation
# 1) Use explicit script scope for the variable 'b'
# 2) After increment (++) do not forget to output it
$script:b = 0
Import-Csv test1.csv |
select Name, #{n='Counter'; e={$script:b++; $script:b}} |
Export-Csv -NoTypeInformation test2.csv
This is a slight variation as I needed this in a string. Roman's curly brackets don't take a string. You need to use parentheses:
`"$($script:b++; $script:b)"`
Ohter option (although not so beautiful) is to use Add-Member. You tried it in your ForEach-Object:
gci |
select Name |
% -beg {$counter=0} {$counter++; Add-Member -in $_ -name Test NoteProperty $counter -pass }