I wrote a script to find GID's in AD, this is working fine, the issue I'm having is filtering out the blank (null lines)
$searcher=[adsisearcher]"(objectCategory=user)"
$result = $searcher.FindAll()
$result | Select-Object #{Name="DN";Expression+{$_.properties.distinguishedname}},#{Name="gid";Expression={$_.properties.gidnumber }} |Sort-Object -Property gid
I find it odd that you would be getting blank lines with that code. Can't think of a scenario where the distinguishedname of a user is null. The one issue with your code that I do see might just be a typo in your first calculated expression:
#{Name="DN";Expression+{$_.properties.distinguishedname}}
should instead be
#{Name="DN";Expression={$_.properties.distinguishedname}}
However that should have just made a syntax error that would be easily caught before execution.
Filtering Blanks
A real easy PowerShell way to deal with this is to use a simple Where-Object clause. For argument sake lets say that the GID could be empty/null.
$result | Select-Object #{Name="DN";Expression={$_.properties.distinguishedname}},#{Name="gid";Expression={$_.properties.gidnumber }} | Where-Object{$_.GID} | Sort-Object -Property gid
A null or empty value evaluates to False in PowerShell. Where-Object{$_.GID} will only allow objects with a populated property for GID to pass as output. You will get similar results from string static methods as well. These would also add readability to your code.
... | Where-Object{[string]::IsNullOrWhiteSpace($_.GID)} | ...
There is also [string]::IsNullOrEmpty()
Sort Object has a -unique property which will remove most of the blank lines, all but one as you have guessed. You could also pipe to a
where-object -ne '`n'
or something similar, let me know if I should elaborate that.
Related
This basically works
foreach ($cprev in $CopyPreventeds) {
Write-Host ("prevented copy $(($cprev)."Name")")
$cprev | Select-Object Path, Name, Length, LastWrite, DestinationNewer | Export-Csv '.\prevented.csv' -NoTypeInformation
}
But only the last output is written to the csv. How could I write all contents to a new csv with an output at the same time for the user in PowerShell.
Maybe I'm missing something?
While I appreciate a solution has already been proposed in the comments, I have to ask, given the narrow scope of the question why are we using an obscure, albeit clever technique? And/or, repeatedly invoking Export-Csv...
The question doesn't mention sparing a variable. Moreover, There doesn't appear to be a need for the ForEach loop.
$CopyPreventeds |
Select-Object Path, Name, Length, LastWrite, DestinationNewer |
Export-Csv '.\prevented.csv' -NoTypeInformation
In the above $CopyPreventeds already exists and remains so, unmolested after the export. You would need only to output it again for the benefit of an interactive user. All taking advantage of PowerShell's intuitive pipeline and features.
Moreover, since the iteration variable $cprev isn't needed you are still less one variable.
Note: You don't need -Append because you are streaming into a single Export-Csv command, as opposed to repeatedly invoking it.
There are at least 2 ways (probably many more) you could conveniently output to an interactive user.
1: Echo a header, something like "The following copies were prevented:" then echo the variable $CopyPreventeds, presumably to a table.
Note: That given multiple points at which you seem only interested in a subset of properties. You may think about trimming those objects beforehand:
$CopyPreventeds =
$CopyPreventeds |
Select-Object Path, Name, Length, LastWrite, DestinationNewer
$CopyPreventeds | Export-Csv '.\prevented.csv' -NoTypeInformation
Write-Host "The following copies were prevented:"
$CopyPreventeds | Format-Table -AutoSize | Out-Host
Note: More than 4 Properties in a [PSCustomObject] (resulting from Select-Object) where custom formatting hasn't been defined will by default output as a list, so use Format-Table to overcome that. Out-Host is then used to prevent pipeline pollution.
2: Return to using a ForEach-Object Loop for the output between the Select-Object and the Export-Csv command.
$CopyPreventeds |
Select-Object Path, Name, Length, LastWrite, DestinationNewer
ForEach-Object{
"Prevented Copy : {0}, {1}, {2}, {3}, {4}" -f $_.Path, $_.Name, $_.Length, $_.LastWrite, $_.DestinationNewer |
Write-Host
$_
} |
Export-Csv '.\prevented.csv' -NoTypeInformation
In this example, when you are done outputting to the screen (admittedly a little messy), you emit $_ from the loop, thus piping it to Export-Csv just the same.
Note: there are a number of ways to construct strings, I choose to use the -f operator here because it's a little cleaning than imbedding numerous $() sub expressions. And, of course this assume you want to prefix on every line Which I personally think is gratuitous, so I'd choose something more like #1..
I have a bug I cannot beat. When I run my script gets to this chunk of code it is incorrectly removing unique values:
import-csv "$LocalPath\A1-$abbrMonth$Year.csv" |
where {$_."CustomerName" -match $Customersregex} |
select "SubmitterID","SubmitterName","JobDate","JobTime",#{Name="Form";Expression={if ($_.FormName -match "Copy"){"C"};if ($_.FormName -match "Letter"){"L"} else {""} }},"TotalDocs",#{Name="AddnPages";Expression={$_.TotalAdditionalPages}},"InputFilename",#{Name="ActualDocs";Expression={[string]([int]$_.RegularDocs + [int]$_.UnqualifiedDocs)}}|
sort "InputFilename" -Unique |
export-csv "$LocalPath\A2-$abbrMonth$Year.csv" -NoTypeInformation
It's occurring during the "sort "InputFilename" -Unique" line, however it will work properly when I cut it up and execute it line by line, but not in the original script.
Is there any other way to remove duplicates based on the value of a column? I've tried using the "-unique" parameter on the Select-Object statement but I can't find a way to limit it to only one column.
EDIT: To clarify the issue I'm having, I have a LARGE list of accounting data. I'm trying to remove duplicate entries by using "Sort -unique". After the above code is running, there are entries missing that should not be because they are unique. I can isolate them in their own CSV, run the above code and all entries are present that should be, however when I run my master CSV file through the above code (and only that code, nothing else) and search for those entries they are missing.
EDIT 2: Looks like it was an issue with the data file. Good grief.
You can always group things, then expand the first item in the group. It's not fast, but it works for what you're doing.
import-csv "$LocalPath\A1-$abbrMonth$Year.csv" |
where {$_."CustomerName" -match $Customersregex} |
group InputFilename |
% { $_.Group[0] } |
select "SubmitterID","SubmitterName","JobDate","JobTime",#{Name="Form";Expression={if ($_.FormName -match "Copy"){"C"};if ($_.FormName -match "Letter"){"L"} else {""} }},"TotalDocs",#{Name="AddnPages";Expression={$_.TotalAdditionalPages}},"InputFilename",#{Name="ActualDocs";Expression={[string]([int]$_.RegularDocs + [int]$_.UnqualifiedDocs)}}|
sort "InputFilename" |
export-csv "$LocalPath\A2-$abbrMonth$Year.csv" -NoTypeInformation
I am trying to write a simple PS script to check large .txt log files for a short string: "SRVE0242I:"
$lines = Select-String -Path $logDir -Pattern "SRVE0242I:" | Select-Object line | Out-String
On output though, it only displays the following:
Line
[28/06/17 13:48:27:839] 00000020 ServletWrappe I SRVE0242I: [User] [User] [com_xxxxxxx_...
And not the full line. Is there a limit to how many characters this pulls? I can't find any info on any restrictions for the Select-String cmdlet. Is there a better way to do this so that I don't a) pull the heading "Line" in my list of lines (Don't really want to create table formatting for such a simple output) and b) get the whole line when I pull the info?
You are seeing it like this because it's displaying the Line property using the default Format-Table view and shortening it to the width of the console.
Do this instead:
$lines = Select-String -Path $logDir -Pattern "SRVE0242I:" | Select-Object -ExpandProperty line
This returns the value of the Line property as a string to the $lines variable. You don't need to use Out-String.
There is! Long story short, Select-Object is doing the truncating here. Here's one way to get the first untruncated line in a Select-String output
$(Select-String -Path $logDir -Pattern "SRVE0242I:")[0].Line
When you run into something like this, you can break down the individual steps to determine what's happening by piping things to Get-Member. Here's what's happening in the code above:
Select-String <# args #> | Get-Member
Select-String gives us a MatchInfo object, which (as you've correctly determined) has a 'Line' property. When run on it's own, Select-String will actually spit out all the information you're looking for, and will not truncate it by default (at least, on v6.0.0-beta). It does give you an array of MatchInfo objects if it finds multiple matches, so you have to index into that array if you just want the first one (like I did above).
Select-String <# args #> | Select-Object Line | Get-Member
Select-Object applies PowerShell's default formatting for objects which, in most cases, will truncate your output for easier viewing. For objects with a bunch of members (like a MatchInfo object), it will try to do one per line by default.
Select-String <# args #> | Select-Object Line | Out-String | Get-Member
Out-String directly translates it's input to a string. That is, rather than trying to cast something to a string or pull a string Property out of an object that's passed to it, it just changes whatever it receives into an object. In this case, it turns the already-formatted MatchInfo output into a string. Nothing happens to the output on the terminal, but Get-Member will reveal a String rather than a MatchInfo object.
It's not directly relevant here, but if you're interested in modifying the default formatting, it's governed by the types.ps1xml file.
I have a text file that contains elements separated by an '=' sign (i.e. color1=red, color2=blue, etc.
I used the import-csv command and provide headers (i.e.
$Import_Cfg = Import-Csv .\Env.cfg -Header Title,Setting -Delimiter =
)
Now this works fine if I want to assign a particular item to another variable if I know the index number and I have used that approach but it won't always work for me because I don't always know what other data will be there.
I thought that by using something like:
$MyColor1 = $Import_Cfg.Setting |where {$_.Title -match "Blue"}
$MyColor2 = $Import_Cfg.Setting |where {$_.Title -match "Red"}
it should work, but I get no returns for either item. When I type in $Import_cfg I can see the entire array (without the "=" signs). If I tell use the command
$MyColor1 = $import_cfg[0].setting
I get the right answer.
Obviously I'm not using colors but a bunch of different items that I need to assign to variables for use elsewhere. Any thoughts on what I'm doing wrong? Everything I've read says that what I have above should work.
Please no flames on why I'm using import-csv vs get-content. I'm sure either will work. This is an approach that I've used and computationally it doesn't matter. If programatically it makes a difference I'm all ears!!!
Thanks for all your help.
The value of the Setting property itself has no Title property.
You need to apply Where before you extract the property value you need (as mentioned in the comments):
$BlueSettings = $Import_Cfg |where {$_.Title -match "Blue"} |Select-Object -ExpandProperty Setting
or, using property enumeration:
$BlueSettings = ($Import_Cfg |where {$_.Title -match "Blue"}).Setting
Question
How do I reformat an unknown CSV column name according to a formula or subroutine (e.g. rename column " Arbitrary Column Name " to "Arbitrary Column Name" by running a trim or regex or something) while maintaining data?
Goal
I'm trying to more or less sanitize columns (the names) in a hand-produced (or at least hand-edited) csv file that needs to be processed by an existing PowerShell script. In this specific case, the columns have spaces that would be removed by a call to [String]::Trim(), or which could be ignored with an appropriate regex, but I can't figure a way to call or use those techniques when importing or processing a CSV.
Short Background
Most files and columns have historically been entered into the CSV properly, but recently a few columns were being dropped during processing; I determined it was because the files contained a space (e.g., Select-Object was being told to get "RFC", but Import-CSV retrieved "RFC ", so no matchy-matchy). Telling the customer to enter it correctly by hand (though preferred and much simpler) is not an option in this case.
Options considered
I could manually process the text of the file, but that is a messy and error prone way to re-invent the wheel. I wonder if there's a syntax with Select-Object that would allow a softer match for column names, but I can't find that info.
The closest I have come conceptually is using a calculated property in the call to Select-Object to rename the column, but I can only find ways to rename a known column to another known column. So, this would require enumerating the columns and matching them exactly (preferred) or a softer match (like comparing after trimming or matching via regex as a fallback) with expected column names, then creating a collection of name mappings to use in constructing calculated properties from that information to select into a new object.
That seems like it would work, but more it's work than I'd prefer, and I can't help but hope that there's a simpler way I haven't been able to find via Google. Maybe I should try Bing?
Sample File
Let's say you have a file.csv like this:
" RFC "
"1"
"2"
"3"
Code
Now try to run the following:
$CSV = Get-Content file.csv -First 2 | ConvertFrom-Csv
$FixedHeaders = $CSV.PSObject.Properties.Name.Trim(' ')
Import-Csv file.csv -Header $FixedHeaders |
Select-Object -Skip 1 -Property RFC
Output
You will get this output:
RFC
---
1
2
3
Explanation
First we use Get-Content with parameter -First 2 to get the first two lines. Piping to ConvertFrom-Csv will allow us to access the headers with PSObject.Properties.Name. Use Import-Csv with the -Header parameter to use the trimmed headers. Pipe to Select-Object and use -Skip 1 to skip the original headers.
I'm not sure about comparisons in terms of efficiency, but I think this is a little more hardened, and imports the CSV only once. You might be able to use #lahell's approach and Get-Content -raw, but this was done and it works, so I'm gonna leave it to the community to determine which is better...
#import the CSV
$rawCSV = Import-Csv $Path
#get actual header names and map to their reformatted versions
$CSVColumns = #{}
$rawCSV |
Get-Member |
Where-Object {$_.MemberType -eq "NoteProperty"} |
Select-Object -ExpandProperty Name |
Foreach-Object {
#add a mapping to the original from a trimmed and whitespace-reduced version of the original
$CSVColumns.Add(($_.Trim() -replace '(\s)\s+', '$1'), "$_")
}
#Create the array of names and calculated properties to pass to Select-Object
$SelectColumns = #()
$CSVColumns.GetEnumerator() |
Foreach-Object {
$SelectColumns += {
if ($CSVColumns.values -contains $_.key) {$_.key}
else { #{Name = $_.key; Expression = $CSVColumns[$_.key]} }
}
}
$FormattedCSV = $rawCSV |
Select-Object $SelectColumns
This was hand-copied to a computer where I don't have the rights to run it, so there might be an error - I tried to copy it correctly
You can use gocsv https://github.com/DataFoxCo/gocsv to see the headers of the csv, you can then rename the headers, behead the file, swap columns, join, merge, any number of transformations you want