I am converting code that currently parse a large csv file once for each agency. Below is what one line looks like:
Import-Csv $HostList | Where-Object {$_."IP Address" -Match "^192.168.532.*" -or $_Domain -eq "MYDOMAIN"`
-and (get-date $_.discovery_timestamp) -gt $PreviousDays} | select-object Hostname","Domain","IP Address","discovery_timestamp","some_version" | `
Export-Csv -NoTypeInformation -Path $out_data"\OBS_"$Days"_days.txt"
write-host "OBS_DONE"
I have about 30 of these.
I want to parse the csv file once, possibly using foreach and import.csv.
I thought I could do something like:
$HostFile = Import-csv .\HostList.csv
foreach ($line in $HostFile)
{
Where-Object {$_."IP Address" -Match "^172.31.52.*"}
write-host $line
#| Export-Csv -NoTypeInformation -Path H:\Case_Scripts\test.csv
I've tried many permutations of the above, and it never is matching on the "Where-Object" like it does on the example functioning script above.
Any guidance and learning opportunities are appreciated.
My good man, you need to be introduced to the Switch cmdlet. This will sort for all of your companies at once.
$CompanyA = #()
$CompanyB = #()
$CompanyD = #()
$Unknown = #()
Switch(Import-CSV .\HostList.csv){
{(($_."IP Address" -match "^192.168.532.*") -or ($_Domain -eq "CompanyA.com")) -and ((get-date $_.discovery_timestamp) -gt $PreviousDays)} {$CompanyA += $_; Continue}
{(($_."IP Address" -match "^192.26.19.*") -or ($_Domain -eq "CompanyB.net")) -and ((get-date $_.discovery_timestamp) -gt $PreviousDays)} {$CompanyB += $_; Continue}
{(($_."IP Address" -match "^94.8.222.*") -or ($_Domain -eq "CompanyC.org")) -and ((get-date $_.discovery_timestamp) -gt $PreviousDays)} {$CompanyC += $_; Continue}
default {$Unknown += $_}
}
$CompanyA | Export-Csv $out_data"\CompanyA"$Days"_days.txt" -NoType
$CompanyB | Export-Csv $out_data"\CompanyB"$Days"_days.txt" -NoType
$CompanyC | Export-Csv $out_data"\CompanyC"$Days"_days.txt" -NoType
If($Unknown.count -gt 0){Write-Host $Unknown.count + " Entries Did Not Match Any Company" -Fore Red
$Unknown}
That will import the CSV, and for each entry try to match it against the criteria for each of the three lines. If it matches the criteria in the first ScriptBlock, it will perform the action in the second ScriptBlock (add that entry to one of the three arrays I created first). Then it outputs each array to it's own text file in CSV format as you had done in your script. The ;Continue just makes it so it stops trying to match once it finds a valid match, and continue's to the next record. If it can't match any of them it will default to adding it to the Unknown array, and at the end it checks if there are any in there it warns the host and lists the unmatched entries.
Edit: Switch's -RegEx argument... Ok, so the purpose of that is if you want a simple regex match such as:
Switch -regex ($ArrayOfData){
".*#.+?\..{2,5}" {"It's an email address"}
"\d{3}(?:\)|\.|-)\d{3}(?:\.|-)\d{4}" {"It's a phone number"}
"\d{3}-\d{2}-\d{4}" {"Probably a social security number"}
}
What this doesn't allow for is -and, or -or statements. If you use a scriptblock to match against (like I did in my top example) you can use the -match operator, which performs a regex match by default, so you can still use regex without having to use the -regex argument for Switch.
Where-Object should have a collection of objects passed to it. In your second script block this isn't happening, so in the filter the $_ variable will be empty.
There are a couple of ways to fix this - the first one that comes to mind is to replace Where-Object with an 'if' statement. For example
if ($line."IP Address" -Match "^172.31.52.*") { write-output $line }
Related
$NameMatches = $Prices | Where-Object 'name' -EQ $sub.OfferName
$TermMatches = $NameMatches | where-object 'itemCode' -match $Term
$BillingFreqMatches = $TermMatches | where-object 'ItemCode' -match $BillingFreq
These 3 lines work, but any syntax I use to put them on one line returns no results.
Seems like you're looking for this:
$Prices | Where-Object {
$_.Name -eq $sub.OfferName -and $_.itemCode -match $Term -and $_.itemCode -match $BillingFreq
}
Note that you cannot use Comparison Statement when doing multiple conditions, you must use Script block. See Description section from the Cmdlet's MS Docs.
I'm fairly new to Powershell and programming in general. I want to search files using Powershell having multiple conditions. I have managed to write this code
$Drives = Get-PSDrive -PSProvider 'FileSystem'
$Filename= 'Result'
$IncludeExt= '*csv,*docx'
$StartDate= '11/1/20'
$EndDate= '1/26/21'
Get-ChildItem -Path $Drives.Root -Recurse |Where-Object {$IncludeExt -match $_.Extension} | Where-Object { $_.BaseName -match $Filename} | Where-Object {$_.lastwritetime -ge $StartDate -AND $_.lastwritetime -le $EndDate} |
foreach{
$Item = $_.Basename
$Path = $_.FullName
$Type = $_.Extension
$Age = $_.CreationTime
$Path | Select-Object `
#{n="Name";e={$Item}},`
#{n="Created";e={$Age}},`
#{n="filePath";e={$Path}},`
#{n="Folder/File";e={if($Folder){"Folder"}else{$Type}}}`
}| Export-Csv D:\FFNew.csv -NoTypeInformation
This works well when the all variables are mentioned. But how do I get this to work when
Case1: If $Filename is empty then it gives all the files with the mentioned extensions and files modified in Range of dates
Case2: If $IncludeExt is left empty then it gives all files with the $Filename mentioned, currently it gives only the folders and files modified in Range of dates
Case 3: If $Filename and $IncludeExt is left empty it gives all the files modified between the $StartDate and $EndDate
Pranay,
[EDITED]
Ok, here's the revised (exact) script with notes and sample output. Note: you'll have to change the items that are specific to my machine!
$Drives = Get-PSDrive -PSProvider 'FileSystem'
$Filename = "*" #for all or "*partial name*"
$IncludeExt = $Null #for no ext. or "*.csv","*.docx",etc...
$StartDate = '01/1/2020' #to ignore this use 1/1/1920
#For latest date use below otherwise specify date.
$EndDate = (Get-Date).ToShortDateString()
#Note: below uses only 3rd drive in the array remove [2] for all.
$GCIArgs = #{Path = $Drives[2].Root
Recurse = $True
}
If ($Null -ne $IncludeExt) {
$GCIArgs.Add("Include",$IncludeExt)
}
Get-ChildItem #GCIArgs |
Where-Object {($_.BaseName -Like $Filename) -and
($_.lastwritetime -ge $StartDate) -and
($_.lastwritetime -le $EndDate) } |
foreach{
$Item = $_.Basename
$Path = $_.FullName
$Type = $_.Extension
$Type = & {if($_.PSIsContainer){"Folder"}else{$_.Extension}}
$Age = $_.CreationTime
$Path | Select-Object #{n="Name" ;e={$Item}},
#{n="Created" ;e={$Age}} ,
#{n="filePath" ;e={$Path}},
#{n="Folder/File";e={$Type}}
} | Export-Csv -LiteralPath 'G:\BEKDocs\FFNew.csv' -NoTypeInformation
Notes:
$IncludeExt is specified as $Null if it is not used and if used the list is like this ".csv",".docx"
$Filename is specified as "*" for all filenames. Also changed the test from -match to -like so partial filenames should include *, e.g. "partial name".
Notice I changed the location of the check for Extensions to use the -Include parameter of the Get-ChildItem vs checking in the Where-Object.
Changed the piping of data to successive Where-Object clauses and replaced with -and operator, same effect and more efficient.
Changed the test for Directories to use the PSIsContainer property, couldn't see where you were getting the value for $Folder.
Removed the continuation characters from the Select-Object as the comma serves that purpose and is cleaner.
Sample output on Single drive (per code shown above) with some lines hidden for space considerations but notice the last line number.
Sample output on all drives (code edited as per comment in code), again lines hidden for space but showing multiple drives and final line number.
HTH
I have a small PowerShell script that outputs two columns:
A string labeled Name and
a collection (in curly braces) labeled Users
I need to output this to a CSV file, but the collection just outputs as error-gobbledygook.
I know I need to convert this to a string or iterate or something, but I am having trouble figuring this out.
$ActiveSharedCapacityWorkspaces = Get-PowerBIWorkspace -Scope Organization -All | Where {($_.IsOnDedicatedCapacity -eq $false) -and ($_.State -eq "Active") -and ($_.Type -ne "PersonalGroup")}
$ActiveSharedCapacityWorkspaceUsers = $ActiveSharedCapacityWorkspaces | Select Name,Users | Get-Unique -AsString
$ExportFile = "C:\Users\xxx\Desktop\ActiveSharedCapacityWorkspaceUsers.csv"
$ActiveSharedCapacityWorkspaceUsers | Export-Csv $ExportFile
Expected Result:
Name:WorkspaceName Users:Joe Schmoe, Billy Bob
Actual Results:
Name:WorkspaceName Users:System.Linq.Enumerable+WhereSelectListIterator`2[Microsoft.PowerBI.Api.V2.Models.GroupUserAccessRight,Microsoft.PowerBI.Common.Api.Workspaces.WorkspaceUser]
Get-PowerBiWorkspace returns an IEnumerable, in your case you're storing a Linq expression on this IEnumerable.
As stated by #LotPings you've to invoke the Where-Linq expression. This is done by enumeration operations, like -join (because the join operator has to enumerate the collection, which causes the Linq query to be resolved/invoked). As #LotOfPings and #AdminOfThings suggested change your code to:
$ActiveSharedCapacityWorkspaces = Get-PowerBIWorkspace -Scope Organization -All | Where {($_.IsOnDedicatedCapacity -eq $false) -and ($_.State -eq "Active") -and ($_.Type -ne "PersonalGroup")}
$ActiveSharedCapacityWorkspaceUsers = $ActiveSharedCapacityWorkspaces | Select-Object Name,#{n="Users';e={$_.Users.UserPrincipalName -join ', '}}
$ExportFile = "C:\Users\xxx\Desktop\ActiveSharedCapacityWorkspaceUsers.csv"
$ActiveSharedCapacityWorkspaceUsers | Export-Csv $ExportFile -NoTypeInformation
The -NoTypeInformation suppresses type information in the 0th line of the generated CSV file. If you're PowerShell 6 type information generation is deactivated per default.
I need to remove a row from a table if I find a certain kind of email address in my CSV table. There are multiple email fields but I only want to do this for one email column, not any which is what I think the code below would currently do.
How can I specify that?
$data = foreach ($line in Get-Content D:\Data\info.csv) {
if ($line -Like '*#LS*' -or $line -Like '*#gmail*') {
} else {
$line
}
}
$data | Set-Content D:\Data\info.csv -Force
Use a Where-Object filter:
$file = 'D:\Data\info.csv'
(Get-Content $file) | Where-Object {
$_ -notlike '*#LS*' -and $_ -notlike '*#gmail*'
} | Set-Content $file
If you want to check a particular field instead of the entire line use Import-Csv/Export-Csv instead of Get-Content/Set-Content:
$file = 'D:\Data\info.csv'
(Import-Csv $file) | Where-Object {
$_.FOO -notlike '*#LS*' -and $_.FOO -notlike '*#gmail*'
} | Export-Csv $file -NoType
Replace FOO with the actual field name.
Instead of the wildcard matches you could also use string operations:
-not $_.FOO.Contains('#LS') -and -not $_.FOO.Contains('#gmail')
or a single regular expression:
$_.FOO -notmatch '#(LS|gmail)'
To complement Ansgar Wiechers' helpful answer:
Select-String allows for a concise solution (PSv3+ syntax) with whole-line matching - as advised by Ansgar, using Import-Csv and limiting matching to the field of interest is preferable:
(Select-String -NotMatch '#LS', '#gmail' D:\Data\info.csv).Line |
Set-Content D:\Data\info.csv -Force
Note how the Select-String call is enclosed in (...) to ensure that the input file is processed in full up front, allowing you to write back to the same file in the same pipeline.
Note, however, that that loads all matching lines into memory at once, and that there's a small risk of data loss if writing back to the input file gets interrupted - both issues could be remedied with more effort.
Using powershell, lets say I have a csv file that contains
fname,lname,id,etc..
Is there a way to use where-object to look through all of the columns instead of just one.
For example, instead of doing:
Import-csv location |where-Object {$_.fname -eq "hi"}
next line:
Import-csv location |where-Object {$_.lname -eq "hi"} and so on.
It would be something like:
Import-csv location |where-Object {ANY -eq "hi"}
Yes, you could iterate over $_.psobject.Properties to inspect every column and return $true if one of them matches:
Import-Csv document.csv |Where-Object {
foreach($p in ($_.psobject.Properties |? {$_.MemberType -eq 'NoteProperty'})) {
if($p.Value -match "hi"){ $true }
}
}
Since the column headings / property names are irrelevant and all you care about is the values, you can handle it as text:
Get-Content c:\somedir\somefile.csv |
Where {$_ -like '*"hi"*'}
If you're just after a count, you can just count the matches. If you want objects, you can run the matched lines through convertfrom-csv.