Comparing two CSV's for similar strings - powershell

I am currently trying to compare two CSV's. doc1 has 5000 strings in it, and doc2 has 100 strings in it. Every string in doc2 ends in "admin". I want to compare doc1 & doc2 and find all the strings that match up to the point it reaches "admin".
so as an example:
a string in doc1
a string in doc1 admin
it will output both of these to a new CSV
results = foreach ($OU in $OUs) {
Get-ADGroup -SearchBase $OU -Filter * -Properties CanonicalName,Created,Description,ManagedBy,Member,MemberOf,Members,ObjectGUID,whenChanged |
? {($_.ManagedBy -like $null) -and ($_.Description -notlike "*owner*") -and ($_.CanonicalName -notlike "*admin")}
}
$results | select CanonicalName,Description,ManagedBy,Member,MemberOf,Members,ObjectGUID,Created,whenChanged |
Export-Csv .\doc1.csv -NoTypeInformation
$results0 = foreach ($OU in $OUs) {
Get-ADGroup -SearchBase $OU -Filter * -Properties CanonicalName,Created,Description,ManagedBy,Member,MemberOf,Members,ObjectGUID,whenChanged |
? {($_.ManagedBy -like $null) -and ($_.Description -notlike "*owner*") -and ($_.CanonicalName -like "*admin")}
}
$results0 | select CanonicalName,Description,ManagedBy,Member,MemberOf,Members,ObjectGUID,Created,whenChanged |
Export-Csv .\doc2.csv -NoTypeInformation
$csvPath1 = ".\doc1.csv"
$csvPath2 = '.\doc2.csv'
$propertyToCompare = 'CanonicalName'
$csv1 = Import-Csv -Path $csvPath1
$csv2 = Import-Csv -Path $csvPath2
$duplicates = Compare-Object $csv1 $csv2 -Property $propertyToCompare -ExcludeDifferent -PassThru | Select-Object -ExpandProperty $propertyToCompare
$csv1 | Where-Object { $_.$propertyToCompare -in $duplicates } | Export-Csv -Path .\AdminsAndNotAdminsInOneFile.csv -NoTypeInformation
With Compare-Object I don't know how to make it ignore the last few characters in the string in doc2. Is there some way for me to modify the string?

You can take advantage of the fact that Compare-Object supports calculated properties as comparison properties (here, only a script block ({...}) is passed, which is the same as passing
#{ Expression = { ... } }):
Compare-Object $csv1 $csv2 -ExcludeDifferent -IncludeEqual -PassThru -Property {
$_.$propertyToCompare -replace ' admin$'
} | Select-Object -ExpandProperty $propertyToCompare
Note that -PassThru ensures that the input objects are passed through, which in the case of objects that compare as equal means that the LHS ($csv1) object is passed through.
Expression $_.$propertyToCompare -replace ' admin$', which is evaluated for each input object, uses regex admin$ to replace ' admin' at the end ($) of the value of property $propertyToCompare; if that string isn't present, the value is used as-is.
In effect, the objects are compared as if the trailing ' admin' weren't present.

Related

PowerShell compare CSV to AD Subnets and update / add if required

I've got a CSV file with headers subnet, site, description. I want to import this into PoSH and then compare what is in AD Sites and Services. Now, this might be "null" or it might already contain subnet values. In either case, I need the subnets in AD compared to the CSV and updated/added as required.
I thought about If/Else, but that doesn't seem to handle the scenario where no subnets exist.
So I moved to try/catch and also Compare-Object.
Anyway, an example of what I've tried:
$csvSubnets = Import-Csv -Path C:\subnets.csv
$adSubnets = Get-ADReplicationSubnet -Filter *
Compare-Object -ReferenceObject #($csvSubnets | Select-Object) -DifferenceObject #($adSubnets | Select-Object) -Property Name | Where-Object {
$_.SideIndicator -eq '<='
} | ForEach-Object {
$csvSubnetName = $_.subnet
$csvSiteName = $_.site
$csvDescription = $_.description
$adSubnetName = $adSubnets.Where({$_.Name -eq $csvSubnetName})
New-ADReplicationSubnet -Site $_.site -Name $_.subnet -Description $_.description
}
This returns the following error:
New-ADReplicationSubnet : Cannot validate argument on parameter 'Site'. The argument is null
Would anyone have a suggestion as to how to fix this, or perhaps a more efficient way of achieving what I need? Doesn't need to be computationally efficient as there's probably never more than 10 sites (subnets)
The main thing I'd point out is you aren't passing the original objects coming out of the comparison to the ForEach-Object loop. Either reference the property $_.InputObject or preferably use the -PassThru parameter on the Compare-Object command:
Compare-Object -ReferenceObject #($csvSubnets | Select-Object) -DifferenceObject #($adSubnets | Select-Object) -Property Name -PassThru |
Where-Object { $_.SideIndicator -eq '<='} |
ForEach-Object {
$csvSubnetName = $_.subnet
$csvSiteName = $_.site
$csvDescription = $_.description
$adSubnetName = $adSubnets.Where({$_.Name -eq $csvSubnetName})
New-ADReplicationSubnet -Site $_.site -Name $_.subnet -Description $_.description
}
An aside:
The Compare-Object command is rather hard to read, given intermingled pipelines. It may be better to establish those as variables separately. However, you aren't specifying any properties in those Select-Object commands. I presume you are trying to level-set the typing to [PSCustomObject]'s in which case you should really only need to do that with the rich objects returned from Get-ADReplicationSubnet Import-Csv will always return [PSCustomObject]. So you could revise a little like:
$csvSubnets = Import-Csv -Path 'C:\subnets.csv'
$adSubnets = Get-ADReplicationSubnet -Filter * | Select-Object *
Compare-Object -ReferenceObject $csvSubnets -DifferenceObject $adSubnets -Property Name -PassThru |
Where-Object { $_.SideIndicator -eq '<='} |
ForEach-Object {
$csvSubnetName = $_.subnet
$csvSiteName = $_.site
$csvDescription = $_.description
$adSubnetName = $adSubnets.Where({$_.Name -eq $csvSubnetName})
New-ADReplicationSubnet -Site $_.site -Name $_.subnet -Description $_.description
}
I can't test your code but after solving the first problem you may have another with this line:
$adSubnetName = $adSubnets.Where({$_.Name -eq $csvSubnetName})
You aren't referencing the .Name property after you've isolated the AD Subnet you're interested in. I might be missing something, but if you do have that problem a quick fix might be something like:
$adSubnetName = ($adSubnets.Where({$_.Name -eq $csvSubnetName})).Name
Also, I like the .Where() method, but I'm a little iffy on it's return types, and on the readability of unnecessarily mixing and matching. You could replace with a regular Where{} clause:
$adSubnetName = ( $adSubnets | Where-Object{ $_.Name -eq $csvSubnetName } ).Name
Update:
It is true that if the difference object is null the Compare-Object command will fail. However, that approach stemmed from your original code. There are actually many ways to extract the difference between 2 lists. Here's a more concise example:
$csvSubnets = Import-Csv -Path 'C:\subnets.csv'
$adSubnets = (Get-ADReplicationSubnet -Filter *).Name
$csvSubnets |
Where-Object{ $_.Name -notin $adSubnets } |
ForEach-Object{
$SubnetParams = #{
Name = $_.subnet
Site = $_.site
Description = $_.description
}
New-ADReplicationSubnet #SubnetParams
}
In this case we use the -notin operator against a list of subnet names. $adSubnets isn't being used for anything else. We don't need it to be objects custom or otherwise. Now for each record in the CSV file just see if the name isn't on the list, and if not proceed to create the subnet.
Speed is not a likely concern in this case however you can actually make the above example a little more efficient using If logic in the loop and eliminating the Where{}.
$csvSubnets = Import-Csv -Path 'C:\subnets.csv'
$adSubnets = (Get-ADReplicationSubnet -Filter *).Name
$csvSubnets |
ForEach-Object{
If($_.Name -notin $adSubnets ) {
$SubnetParams = #{
Name = $_.subnet
Site = $_.site
Description = $_.description
}
New-ADReplicationSubnet #SubnetParams
}
}
In either example the key point is that -notin will work fine even if the right hand side of the comparison is null. It's simply return false and therefore the subnet will get created.
Firstly, thank you to the other responses here which lead me down a slightly different path to a solution.
$VerbosePreference = 'Continue'
$csvSubnets = Import-Csv -Delimiter "," -Path C:\subnets.csv
foreach ($subnet in $csvSubnets) {
$ADSubnetName = $subnet.subnet
if ($S = Get-ADReplicationSubnet -Filter {Name -eq $ADSubnetName}) {
Write-Verbose "Subnet $($ADSubnetName) already present"
Continue # to the next subnet
}
$SubnetParams = #{
Name = $subnet.subnet
Site = $subnet.site
Description = $subnet.description
}
Write-Verbose "Created subnet $($ADSubnetName)"
New-ADReplicationSubnet #SubnetParams
}

How to use GetEnumerator() to export hash table to a csv file in PowerShell?

I wrote a search function, which searches for some active directory attributes. I now want to export the content to a .csv file. If I see it correctly, the whole output is a hash table. But when I try to use the GetEnumerator() function, it doesn't do anything.
Can someone help and maybe explain what I did wrong? The code is below. Thanks in advance!
Import-Module ActiveDirectory
$users = Get-ADUser -Filter { employeeNumber -notlike '*' } -Properties memberOf, samAccountName | ForEach-Object -Process { #{ $_.samAccountName = $_.memberOf | Where-Object { $_ -like '*DAT*' } } }
$users.GetEnumerator() |
Select-Object -Property #{N='AD1';E={$_.Key}},
#{N='AD2';E={$_.Value}} |
Export-Csv -NoTypeInformation -Path H:\test123.csv
If you look at your code, you are creating a list of hashtables that contains your SamAccountName and Groups. But, when you use the enumerator, you are only thinking about the hashtable and not the list you created.
This is how you can iterate through a hashtable. You first create a hash table and add all elements to it.
$hash = #{}
Get-ADUser -Filter { employeeNumber -notlike '*' } -Properties memberOf, samAccountName | ForEach-Object -Process { $hash.Add($_.samAccountName, ($_.memberOf | Where-Object { $_ -like '*DAT*' })) }
$hash.GetEnumerator() |
Select-Object -Property #{N='AD1';E={$_.Key}},
#{N='AD2';E={$_.Value}} |
Export-Csv -NoTypeInformation -Path H:\test123.csv
Or you can continue with the list of hashtables but change how you are accessing the data. Each element of your list is a single hashtable with Keys (only one in it).
$users = Get-ADUser -Filter { employeeNumber -notlike '*' } -Properties memberOf, samAccountName | ForEach-Object -Process { #{ $_.samAccountName = $_.memberOf | Where-Object { $_ -like '*DAT*' } } }
$users.GetEnumerator() |
Select-Object -Property #{N='AD1';E={$_.Keys[0]}},
#{N='AD2';E={$_.Values[0]}} |
Export-Csv -NoTypeInformation -Path H:\test123.csv

Run AD search for users not in Group excluding OU's

Good Afternoon
I am trying to create a PS script which pulls all users not in a certain Security group. I have managed to get this to work fine. However i require it to omit certain OU's as i don't want certain accounts included in this process like terminated users and support accounts for examples.
So i created the below to do this but it seems to fail. Its where i have tried to add some filtering. Can someone help put this in the right direction?
import-Module activedirectory
$results = #()
$users = Get-ADUser -Properties memberof -Filter {enabled -eq $true} | ? {$_.DistinguishedName -notlike "*,OU=Exchange,OU=Support Accounts,OU=Terminated Users and Computers do not use,OU=TerminatedEmployeesContractors,OU=TestAccounts*"} *
$ExportPath = 'c:\app\users_in_ou1.csv'
foreach ($user in $users) {
$groups = $user.memberof -join ';'
$results += New-Object psObject -Property #{'User'=$user.name;'Groups'= $groups}
}
$results | Where-Object { $_.groups -notmatch 'SG_XXXXXXXXXXX' } | Select-Object user | export-csv $ExportPath
Thanks
I would build a regex from all OUs that should be excluded from the search by joining the strings with the regex 'OR' character (|) and use the -notmatch operator.
Because there may be characters in these strings that have special meaning in regex, use [Regex]::Escape() on each before joining them.
Something like below:
Import-Module ActiveDirectory
# create a regex from an array of OUs to exclude by 'OR-ing' them with the pipe character
$excludeOUs = ('OU=Exchange','OU=Support Accounts','OU=Terminated Users and Computers do not use',
'OU=TerminatedEmployeesContractors','OU=TestAccounts' | ForEach-Object {[Regex]::Escape($_)}) -join '|'
$ExportPath = 'c:\app\users_in_ou1.csv'
# get a list of objects not having any of the excluded OUs in their DistinguishedName
# and at the same time output objects with properties 'User' and 'Groups'
$users = Get-ADUser -Properties Name, MemberOf -Filter 'Enabled -eq $true' |
Where-Object {$_.DistinguishedName -notmatch $excludeOUs} |
Select-Object #{Name = 'User'; Expression = {$_.Name}},
#{Name = 'Groups'; Expression = {($_.MemberOf -join ';')}}
# next filter this out further by excluding a certain group and export to Csv
$users | Where-Object { $_.Groups -notmatch 'SG_XXXXXXXXXXX' } | Export-Csv $ExportPath -NoTypeInformation

How to save Active Directory list of all users in all e-mail distributions to .CSV?

I found this example but I am not sure how I can properly save the output to a .csv.
Import-Module ActiveDirectory
$Groups = Get-ADGroup -Filter {GroupCategory -eq "Distribution"} -Properties Members
ForEach ($g in $Groups) {
Write-Host $g.name
Write-Host $g.members `n
}
I have tried something such as:
Import-Module ActiveDirectory
$Groups = Get-ADGroup -Filter {GroupCategory -eq "Distribution"} -Properties Members
ForEach ($g in $Groups) {
$g.name | Export-CSV C:\log.csv -notypeinformation -Append
$g.members | Export-CSV C:\log.csv -notypeinformation -Append
}
It only saves 1 column to the CSV which is called length.
This also makes me remove the 'n at the end of Write-Host $g.members `n
Is there a way that I can grab this data and save it to .csv properly?
UPDATE
With help from TheMadTechnician and this link https://blogs.technet.microsoft.com/heyscriptingguy/2013/07/22/export-user-names-and-proxy-addresses-to-csv-file/ I was able to get closer to what I want.
Import-Module ActiveDirectory
$Groups = Get-ADGroup -Filter {GroupCategory -eq "Distribution"} -Properties Members
ForEach ($g in $Groups) {
$g.name | Export-CSV C:\log.csv -notypeinformation -Append
$g.members | Export-CSV C:\log.csv -notypeinformation -Append
}
$Groups | Select Name,#{L='Members_1'; E={$_.members[0]}}, #{L='Members_2';E={$_.Members[1]}}, #{L='Members_3';E={$_.Members[2]}}, #{L='Members_4';E={$_.Members[3gq]}} | Export-Csv C:\log.csv -notype
This gives me an output of the below in my CSV:
Name Members_1 Members_2 ETC...
NameOfGroup CN=Stormy Daniels,OU=IT,DC=DomainName,DC=com CN=Joe Bob,OU=IT,DC=DomainName,DC=com
Now the list of users can be huge so I would have to continue creating Members_3, Members_4, etc...
I'm not sure if there is a way I can specify all users or loop
#{L='Members_1'; E={$_.members[0]}}
and increment the number until all users are displayed.
I also only need the CN with the name. I don't need the Ou= or Dc=.
Ah this proved harder than I expected - due to the member counting (you have to do a count which can be comparable to integer). I have added a possibility to limit result size as for large queries the active directory produces timeouts.
$limit_result_size = 10
$group_name = Get-ADGroup -Filter {GroupCategory -eq "Distribution"} -Properties Name, Members -ResultSetSize:$limit_result_size | Select-object name
ForEach ($name in $group_name.name) {
If ((![String]::IsNullOrEmpty("$name")) -And ("$name" -notlike 'index')) {
$count_members = Get-ADGroupMember -Identity "$name" | Measure-Object | Select-Object Count
Write-Output "The AD group $name has $($count_members.Count) members.`n"
For($counter = 0; $counter -lt $count_members.Count; $counter++) {
$person = Get-ADGroup -Filter {Name -eq $name} -Properties Name, Members | Select-Object Name, #{N='Members';E={$_.Members[$counter]}}
$person.Members = $person.Members | Select-String 'CN=[0-9a-zA-Z]+' -AllMatches | % { $_.Matches } | % { $_.Value }
$person | export-csv -NoTypeInformation -Append -Path '<your_path>\log.csv'
}
}
}
Short description:
(![String]::IsNullOrEmpty("$name")) -And ("$name" -notlike 'index')) conditions which the AD group should satisfy.
Select-String 'CN=[0-9a-zA-Z]+' -AllMatches | % { $_.Matches } | % { $_.Value } Selects only CN=string_with_numbers. You could replace it with CN=\w+ if you prefer.
The script produces a pair in CV AD group and the CN=user_name. If anything else is unclear please ask.
EDIT
If you have spaces in the names of the Common Names (CN) you have to adjust the regexp to CN=[0-9a-zA-Z\s]+.
EDIT 2 Adding user's email addresses.
Since your question has in the title request for emails I'll answer here without new question. Note that this solution uses lookbehind in regexp to exclude the CN= from the output so it can be used as source for the user query. It also uses a PSCustomObject which gathers all the information together. I have renamed some variables to make better sense in the context of user details.
$limit_result_size = 10
$group_name = Get-ADGroup -Filter {GroupCategory -eq "Distribution"} -Properties Name, Members -ResultSetSize:$limit_result_size | Select-object name
ForEach ($name in $group_name.name) {
If ((![String]::IsNullOrEmpty("$name")) -And ("$name" -notlike 'index')) {
$count_members = Get-ADGroupMember -Identity "$name" | Measure-Object | Select-Object Count
Write-Output "The AD group $name has $($count_members.Count) members.`n"
For($counter = 0; $counter -lt $count_members.Count; $counter++) {
$person = Get-ADGroup -Filter {Name -eq $name} -Properties Name, Members | Select-Object Name, #{N='Members';E={$_.Members[$counter]}}
$person.Members = $person.Members | Select-String '(?<=CN=)[0-9a-zA-Z\s]+' -AllMatches | % { $_.Matches } | % { $_.Value }
$person_details = Get-AdUser -filter {name -eq $member} -Properties mail | Select-Object mail
$person_additional_details = [PSCustomObject]#{ group_name = $group.Name
user_name = $group.Members
email = $person_details.mail
}
If ([String]::IsNullOrEmpty($($person_additional_details.email))) {
$person_additional_details.psobject.properties["email"].value = '<empty>'
}
# For user to see the written data
Write-Output "AD Group: $($person_additional_details.group_name) `
AD User: $($person_additional_details.user_name) `
Users`'s email: $($person_additional_details.email)`n"
# writing into the CSV file
$person_additional_details | export-csv -NoTypeInformation -Append -Path '<your_path>\log.csv'
}
}
}

Compare CSV to Array

I have an empty array that's storing all my windows services that start with certain strings such as OPS-AmazonServer,not included in the code I provided is where I parse the service to just say it's application name.
I then have a CSV file with a list of service names labeled under 'Application Name'. It looks like this
ApplicationName,Instance,Priority
AuthBridge,,1
AmazonServer,,1
AmexEC,,1
What I want to do is compare the service stored in the array to the CSV list but I can't seem to figure out how the logic flows.
$services = get-service Centinel* -ComputerName $serverName | select -expand name
$centinelServices = #()
$services = get-service OPS* -ComputerName $serverName | select -expand name
$opsServices = #()
$services = #()
foreach($service in $centinelServices) {
$services += $service
}
foreach($service in $opsServices) {
$services += $service
}
$csvLocation = "\\logserver\Cardinal\OPS\QA\Task\conf\Centinel\app-restart.csv"
$masterList = import-csv $csvLocation
$applications = #()
$masterList | ForEach-Object {$applications += $_.ApplicationName}
forEach($service in $services){
forEach($application in $applications){
if($service -eq $application){
"$service match found"
}
else {
"$service match not found"
}
}
Ok, easiest way to do this is to use Compare-Object, and a little magic with Select.
I'm going to assume that the ApplicationName column in your CSV is a list of strings that match up with the Name property in your Windows Services list. So let's start by importing that CSV, and changing the property name of ApplicationName to just Name, so that it matches the related property on your Windows Service objects.
$masterList = Import-Csv $csvLocation | Select #{l='Name';e={$_.ApplicationName}}
Then we simply use Compare-Object to see what's in both lists:
Compare-Object (Get-Service) -DifferenceObject $masterList -Property Name -IncludeEqual
If you wanted to parse that you can always pipe it to a Where clause, or use combinations of -IncludeEqual and -ExcludeDifferent parameters:
$masterList = Import-Csv $csvLocation | Select #{l='Name';e={$_.ApplicationName}}
$myServices = Get-Service
$foundServices = Compare-Object $myServices -DifferenceObject $masterList -Property Name -IncludeEqual -ExcludeDifferent
$servicesNotInMaster = Compare-Object $myServices -DifferenceObject $masterList -Property Name | Where {$_.SideIndicator -eq '<='}
$servicesNotFoundLocally = Compare-Object $myServices -DifferenceObject $masterList -Property Name | Where {$_.SideIndicator -eq '=>'}
Or using the Switch cmdlet to do it all in one go:
$masterList = Import-Csv $csvLocation | Select #{l='Name';e={$_.ApplicationName}}
$myServices = Get-Service
Switch(Compare-Object $myServices -dif $masterList -prop Name -includeequal -PassThru){
{$_.SideIndicator -eq '<='} {[array]$servicesNotInMaster += $_}
{$_.SideIndicator -eq '=>'} {[array]$servicesNotFoundLocally += $_}
{$_.SideIndicator -eq '=='} {[array]$foundServices += $_}
}
Edit: Ok, updating from your addition to the OP. Looks like you could be well served by simply using a Where clause rather than getting services over and over.
$services = Get-Service -ComputerName $serverName | Where{$_.Name -like 'ops*' -or $_.Name -like 'Centinel*'} | Select -Expand Name
Then you import your CSV, and use Select -Expand again to get the value of the property, rather than looping through it like you were before.
$masterList = Import-Csv $csvLocation | Select -Expand ApplicationName
Now you just have two arrays of strings, so this actually gets even simpler than comparing objects... You can use the -in operator in a Where statement like this:
$services | Where{$_ -in $masterList} | ForEach{"$_ match found"}
That basically filters the $services array to look for any strings that are in the $masterList array. This will only work for exact matches though! So if the service is listed as 'OPS-AmazonServer', but in your CSV file it is listed at just 'AmazonServer' it will not work! I use that example specifically because you have that in your example in your question. You specifically call out the service named 'OPS-AmazonServer' and then in your CSV sample you list just 'AmazonServer'.
If the listings in the CSV are partial strings that you want to match against you could use RegEx to do it. This will probably make less sense if you aren't familiar with RegEx, but this would work:
$services = Get-Service -ComputerName $serverName | Where{$_.Name -like 'ops*' -or $_.Name -like 'Centinel*'} | Select -Expand Name
$masterList = (Import-Csv $csvLocation | ForEach{[regex]::escape($_.ApplicationName)}) -join '|'
$services | Where{ $_ -match $masterList } | ForEach{"$_ match found"}