Compare two csv files and find discrepancies - powershell

I am new in powershell world , I got some project on powershell for inventory reconciling .
I am not sure how to proceed on this , I tried some basic steps and I am able to export
users/group/group membership.
Following are the requirements :
Now What I have achieved - Thanks to Govnah
AD query:
Get-QADUser -searchRoot $OuDomain -SizeLimit 0 |
Select-Object dn, sAMAccountName, #{Name="Groups";Expression={(Get-QADMemberOf $_ | Select-Object -expandProperty Name) -join ";"}} |
Sort-Object SamAccountName |
export-csv $FilePath
I have now two csv files likes AD_users.csv and Oracle_users.csv
I want to compare both files and redirect the difference like
AD users does not exist in Oracle
Oracle User does not exist in AD
Sample data
AD_users.csv
u388848993
K847447388
u994888484
Oracle_users.csv
k095848889
u388848993
I can query oracle database , AD query is also fine the only concern is that I am not able to compare the output.

I did it something like this in a script I wrote:
[System.Collections.ArrayList]$adlist = Get-Content c:\users\sverker\desktop\ad.csv |Sort-Object
[System.Collections.ArrayList]$oraclelist = Get-Content c:\users\sverker\desktop\oracle.csv |Sort-Object
$Matching_numbers = #()
ForEach ($number in $adlist)
{
if ($oraclelist.Contains($number))
{
$Matching_numbers += $number
}
}
ForEach ($number in $Matching_numbers)
{
$adlist.Remove($number)
$oraclelist.Remove($number)
}
now $Matching_numbers now contains the matching numbers
and $adlist contains only numbers from AD
and $oraclelist only numbers from Oracle
you can then loop through the list and display values:
Write-Host "Matches:"
ForEach ($value in $Matching_numbers)
{
$Message += $value + [Environment]::NewLine
}
Write-Host $Message
Write-Host "AD only:"
ForEach ($value in $adlist)
{
$MessageAd += $value + [Environment]::NewLine
}
Write-Host $MessageAd
Write-Host "Oracle only:"
ForEach ($value in $oraclelist)
{
$MessageOracle += $value + [Environment]::NewLine
}
Write-Host $MessageOracle
or simply by writing
$Matching_numbers
will output the list to console
You can output the $Message variables to a file or so..
No doubt, there is a nicer way to do it, but this worked for me for a certain type of file.

Related

How to speed up the process in PowerShell

I've a simple script that generate orphan OneDrive report. It's working but it's very slow so I'm just wondering how could I modify my script to speed up the process. Any help or suggestion would be really appreciated.
Basically this is what I'm doing:
Get the owner email from "Owner" column and check it using AzureAD to see if I get any error
If I get an error then check it in on-prem ADGroup
If that owner is existed in the on-prem ADGroup then it's an orphan
Export only that user to a new csv file
$ImportData = "E:\scripts\AllOneDriveUser.csv"
$Report = "E:\scripts\OrphanOneDrive.csv"
$CSVImport = Import-CSV $ImportData
ForEach ($CSVLine in $CSVImport) {
$CSVOwner = $CSVLine.Owner
try{
Get-AzureADUser -ObjectId $CSVOwner
}catch{
$StatusMessage = $_.Exception.Message
if($Null -eq $StatusMessage){
Write-Host "User Found, Ignore from orphan list."
}else{
#Owner not found in AzureAD
$group = 'TargetGroup'
$filter = '(memberof={0})' -f (Get-ADGroup $group).DistinguishedName
$filterName = Get-ADUser -LDAPFilter $filter
$ModifiedOwner = $CSVOwner -split"#"[0]
if( $ModifiedOwner[0] -in $filterName.Name ){
Write-host "Adding it into orphaned list"
$CSVLine | Export-Csv $Report -Append -notypeinformation -force
}else{
Write-Host "Not orphaned"
}
}
}
}
I have over 8000 record in my import csv file and over 5000 member in my on-prem AD group so it taking very long.
You can greatly improve your script by using a HashSet<T> in this case, but also the main issue of your code is that you're querying the same group over and over, it should be outside the loop!
There is also the use of Export-Csv -Append, appending to a file per loop iteration is very slow, better to streamline the process with the pipelines so Export-Csv receives the objects and exports only once instead of opening and closing the FileStream everytime.
Hope the inline comments explain the logic you can follow to improve it.
$ImportData = "E:\scripts\AllOneDriveUser.csv"
$Report = "E:\scripts\OrphanOneDrive.csv"
# we only need to query this once! outside the try \ catch
# a HashSet<T> enables for faster lookups,
# much faster than `-in` or `-contains`
$filter = '(memberof={0})' -f (Get-ADGroup 'GroupName').DistinguishedName
$members = [Collections.Generic.HashSet[string]]::new(
[string[]] (Get-ADUser -LDAPFilter $filter).UserPrincipalName,
[System.StringComparer]::OrdinalIgnoreCase
)
Import-CSV $ImportData | ForEach-Object {
# hitting multiple times a `catch` block is expensive,
# better use `-Filter` here and an `if` condition
$CSVOwner = $_.Owner
if(Get-AzureADUser -Filter "userprincipalname eq '$CSVOwner'") {
# we know this user exists in Azure, so go next user
return
}
# here is for user not found in Azure
# no need to split the UserPrincipalName, HashSet already has
# a unique list of UserPrincipalNames
if($hash.Contains($CSVOwner)) {
# here is if the UPN exists as member of AD Group
# so output this line
$_
}
} | Export-Csv $Report -NoTypeInformation

Powershell script with Contains operator not working

I am completely stuck! Hoping someone can help point out what I'm doing wrong. PS script below.
For testing I have 2 users in "newusers.csv" and the same 2 users in "currentmembers.csv". These are teh same file just renamed. Results of script are that the 2 users in "newusers.csv" are NOT members of "currentmembers.csv". This is obviously wrong but I can't figure out what is wrong with the script.
$list = Import-Csv newusers.csv
$members = Import-Csv currentmembers.csv
foreach ($UPN in $list) {
If ($members-Contains $UPN.upn) {
Write-Host $UPN.upn "is member"
} Else {
Write-Host $UPN.upn "is not a member"
}}
Results
user 1 is not a member
user 2 is not a member
Your two Import-CSV calls are creating an array of objects, each containing two objects with 1 value (UPN). Your use of -contains will not work here as, you are not comparing directly to a string, more info here.
You have two options.
#1 you can use -in
foreach ($UPN in $list) {
If ($UPN.upn -in $members.upn) {
Write-Host $UPN "is member"
}
Else {
Write-Host $UPN "is not a member"
}
}
#2 You can use -match
foreach ($UPN in $list) {
If ($members -match $UPN.upn) {
Write-Host $UPN.upn "is member"
}
Else {
Write-Host $UPN.upn "is not a member"
}
}
Completely agree with Owain's answer, but I got halfway through mine so I thought I'd finish it. You can try playing with sample data directly in PowerShell as follows, using ConvertFrom-Csv, which saves you editing files and then hopping back into PowerShell:
$list = #'
upn
user 1
user 2
user 7
user 99
'# | ConvertFrom-Csv
$members = #'
upn
user 2
user 1
user 4
'# | ConvertFrom-Csv
foreach ($UPN in $list) {
If ($UPN.upn -in $members.upn) {
Write-Host $UPN.upn "is member"
} Else {
Write-Host $UPN.upn "is not a member"
}}
As Owain said, you need to alter the comparison part to compare a string to an array.
Your $member should be an array of strings, not array of objects. Same thing as you are comparing $upn.UPN - the $upn is a object and you're accessing UPN member of it.
Hope that I explained this clearly
$members = Import-Csv currentmembers.csv | select -expandproperty ColumnName

Check a username list and if they already exist Powershell

I have a CSV with 100 Usernames i have to check now if they already exist. Whats the best way to do that?
And is there a possibility that if the Username "marmar" is already used the programm checks by its own if username "marmar1" or if that is used aswell "marmar2" is free?
Is it easier to read the Usernames through the csv or should i copy them into Powershell?
examples of Usernames:
marmar
langas
ianmow
lowbob
berret
lawpaw1
etc.
Open for ideas and tipps.$
Thanks very much
If your CSV file looks anything like
"UserName","OtherStuff"
"marmar","blah"
"langas2","blahblah"
"ianmow","blahblahblah"
"lowbob","blahblahblahblah"
"berret","blahblahblahblahblah"
"lawpaw1","blahblahblahblahblahblah"
You can do that with something like this:
$freeUserNames = Import-csv 'D:\UsersNamesToCheck.csv' | ForEach-Object {
#Check to see if the user already exists in AD
$name = $_.UserName
$dupes = Get-ADUser -Filter "SamAccountName -like '$name*'" -ErrorAction SilentlyContinue | Select-Object -ExpandProperty SamAccountName
$index = 1
# check if the username is already in use and if so, increase the index number and test again
while ($dupes -contains $name) {
$name = '{0}{1}' -f ($name -replace '\d+$'), $index++
}
$name
}
# output on screen
Write-Host "Unused usernames:`r`n"
$freeUserNames
Possible output:
Unused usernames:
marmar
langas1
ianmow1
lowbob
berret1
lawpaw2

How to seperate CSV values within a CSV into new rows in PowerShell

I'm receiving an automated report from a system that cannot be modified as a CSV. I am using PowerShell to split the CSV into multiple files and parse out the specific data needed. The CSV contains columns that may contain no data, 1 value, or multiple values that are comma separated within the CSV file itself.
Example(UPDATED FOR CLARITY):
"Group","Members"
"Event","362403"
"Risk","324542, 340668, 292196"
"Approval","AA-334454, 344366, 323570, 322827, 360225, 358850, 345935"
"ITS","345935, 358850"
"Services",""
I want the data to have one entry per line like this (UPDATED FOR CLARITY):
"Group","Members"
"Event","362403"
"Risk","324542"
"Risk","340668"
"Risk","292196"
#etc.
I've tried splitting the data and I just get an unknown number of columns at the end.
I tried a foreach loop, but can't seem to get it right (pseudocode below):
Import-CSV $Groups
ForEach ($line in $Groups){
If($_.'Members'.count -gt 1, add-content "$_.Group,$_.Members[2]",)}
I appreciate any help you can provide. I've searched all the stackexchange posts and used Google but haven't been able to find something that addresses this exact issue.
Import-Csv .\input.csv | ForEach-Object {
ForEach ($Member in ($_.Members -Split ',')) {
[PSCustomObject]#{Group = $_.Group; Member = $Member.Trim()}
}
} | Export-Csv .\output.csv -NoTypeInformation
# Get the raw text contents
$CsvContents = Get-Content "\path\to\file.csv"
# Convert it to a table object
$CsvData = ConvertFrom-CSV -InputObject $CsvContents
# Iterate through the records in the table
ForEach ($Record in $CsvData) {
# Create array from the members values at commas & trim whitespace
$Record.Members -Split "," | % {
$MemberCount = $_.Trim()
# Check if the count is greater than 1
if($MemberCount -gt 1) {
# Create our output string
$OutputString = "$($Record.Group), $MemberCount"
# Write our output string to a file
Add-Content -Path "\path\to\output.txt" -Value $OutputString
}
}
}
This should work, you had the right idea but I think you may have been encountering some syntax issues. Let me know if you have questions :)
Revised the code as per your updated question,
$List = Import-Csv "\path\to\input.csv"
foreach ($row in $List) {
$Group = $row.Group
$Members = $row.Members -split ","
# Process for each value in Members
foreach ($MemberValue in $Members) {
# PS v3 and above
$Group + "," + $MemberValue | Export-Csv "\path\to\output.csv" -NoTypeInformation -Append
# PS v2
# $Group + "," + $MemberValue | Out-File "\path\to\output.csv" -Append
}
}

Multiple variables in Foreach loop [PowerShell]

Is it possible to pull two variables into a Foreach loop?
The following is coded for the PowerShell ASP. The syntax is incorrect on my Foreach loop, but you should be able to decipher the logic I'm attempting to make.
$list = Get-QADUser $userid -includeAllProperties | Select-Object -expandproperty name
$userList = Get-QADUser $userid -includeAllProperties | Select-Object -expandproperty LogonName
if ($list.Count -ge 2)
{
Write-host "Please select the appropriate user.<br>"
Foreach ($a in $list & $b in $userList)
{
Write-host "<a href=default.ps1x?UserID=$b&domain=$domain>$b - $a</a><br>"}
}
}
Christian's answer is what you should do in your situation. There is no need to get the two lists. Remember one thing in PowerShell - operate with the objects till the last step. Don't try to get their properties, etc. until the point where you actually use them.
But, for the general case, when you do have two lists and want to have a Foreach over the two:
You can either do what the Foreach does yourself:
$a = 1, 2, 3
$b = "one", "two", "three"
$ae = $a.getenumerator()
$be = $b.getenumerator()
while ($ae.MoveNext() -and $be.MoveNext()) {
Write-Host $ae.current $be.current
}
Or use a normal for loop with $a.length, etc.
Try like the following. You don't need two variables at all:
$list = Get-QADUser $userid -includeAllProperties
if ($list.Count -ge 2)
{
Write-Host "Please select the appropriate user.<br>"
Foreach ($a in $list)
{
Write-Host "<a href=default.ps1x?UserID=$a.LogonName&domain=$domain>$a.logonname - $a.name</a><br>"
}
}