Powershell Shadow Groups - Inconsistent Group Membership - powershell

Question

I am using Powershell to maintain several shadow groups (a group that mirrors all the users in a specific OU.)
I have three OUs, Staff, Faculty, and Administration. Each has a group associated with it, with the Staff group encompassing all three groups.
Here is the code:
$server="win-ad1.example.com"
#Staff
$AdministrationOU="OU=Administration,OU=Accounts,DC=example,DC=com"
$FacultyOU="OU=Faculty,OU=Accounts,DC=example,DC=com"
$StaffOU="OU=Staff,OU=Accounts,DC=example,DC=com"
$ShadowGroup="CN=Staff,OU=User Groups,OU=Accounts,DC=example,DC=com"
Get-ADGroupMember -Server $server –Identity $ShadowGroup `
| Where-Object {($_.distinguishedName –NotMatch $AdministrationOU) `
-OR ($_.distinguishedName –NotMatch $FacultyOU) `
-OR ($_.distinguishedName –NotMatch $StaffOU)} `
| ForEach-Object {Remove-ADPrincipalGroupMembership -Server $server –Identity $_ –MemberOf $ShadowGroup –Confirm:$false}
Sleep -Seconds 2
Get-ADUser -Server $server –SearchBase $AdministrationOU –SearchScope OneLevel –LDAPFilter "(!(memberOf=$ShadowGroup))" `
| ForEach-Object {Add-ADPrincipalGroupMembership -Server $server –Identity $_ –MemberOf $ShadowGroup}
Get-ADUser -Server $server –SearchBase $FacultyOU –SearchScope OneLevel –LDAPFilter "(!(memberOf=$ShadowGroup))" `
| ForEach-Object {Add-ADPrincipalGroupMembership -Server $server –Identity $_ –MemberOf $ShadowGroup}
Get-ADUser -Server $server –SearchBase $StaffOU –SearchScope OneLevel –LDAPFilter "(!(memberOf=$ShadowGroup))" `
| ForEach-Object {Add-ADPrincipalGroupMembership -Server $server –Identity $_ –MemberOf $ShadowGroup}
#Administration
$ShadowGroup="CN=Administration,OU=User Groups,OU=Accounts,DC=example,DC=com"
Get-ADGroupMember -Server $server –Identity $ShadowGroup `
| Where-Object {$_.distinguishedName –NotMatch $AdministrationOU} `
| ForEach-Object {Remove-ADPrincipalGroupMembership -Server $server –Identity $_ –MemberOf $ShadowGroup –Confirm:$false}
Sleep -Seconds 2
Get-ADUser -Server $server –SearchBase $AdministrationOU –SearchScope OneLevel –LDAPFilter "(!(memberOf=$ShadowGroup))" `
| ForEach-Object {Add-ADPrincipalGroupMembership -Server $server –Identity $_ –MemberOf $ShadowGroup}
Get-ADGroupMember -Server $server –Identity $ShadowGroup `
| Enable-ADAccount -Server $server
....
#same code for Faculty
This is (I think) pretty standard code, but I admit to writing very little powershell. The code should remove all the users in the group that aren't in the OU, then add all the users (not already in the group) that are in the OU to the group.
In theory, this works great, however in practice adding/removing users to the Staff group can go massively wrong. In one run of the script, it will suddenly remove swathes of (random) users from the group, despite them obviously being in one of the OUs. A run of the script again may add them back or take out other users. Many time the script runs perfectly.
I had initially chalked it up to the script talking to different domain controllers, so I added the -Server argument to each Active Directory call, so that it would hit the same (primary) domain controller each time. The problem still occurs, and in all of my searching I can't find anyone with the same issue (probably because I'm not really sure how to phrase what is happening.)
Any help would be appreciated.
Note, for scale here are the size of the OUs:
Staff: 199
Faculty: 194
Administration: 32