longer execution time after Get-ADUser when a server is specified - powershell

Question

I have a script that queries AD and processes the result:
$users=Get-ADUser -filter {...} -Properties ...
try {
foreach($user in $users) {
$obj=new-object PSObject #property {...}
[void]($arraylist.add($obj))
}
}
catch {...}
the Get-ADUser takes about 1.5sec and returns 1000+ rows, and the whole try block takes about 0.5sec.
the #Properties {...} part only processes $user fields and has no further interaction with the AD or any other resource. the try block runs without error.
if now I change the first line to
$users=Get-ADUser -filter {...} -Properties ... -Server <someDC>
just adding a -Server parameter, the Get-ADUsercommand takes about 3sec and the try block 20+sec
I tried many times to be sure that the remote connection is well established without any improvement. I tried with each of the available DCs and got the very same execution time each time. If I remove the -Server parameter, I get the original execution time back.
the problem has similarities with the one described in this post but the my problem is with the code executed after the Get-ADUser, not with the Get-ADUser by itself.
what is very disturbing is that it's the code after which get slower.
do you have any clue?
DC: Win2008R2
Client: Win7, Powershell5.1, AD Management Tools locally installed.