Set-S3Acl Powershell cmdlet not working post 20-30k objects - powershell

Question

My client had an issue, i.e., they accidentally copied 13 million Objects (files) to S3 bucket with wrong permissions. They have asked my team to fix it. We have to update each 13 million files in the S3 bucket with correct ACLs. We are using below powershell script to fix it. However, when the script runs on a folder with more than 20-30k objects, it fails to set the ACLs. [It iterates thru the loop, but it wont set the permission post 20-30k objects, no exception either]
I am suspecting that the requests might be getting throttled. Have any one of you came across such issue. Please help me on how to proceed.
I am looking for answers for the below questions:
1. If the API calls are getting throttled # 20-30k objects, how can I modify my script to overcome it.
2. What is the best practice in terms of scripting to "modify" AWS resources (like set ACL permission to S3 objects) for millions of objects
(I am not looking for the "BucketPolicy" approach, as we have to do it with a script and apply the ACLs to every S3 object)
Param (
[Parameter(Position=0,Mandatory=$true)]
[string]$profile,
[Parameter(Position=1,Mandatory=$true)]
[string]$switchToAccount,
[Parameter(Position=2,Mandatory=$true)]
[string]$roleName,
[Parameter(Position=3,Mandatory=$true)]
[string]$keyPrefix
)
#Set base AWS credentials
Set-AWSCredentials -ProfileName $profile
Set-DefaultAWSRegion -Region $region
#Get and set MFA device ARN
$userName = (Get-IAMUser).UserName
$mfaArn = "arn:aws:iam::xxxxxxxxx:mfa/" + "$userName"
#Configure CAA roles
$roleArn = "arn:aws:iam::" + "$switchToAccount" + ":role/" + "$roleName"
$roleSessionName = "xxxxxxxxxxxx"
#Prompt for MFA token and perform CAA request
$tokenCode = Read-Host -Prompt "Enter MFA token for $accountNumber"
$switchRole = Use-STSRole -RoleSessionName $roleSessionName -RoleArn $roleArn -TokenCode $tokenCode -SerialNumber $mfaArn
#Set new role for CAA
Set-AWSCredentials -Credential $switchRole.Credentials
#Declare access level for S3 Object ACL grantees
$FULL_CONTROL = [Amazon.S3.S3Permission]::FULL_CONTROL
$grants = #();
#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxxxxxx
$grantee1 = New-Object -TypeName Amazon.S3.Model.S3Grantee
$grantee1.EmailAddress = "xxxxxxxxxxxxxxxxxxx"
#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxx
$grantee2 = New-Object -TypeName Amazon.S3.Model.S3Grantee
$grantee2.EmailAddress = "xxxxxxxxxxxxxxxxxxx"
#Grant FULL_CONTROL access to xxxxxxxxxxxxxxxxxxxx
$grantee3 = New-Object -TypeName Amazon.S3.Model.S3Grantee
$grantee3.EmailAddress = "xxxxxxxxxxxxxxxxxxxxx"
#Create grant and add to grant list
$grant1 = New-Object -TypeName Amazon.S3.Model.S3Grant
$grant1.Grantee = $grantee1
$grant1.Permission = $FULL_CONTROL
$grants += $grant1
#Create grant and add to grant list
$grant2 = New-Object -TypeName Amazon.S3.Model.S3Grant
$grant2.Grantee = $grantee2
$grant2.Permission = $FULL_CONTROL
$grants += $grant2
#Create grant and add to grant list
$grant3 = New-Object -TypeName Amazon.S3.Model.S3Grant
$grant3.Grantee = $grantee3
$grant3.Permission = $FULL_CONTROL
$grants += $grant3
#Set bucket name for S3 objects
$bucketName = "xxxxxxxxxxxxxxxxxxxxxxxxx"
#Get all S3 Objects in specified bucket
$s3Objects = Get-S3Object -BucketName $bucketName -KeyPrefix $keyPrefix
#Count for progress bar
$totalObjects = $s3Objects.length
$i = 1
$fail_count = 0
$current_count = 0
$file_path = "C:\Users\Administrator\Desktop\Failed_Objects_new\" + $keyPrefix.Replace("/","_") + ".txt"
$file_path_retry = "C:\Users\Administrator\Desktop\Failed_Objects_new_retry\" + $keyPrefix.Replace("/","_") + ".txt"
new-item $file_path -ItemType file
new-item $file_path_retry -ItemType file
"Total Object Count:" + $totalObjects + "`n" | Out-File $file_path -Append
foreach($s3Object in $s3Objects){
$owner = $s3Object.owner.id
$s3Object.name | Write-Output
$current_count++
#Extracts Key for each S3 object in bucket
$key = $s3Object.Key
#Logging
Write-Host "Setting $bucketName | $key | $grants"
# Pick objects that were modified on or before July 15th
try {
if (($s3Object.LastModified.month -lt 7)) {
Set-S3ACL -BucketName $bucketName -Key $key -Grant $grants -OwnerId $owner
$owner | Write-Host
}
elseif(($s3Object.LastModified.month -eq 7) -and ($s3Object.LastModified.day -le 15)) {
Set-S3ACL -BucketName $bucketName -Key $key -Grant $grants -OwnerId $owner
$owner | Write-Host
}
}catch{
"Failed $bucketName | $key | $grants" | out-file $file_path -Append
$key | Out-File $file_path_retry -Append
$fail_count++
}
Write-Host "progress: " $current_count "/" $totalObjects
#Update progress bar
$percentComplete = $i/$totalObjects
Write-Progress -Activity "Setting S3 Object ACL's" -Status "$i% complete" -PercentComplete $percentComplete
$i++
}
"`n`n Total Fail Count:" + $fail_count | Out-File $file_path -Append

Answer 1

Steps to debug the problem:
Make sure if it is throttling issue. In for loop; break after 10k objects and see if everything works fine.
Also, put print statements inside try block both if and else.. to make sure if its reaching there or not; and when is it failing.