unity network variabe occasional writing permission error - unity3d

Question

I am new to unity netcode and I am trying to implement a grabbing/dropping mechanism for a 3rd person game.
But my network variable occasionally throws a confusing error out at the host side when client side "drop" something:
[Netcode] Client wrote to NetworkVariable`1 without permission. No more variables can be read. This is critical. => NetworkObjectId: 5 - GetNetworkBehaviourOrderIndex(): 1 - VariableIndex: 0
Sometimes it throws 1 of this error, sometimes it throws 2 and in rare case it doesn't throw error at all.
What is the possible reason for this?
Basically my approach is:
(below using clientNetworkTransform)
Grab:
calling grabServerRpc( callerClientId ) in the Grabbable object by the local player
passing the ownership of the Grabbable setting the network variable grabbedClientId to callerClientId in grabServerRpc()
in the OnValueChange delegate for every player, check if LocalPlayerId == grabbedClientId
if equal, set the local targetTransform to the grabbing point of the player and serval more properties to complete the grab
(the OnValueChange also check if grabbedClientId == Default value before doing anything to prevent execution due to dropServerRpc())
Drop:
unset the properties to drop it.
call the dropServerRpc() in the Grabbable
removeOwnership() in the dropServerRpc()
restore the grabbedClientId to default value to complete the drop
public void grab(ulong grabbedClientId){
this.grabServerRpc(grabbedClientId);
}
[ServerRpc(RequireOwnership = false)]
private void grabServerRpc(ulong grabClientId){
if (this.grabbedClientId != TPNetworkGrabbable.GRABBED_CLIENT_DEFAULT) return;
this.networkObject.ChangeOwnership(grabClientId);
this.grabbedClientId = grabClientId;
print("grabbed");
return;
}
private void OnGrab(ulong previous, ulong current){ // OnValueChange for grabbedClientId
if (this.grabbedClientId == TPNetworkGrabbable.GRABBED_CLIENT_DEFAULT) return; // prevent calling due to changes in drop()
if (this.grabbedClientId != NetworkManager.LocalClientId) return;
TPNetworkGrabbingControl grabbingControl = NetworkManager.LocalClient.PlayerObject.GetComponent<TPNetworkGrabbingControl>();
this._targetTransform = grabbingControl.grabPosition;
this.rigidbody.useGravity = false;
this.transform.rotation = Quaternion.Euler(0, 0, 0);
this.rigidbody.freezeRotation = true;
}
public void drop(){
if (this.grabbedClientId != NetworkManager.LocalClientId) return;
if (this.grabbedClientId == TPNetworkGrabbable.GRABBED_CLIENT_DEFAULT) return;
this._targetTransform = null;
this.rigidbody.useGravity = true;
this.rigidbody.freezeRotation = false;
this.dropServerRpc();
}
[ServerRpc(RequireOwnership = false)]
private void dropServerRpc(){
this.networkObject.RemoveOwnership();
this.grabbedClientId = TPNetworkGrabbable.GRABBED_CLIENT_DEFAULT;
print("dropped");
}
what I have done:
global searched my project and every single modification to any network variable are already done in [ServerRpc] and they are server write authoritive by default.
Googled and found basically nothing related to this specific error, except this:
though the solution seems to not apply to my case and the example does not exist anymore.