NetworkVariable ownership write permissions and ChangeOwnership race condition

[zachstronaut]

I have a NetworkVariable with writePerm: NetworkVariableWritePermission.Owner in one NetworkBehaviour and on that same NetworkObject I have another NetworkBehaviour that uses ChangeOwnership from within a coroutine.

When ownership changes, I notice I get this error sometimes on the host:

[Netcode] Client wrote to NetworkVariable`1 without permission. No more variables can be read. This is critical. => NetworkObjectId: 95 - GetNetworkBehaviourOrderIndex(): 0 - VariableIndex: 0

The code writing to the NetworkVariable is in Update() and I have tried wrapping it with IsOwner and NetworkObject.IsOwner and neither of those prevent this error message.

It seems like maybe there is a race condition here with netcode internals?

I am using netcode 2.4.2 with a Host-Client topology. In this case a host with one additional client.

[NoelStephensUnity]

@zachstronaut
Could you provide the network topology you are using in the description as it helps to know which authority model the issue could be occurring (i.e. client-server or distributed authority)? However, it seems as if you are most likely using a client-server network topology due to the nature of the error message.

That error message typically means that the server is receiving changes to a NetworkVariable after having already changed the owner of the NetworkObject. This message would not normally be logged with the log level set to Normal or higher.

If I understand correctly, the issue is something along the lines of:
Host, Client-A, Client-B, and NetworkBehaviour-A

• Client-A starts as the owner.
• Client-A makes changes to a NetworkVariable on NetworkBehaviour-A
  • Delta message is sent and considered "in-flight".
• Host changes ownership to Client-B while the delta message is still in-flight.
• Client-A receives the ownership change shortly after having sent the delta message.
• Host receives the delta message from Client-A
  • Host basically ignores this change since Client-B is now the owner.
  • If the LogLevel is set to Developer then the "No more variables can be read." message is logged to let you know that there was still an in-flight message when the ownership changed and you might have lost a delta from the previous owner.

Since ownership changes are immediate (especially on the Host side), then you could run into this issue where the end result is that the previous owner's last deltas were not synchronized by the host to all other clients since the new owner is dictating the state of that NetworkVariable.

The frequency in which this could happen depends upon how often changes to the NetworkVariable could be made (i.e. if every tick then a higher chance it could occur than if every 6+ ticks). The over-all assumption is that if the server changes ownership then the state of the NetworkVariable(s) on the server side instance are the last known states of the previous owner and from that point forward the new owner dictates the states of those NetworkVariables. Any in-flight delta messages from the previous owner will be ignored.

If you would like us to provide feedback specific to your project (since this seems to be a timing related issue), then if you were to submit a replication project or file an issue within the editor (and post the IN number here) we would be better able to narrow down the best solution to avoid this from happening....but in reality client write NetworkVariables can have this kind of potential "in-flight" race condition from time to time depending upon latencies.

As a side note: I think this error message needs to be updated to better describe the scenario taking place so we will definitely be taking care of that.

[zachstronaut]

Hello! I have a host and one additional client in this bug case. I am seeing this with two editors and parrelsync. I often test with more realistic modest simulated pings like ~200ms. I do have my LogLevel set to Developer.

There is a NetworkObject with NetworkBehaviours that starts as a scene object owned by the host. One of the NetworkBehaviours is tracking the velocity of the Rigidbody on that object. (We're not using NetworkRigidbody for reasons, and a kinematic Rigidbody on the non-owner has no velocity, so we share it with the non-owner as a NetworkVariable.) The velocity NetworkVariable gets set every Update() by the object owner, and so whenever there is a tick whatever the last value written to it in Update() is what is gonna get sent.

The host can detect a scenario where ownership needs to change (can go from host to client, or from client back to host). It's when the host makes this change that the reported error happens even though the write to the NetworkVariable is wrapped in IsOwner.

Sounds like maybe this error isn't a big deal?

[NoelStephensUnity]

There is a NetworkObject with NetworkBehaviours that starts as a scene object owned by the host. One of the NetworkBehaviours is tracking the velocity of the Rigidbody on that object. (We're not using NetworkRigidbody for reasons, and a kinematic Rigidbody on the non-owner has no velocity, so we share it with the non-owner as a NetworkVariable.) The velocity NetworkVariable gets set every Update() by the object owner, and so whenever there is a tick whatever the last value written to it in Update() is what is gonna get sent.
The host can detect a scenario where ownership needs to change (can go from host to client, or from client back to host). It's when the host makes this change that the reported error happens even though the write to the NetworkVariable is wrapped in IsOwner.

Since it sounds like it could need to have varying adjustments and you are dealing with order of operation issues while changing ownership, it sounds like you might want to use more of a synchronized RPC driven property?

Have you tried something like this?

using System.Runtime.CompilerServices;
using Unity.Netcode;
using UnityEngine;

public class VelocitySynchronizer : NetworkBehaviour, INetworkUpdateSystem
{
    public float VelocityDelta = 0.1f;

    protected Vector3 Velocity
    {
        get
        {
            return m_Velocity;
        }
    }
    private Vector3 m_Velocity;

    protected Rigidbody Rigidbody;

    /// <summary>
    /// Synchronize late joining clients to the RPC driven local <see cref="m_Velocity"/> property
    /// </summary>
    protected override void OnSynchronize<T>(ref BufferSerializer<T> serializer)
    {
        serializer.SerializeValue(ref m_Velocity);
        base.OnSynchronize(ref serializer);
    }

    private void Awake()
    {
        Rigidbody = GetComponent<Rigidbody>();
    }

    protected override void OnNetworkPostSpawn()
    {
        ConfigureOwnerUpdate();
        base.OnNetworkPostSpawn();
    }

    protected override void OnOwnershipChanged(ulong previous, ulong current)
    {
        var localClientId = NetworkManager.LocalClientId;
        if (previous == localClientId || current == localClientId)
        {
            ConfigureOwnerUpdate();
        }
        base.OnOwnershipChanged(previous, current);
    }

    private void ConfigureOwnerUpdate()
    {
        NetworkUpdateLoop.UnregisterNetworkUpdate(this);
        if (IsOwner)
        {
            NetworkUpdateLoop.RegisterNetworkUpdate(this, NetworkUpdateStage.FixedUpdate);
        }
    }

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    protected bool Approximately(Vector3 a, Vector3 b)
    {
        return System.Math.Round(Mathf.Abs(a.x - b.x), 3) <= VelocityDelta &&
            System.Math.Round(Mathf.Abs(a.y - b.y), 3) <= VelocityDelta &&
            System.Math.Round(Mathf.Abs(a.z - b.z), 3) <= VelocityDelta;
    }

    public void NetworkUpdate(NetworkUpdateStage updateStage)
    {
        if (!Approximately(m_Velocity, Rigidbody.linearVelocity))
        {
            OnUpdateVelocity(Rigidbody.linearVelocity);
        }
    }

    protected virtual void OnVelocityUpdated()
    {

    }

    private void OnUpdateVelocity(Vector3 velocity)
    {
        m_Velocity = velocity;
        if (IsOwner)
        {
            SynchronizeVelocityRpc(velocity);
        }
    }

    [Rpc(SendTo.NotMe)]
    private void SynchronizeVelocityRpc(Vector3 velocity, RpcParams rpcParams = default)
    {
        // If an edge case occurs where ownership changed while there were in-flight packets
        // then just ignore the old owner's update (or optionally you could apply it depending
        // upon what seems to work out the best for you)
        if (rpcParams.Receive.SenderClientId != OwnerClientId)
        {
            return;
        }
        OnUpdateVelocity(velocity);
    }
}

Sounds like maybe this error isn't a big deal?

Well... you could run into minor issues on the previous owner side but yeah your gut instinct on severity is correct.
Of course, for things that could require frequent updates I would recommend using more of a synchronized RPC driven local property approach where you have a more granular control over how much of a change is enough to trigger synchronizing the change and which instance is being invoked every fixed update.

[zachstronaut]

Cool, thanks for that. I guess I'll make a decision whether I'm content with my barebones solution with rough edges or if I should go with a more thorough handling of the edge cases like you provided.