Senior Network Gameplay Programmer

Replication Graph – How To Reduce Network Bandwidth In Unreal

on

One of the main metrics we need to keep under control when developing networked games is how much bandwidth is required for a stable gameplay experience. If we try to send or receive too much data then we can saturate the weakest players connection and therefore degrade the player experience, as we need to prioritise some information and delay others. As you’d expect this is a big issue in Fortnite due to having 100 players in a map, so Epic have provided the Replication Graph as a solution.

Bandwidth limits are often fairly easy to stay under when you’ve only got one or two players in a session, but as you increase the number of connected clients you also increase the amount of data that must be sent to keep them all in sync. This effect is exponential as there’s not just more data to send but you also need to send it to more destinations.

One of the core systems that Unreal provides for handling this problem is Actor Relevancy. I’d recommend reading the linked article for a more in depth explanation, but in short it’s a system that will remove an actor from a client’s world while they are too far away from the players view. This means that the client will no longer require any updates when data is changed on that actor and therefore eliminates the need to send any packets.

Unfortunately distance based culling can only get you so far. Fog of war is a common gameplay feature where you can only see enemy team members if one of your team members has line of sight. The Replication Graph is a feature that can be used to specify more complex rules on what each player should receive. In the case of fog of war games, this benefit can double up to prevent having any of this data available on the client which could be used for cheats such as wall hacks. As the server frees up some of its time spent sending packets, you’ll also see substantial CPU time gains from this optimisation.

Setting Up The Replication Graph

First of all we need to enable the plugin like any other plugin, by adding “ReplicationGraph” to PrivateDependencyModuleNames in your build.cs and adding the plugin to enabled plugins in your .uproject.

"Plugins": [
		{
			"Name": "ReplicationGraph",
			"Enabled": true
		}
	]

We can then go ahead to create a new class that inherits from UReplicationGraph.

#pragma once

#include "CoreMinimal.h"
#include "ReplicationGraph.h"
#include "TutorialRepGraph.generated.h"

UCLASS(Blueprintable)
class REPGRAPH_API UTutorialRepGraph : public UReplicationGraph
{
	GENERATED_BODY()
};

Finally we need to add a line to our DefaultEngine.ini which will tell the NetDriver which replication graph to use. I’m going to create a blueprint child of our new UTutorialRepGraph class to make it a bit easier to set up any references or settings so my DefaultEngine.ini will point to that. There is also an option to bind to a delegate if you require different rep graphs for different game modes which you can read more about in Unreal’s documentation.

[/Script/OnlineSubsystemUtils.IpNetDriver]
ReplicationDriverClassName="/Game/BP_TutorialRepGraph.BP_TutorialRepGraph_C"

Setting Up The Connection Manager And Nodes

A node is how we specify rules for who should receive what data and we can add or remove actors to these nodes as we need. Each connection is assigned their own connection manager which holds the nodes for any unique rules. For this example I’d like to recreate a fog of war scenario where players can always see other members of their own team, but can only see the opposing players when they have entered the player’s line of sight. To visualise this I have set up characters which get assigned to a random team when spawned and coloured to match.

The first step is to set up the different type of nodes we will need to make this happen. We have two functions we must override to do this: InitGlobalGraphNodes and InitConnectionGraphNodes. These functions give us the opportunity to set up nodes that all connections and actors will require and nodes that specify rules per player.

void UTutorialRepGraph::InitGlobalGraphNodes()
{
	Super::InitGlobalGraphNodes();

  // Create the always relevant node
	AlwaysRelevantNode = CreateNewNode<UReplicationGraphNode_AlwaysRelevant_WithPending>();
	AddGlobalGraphNode(AlwaysRelevantNode);
}

void UTutorialRepGraph::InitConnectionGraphNodes(UNetReplicationGraphConnection* ConnectionManager)
{
	Super::InitConnectionGraphNodes(ConnectionManager);
	
	// Create the connection graph for the incoming connection
	UTutorialConnectionManager* TutorialConnectionManager = Cast<UTutorialConnectionManager>(ConnectionManager);

	if (ensure(TutorialRepGraph))
	{
		TutorialConnectionManager->AlwaysRelevantForConnectionNode = CreateNewNode<UReplicationGraphNode_AlwaysRelevant_ForConnection>();
		AddConnectionGraphNode(TutorialRepGraph->AlwaysRelevantForConnectionNode, ConnectionManager);
		
		TutorialConnectionManager->TeamConnectionNode = CreateNewNode<UReplicationGraphNode_AlwaysRelevant_ForTeam>();
		AddConnectionGraphNode(TutorialRepGraph->TeamConnectionNode, ConnectionManager);
	}
}

We can add all actors to the AlwaysRelevantNode that every player needs access to, such as the game state. There is also a bool here, bRequiresPrepareForReplicationCall which will enable calls to PrepareForReplication before a connection is ready to replicate. We can talk more about that later.

UReplicationGraphNode_AlwaysRelevant_WithPending::UReplicationGraphNode_AlwaysRelevant_WithPending()
{
  // Call PrepareForReplication before replication once per frame
	bRequiresPrepareForReplicationCall = true;
}

void UReplicationGraphNode_AlwaysRelevant_WithPending::PrepareForReplication()
{
	UTutorialRepGraph* ReplicationGraph = Cast<UTutorialRepGraph>(GetOuter());
	ReplicationGraph->HandlePendingActorsAndTeamRequests();
}

Each connection has its own connection manager in which we specify the class in the constructor and it is also added to its own UReplicationGraphNode_AlwaysRelevant_ForConnection and UReplicationGraphNode_AlwaysRelevant_ForTeam nodes. The _ForConnection node is used for actors which are only relevant to this connection like the PlayerController. The _ForTeam node is what we will use to show other members of the same team. We’re using GatherActorListsForConnection here which will be called every tick to provide a list of nodes with the same team index as itself and will therefore be relevant.

UTutorialRepGraph::UTutorialRepGraph()
{
  // Specify the connection graph class to use
	ReplicationConnectionManagerClass = UTutorialConnectionManager::StaticClass();
}

UCLASS()
class UReplicationGraphNode_AlwaysRelevant_ForTeam : public UReplicationGraphNode_ActorList
{
	GENERATED_BODY()

	virtual void GatherActorListsForConnection(const FConnectionGatherActorListParameters& Params) override;
	virtual void GatherActorListsForConnectionDefault(const FConnectionGatherActorListParameters& Params);
};

UCLASS()
class UTutorialConnectionManager : public UNetReplicationGraphConnection
{
	GENERATED_BODY()

public:
  // UReplicationGraphNode_AlwaysRelevant_ForConnection is a node type provided by Epic for actors always relevant to a connection
	UPROPERTY()
	UReplicationGraphNode_AlwaysRelevant_ForConnection* AlwaysRelevantForConnectionNode;
	
	UPROPERTY()
	UReplicationGraphNode_AlwaysRelevant_ForTeam* TeamConnectionNode;

	int32 Team = -1;
};

void UReplicationGraphNode_AlwaysRelevant_ForTeam::GatherActorListsForConnection(
	const FConnectionGatherActorListParameters& Params)
{
	// Get all other team members with the same team ID from ReplicationGraph->TeamConnectionListMap
	UTutorialRepGraph* ReplicationGraph = Cast<UTutorialRepGraph>(GetOuter());
	const UTutorialConnectionManager* ConnectionManager = Cast<UTutorialConnectionManager>(&Params.ConnectionManager);
	if (ReplicationGraph && ConnectionManager && ConnectionManager->Team != -1)
	{
		if (TArray<UTutorialConnectionManager*>* TeamConnections = ReplicationGraph->TeamConnectionListMap.GetConnectionArrayForTeam(ConnectionManager->Team))
		{
			for (const UTutorialConnectionManager* TeamMember : *TeamConnections)
			{
				TeamMember->TeamConnectionNode->GatherActorListsForConnectionDefault(Params);
			}
		}
	}
	else
	{
		Super::GatherActorListsForConnection(Params);
	}
}

void UReplicationGraphNode_AlwaysRelevant_ForTeam::GatherActorListsForConnectionDefault(
	const FConnectionGatherActorListParameters& Params)
{
	Super::GatherActorListsForConnection(Params);
}

Adding Each Actor To The Correct Node

Now that we’re set up and each connection gets its own connection manager we need to add each actor to the correct node. Every actor will pass through the RouteAddNetworkActorToNodes function where we can then call NotifyAddNetworkActor if that actor needs a specific rule. Below is a simple example of what you can do to get things working but you’ll likely have many actors with different rules and would benefit from creating a better system for this (as shown in the Locus Replication Graph example).

void UTutorialRepGraph::RouteAddNetworkActorToNodes(const FNewReplicatedActorInfo& ActorInfo,
                                                    FGlobalActorReplicationInfo& GlobalInfo)
{
	// All clients must receive game states and player states
	if (ActorInfo.Class->IsChildOf(AGameStateBase::StaticClass()) || ActorInfo.Class->IsChildOf(APlayerState::StaticClass()))
	{
		AlwaysRelevantNode->NotifyAddNetworkActor(ActorInfo);
	}
	// If not we see if it belongs to a connection
	else if (UTutorialConnectionManager* ConnectionManager = GetTutorialConnectionManagerFromActor(ActorInfo.GetActor()))
	{
		if (ActorInfo.Actor->bOnlyRelevantToOwner)
		{
			ConnectionManager->AlwaysRelevantForConnectionNode->NotifyAddNetworkActor(ActorInfo);
		}
		else
		{
			ConnectionManager->TeamConnectionNode->NotifyAddNetworkActor(ActorInfo);
		}
	}
	else if(ActorInfo.Actor->GetNetOwner())
	{
	  // Add to PendingConnectionActors if the net connection is not ready yet
		PendingConnectionActors.Add(ActorInfo.GetActor());
	}
}

GetTutorialConnectionManagerFromActor is a simple function that will either get or create a connection manager for any actor with a net connection. 

UTutorialConnectionManager* UTutorialRepGraph::GetTutorialConnectionManagerFromActor(const AActor* Actor)
{
	if (Actor)
	{
		if (UNetConnection* NetConnection = Actor->GetNetConnection())
		{
			if (UTutorialConnectionManager* ConnectionManager = Cast<UTutorialConnectionManager>(FindOrAddConnectionManager(NetConnection)))
			{
				return ConnectionManager;
			}
		}
	}
	
	return nullptr;
}

Unfortunately there is some time where the net connection is not available yet so this function will return a nullptr. You can see above in RouteAddNetworkActorToNodes we’re catching that case if the actor has a net owner and adding it to the PendingConnectionActors array so that we can process it later. 

void UTutorialRepGraph::HandlePendingActorsAndTeamRequests()
{	
  // Set up all pending connections
	if (PendingConnectionActors.Num() > 0)
	{
		TArray<AActor*> PendingActors = MoveTemp(PendingConnectionActors);

		for (AActor* Actor : PendingActors)
		{
			if (IsValid(Actor))
			{
				FGlobalActorReplicationInfo& GlobalInfo = GlobalActorReplicationInfoMap.Get(Actor);
				RouteAddNetworkActorToNodes(FNewReplicatedActorInfo(Actor), GlobalInfo);
			}
		}
	}
}

This function is called by the previous PrepareForReplication call and will just repeat any setup logic for each of the PendingConnectionActors to make sure they’re properly set up and assigned to the correct nodes.

This is the bulk of the player setup but it does leave out one critical piece of the puzzle which is setting the team index in the connection graph.

Setting The Player’s Team

void UTutorialRepGraph::SetTeamForPlayerController(APlayerController* PlayerController, int32 Team)
{
	if (PlayerController)
	{
		if (UTutorialConnectionManager* ConnectionManager = GetTutorialConnectionManagerFromActor(PlayerController))
		{
			const int32 CurrentTeam = ConnectionManager->Team;
			if (CurrentTeam != Team)
			{
			  // Remove the connection to the old team list
				if (CurrentTeam != -1)
				{
					TeamConnectionListMap.RemoveConnectionFromTeam(CurrentTeam, ConnectionManager);
				}

        // Add the graph to the new team list
				if (Team != -1)
				{
					TeamConnectionListMap.AddConnectionToTeam(Team, ConnectionManager);
				}
				
				ConnectionManager->Team = Team;
			}
		}
		else
		{
		  // Add to PendingTeamRequests if the net connection is not ready yet
			PendingTeamRequests.Emplace(Team, PlayerController);
		}
	}
}

SetTeamForPlayerController is a function that adds the PlayerController to the correct team index in the TeamConnectionListMap. The TeamConnectionListMap is just a map with the team index as the key and a list of each connection graph assigned to that team as the value, so that we can quickly query it as needed (and we’ve already used it in GatherActorListsForConnection). You can see here we may have the same issue as earlier where we try to set the team before the net connection is available, so we also have to store an array of pending team requests and also add that setup functionality to HandlePendingActorsAndTeamRequests.

void UTutorialRepGraph::HandlePendingActorsAndTeamRequests()
{
  // Setup all pending team requests
	if(PendingTeamRequests.Num() > 0)
	{
		TArray<TTuple<int32, APlayerController*>> TempRequests = MoveTemp(PendingTeamRequests);

		for (const TTuple<int32, APlayerController*>& Request : TempRequests)
		{
			if (IsValid(Request.Value))
			{
				SetTeamForPlayerController(Request.Value, Request.Key);
			}
		}
	}
	
	//... Handle Pending Actors
}

The FTeamConnectionListMap has a few functions to improve the usability of adding and removing connection graphs from the map.

struct FTeamConnectionListMap : TMap<int32, TArray<UTutorialConnectionManager*>>
{
	TArray<UTutorialConnectionManager*>* GetConnectionArrayForTeam(int32 Team);
	
	void AddConnectionToTeam(int32 Team, UTutorialConnectionManager* ConnManager);
	void RemoveConnectionFromTeam(int32 Team, UTutorialConnectionManager* ConnManager);
};

TArray<UTutorialConnectionManager*>* FTeamConnectionListMap::GetConnectionArrayForTeam(int32 Team)
{
	return Find(Team);
}

void FTeamConnectionListMap::AddConnectionToTeam(int32 Team, UTutorialConnectionManager* ConnManager)
{
	TArray<UTutorialConnectionManager*>& TeamList = FindOrAdd(Team);
	TeamList.Add(ConnManager);
}

void FTeamConnectionListMap::RemoveConnectionFromTeam(int32 Team, UTutorialConnectionManager* ConnManager)
{
	if (TArray<UTutorialConnectionManager*>* TeamList = Find(Team))
	{
		TeamList->RemoveSwap(ConnManager);
		
		// Remove the team from the map if there are no more connections
		if (TeamList->Num() == 0)
		{
			Remove(Team);
		}
	}
}

And finally, outside of the connection graph we need to call into SetTeamForPlayerController whenever the team index changes. As this is a simple example we just do it once in the BeginPlay of the PlayerState.

void ATutorialPlayerState::BeginPlay()
{
	Super::BeginPlay();

	if (HasAuthority())
	{
		Team = FMath::RandBool() ? 0 : 1;
		
		if (const UWorld* World = GetWorld())
		{
			if (const UNetDriver* NetworkDriver = World->GetNetDriver())
			{
				if (UTutorialRepGraph* RepGraph = NetworkDriver->GetReplicationDriver<UTutorialRepGraph>())
				{
					RepGraph->SetTeamForPlayerController(GetPlayerController(), Team);
				}
			}
		}
	}
}

The result of this work is that each player can only see others in the same team. Great for a first step but only half way to where we want to be!

Cleaning Up

Before we move on there are also a few functions to implement so that we clean up all state properly if an actor or connection is destroyed. First up is RemoveClientConnection which is pretty self explanatory, we just need to remove the connection graph from the TeamConnectionListMap.

void UTutorialRepGraph::RemoveClientConnection(UNetConnection* NetConnection)
{
	int32 ConnectionId = 0;
	bool bFound = false;
	
	auto UpdateList = [&](TArray<TObjectPtr<UNetReplicationGraphConnection>>& List)
	{
		for (int32 idx = 0; idx < List.Num(); ++idx)
		{
			UTutorialConnectionManager* ConnectionManager = Cast<UTutorialConnectionManager>(Connections[idx]);
			repCheck(ConnectionManager);

			if (ConnectionManager->NetConnection == NetConnection)
			{
				ensure(!bFound);

        // Remove the connection from the team node if the team is valid
				if (ConnectionManager->Team != -1)
				{
					TeamConnectionListMap.RemoveConnectionFromTeam(ConnectionManager->Team, ConnectionManager);
				}

        // Also remove it from the input list
				List.RemoveAtSwap(idx, 1, false);
				bFound = true;
			}
			else
			{
				ConnectionManager->ConnectionOrderNum = ConnectionId++;
			}
		}
	};

	UpdateList(Connections);
	UpdateList(PendingConnections);
}

Then there is RouteRemoveNetworkActorToNodes which is pretty much the reverse of RouteAddNetworkActorToNodes and just calls NotifyRemoveNetworkActor on each node that the actor was added to.

void UTutorialRepGraph::RouteRemoveNetworkActorToNodes(const FNewReplicatedActorInfo& ActorInfo)
{
	if (ActorInfo.Class->IsChildOf(AGameStateBase::StaticClass()) || ActorInfo.Class->IsChildOf(APlayerState::StaticClass()))
	{
		AlwaysRelevantNode->NotifyRemoveNetworkActor(ActorInfo);
	}
	else if (const UTutorialConnectionManager* ConnectionManager = GetTutorialConnectionManagerFromActor(ActorInfo.GetActor()))
	{
		if (ActorInfo.Actor->bOnlyRelevantToOwner)
		{
			ConnectionManager->AlwaysRelevantForConnectionNode->NotifyRemoveNetworkActor(ActorInfo);
		}
		else
		{
			ConnectionManager->TeamConnectionNode->NotifyRemoveNetworkActor(ActorInfo);
		}
	}
	else if (ActorInfo.Actor->GetNetOwner())
	{
		PendingConnectionActors.Remove(ActorInfo.GetActor());
	}
}

Finally we can use ResetGameWorldState to clean up the graph when seamless travelling. This is only needed for seamless travel as non-seamless connections will be destroyed and recreated when travelling and should correctly clean up via RemoveClientConnection and RouteRemoveNetworkActorToNodes.

void UTutorialRepGraph::ResetGameWorldState()
{
	Super::ResetGameWorldState();

	PendingConnectionActors.Reset();
	PendingTeamRequests.Reset();
	
	auto EmptyConnectionNode = [](TArray<TObjectPtr<UNetReplicationGraphConnection>>& GraphConnections)
	{
		for (UNetReplicationGraphConnection* GraphConnection : GraphConnections)
		{
			if (const UTutorialConnectionManager* TutorialConnectionManager = Cast<UTutorialConnectionManager>(GraphConnection))
			{
			  // Clear out all always relevant actors
				// Seamless travel means that the team connections will still be relevant due to the controllers not being destroyed
				TutorialConnectionManager->AlwaysRelevantForConnectionNode->NotifyResetAllNetworkActors();
			}
		}
	};

	EmptyConnectionNode(PendingConnections);
	EmptyConnectionNode(Connections);
}

Extending The Replication Graph To Use Visibility

That took a lot of code to set up but now that we have got a base it should be easy enough to extend. To continue the example we’re going to add some raycasts so that it is possible to see the other teams if their actor is within line of sight.

The first thing we need to do is cache the connection’s pawn when it is added to the node so that we can access it quickly each time we need to raycast. This should be fine for this example but typical projects may have more than one type of pawn per connection and may need more some more complex logic to properly handle changing pawns.

void UTutorialRepGraph::RouteAddNetworkActorToNodes(const FNewReplicatedActorInfo& ActorInfo,
                                                    FGlobalActorReplicationInfo& GlobalInfo)
{
  // ...
  else
	{
		ConnectionManager->TeamConnectionNode->NotifyAddNetworkActor(ActorInfo);

		if (APawn* Pawn = Cast<APawn>(ActorInfo.GetActor()))
		{
			ConnectionManager->Pawn = Pawn;
		}
	}
	// ...
}

We can then add some functionality in our UReplicationGraphNode_AlwaysRelevant_ForTeam::GatherActorListsForConnection function to loop through each non team member connection graph and raycast to each of their pawns. If the raycast does not hit anything it means we have a line of sight to the target and we can add it to the list of actors to replicate.

void UReplicationGraphNode_AlwaysRelevant_ForTeam::GatherActorListsForConnection(
	const FConnectionGatherActorListParameters& Params)
{

	{
		// ... Add all team members

		// Add all visible non-team actors to the list
		const TArray<UTutorialConnectionManager*>& NonTeamConnections = ReplicationGraph->TeamConnectionListMap.GetVisibleConnectionArrayForNonTeam(ConnectionManager->Pawn.Get(), ConnectionManager->Team);

		for (const UTutorialConnectionManager* NonTeamMember : NonTeamConnections)
		{
			NonTeamMember->TeamConnectionNode->GatherActorListsForConnectionDefault(Params);
		}
		
		// ...
	}
}

TArray<UTutorialConnectionManager*> FTeamConnectionListMap::GetVisibleConnectionArrayForNonTeam(const APawn* Pawn, int32 Team)
{	
	TArray<UTutorialConnectionManager*> NonTeamConnections;

	if (!IsValid(Pawn))
	{
		return NonTeamConnections;
	}

	// Setup query params and ignore all team members
	TArray<UTutorialConnectionManager*>* TeamMembers = GetConnectionArrayForTeam(Team);
		
	FCollisionQueryParams TraceParams;
	if (TeamMembers)
	{
		for (const UTutorialConnectionManager* ConnectionManager: *TeamMembers)
		{
			TraceParams.AddIgnoredActor(ConnectionManager->Pawn.Get());
		}
	}
	else
	{
		TraceParams.AddIgnoredActor(Pawn);
	}

	// Iterate over all teams that do not match the input team
	TArray<int32> Teams;
	GetKeys(Teams);

	const UWorld* World = Pawn->GetWorld();
	const FVector TraceOffset = FVector(0.0f, 0.0f, 180.0f);
	const FVector TraceStart = Pawn->GetActorLocation() + TraceOffset;
	for (int32 i = 0; i < Teams.Num(); i++)
	{
		const int32 TeamID = Teams[i];
		if (TeamID != Team)
		{
			const TArray<UTutorialConnectionManager*>* OtherTeamMembers = GetConnectionArrayForTeam(TeamID);

			if (OtherTeamMembers)
			{
				for (UTutorialConnectionManager* ConnectionManager: *OtherTeamMembers)
				{
					if (!ConnectionManager->Pawn.IsValid())
					{
						continue;
					}
					
					// Raycast between our pawn and the other. If we hit anything then we do not have line of sight
					FHitResult OutHit;
					const FVector TraceEnd = ConnectionManager->Pawn.Get()->GetActorLocation() + TraceOffset;
					if (!World->LineTraceSingleByChannel(OutHit, TraceStart, TraceEnd, ECC_GameTraceChannel1, TraceParams))
					{
						NonTeamConnections.Add(ConnectionManager);		
					}
				}
			}
		}
	}

	return NonTeamConnections;
}

That’s it! GatherActorListsForConnection will regularly update our visible actors which means the other team will pop in and out of visibility, and when they’re not visible we no longer need to send any info about the pawn.

Further Reading

Hopefully this has been a nice introduction to how powerful the replication graph can be but we’ve only really scratched the surface. I’ve made the repository for this example public at Github if you’d prefer to read through it in your own time.

Much of my knowledge is built from the LocusReplicationGraph example. This provides more functionality for creating rules for specific actors through the replication graph blueprint and is a great place to go after this article. There’s also a great example in the Lyra sample (make sure you’ve connected your Epic account to Github!).

Alongside this, Epic have provided us with quite a few premade node types which we can inherit from and are worth knowing about. If they don’t fit what you’re looking for you should hopefully at least be able to use them as a reference for your own types!

  • UReplicationGraphNode_ActorList
  • UReplicationGraphNode_ActorListFrequencyBuckets
  • UReplicationGraphNode_DynamicSpatialFrequency
  • UReplicationGraphNode_ConnectionDormancyNode
  • UReplicationGraphNode_DormancyNode
  • UReplicationGraphNode_GridCell
  • UReplicationGraphNode_GridSpatialization2D
  • UReplicationGraphNode_AlwaysRelevant
  • UReplicationGraphNode_AlwaysRelevant_ForConnection
  • UReplicationGraphNode_TearOff_ForConnection

And last but not least, I’ve got a few other articles focused on Unreal network issues if you haven’t seen them yet and more in the pipeline!

Discover more from Kieran Newland

Subscribe to get the latest posts sent to your email.

Tags:
Blog, Unreal Engine

Leave a Reply

Your email address will not be published. Required fields are marked *