When using preferred networks you might find your backup jobs take longer to initalise, commonly when you’ve got distributed backup components such as a physical proxy/repository and your Veeam Backup & Replication server isn’t performing any backup processing. This is because your Veeam Backup & Replication Server is also required to use your preferred networks configuration, even if it doesn’t have any network connectivity that resides within your preferred networks.
This causes the job initialisation to take a lot longer than usual, and this grows even slower the more distributed your network topology and the more preferred networks you have. This is because your Veeam Backup & Replication server has to work through every preferred network match, before moving onto the rest of your network interfaces.
Now, a quick reminder that the scenario above is just one example of why isolated networks might exist, but certainly not the only one. If you’re attempting to use preferred networks to send cross-site backup traffic over specific routes, this is another example whereby you could be leveraging an isolated preferred network, and encountering this problem as a result.
Blocks & Drops
When the Veeam Backup & Replication Server is unable to route its packet to the preferred network directly, it passes it to the default gateway, and if that can’t deliver the packet based on its route table, it passes it onto its own default gateway, and so on. That’s how the internet works. However eventually this packet reaches your ISP, who would see a private IP, and drop the traffic. If you were to check your Veeam logs, you’d see lots of messages like the below:
<38> Error Failed to connect to agent's endpoint 'XXX.XXX.XXX.XXX:2500'. Host: 'FQDN.COMPANYDOMAIN.TLD'.
<38> Error A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond XXX.XXX.XXX.XXX:2500 (System.Net.Sockets.SocketException)
<38> Error at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
<38> Error at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
<38> Error at Veeam.Backup.Common.CNetSocket.Connect(IPEndPoint remoteEp)
<38> Error at Veeam.Backup.AgentProvider.CAgentEndpointConnecter.ConnectToAgentEndpoint(ISocket socket, IAgentEndPoint endPoint)So, what’s happening here? Because the traffic isn’t being rejected, and instead being silently dropped, Veeam is waiting for the default timeout of 20 seconds for a response to its packet. The logs frequently show connection retries as well, and we have to exhaust the list of preferred network IP addresses, before we move onto a ‘non-preferred’ network, that will typically get us where we need to be. This compounds into a lot of time, I’ve seen start up delays of 10-15 minutes in more complex networks due to this.
There is a solution however that doesn’t cost, nor take long to set up!
Windows Firewall Blocking Traffic – A Good Thing?
We can use Windows Firewall to filter our outbound traffic, and create a specific block rule for the IP addresses within the preferred networks. By creating a block rule, the packets that Veeam crafts to send to the IP addresses on the preferred networks are immediately rejected on egress, forcing Veeam to move on much faster. You’ll know its working because your backups will initialise faster, and you’ll see the following in your logs instead.
<66> Error Failed to connect to agent's endpoint 'XXX.XXX.XXX.XXX:2500'. Host: 'FQDN.COMPANYDOMAIN.TLD'.
<66> Error An attempt was made to access a socket in a way forbidden by its access permissions XXX.XXX.XXX.XXX:2500 (System.Net.Sockets.SocketException)
<66> Error at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
<66> Error at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
<66> Error at Veeam.Backup.Common.CNetSocket.Connect(IPEndPoint remoteEp)
<66> Error at Veeam.Backup.AgentProvider.CAgentEndpointConnecter.ConnectToAgentEndpoint(ISocket socket, IAgentEndPoint endPoint)You don’t need to use Windows Firewall necessarily, but you do need to actively block the traffic to benefit from these faster initalisation times, even if it’s at your edge firewalls, to prevent your ISP from silently dropping the traffic.
If this isn’t a good solution for you, your other solution is to add your Veeam Backup & Replication server to the preferred networks, and allow the traffic to flow normally.
