Hi,

So this is either routing or firewalls I’d say.

You mention azure so is this server a VM running in Azure?

Are the devices within the same subnet/vnet? If not then I would explore routing, you mentioned pinging works but ping doesn’t confirm the device is what you think. I’m not saying this is your issue but I’ve had a few times where someone has made an IP conflict that has broken communication.

What you need to know is regardless of default gateway or not, in a multi NIC configuration, VBR and the endpoint will attempt to communicate across ALL interfaces, you can enforce a processing order for this with the preferred networks setting, but this would only delay backup tasks not stop them working as it would eventually failover to a working configuration.

Your error indicates an inability to communicate or a firewall dropping the traffic silently instead of denying

Thank you for your response.

The Azure matter is a red herring. Sorry for any confusion I have caused. I’ll clarify.

The backup server has LAN IP 192.1YY.1.62/24 and is domain joined on untagged on VLAN 1.

The backup server is a VM on the physical host with IP 192.1YY.1.3/24 and is domain joined on VLAN 1.

All other VMs are also domain joined with IP in range 192.1YY.1.0/24 and are untagged on VLAN 1.

All physical machines are also domain joined with IP in the range 192.1YY.1.0/24 and are untagged on VLAN 1.

All of the VMs and physical machines can ping and nslookup successfully. They all have domain network discovery set to on.

In the 192.1YY.1.0/24 network all physical and virtual devices are connected at Layer 2, so there are no firewalls and Layer 3 devices in the network path between devices. All of the devices are Microsoft (Server 2022 or Windows 10 22H2) and have Windows Defender Advanced Firewall with Veeam deployed inbound and outbound rules. I have even specifically added rules inbound and outbound for port 10005 on the backup server and each of the physical hosts I have tried to back up with agents. 

This picture shows the registry key “Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam EndPoint Backup” with the only confusing entry being the SqlServerName which is “(localdb)” but the real SQL server is 192.1YY.1.58/24.

Short of installing WireShark, how can I explore whether there is a silent firewall drop? I’d still like to know how the Agent is configured, and whether it gets settings from some fort of configuration file.

Hi @DrThirsty can you check this KB https://www.veeam.com/kb4557

Hi @DrThirsty -

Did you attempt to disable Windows Firewall on the client devices to see if it is indeed the issue? Do you have UAC enabled anywhere? This would also cause a problem. The following are ports VBR uses with Win Agent:

https://helpcenter.veeam.com/docs/agentforwindows/userguide/ports.html?ver=60

Veeam also has a KB of that very error DrThirsty. Give it a look over and try the steps Veeam suggests to see if any resolve your issue:

https://www.veeam.com/kb4557

Hope that helps.

Thanks for this, I asked about Azure because you could have NSGs in the way but if these are on-prem then that’s not a problem…

The KB everyone is sharing is a good run book, I would be tempted to do a quick PCAP on both ends to see if the traffic is getting there or not to rule whether this is a connectivity issue or a software issue.

Thank you everyone, I will be following your suggestions this afternoon.

Sorry for any delay in replying to your postings. I am in the UK and it is currently British Summer Time (UTC/GMT+1) so time zone differences may sometimes apparently delay my replies and responses.

I had some thoughts as well. The agents are successfully deployed from server to endpoint (client) and rescans work fine. So there is connection from server to client. The client can connect to the server ports 80, 443, 445 and many others, but not port 10005 although the port is listening. Both server and client are running Sophos Intercept X antivirus and I shall take a look at the Sophos logs to see whether it is causing the problems. Are there any known issues with Veeam 12 and Sophos AV, or any sources of articles on that topic?

Also @Andanet and @coolsport00 re KB https://www.veeam.com/kb4557 

I have already done everything proposed in that article EXCEPT at the foot of the reference where it says “Update configuration using the Configurator” which I will do if all else fails and I can’t find some other logical reason for the problem.

Hey! Fellow Brit here

Id expect Sophos/Intercept-X to impact performance more than anything else, KB1999 (https://www.veeam.com/kb1999) has the details of required/recommended exclusions so I would check those. Only way I’d see intercept x getting involved is if you’re using any of the network protection options that restrict client to client communication when a client is unhealthy.

Id try the PowerShell command from the KB article others have shared to confirm that the 10005 port is bound to all interfaces not just randomly your WAN/DMZ one. I wouldn’t expect this but again just looking for clear explanations.

I’d also suggest checking that your windows firewall rules aren’t scoped too narrow such as explicit remote/local IP addresses only for port 10005, or even bound to the wrong binary, I’d start with your VBR server as you said this is common across multiple endpoints.

Also, I completely understand not wanting to install wireshark on all your endpoints and VBR, you can take a PCAP without it (kinda). You can use “netsh trace start” with some other parameter options to generate an ETL file and then use etl2pcapng to convert to something readable by wireshark: https://github.com/microsoft/etl2pcapng/releases

@DrThirsty - yes, some folks have posted here having connectivity issues who use Sophos, so do look over the KB Michael shared.

Keep us posted. 

Hi Experts,

I have followed https://www.veeam.com/kb1999 instructions, adding exclusions to Sophos Intercept X for the backup server. That made no difference.

Here is the result of netstat -na | findstr “10005” which I think shows all NICs listening and that the Veeam.BackupService is bound to the port.

Here’s all of the automatically installed Windows Defender Advanced Firewall inbound rules created when Veeam is installed, plus a specific inbound for port 10005.

Here’s the latest result of Test-NetConnection from a physical endpoint with the agent successfully installed.

 I’m next going to try some sort of WireShark-ing… Any further suggestions most welcomed. Thanks in advance.

...and here’s an update.

The backup server has 2x NICs. The LAN NIC has address 192.1YY.1.62/24 and ipconfig shows the following.

• IP address 192.1YY.1.62
• Subnet 255.255.255.0
• Default Gateway - none
• DNS server 192.1YY.1.2
• Untagged VLAN 1

The WAN NIC has address 172.16.2.2/24 and ipconfig shows the following.

• IP address 172.16.2.2
• Subnet 255.255.255.0
• Default Gateway 172.16.2.1
• DNS server - none
• Untagged VLAN 100

The backup server is called STORM-SHIELD and if it is ping’d or nslookup the return the LAN IP address.

• Test-NetConnection -ComputerName STORM-SHIELD -Port 10005 fails.
• Test-NetConnection -ComputerName 172.16.2.2 -Port 10005 succeeds!!

The Veeam installation has bound the owning process to the wrong NIC, and devices on the LAN can’t communicate with the WAN NIC as they are on different VLANs.

How do I fix this?

Hi @DrThirsty -

From the Global/Main Menu in the Console, you can try to set your Preferred Network to see if that works for you:

https://helpcenter.veeam.com/docs/backup/vsphere/select_backup_network.html?zoom_highlight=preferred+network&ver=120

Hi,

We confirmed earlier that Veeam isn’t binding to a specific interface, Veeam doesn’t give you the option to do this either. I would test with firewall disabled and if it starts working then I would recreate the 10005 rule, it seems to be working via the other interface which makes me think that there’s a particular setting on the firewall rule that isn’t working right.Alternarively, I did think you might have a deny rule taking precedence here, something specific to your LAN network as it’s not impacting your WAN connection

Investigating further, ALL of the Veeam ports (8543, 9402, 20443, 33034,33035, 9401,9419… all of them) are bound to the WAN NIC (i.e. the WRONG) NIC. 

I have tried to specify the Preferred Network in the Veeam management console. Made no difference, even after restarting all the Veeam services.

This must be something that is configured at the time of installation.

I can find no relevant references to port 10005 in the registry. So, there must be a setup or configuration file (.config or .xml or something else) in which the bindings are defined at the time of installation. Any thoughts?

My (..I really don’t want to do this unless I must...) action of last resort is to completely uninstall Veeam, delete all the registry keys, delete all of the Veeam folders and the config in the AppData\Local\Veeam folder of the installation account. Disable the WAN NIC, then clean install Veeam to see if it binds to the (correct) LAN NIC.

There are no “deny” rules on any firewalls.

Does Veeam have something like a hosts or routing table somewhere? Everything is pointing to the wrong NIC.

I can’t be the first person to have tripped into this issue. I am running out of ideas...

Hi @DrThirsty -

I actually was going to suggest that to you earlier...uninstall Veeam/reinstall it, to see if it then binds to the new NIC. This is a VM, correct? If so, at the least, you can take a snapshot of the VBR server, temporarily disable and/or remove your WAN NIC, and attempt the resinstall to see if it then works. If not, you can revert to your snapshot to be back to where you started.

Before attempting the reinstall, I would recommend getting ahold of Veeam Support to see if they can address this issue for you in a more ‘clean’ way. 

Hi @coolsport00 

Thanks for confirming that my action of last resort may be a way to fix this. I am currently using a 30 day evaluation license, so don’t know whether I am in a position to raise a ticket with Veeam Support. I will look into this.

I should also mention, that this is all being done in my company’s Test Environment (not the Production Environment). Our intention is to evaluate Veeam and if it does what we need, then we would be buying about 10x full licenses; one for HQ, one for R&D Centre, one for each of the Manufacturing Hubs etc. Each location has a dual NIC DMZ server with a LAN NIC and a WAN NIC where the WAN NIC sits behind firewalls and load balancers to connect to the internet and, particularly, to Azure storage blobs. We have to be able to install Veeam onto these DMZ servers without disabling the WAN NICs because we need the WAN NIC up so that we can IKEv2 VPN connect through the DMZ tunnel between the IT Department at HQ and all the other locations…

That’s why I am persisting with investigations in the registry and in the Veeam application folders. If it is possible to modify a .reg key or a .config file to fix a binding, we can create a script to do that overnight without disrupting operations.

Hi @DrThirsty -

Thanks for the additional info. Yes, you can open up a support ticket with Veeam during evaluation phase.

Also, though we do try to help with issues Community members inquire about here, please keep in mind we are *not* Support. We only offer suggestions. Just FYI 

Hi @coolsport00 

Don’t worry, I completely understand how communities work I am a member of several myself. I tend to prefer community sourced advice because, in my experience, someone has often encountered a problem and has direct hands-on knowledge of the steps to put things straight. Sometimes the formal support routes read from scripts of the theory of how to fix something, rather than hands-on practice. (I have had some pretty terrible experiences with paid for support from Microsoft...).

If I can’t resolve this myself, and before I go down the uninstall-reinstall route, I’ll try the Veeam Support Service route.

TTFN

Hi @DrThirsty - completely understand. I just wanted to make sure of that though...some don’t realize this 

Hello Experts,

Here are the steps I ended up following to resolve this problem.

1. Follow the migration process at https://helpcenter.veeam.com/docs/backup/vsphere/vbr_config_migrate.html?ver=120#step3
2. At step (4) do the following instead…
3. Uninstall Veeam B&R from the multi-NIC Server.
4. Delete ALL the Veeam folders in Program Files, Program Files (x86), ProgramData, <installUser>\Appdata\Local\Veeam etc.
5. Delete ALL the Veeam registry keys. (NOTE: Take a full backup of your registry before you do this. Changes to the registry can really mess up your system.)
6. Reboot the multi-NIC Server.
7. Disable the WAN facing NIC (i.e. the one with the lower IPv4 or IPv6 metric which in my case also had a default gateway whilst my LAN facing NIC doesn’t).
8. Mount the Veeam .iso and perform a clean installation, noting the instruction that you need to, “...you need to specify a new database name to store the configuration data...” in step (4) of the reference in item (1).
9. Follow step (5) onwards.
10. From a physical endpoint you want to back up using an Agent, check the connection is working by running, “Test-NetConnection -ComputerName <BackupServerName> -Port “10005” -InformationLevel Detailed” and check that it does not fail.
11. Enable the WAN facing NIC and repeat step (10).

Hopefully, you have now installed Veeam backup and recovery on a multi-NIC server with the necessary Veeam ports bound to the correct NIC. This worked for me.

I tried to find a configuration file with the bindings, or a registry key with the settings, but I just couldn’t find anything relevant. 

I think it is a bit shoddy that the Veeam installation does not prompt an installation administrator for choices about NIC bindings. The installer seems to assume that a NIC which has a default gateway, even if it does not have a nominated DNS server, is the one to bind the ports onto. That’s not necessarily the right assumption if the Veeam server is being installed on a server in a DMZ.

This process worked for me. However, that is in a Test Environment, and it isn’t a good way to do something like this for a Production Environment. If you have that funds to license another single purpose MS Server, I would spin up a Veeam specific server on the LAN side of a DMZ and create the necessary firewall rules so that it can communicate through to the internet so that you can leverage S3 storage in Azure/AWS/Google if that is part of your BCDR plan.

Good luck...

Hi @DrThirsty -

Thank you for sharing what worked for you. Again, there may be a cleaner way to do what you want by contacting Support. At the very least, I would recommend you going over to the Forum to share your dilemma with the Product Mgmt team. They may have a better solution for you. They’re pretty good at responding to queries.

Best.