Solved

Backup stopped working


Userlevel 1

Hello,

 

I have a Veeam server that backs up a Win server 2016 server located at another location.  Veeam connects to this VM over a SDWAN connection (Zerotier)  This has worked perfectly for years.  All of a sudden, a few days ago, it started failing.  I uninstalled all components (Veeam Agent) from the server, then redeployed the agent.  I can ping the remote server, install the veeam agent, etc on it.  But when I run the job, I get an error:  “Error: Failed to connect to remote backup service.”  

 

Any ideas what could cause this?

icon

Best answer by Iams3le 28 February 2022, 20:30

View original

10 comments

Userlevel 7
Badge +6

I’d immediately be thinking it’s either firewall or NAT. Can you provide more information on these categories to confirm you’ve met the port requirements here:

 

https://helpcenter.veeam.com/docs/backup/vsphere/used_ports.html?ver=110

Userlevel 7
Badge +6

We will need more information to help further -

  1. What version is Veeam on the backup server?
  2. What is the agent version?
  3. What do the logs say which you can find here on the Backup server - C:\ProgramData\Veeam

To me the error means the backup service on the agent could possibly be down but that is just a guess at this point.

Userlevel 7
Badge +4

If it‘s NAT, then I wonder how it have worked until now.

NAT for Veeam Agents is not supported by veeam.

You need either a VPN connection between the agent and the backup server or Veeam Cloud Connect (as a Service provider or Veeam Cloud Connect for Enterprise if you are an eligible customer).

Userlevel 1

I am using Backup & Replication 11 - Community edition

The Veeam agent version is whatever version that is pushed out by the backup server.  (How do I check t his?)

 

I can see logs. Is there a specific one you are looking for?

 

I do see this in one of the logs:

 

[28.02.2022 13:43:05] <159> Error        Failed to exec command from foreign socket
[28.02.2022 13:43:05] <159> Error Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. (System.IO.IOException)
[28.02.2022 13:43:05] <159> Error at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
[28.02.2022 13:43:05] <159> Error at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
[28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
[28.02.2022 13:43:05] <159> Error at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
[28.02.2022 13:43:05] <159> Error at System.IO.BinaryReader.ReadInt32()
[28.02.2022 13:43:05] <159> Error at Veeam.Backup.Common.BinaryReaderExm.ReadCompressedString(BinaryReader self, Int32 maxSize, UInt32 uncompressedSize, Encoding encoding)
[28.02.2022 13:43:05] <159> Error at Veeam.Backup.ServiceLib.CInvokerServer.ExecThreadProc(Object parameters)
[28.02.2022 13:43:05] <159> Error An existing connection was forcibly closed by the remote host (System.Net.Sockets.SocketException)
[28.02.2022 13:43:05] <159> Error at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
[28.02.2022 13:43:05] <159> Info Response: [2816] <RIResponse Exception="AAEAAAD/////AQAAAAAAAAAMAgAAAFdWZWVhbS5CYWNrdXAuQ29tbW9uLCBWZXJzaW9uPTExLjAuMC4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2VuPWJmZDY4NGRlMjI3Njc4M2EFAQAAAChWZWVhbS5CYWNrdXAuQ29tbW9uLkN1c3RvbVN0YWNrRXhjZXB0aW9uDgAAAAlDbGFzc05hbWUHTWVzc2FnZQREYXRhDklubmVyRXhjZXB0aW9uB0hlbHBVUkwQU3RhY2tUcmFjZVN0cmluZxZSZW1vdGVTdGFja1RyYWNlU3RyaW5nEFJlbW90ZVN0YWNrSW5kZXgPRXhjZXB0aW9uTWV0aG9kB0hSZXN1bHQGU291cmNlDVdhdHNvbkJ1Y2tldHMMRnVsbFR5cGVOYW1lEEN1c3RvbVN0YWNrVHJhY2UBAQMEAQEBAAEAAQcBAR5TeXN0ZW0uQ29sbGVjdGlvbnMuSURpY3Rpb25hcnkoVmVlYW0uQmFja3VwLkNvbW1vbi5DdXN0b21TdGFja0V4Y2VwdGlvbgIAAAAICAICAAAABgMAAAAoVmVlYW0uQmFja3VwLkNvbW1vbi5DdXN0b21TdGFja0V4Y2VwdGlvbgYEAAAAcVVuYWJsZSB0byByZWFkIGRhdGEgZnJvbSB0aGUgdHJhbnNwb3J0IGNvbm5lY3Rpb246IEFuIGV4aXN0aW5nIGNvbm5lY3Rpb24gd2FzIGZvcmNpYmx5IGNsb3NlZCBieSB0aGUgcmVtb3RlIGhvc3QuCgkFAAAACgoKAAAAAAoAFROACgoGBgAAABVTeXN0ZW0uSU8uSU9FeGNlcHRpb24GBwAAAMMHICAgYXQgU3lzdGVtLk5ldC5Tb2NrZXRzLk5ldHdvcmtTdHJlYW0uUmVhZChCeXRlW10gYnVmZmVyLCBJbnQzMiBvZmZzZXQsIEludDMyIHNpemUpDQogICBhdCBTeXN0ZW0uTmV0LkZpeGVkU2l6ZVJlYWRlci5SZWFkUGFja2V0KEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgY291bnQpDQogICBhdCBTeXN0ZW0uTmV0LlNlY3VyaXR5Ll9Tc2xTdHJlYW0uU3RhcnRGcmFtZUhlYWRlcihCeXRlW10gYnVmZmVyLCBJbnQzMiBvZmZzZXQsIEludDMyIGNvdW50LCBBc3luY1Byb3RvY29sUmVxdWVzdCBhc3luY1JlcXVlc3QpDQogICBhdCBTeXN0ZW0uTmV0LlNlY3VyaXR5Ll9Tc2xTdHJlYW0uU3RhcnRSZWFkaW5nKEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgY291bnQsIEFzeW5jUHJvdG9jb2xSZXF1ZXN0IGFzeW5jUmVxdWVzdCkNCiAgIGF0IFN5c3RlbS5OZXQuU2VjdXJpdHkuX1NzbFN0cmVhbS5Qcm9jZXNzUmVhZChCeXRlW10gYnVmZmVyLCBJbnQzMiBvZmZzZXQsIEludDMyIGNvdW50LCBBc3luY1Byb3RvY29sUmVxdWVzdCBhc3luY1JlcXVlc3QpDQogICBhdCBTeXN0ZW0uTmV0LlNlY3VyaXR5LlNzbFN0cmVhbS5SZWFkKEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgY291bnQpDQogICBhdCBTeXN0ZW0uSU8uQmluYXJ5UmVhZGVyLkZpbGxCdWZmZXIoSW50MzIgbnVtQnl0ZXMpDQogICBhdCBTeXN0ZW0uSU8uQmluYXJ5UmVhZGVyLlJlYWRJbnQzMigpDQogICBhdCBWZWVhbS5CYWNrdXAuQ29tbW9uLkJpbmFyeVJlYWRlckV4bS5SZWFkQ29tcHJlc3NlZFN0cmluZyhCaW5hcnlSZWFkZXIgc2VsZiwgSW50MzIgbWF4U2l6ZSwgVUludDMyIHVuY29tcHJlc3NlZFNpemUsIEVuY29kaW5nIGVuY29kaW5nKQ0KICAgYXQgVmVlYW0uQmFja3VwLlNlcnZpY2VMaWIuQ0ludm9rZXJTZXJ2ZXIuRXhlY1RocmVhZFByb2MoT2JqZWN0IHBhcmFtZXRlcnMpAQUAAAABAAAACQMAAAAGCQAAAD1BbiBleGlzdGluZyBjb25uZWN0aW9uIHdhcyBmb3JjaWJseSBjbG9zZWQgYnkgdGhlIHJlbW90ZSBob3N0CgoKCgoAAAAACgAVE4AKCgYKAAAAIlN5c3RlbS5OZXQuU29ja2V0cy5Tb2NrZXRFeGNlcHRpb24GCwAAAFQgICBhdCBTeXN0ZW0uTmV0LlNvY2tldHMuTmV0d29ya1N0cmVhbS5SZWFkKEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgc2l6ZSkL"><PlainException><KeyValue Key="Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host." Value="System.IO.IOException" /><KeyValue Key="An existing connection was forcibly closed by the remote host" Value="System.Net.Sockets.SocketException" /></PlainException></RIResponse> (to host 'IP REMOVED: 61576')
[28.02.2022 13:43:05] <159> Error Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. (System.IO.IOException)
[28.02.2022 13:43:05] <159> Error at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
[28.02.2022 13:43:05] <159> Error at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
[28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
[28.02.2022 13:43:05] <159> Error at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
[28.02.2022 13:43:05] <159> Error at System.IO.BinaryReader.ReadInt32()
[28.02.2022 13:43:05] <159> Error at Veeam.Backup.Common.BinaryReaderExm.ReadCompressedString(BinaryReader self, Int32 maxSize, UInt32 uncompressedSize, Encoding encoding)
[28.02.2022 13:43:05] <159> Error at Veeam.Backup.ServiceLib.CInvokerServer.ExecThreadProc(Object parameters)
[28.02.2022 13:43:05] <159> Error An existing connection was forcibly closed by the remote host (System.Net.Sockets.SocketException)
[28.02.2022 13:43:05] <159> Error at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

 

Userlevel 7
Badge +5

> from the transport connection: An existing connection was forcibly closed by the remote host.

This output looks like a firewall issue at a first glance. Was there a recent change in the network? I asked because of this clause “  All of a sudden, a few days ago, it started failing”. Please check this!

> I can ping the remote server, install the veeam agent, etc on it.

The target machine is running, no doubt from your remark! But the service that you're trying to connect to is not available.

 

Userlevel 7
Badge +6

Under C:\ProgramData\Veeam\Backup -- there should be a folder for the Agent job and then logs inside there.  This might tell you what the issue possibly is.

Userlevel 1

Bingo!  Thank you gentlemen.  Sometimes all it takes is a tip to get you back on track.  I DID in fact change something right around that time. All is well now and working.  It was related to a NIC card.

 

Thanks again.

Userlevel 7
Badge +6

Bingo!  Thank you gentlemen.  Sometimes all it takes is a tip to get you back on track.  I DID in fact change something right around that time. All is well now and working.  It was related to a NIC card.

 

Thanks again.

Glad to hear you figured it out.

Userlevel 7
Badge +5

Bingo!  Thank you gentlemen.  Sometimes all it takes is a tip to get you back on track.  I DID in fact change something right around that time. All is well now and working.  It was related to a NIC card.

 

Thanks again.

Great! I am glad you found the comments useful. Kindly mark the answer you found most useful, thank you.

Userlevel 1

I had more than one helpful answer, but had to pick one.  :)  Thanks again gents.

Comment