Skip to main content
Solved

Backup stopped working

A

Hello,

 

I have a Veeam server that backs up a Win server 2016 server located at another location.  Veeam connects to this VM over a SDWAN connection (Zerotier)  This has worked perfectly for years.  All of a sudden, a few days ago, it started failing.  I uninstalled all components (Veeam Agent) from the server, then redeployed the agent.  I can ping the remote server, install the veeam agent, etc on it.  But when I run the job, I get an error:  “Error: Failed to connect to remote backup service.”  

 

Any ideas what could cause this?

Best answer by Iams3le

> from the transport connection: An existing connection was forcibly closed by the remote host.

This output looks like a firewall issue at a first glance. Was there a recent change in the network? I asked because of this clause “  All of a sudden, a few days ago, it started failing”. Please check this!

> I can ping the remote server, install the veeam agent, etc on it.

The target machine is running, no doubt from your remark! But the service that you're trying to connect to is not available.

 

View original
    Chris.Childerhose
    MicoolPaul
    ashanpro
    3 people like this
    • Chris.Childerhose
      Chris.Childerhose
    • MicoolPaul
      MicoolPaul
    • ashanpro
      ashanpro
    Did this topic help you find an answer to your question?

    10 comments

    MicoolPaul
    Forum|alt.badge.img+23
    • MicoolPaul
    • 2363 comments
    • February 28, 2022

    I’d immediately be thinking it’s either firewall or NAT. Can you provide more information on these categories to confirm you’ve met the port requirements here:

     

    https://helpcenter.veeam.com/docs/backup/vsphere/used_ports.html?ver=110

    Michael Paul - Opinions are my own and do not necessarily reflect the opinion of Veeam | https://micoolpaul.com | Mastodon: @micoolpaul@masto.nu | Bluesky: @micoolpaul.com
    Chris.Childerhose
    Forum|alt.badge.img+21
    • Chris.Childerhose
    • Veeam Legend, Veeam Vanguard
    • 8524 comments
    • February 28, 2022

    We will need more information to help further -

    1. What version is Veeam on the backup server?
    2. What is the agent version?
    3. What do the logs say which you can find here on the Backup server - C:\ProgramData\Veeam

    To me the error means the backup service on the agent could possibly be down but that is just a guess at this point.

    Chris Childerhose - VMCA2024 | VMCE2023 | VMCE-SP2024 | Veeam Vanguard 8* | Veeam Legend 5* | vExpert 6* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
    MicoolPaul
    1 person likes this
    • MicoolPaul
      MicoolPaul
    Mildur
    Forum|alt.badge.img+12
    • Mildur
    • Influencer
    • 1035 comments
    • February 28, 2022

    If it‘s NAT, then I wonder how it have worked until now.

    NAT for Veeam Agents is not supported by veeam.

    You need either a VPN connection between the agent and the backup server or Veeam Cloud Connect (as a Service provider or Veeam Cloud Connect for Enterprise if you are an eligible customer).

    Product Management Analyst @ Veeam Software
    Chris.Childerhose
    MicoolPaul
    2 people like this
    • Chris.Childerhose
      Chris.Childerhose
    • MicoolPaul
      MicoolPaul
    A
    • adam-int
    • Author
    • New Here
    • 3 comments
    • February 28, 2022

    I am using Backup & Replication 11 - Community edition

    The Veeam agent version is whatever version that is pushed out by the backup server.  (How do I check t his?)

     

    I can see logs. Is there a specific one you are looking for?

     

    I do see this in one of the logs:

     

    [28.02.2022 13:43:05] <159> Error        Failed to exec command from foreign socket
[28.02.2022 13:43:05] <159> Error        Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. (System.IO.IOException)
[28.02.2022 13:43:05] <159> Error           at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
[28.02.2022 13:43:05] <159> Error           at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
[28.02.2022 13:43:05] <159> Error           at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error           at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error           at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error           at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
[28.02.2022 13:43:05] <159> Error           at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
[28.02.2022 13:43:05] <159> Error           at System.IO.BinaryReader.ReadInt32()
[28.02.2022 13:43:05] <159> Error           at Veeam.Backup.Common.BinaryReaderExm.ReadCompressedString(BinaryReader self, Int32 maxSize, UInt32 uncompressedSize, Encoding encoding)
[28.02.2022 13:43:05] <159> Error           at Veeam.Backup.ServiceLib.CInvokerServer.ExecThreadProc(Object parameters)
[28.02.2022 13:43:05] <159> Error        An existing connection was forcibly closed by the remote host (System.Net.Sockets.SocketException)
[28.02.2022 13:43:05] <159> Error           at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
[28.02.2022 13:43:05] <159> Info         Response: [2816] <RIResponse Exception="AAEAAAD/////AQAAAAAAAAAMAgAAAFdWZWVhbS5CYWNrdXAuQ29tbW9uLCBWZXJzaW9uPTExLjAuMC4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2VuPWJmZDY4NGRlMjI3Njc4M2EFAQAAAChWZWVhbS5CYWNrdXAuQ29tbW9uLkN1c3RvbVN0YWNrRXhjZXB0aW9uDgAAAAlDbGFzc05hbWUHTWVzc2FnZQREYXRhDklubmVyRXhjZXB0aW9uB0hlbHBVUkwQU3RhY2tUcmFjZVN0cmluZxZSZW1vdGVTdGFja1RyYWNlU3RyaW5nEFJlbW90ZVN0YWNrSW5kZXgPRXhjZXB0aW9uTWV0aG9kB0hSZXN1bHQGU291cmNlDVdhdHNvbkJ1Y2tldHMMRnVsbFR5cGVOYW1lEEN1c3RvbVN0YWNrVHJhY2UBAQMEAQEBAAEAAQcBAR5TeXN0ZW0uQ29sbGVjdGlvbnMuSURpY3Rpb25hcnkoVmVlYW0uQmFja3VwLkNvbW1vbi5DdXN0b21TdGFja0V4Y2VwdGlvbgIAAAAICAICAAAABgMAAAAoVmVlYW0uQmFja3VwLkNvbW1vbi5DdXN0b21TdGFja0V4Y2VwdGlvbgYEAAAAcVVuYWJsZSB0byByZWFkIGRhdGEgZnJvbSB0aGUgdHJhbnNwb3J0IGNvbm5lY3Rpb246IEFuIGV4aXN0aW5nIGNvbm5lY3Rpb24gd2FzIGZvcmNpYmx5IGNsb3NlZCBieSB0aGUgcmVtb3RlIGhvc3QuCgkFAAAACgoKAAAAAAoAFROACgoGBgAAABVTeXN0ZW0uSU8uSU9FeGNlcHRpb24GBwAAAMMHICAgYXQgU3lzdGVtLk5ldC5Tb2NrZXRzLk5ldHdvcmtTdHJlYW0uUmVhZChCeXRlW10gYnVmZmVyLCBJbnQzMiBvZmZzZXQsIEludDMyIHNpemUpDQogICBhdCBTeXN0ZW0uTmV0LkZpeGVkU2l6ZVJlYWRlci5SZWFkUGFja2V0KEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgY291bnQpDQogICBhdCBTeXN0ZW0uTmV0LlNlY3VyaXR5Ll9Tc2xTdHJlYW0uU3RhcnRGcmFtZUhlYWRlcihCeXRlW10gYnVmZmVyLCBJbnQzMiBvZmZzZXQsIEludDMyIGNvdW50LCBBc3luY1Byb3RvY29sUmVxdWVzdCBhc3luY1JlcXVlc3QpDQogICBhdCBTeXN0ZW0uTmV0LlNlY3VyaXR5Ll9Tc2xTdHJlYW0uU3RhcnRSZWFkaW5nKEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgY291bnQsIEFzeW5jUHJvdG9jb2xSZXF1ZXN0IGFzeW5jUmVxdWVzdCkNCiAgIGF0IFN5c3RlbS5OZXQuU2VjdXJpdHkuX1NzbFN0cmVhbS5Qcm9jZXNzUmVhZChCeXRlW10gYnVmZmVyLCBJbnQzMiBvZmZzZXQsIEludDMyIGNvdW50LCBBc3luY1Byb3RvY29sUmVxdWVzdCBhc3luY1JlcXVlc3QpDQogICBhdCBTeXN0ZW0uTmV0LlNlY3VyaXR5LlNzbFN0cmVhbS5SZWFkKEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgY291bnQpDQogICBhdCBTeXN0ZW0uSU8uQmluYXJ5UmVhZGVyLkZpbGxCdWZmZXIoSW50MzIgbnVtQnl0ZXMpDQogICBhdCBTeXN0ZW0uSU8uQmluYXJ5UmVhZGVyLlJlYWRJbnQzMigpDQogICBhdCBWZWVhbS5CYWNrdXAuQ29tbW9uLkJpbmFyeVJlYWRlckV4bS5SZWFkQ29tcHJlc3NlZFN0cmluZyhCaW5hcnlSZWFkZXIgc2VsZiwgSW50MzIgbWF4U2l6ZSwgVUludDMyIHVuY29tcHJlc3NlZFNpemUsIEVuY29kaW5nIGVuY29kaW5nKQ0KICAgYXQgVmVlYW0uQmFja3VwLlNlcnZpY2VMaWIuQ0ludm9rZXJTZXJ2ZXIuRXhlY1RocmVhZFByb2MoT2JqZWN0IHBhcmFtZXRlcnMpAQUAAAABAAAACQMAAAAGCQAAAD1BbiBleGlzdGluZyBjb25uZWN0aW9uIHdhcyBmb3JjaWJseSBjbG9zZWQgYnkgdGhlIHJlbW90ZSBob3N0CgoKCgoAAAAACgAVE4AKCgYKAAAAIlN5c3RlbS5OZXQuU29ja2V0cy5Tb2NrZXRFeGNlcHRpb24GCwAAAFQgICBhdCBTeXN0ZW0uTmV0LlNvY2tldHMuTmV0d29ya1N0cmVhbS5SZWFkKEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgc2l6ZSkL"><PlainException><KeyValue Key="Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host." Value="System.IO.IOException" /><KeyValue Key="An existing connection was forcibly closed by the remote host" Value="System.Net.Sockets.SocketException" /></PlainException></RIResponse> (to host 'IP REMOVED: 61576')
[28.02.2022 13:43:05] <159> Error        Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. (System.IO.IOException)
[28.02.2022 13:43:05] <159> Error           at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
[28.02.2022 13:43:05] <159> Error           at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
[28.02.2022 13:43:05] <159> Error           at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error           at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error           at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
[28.02.2022 13:43:05] <159> Error           at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
[28.02.2022 13:43:05] <159> Error           at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
[28.02.2022 13:43:05] <159> Error           at System.IO.BinaryReader.ReadInt32()
[28.02.2022 13:43:05] <159> Error           at Veeam.Backup.Common.BinaryReaderExm.ReadCompressedString(BinaryReader self, Int32 maxSize, UInt32 uncompressedSize, Encoding encoding)
[28.02.2022 13:43:05] <159> Error           at Veeam.Backup.ServiceLib.CInvokerServer.ExecThreadProc(Object parameters)
[28.02.2022 13:43:05] <159> Error        An existing connection was forcibly closed by the remote host (System.Net.Sockets.SocketException)
[28.02.2022 13:43:05] <159> Error           at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

     

    Chris.Childerhose
    1 person likes this
    • Chris.Childerhose
      Chris.Childerhose
    Iams3le
    Forum|alt.badge.img+11
    • Iams3le
    • Veeam Legend
    • 1395 comments
    • Answer
    • February 28, 2022

    > from the transport connection: An existing connection was forcibly closed by the remote host.

    This output looks like a firewall issue at a first glance. Was there a recent change in the network? I asked because of this clause “  All of a sudden, a few days ago, it started failing”. Please check this!

    > I can ping the remote server, install the veeam agent, etc on it.

    The target machine is running, no doubt from your remark! But the service that you're trying to connect to is not available.

     

    [Microsoft MVP | Veeam Legend (2021 -2025) | VMware vExpert 5x | Cisco Champion 3x | Object First Ace 2x]
    Chris.Childerhose
    1 person likes this
    • Chris.Childerhose
      Chris.Childerhose
    Chris.Childerhose
    Forum|alt.badge.img+21
    • Chris.Childerhose
    • Veeam Legend, Veeam Vanguard
    • 8524 comments
    • February 28, 2022

    Under C:\ProgramData\Veeam\Backup -- there should be a folder for the Agent job and then logs inside there.  This might tell you what the issue possibly is.

    Chris Childerhose - VMCA2024 | VMCE2023 | VMCE-SP2024 | Veeam Vanguard 8* | Veeam Legend 5* | vExpert 6* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
    Iams3le
    A
    2 people like this
    • Iams3le
      Iams3le
    • A
      adam-int
    A
    • adam-int
    • Author
    • New Here
    • 3 comments
    • February 28, 2022

    Bingo!  Thank you gentlemen.  Sometimes all it takes is a tip to get you back on track.  I DID in fact change something right around that time. All is well now and working.  It was related to a NIC card.

     

    Thanks again.

    Chris.Childerhose
    MicoolPaul
    Iams3le
    3 people like this
    • Chris.Childerhose
      Chris.Childerhose
    • MicoolPaul
      MicoolPaul
    • Iams3le
      Iams3le
    Chris.Childerhose
    Forum|alt.badge.img+21
    • Chris.Childerhose
    • Veeam Legend, Veeam Vanguard
    • 8524 comments
    • February 28, 2022
    adam-int wrote:

    Bingo!  Thank you gentlemen.  Sometimes all it takes is a tip to get you back on track.  I DID in fact change something right around that time. All is well now and working.  It was related to a NIC card.

     

    Thanks again.

    Glad to hear you figured it out.

    Chris Childerhose - VMCA2024 | VMCE2023 | VMCE-SP2024 | Veeam Vanguard 8* | Veeam Legend 5* | vExpert 6* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
    Iams3le
    A
    2 people like this
    • Iams3le
      Iams3le
    • A
      adam-int
    Iams3le
    Forum|alt.badge.img+11
    • Iams3le
    • Veeam Legend
    • 1395 comments
    • February 28, 2022
    adam-int wrote:

    Bingo!  Thank you gentlemen.  Sometimes all it takes is a tip to get you back on track.  I DID in fact change something right around that time. All is well now and working.  It was related to a NIC card.

     

    Thanks again.

    Great! I am glad you found the comments useful. Kindly mark the answer you found most useful, thank you.

    [Microsoft MVP | Veeam Legend (2021 -2025) | VMware vExpert 5x | Cisco Champion 3x | Object First Ace 2x]
    Chris.Childerhose
    1 person likes this
    • Chris.Childerhose
      Chris.Childerhose
    A
    • adam-int
    • Author
    • New Here
    • 3 comments
    • March 1, 2022

    I had more than one helpful answer, but had to pick one.  :)  Thanks again gents.

    Chris.Childerhose
    Iams3le
    2 people like this
    • Chris.Childerhose
      Chris.Childerhose
    • Iams3le
      Iams3le

    Comment


    Terms & Conditions