I’d immediately be thinking it’s either firewall or NAT. Can you provide more information on these categories to confirm you’ve met the port requirements here:
https://helpcenter.veeam.com/docs/backup/vsphere/used_ports.html?ver=110
We will need more information to help further -
- What version is Veeam on the backup server?
- What is the agent version?
- What do the logs say which you can find here on the Backup server - C:\ProgramData\Veeam
To me the error means the backup service on the agent could possibly be down but that is just a guess at this point.
If it‘s NAT, then I wonder how it have worked until now.
NAT for Veeam Agents is not supported by veeam.
You need either a VPN connection between the agent and the backup server or Veeam Cloud Connect (as a Service provider or Veeam Cloud Connect for Enterprise if you are an eligible customer).
I am using Backup & Replication 11 - Community edition
The Veeam agent version is whatever version that is pushed out by the backup server. (How do I check t his?)
I can see logs. Is there a specific one you are looking for?
I do see this in one of the logs:
o28.02.2022 13:43:05] <159> Error Failed to exec command from foreign socket
r28.02.2022 13:43:05] <159> Error Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. (System.IO.IOException)
r28.02.2022 13:43:05] <159> Error at System.Net.Sockets.NetworkStream.Read(ByteB] buffer, Int32 offset, Int32 size)
r28.02.2022 13:43:05] <159> Error at System.Net.FixedSizeReader.ReadPacket(ByteB] buffer, Int32 offset, Int32 count)
r28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.StartFrameHeader(ByteB] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
r28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.StartReading(ByteB] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
r28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.ProcessRead(ByteB] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
r28.02.2022 13:43:05] <159> Error at System.Net.Security.SslStream.Read(ByteB] buffer, Int32 offset, Int32 count)
r28.02.2022 13:43:05] <159> Error at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
r28.02.2022 13:43:05] <159> Error at System.IO.BinaryReader.ReadInt32()
r28.02.2022 13:43:05] <159> Error at Veeam.Backup.Common.BinaryReaderExm.ReadCompressedString(BinaryReader self, Int32 maxSize, UInt32 uncompressedSize, Encoding encoding)
r28.02.2022 13:43:05] <159> Error at Veeam.Backup.ServiceLib.CInvokerServer.ExecThreadProc(Object parameters)
r28.02.2022 13:43:05] <159> Error An existing connection was forcibly closed by the remote host (System.Net.Sockets.SocketException)
r28.02.2022 13:43:05] <159> Error at System.Net.Sockets.NetworkStream.Read(ByteB] buffer, Int32 offset, Int32 size)
r28.02.2022 13:43:05] <159> Info Response: s2816] <RIResponse Exception="AAEAAAD/////AQAAAAAAAAAMAgAAAFdWZWVhbS5CYWNrdXAuQ29tbW9uLCBWZXJzaW9uPTExLjAuMC4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2VuPWJmZDY4NGRlMjI3Njc4M2EFAQAAAChWZWVhbS5CYWNrdXAuQ29tbW9uLkN1c3RvbVN0YWNrRXhjZXB0aW9uDgAAAAlDbGFzc05hbWUHTWVzc2FnZQREYXRhDklubmVyRXhjZXB0aW9uB0hlbHBVUkwQU3RhY2tUcmFjZVN0cmluZxZSZW1vdGVTdGFja1RyYWNlU3RyaW5nEFJlbW90ZVN0YWNrSW5kZXgPRXhjZXB0aW9uTWV0aG9kB0hSZXN1bHQGU291cmNlDVdhdHNvbkJ1Y2tldHMMRnVsbFR5cGVOYW1lEEN1c3RvbVN0YWNrVHJhY2UBAQMEAQEBAAEAAQcBAR5TeXN0ZW0uQ29sbGVjdGlvbnMuSURpY3Rpb25hcnkoVmVlYW0uQmFja3VwLkNvbW1vbi5DdXN0b21TdGFja0V4Y2VwdGlvbgIAAAAICAICAAAABgMAAAAoVmVlYW0uQmFja3VwLkNvbW1vbi5DdXN0b21TdGFja0V4Y2VwdGlvbgYEAAAAcVVuYWJsZSB0byByZWFkIGRhdGEgZnJvbSB0aGUgdHJhbnNwb3J0IGNvbm5lY3Rpb246IEFuIGV4aXN0aW5nIGNvbm5lY3Rpb24gd2FzIGZvcmNpYmx5IGNsb3NlZCBieSB0aGUgcmVtb3RlIGhvc3QuCgkFAAAACgoKAAAAAAoAFROACgoGBgAAABVTeXN0ZW0uSU8uSU9FeGNlcHRpb24GBwAAAMMHICAgYXQgU3lzdGVtLk5ldC5Tb2NrZXRzLk5ldHdvcmtTdHJlYW0uUmVhZChCeXRlW10gYnVmZmVyLCBJbnQzMiBvZmZzZXQsIEludDMyIHNpemUpDQogICBhdCBTeXN0ZW0uTmV0LkZpeGVkU2l6ZVJlYWRlci5SZWFkUGFja2V0KEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgY291bnQpDQogICBhdCBTeXN0ZW0uTmV0LlNlY3VyaXR5Ll9Tc2xTdHJlYW0uU3RhcnRGcmFtZUhlYWRlcihCeXRlW10gYnVmZmVyLCBJbnQzMiBvZmZzZXQsIEludDMyIGNvdW50LCBBc3luY1Byb3RvY29sUmVxdWVzdCBhc3luY1JlcXVlc3QpDQogICBhdCBTeXN0ZW0uTmV0LlNlY3VyaXR5Ll9Tc2xTdHJlYW0uU3RhcnRSZWFkaW5nKEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgY291bnQsIEFzeW5jUHJvdG9jb2xSZXF1ZXN0IGFzeW5jUmVxdWVzdCkNCiAgIGF0IFN5c3RlbS5OZXQuU2VjdXJpdHkuX1NzbFN0cmVhbS5Qcm9jZXNzUmVhZChCeXRlW10gYnVmZmVyLCBJbnQzMiBvZmZzZXQsIEludDMyIGNvdW50LCBBc3luY1Byb3RvY29sUmVxdWVzdCBhc3luY1JlcXVlc3QpDQogICBhdCBTeXN0ZW0uTmV0LlNlY3VyaXR5LlNzbFN0cmVhbS5SZWFkKEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgY291bnQpDQogICBhdCBTeXN0ZW0uSU8uQmluYXJ5UmVhZGVyLkZpbGxCdWZmZXIoSW50MzIgbnVtQnl0ZXMpDQogICBhdCBTeXN0ZW0uSU8uQmluYXJ5UmVhZGVyLlJlYWRJbnQzMigpDQogICBhdCBWZWVhbS5CYWNrdXAuQ29tbW9uLkJpbmFyeVJlYWRlckV4bS5SZWFkQ29tcHJlc3NlZFN0cmluZyhCaW5hcnlSZWFkZXIgc2VsZiwgSW50MzIgbWF4U2l6ZSwgVUludDMyIHVuY29tcHJlc3NlZFNpemUsIEVuY29kaW5nIGVuY29kaW5nKQ0KICAgYXQgVmVlYW0uQmFja3VwLlNlcnZpY2VMaWIuQ0ludm9rZXJTZXJ2ZXIuRXhlY1RocmVhZFByb2MoT2JqZWN0IHBhcmFtZXRlcnMpAQUAAAABAAAACQMAAAAGCQAAAD1BbiBleGlzdGluZyBjb25uZWN0aW9uIHdhcyBmb3JjaWJseSBjbG9zZWQgYnkgdGhlIHJlbW90ZSBob3N0CgoKCgoAAAAACgAVE4AKCgYKAAAAIlN5c3RlbS5OZXQuU29ja2V0cy5Tb2NrZXRFeGNlcHRpb24GCwAAAFQgICBhdCBTeXN0ZW0uTmV0LlNvY2tldHMuTmV0d29ya1N0cmVhbS5SZWFkKEJ5dGVbXSBidWZmZXIsIEludDMyIG9mZnNldCwgSW50MzIgc2l6ZSkL"><PlainException><KeyValue Key="Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host." Value="System.IO.IOException" /><KeyValue Key="An existing connection was forcibly closed by the remote host" Value="System.Net.Sockets.SocketException" /></PlainException></RIResponse> (to host 'IP REMOVED: 61576')
r28.02.2022 13:43:05] <159> Error Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. (System.IO.IOException)
r28.02.2022 13:43:05] <159> Error at System.Net.Sockets.NetworkStream.Read(ByteB] buffer, Int32 offset, Int32 size)
r28.02.2022 13:43:05] <159> Error at System.Net.FixedSizeReader.ReadPacket(ByteB] buffer, Int32 offset, Int32 count)
r28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.StartFrameHeader(ByteB] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
r28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.StartReading(ByteB] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
r28.02.2022 13:43:05] <159> Error at System.Net.Security._SslStream.ProcessRead(ByteB] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
r28.02.2022 13:43:05] <159> Error at System.Net.Security.SslStream.Read(ByteB] buffer, Int32 offset, Int32 count)
r28.02.2022 13:43:05] <159> Error at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
r28.02.2022 13:43:05] <159> Error at System.IO.BinaryReader.ReadInt32()
r28.02.2022 13:43:05] <159> Error at Veeam.Backup.Common.BinaryReaderExm.ReadCompressedString(BinaryReader self, Int32 maxSize, UInt32 uncompressedSize, Encoding encoding)
r28.02.2022 13:43:05] <159> Error at Veeam.Backup.ServiceLib.CInvokerServer.ExecThreadProc(Object parameters)
r28.02.2022 13:43:05] <159> Error An existing connection was forcibly closed by the remote host (System.Net.Sockets.SocketException)
r28.02.2022 13:43:05] <159> Error at System.Net.Sockets.NetworkStream.Read(ByteB] buffer, Int32 offset, Int32 size)
> from the transport connection: An existing connection was forcibly closed by the remote host.
This output looks like a firewall issue at a first glance. Was there a recent change in the network? I asked because of this clause “ All of a sudden, a few days ago, it started failing”. Please check this!
> I can ping the remote server, install the veeam agent, etc on it.
The target machine is running, no doubt from your remark! But the service that you're trying to connect to is not available.
Under C:\ProgramData\Veeam\Backup -- there should be a folder for the Agent job and then logs inside there. This might tell you what the issue possibly is.
Bingo! Thank you gentlemen. Sometimes all it takes is a tip to get you back on track. I DID in fact change something right around that time. All is well now and working. It was related to a NIC card.
Thanks again.
Bingo! Thank you gentlemen. Sometimes all it takes is a tip to get you back on track. I DID in fact change something right around that time. All is well now and working. It was related to a NIC card.
Thanks again.
Glad to hear you figured it out.
Bingo! Thank you gentlemen. Sometimes all it takes is a tip to get you back on track. I DID in fact change something right around that time. All is well now and working. It was related to a NIC card.
Thanks again.
Great! I am glad you found the comments useful. Kindly mark the answer you found most useful, thank you.
I had more than one helpful answer, but had to pick one. :) Thanks again gents.