When I using Veeam explorer to restore mailbox items with attachments to a user’s mailbox on On- premise Exchange Server the restore job fail with the below error:
"Failed to create items. The underlying connection was closed: An unexpected error occurred on a send. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host."
Any ideas?
Best answer by Arin
View original
+20
Need more details other than that error as it does not help to narrow things down. What version of VB365 are you using? Latest patch?
Check the logs - C:\ProgramData\Veeam\xxxxxx
+17
My first thought here is networking issue of some kind. Are you restoring over a WAN connection? Or, to a location where connection can be sketchy? Veeam has a KB on part of this error:
I’d try a ‘perpetual ping’ to see if you have any dropped packets to the location you’re restoring to for starters. You can also look at Veeam logs to see if anything more definitive shows there: C:\ProgramData\Veeam\Backup. Another potential issue could be Antivirus software?
+20
Unsure if this will help but was related to SSL expired certs (may not be related but comes up during search of the main error you mention in the OP) - (1) The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel (veeam.com)
VBR 12.1420. Target is on-premise Exchange 2016 mailbox.
I can confirm its any item with attachment that is failing with the same error. Tried different mailboxes, even tried restoring from an entirely new backup. Restore is over the LAN. If I restore to .pst it works fine, the items are all there. If I restore to the mailbox, once the item has an attachment it fails..
+20
Check your VBR logs for restoration information but also check the logs and health of your CAS server.
The restore data flow is port 443 to the EWS connecter on the CAS server you’re targeting.
+20
I get 12.0.1420 but which patch based on this KB - KB2680: Build Numbers and Versions of Veeam Backup & Replication
Also check the logs as mentioned as well to get further details. This might come down to a Support ticket, but we will try to help our best.
+17
Hmm..yeah, not sure what else to suggest. May need to hit up support at this point.
+7
What’s the vbr license? Restore of an entire mailbox or selected mailbox items to the original location is available with Veeam Universal License. When using a legacy socket-based license, the Enterprise or Enterprise Plus editions of Veeam Backup & Replication is required.
Also, there are some restore limitations when you are using the Veeam explorer for Microsoft exchange to restore.
https://helpcenter.veeam.com/docs/backup/explorers/vex_considerations.html?ver=120
Enterprise plus edition.
Is there a name for the restore log file? Im checking programdata\veeam but not seeing it in particular..
+20
Enterprise plus edition.
Is there a name for the restore log file? Im checking programdata\veeam but not seeing it in particular..
Under the backup folder.
+7
Enterprise plus edition.
Is there a name for the restore log file? Im checking programdata\veeam but not seeing it in particular..
%ProgramData%\Veeam\Backup\<Veeam_Explorer_Name>\Log
Have you checked that the ports on the firewall are getting through? In this case 443?
https://helpcenter.veeam.com/docs/backup/explorers/vex_ports.html?ver=120
I dont think its a firewall issue since all mailbox items that do not contain attachments are restored fine.
Log file output -
13:27:11.176] 1 (40672) Use specified Exchange server: owa.companyx.com
[ 13:27:11.239] 14 (45580) Initializing Exchange Web Services...
[ 13:27:11.239] 14 (45580) Connecting to Exchange Web Services (server: owa.companyx.com, account: <null>, ID: 248d9923-313a-4da9-a75a-52a8381a7f22)...
[ 13:27:11.239] 14 (45580) Microsoft Exchange Web Services binding path: https://owa.companyx.com/EWS/Exchange.asmx
[ 13:27:11.239] 14 (45580) Opening root folder using impersonation...
[ 13:27:11.266] 14 (45580) Root folder opened successfully
[ 13:27:11.266] 14 (45580) Exchange Web Services initialized
[ 13:27:11.273] 14 (45580) Counting items...
[ 13:27:11.276] 14 (45580) found 1 item(s)
[ 13:27:11.276] 14 (45580) Restoring items...
[ 13:27:11.561] 14 (45580) Creating 1 items (total size: 871991 bytes)...
[ 13:27:11.608] 14 (45580) Error: Failed to create items.
[ 13:27:11.608] 14 (45580) Type: Veeam.Ews.Internal.ExNetworkException
[ 13:27:11.608] 14 (45580) Stack:
[ 13:27:11.608] 14 (45580) at Veeam.Ews.ExError.Throw(WebException error, String format, Object[] args)
[ 13:27:11.608] 14 (45580) at Veeam.Ews.ExError.Catch(Action action, String format, Object[] args)
[ 13:27:11.608] 14 (45580) at Veeam.Ews.ExError.Catch[T](Func`1 func, String format, Object[] args)
[ 13:27:11.608] 14 (45580) at Veeam.Ews.ExItemOperations.Execute[T](Func`1 action, String errorMessage, String logFailedFormat, Int32 expectedRespNum)
[ 13:27:11.608] 14 (45580) at Veeam.Ews.ExItemOperations.CreateItems(IEnumerable`1 data)
[ 13:27:11.608] 14 (45580) at Veeam.Exchange.Restore.Restore.Retry.ExFolderRetry.CreateItems(IEnumerable`1 data)
[ 13:27:11.608] 14 (45580) at Veeam.Exchange.Restore.ItemsPageCreator.CreateItems(ItemsPage itemsPage, IExFolder targetFolder, Dictionary`2 timedOutItems, Boolean perItemMode, Boolean& timeoutOccurred)
[ 13:27:11.608] 14 (45580) Error: The underlying connection was closed: An unexpected error occurred on a send.
[ 13:27:11.608] 14 (45580) Type: System.Net.WebException
[ 13:27:11.608] 14 (45580) Stack:
[ 13:27:11.608] 14 (45580) at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
[ 13:27:11.608] 14 (45580) at Veeam.Ews.ExBinding.GetWebResponse(WebRequest request)
[ 13:27:11.608] 14 (45580) at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
[ 13:27:11.608] 14 (45580) at EWServices.ExchangeServiceBinding.UploadItems(UploadItemsType UploadItems1)
[ 13:27:11.608] 14 (45580) at Veeam.Ews.ExItemOperations.<>c__DisplayClass10_0.<CreateItems>b__1()
[ 13:27:11.608] 14 (45580) at Veeam.Ews.ExError.<>c__DisplayClass7_0`1.<Catch>b__0()
[ 13:27:11.608] 14 (45580) at Veeam.Ews.ExError.Catch(Action action, String format, Object[] args)
[ 13:27:11.608] 14 (45580) Error: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
[ 13:27:11.608] 14 (45580) Type: System.IO.IOException
[ 13:27:11.608] 14 (45580) Stack:
[ 13:27:11.608] 14 (45580) at System.Net.Sockets.NetworkStream.MultipleWrite(BufferOffsetSize[] buffers)
[ 13:27:11.608] 14 (45580) at System.Net.Security._SslStream.StartWriting(SplitWritesState splitWrite, SplitWriteAsyncProtocolRequest asyncRequest)
[ 13:27:11.608] 14 (45580) at System.Net.Security._SslStream.ProcessWrite(BufferOffsetSize[] buffers, SplitWriteAsyncProtocolRequest asyncRequest)
[ 13:27:11.608] 14 (45580) at System.Net.TlsStream.MultipleWrite(BufferOffsetSize[] buffers)
[ 13:27:11.608] 14 (45580) at System.Net.Connection.Write(ScatterGatherBuffers writeBuffer)
[ 13:27:11.608] 14 (45580) at System.Net.ConnectStream.ResubmitWrite(ConnectStream oldStream, Boolean suppressWrite)
[ 13:27:11.608] 14 (45580) Error: An existing connection was forcibly closed by the remote host
[ 13:27:11.608] 14 (45580) Type: System.Net.Sockets.SocketException
[ 13:27:11.608] 14 (45580) Stack:
[ 13:27:11.608] 14 (45580) at System.Net.Sockets.Socket.MultipleSend(BufferOffsetSize[] buffers, SocketFlags socketFlags)
[ 13:27:11.608] 14 (45580) at System.Net.Sockets.NetworkStream.MultipleWrite(BufferOffsetSize[] buffers)
[ 13:27:11.608] 14 (45580) Connection closed by the remote host. Starting item-by-item creation for current items page...
[ 13:27:11.611] 14 (45580) Connection 248d9923-313a-4da9-a75a-52a8381a7f22 is closed
[ 13:27:11.611] 14 (45580) Connecting to Exchange Web Services (server: owa.companyx.com, account: <null>, ID: 011b92ad-2069-469e-9528-0671ff2e7d4a)...
[ 13:27:11.611] 14 (45580) Creating 1 items (total size: 871991 bytes)...
[ 13:27:11.619] 14 (45580) Warning: Exchange Web Services request failed
[ 13:27:11.619] 14 (45580) Warning: Failed to create items.
[ 13:27:11.619] 14 (45580) Warning: The underlying connection was closed: An unexpected error occurred on a send.
[ 13:27:11.619] 14 (45580) Warning: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
[ 13:27:11.619] 14 (45580) Warning: An existing connection was forcibly closed by the remote host
+20
I think at this point your best option is to contact Support as there is not going to be much more, we can do here on the community site.
Investigating further,
attachments in drafts and sent items are able to be restored.
It seems only attachments that are in the inbox give the error.
Yes indeed, looking like something Veeam support needs to look at for sure
+17
Keep us posted what they find out.
Thanks.
So I managed to solve this one. The problem was that the connection was going over a load balancer which does SSL offloading and then connects to exchange over port 80.
If I enter the hostname of the Exchange server itself in the “restore to..” wizard, to attachment restore works fine- from backup server directly to the exchange server.
+17
Great investigative work
+17
Hi
Thanks.
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
