• EN
Solved

Issue with NAS backup to object storage at iDRIVE

Nico Losschaert
Userlevel 7
Badge +11

Hi guys

I’m having a problem that I can’t seem to resolve.

Perhaps someone already have had the same error or experience.

Environment :

  • It’s not on a production environment, but on my personal lab
  • I’m using NAS backup of a big file-share to an on-premise repository without any issues
  • I’m using a secondary copy of this backup to another repository without any issues
  • I want to have next to that also a secondary copy of this backup to object storage in the cloud

Since version 12 of VBR that is possible, before it was only possible to use object storage for the archive backup of a NAS-backup.

I’m having an account at iDRIVE for E2 Immutable object storage.

With that latest copy I’m having an issue that I can’t seem to resolve.

Because it’s on my lab with my NFR license, I can’t create a support case for this 🤣

 

Issue : 

The job start, but after a while I’m getting the following error : 

 

HTTP exception: WinHttpReceiveResponse_callback_timeout: 12152: The server returned an invalid or unrecognized response

 

, error code: 12152

 

Failed to copy restore point Error: Agent: Failed to process method {NasMaster.CopyPoint}: HTTP exception: WinHttpReceiveResponse_callback_timeout: 12152: The server returned an invalid or unrecognized response

 

When I search on the Veeam forums, I can find 1 hit, but it’s from a long time ago without a real solution.

Capacity Tier Offload - Failure on some VMs - R&D Forums (veeam.com)

I can also find a similar hit on the blog of Rhys Hammond, but another error and regarding Azure with another errorcode.

WinHttpWriteData: 12002: The operation timed out | rhyshammond.com

 

Someone else an idea?

Anybody already tried to perform a copy of a NAS fileshare of 2TB to object storage on the cloud of iDRIVE or another vendor?

 

 

Thx for any feedback.

 

regards

Nico

 

 

 

icon

Best answer by Rick Vanover 16 August 2023, 16:37

View original
#Veeam Legend 5* | VMCA 2* | VMCE 3* | VMTSP+ 2* | VMSP 2* | DCIE 2* | Loves Best Practices | Twitter : @NLosschaert

15 comments

Chris.Childerhose
Userlevel 7
Badge +20

I don't use NAS backups but have not had any issues going to Wasabi at all. Are you doing a copy job or direct to object for your NAS?  Is there limits or throttles on iDrive?

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
Chris.Childerhose
Userlevel 7
Badge +20

Are you using the VBR server to go direct to iDrive or another server as GW?

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
dloseke
Userlevel 7
Badge +6

I was going to say, I haven’t tried IDrive but I’d make sure I can write other data there first.  I also don’t do NAS, but I can’t imagine there’d be as much likelihood for errors there.

Derek M. Loseke, Senior Systems Engineer | Veeam Legend 2022-2023 | VMSP/VMTSP | VCP6-DCV | VSP/VTSP | CCNA | https://technotesanddadjokes.com | @dloseke
Nico Losschaert
Userlevel 7
Badge +11

@Chris.Childerhose : it’s a copy job I’m using in direct mode, so no specific gateway is being used

#Veeam Legend 5* | VMCA 2* | VMCE 3* | VMTSP+ 2* | VMSP 2* | DCIE 2* | Loves Best Practices | Twitter : @NLosschaert
Nico Losschaert
Userlevel 7
Badge +11

@dloseke  : I will try to use a regular copy VM job to the same vendor, to see if I’m having the same issues and try to copy the NAS copy job to Wasabi. This to rule out some things. Thx @dloseke and @Chris.Childerhose for your feedback.

#Veeam Legend 5* | VMCA 2* | VMCE 3* | VMTSP+ 2* | VMSP 2* | DCIE 2* | Loves Best Practices | Twitter : @NLosschaert
Chris.Childerhose
Userlevel 7
Badge +20

@dloseke  : I will try to use a regular copy VM job to the same vendor, to see if I’m having the same issues and try to copy the NAS copy job to Wasabi. This to rule out some things. Thx @dloseke and @Chris.Childerhose for your feedback.

No problem keep us posted. 👍

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
dips
Userlevel 7
Badge +7

Hey @Nico Losschaert do you have a firewall deployed on your lab that might be carrying out some sort of inspection?

Dipen N. K. | IT Security Specialist | Veeam Legend 2022 - 2023 | www.dipen.co.uk
Rick Vanover
Userlevel 7
Badge +10

Hey @Nico Losschaert → I do not believe that iDrive has the Veeam Ready Object with Immutability qualification for the object storage, is that the underlying object storage? Here is the list:

Veeam Alliance Partner Technical Programs

Twitter @RickVanover | Email: rick.vanover@veeam.com
Nico Losschaert
Userlevel 7
Badge +11

Hey @Nico Losschaert → I do not believe that iDrive has the Veeam Ready Object with Immutability qualification for the object storage, is that the underlying object storage? Here is the list:

Veeam Alliance Partner Technical Programs

Hey @Rick Vanover - thx for your feedback. I really appreciate that!!!

First of all : of course I know the Veeam Alliance Partner Technical Program list and professionally I will never recommend a solution that is not on the list.

But this for my personal use.

I was before customer of iDRIVE using just regular storage using a Synology APP.

Because now with version V12 it also possible to write directly to object storage I saw the solution to have a secondary copy of my NAS share to the cloud.

Because I was already a customer of iDRIVE and I saw the fact they were promoting Veeam in using together with their E2 S3 object storage, I forgot to look to the Veeam Alliance list 🤗.

So in that you already deserve the best answer !!!!

But I still want to test things further. I already tested to copy a VM to the same storage and it works. Also another datacenter/region of iDRIVE doesn’t change anything.

Perhaps it’s something in combination with a NAS backup.

But I will further test to find the outcome.

Also I will probably create a case at iDRIVE, but I don’t think they will come with a solution.

A good thing about iDRIVE is also they don’t ask egress costs like Wasabi, but is in fact still a lot cheaper than Wasabi.

 

But of course your answer is correct and I didn’t check 😌.

I will keep you posted.

#Veeam Legend 5* | VMCA 2* | VMCE 3* | VMTSP+ 2* | VMSP 2* | DCIE 2* | Loves Best Practices | Twitter : @NLosschaert
Nico Losschaert
Userlevel 7
Badge +11

Hey @Nico Losschaert do you have a firewall deployed on your lab that might be carrying out some sort of inspection?

Thx @dips for the idea, but that I checked before and didn’t help 😅

#Veeam Legend 5* | VMCA 2* | VMCE 3* | VMTSP+ 2* | VMSP 2* | DCIE 2* | Loves Best Practices | Twitter : @NLosschaert
dips
Userlevel 7
Badge +7

Hey @Nico Losschaert do you have a firewall deployed on your lab that might be carrying out some sort of inspection?

Thx @dips for the idea, but that I checked before and didn’t help 😅

No problem! I’ve come across it before where the firewall would perform DPI and cause the streaming backup to fail

Dipen N. K. | IT Security Specialist | Veeam Legend 2022 - 2023 | www.dipen.co.uk
Nico Losschaert
Userlevel 7
Badge +11

Well guys

I have done more tests and have some feedback for you.

Tests I have done yet : 

  • I tested with a small share (12.5GB with 3.000 files) and a NAS backup-job directly to 2 regions of iDrive : the job starts, but nothing is getting transferred, after +/- 30 minutes I’m getting the next error

 

  • I tested with a very tiny Linux VM and a vSphere backup-job directly to 2 regions of iDrive : everything is going perfectly

 

  • I tested the same small share with a clone the NAS backup-job directly to Wasabi : the job starts, but nothgin is getting transferred, after +/- 30 minutes I’m getting the same error as with iDrive

 

  • I tried the NAS backup-jobs again after disabling my virtual Veeam proxy, so it has to use the backup-server itselves. The seem result.

The window firewall was not enabled on the backup-server.

The physical firewall is not blocking anything.

 

So it seems not be an issue with iDrive, because the same thing is occuring with Wasabi.

Also it seems to be only with the NAS backup-jobs, not with the vSphere backup-jobs.

Also it has nothing to do with being a large NAS share, because with a small share I’m having the same issue.

 

Does anyone else tried a NAS backup-job directly to immutable object-storage on the cloud without having an issue of having the same issue?

As already mentioned I’m doing this in my home test-lab so can’t create any case at Veeam.

@Rick Vanover : apologies but your answer regarding iDrive not being on the Veeam Alliance Partner list is of course correct, but not the cause of this issue...

 

regards

Nico

#Veeam Legend 5* | VMCA 2* | VMCE 3* | VMTSP+ 2* | VMSP 2* | DCIE 2* | Loves Best Practices | Twitter : @NLosschaert
Chris.Childerhose
Userlevel 7
Badge +20

Well guys

I have done more tests and have some feedback for you.

Tests I have done yet : 

  • I tested with a small share (12.5GB with 3.000 files) and a NAS backup-job directly to 2 regions of iDrive : the job starts, but nothing is getting transferred, after +/- 30 minutes I’m getting the next error

 

  • I tested with a very tiny Linux VM and a vSphere backup-job directly to 2 regions of iDrive : everything is going perfectly

 

  • I tested the same small share with a clone the NAS backup-job directly to Wasabi : the job starts, but nothgin is getting transferred, after +/- 30 minutes I’m getting the same error as with iDrive

 

  • I tried the NAS backup-jobs again after disabling my virtual Veeam proxy, so it has to use the backup-server itselves. The seem result.

The window firewall was not enabled on the backup-server.

The physical firewall is not blocking anything.

 

So it seems not be an issue with iDrive, because the same thing is occuring with Wasabi.

Also it seems to be only with the NAS backup-jobs, not with the vSphere backup-jobs.

Also it has nothing to do with being a large NAS share, because with a small share I’m having the same issue.

 

Does anyone else tried a NAS backup-job directly to immutable object-storage on the cloud without having an issue of having the same issue?

As already mentioned I’m doing this in my home test-lab so can’t create any case at Veeam.

@Rick Vanover : apologies but your answer regarding iDrive not being on the Veeam Alliance Partner list is of course correct, but not the cause of this issue...

 

regards

Nico

According to this page object storage is supported for NAS backup - https://helpcenter.veeam.com/docs/backup/vsphere/file_share_support.html?ver=120

 

Chris Childerhose - VMCA2022 | VMCE2023 | Veeam Vanguard 7* | Veeam Legend 3* | vExpert 5* | VCAP-DCV/VCP-DCV | Object First Ace | Cisco Champion | Twitter: @cchilderhose | Blog Site – https://just-virtualization.tech
J

Veeam Case 06316173
Wasabi Case 105257

Error: Failed to pre-process the job Error: HTTP exception: WinHttpQueryDataAvaliable: 12152

Summary of Problem: On Sept. 9th, 21 of 35 clients Backup Copy jobs failed with the above error that was using Wasabi us-central-1.  These clients have Backup copy Jobs that kick off between 10:30 PM PST and 12:50 PM PST.  A Sync Now Latest will return a success backup, but this was a daily error for these clients for 3 weeks until bellow fix. 

Environment: 

Constants: 

  • Veeam B&R 12 (Build 12.0.0.1420 P20230718)
  • Backup Copy Jobs
  • Backup times between 10:30 PM and 12:50 PM
  • Wasabi US-Central-1
  • No Window Updates installed
  • Failed on 9/9/2023

Variables: 

  • VBR servers running on Windows Server 2012r2 - 2022
  • ISP bandwidth 30/30 to 1000/1000 and different Venders and Firewalls 
  • Backup Repository volumes on Raid 5, 6, 10 running on HHD and SSD drives
  • Concurrency tasks range from 2 - 16+ 
  • Repository size range from less than 1 TB to 6 TB
  • Anti-virus ESET, SentinelOne, ThreatLocker, Bitdefender 

Fix: Registry Value Changes was recommend by Wasabi. ( If these registry fixes are removed from this  comment by the moderator.  Ask your amazing Veeam Support Engineer to see if this will fix your issue. ) 

Location: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
Name: S3ConcurrentTaskLimit
Type: REG_DWORD
Default value: 64
Value we should set it to (decimal): 4
*Description: Amount of parallel HTTP requests for data upload (tasks) to the S3-compatible object storage (archive tier).
*Does not affect number of offload tasks, only controls HTTP requests within a given task.

Location: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
Name: S3RequestTimeoutSec
Type: REG_DWORD
Default value: 120 (2 minutes)
Value we should set it to (decimal): 900 (15 minutes)
*Description: Timeout for a single S3 request.

Location: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
Name: S3RequestRetryTotalTimeoutSec
Type: REG_DWORD
Default value: 1800 (30 minutes)
Value we should set it to (decimal): 7200 (2 hours) - you could set this to 9000 as well, it'd be 2.5 hours then.
*Description: Timeout for all the s3 request retries. This is a cumulative timeout that each retry counts against.

CMD line Fix:  ( Note: the /F forces the updated value, remove /F if you want to be prompted on replacing that value. i.e. if you want to see if you have already done this fix. These registry entries do not exist already in this release of VBR )
 

reg add "HKLM\Software\Veeam\Veeam Backup and Replication" /v S3RequestTimeoutSec /t REG_DWORD /d 900 /f 
reg add "HKLM\Software\Veeam\Veeam Backup and Replication" /v S3RequestRetryTotalTimeoutSec /t REG_DWORD /d 7200 /f

reg add "HKLM\Software\Veeam\Veeam Backup and Replication" /v S3ConcurrentTaskLimit /t REG_DWORD /d 4 /f 

Caveat: If you have multiple S3 capable providers this could effect its performance. i.e. This change WILL effect ALL S3 jobs

Opinion: I would consider this a temp Fix. Lengthening the timeouts and reducing the concurrent task limits, might not fit within your backup windows. Considering that this appears to be a bottleneck/performance issue on the off-site storage providers side.  

This provider is baked into Veeam B&R and is a Veeam Alliance Partner.  There are standards that should be kept.  i.e. an up tic in errors like 12152 auto provision more resources to the effected resource, until the alerts are gone or roll back the changes.  3 Weeks of errors did not result in provision more resources for the effected site nor did it trigger a status update for that site.  Other S3 capable providers did not get name recognition as Wasabi did, standards should be kept. 

 

Nico Losschaert
Userlevel 7
Badge +11

Apologies for this late feedback.

In the meanwhile I did what @johnwatson suggested, I created the 3 registry keys and the problem is solved. 1TB has already been uploaded to iDRIVE.

 

Thx much @johnwatson for this suggestion, probably the same issue as with Wasabi.

#Veeam Legend 5* | VMCA 2* | VMCE 3* | VMTSP+ 2* | VMSP 2* | DCIE 2* | Loves Best Practices | Twitter : @NLosschaert

Comment


Terms & Conditions