SOBR Offload Failing - Not Enough Disk space


Userlevel 7
Badge +8

I recently ran in to an issue where we had a few of our SOBRs offloading to OBS which was added using a Gateway server for access.  There was an issue where it told me that we were out of disk space but thinking to myself that cannot be as we have Petabytes of storage for Object.

I investigated the logs and even worked with support where we found out the issue was on the GW server which uses the C:\Windows\Temp folder by default to store data before offloading to the Object Store.

Well it is nice to see there is now a KB article that shows how this can be fixed in three ways.  If you have this issue check it out here - KB4283: Scale-Out Backup Repository Offload task fails with "There is not enough space on the disk" (veeam.com)

The options are -

  1. Increase the space on the GW server - so your C drive
  2. Move to another GW server with more space for C drive
  3. Change the VeeamBackupTemp location to another drive with lots of space

5 comments

Userlevel 5
Badge

For Better Understanding the Veeam Scale-Out Backup Repository

 

The Veeam Scale-out Backup Repository provides horizontal-scaling support for storing data in multiple tiers. This repository contains one or more backup repositories called performance extents, and it can be expanded with cloud-based repositories or local object repositories.

All the storage devices and systems in the scale-out backup repository are joined into a system. This feature:

  • Provides a convenient way to manage backup storage.
  • Provides an easy way to extend repositories, by adding a performance extent to the existing repository. You can expand the scale-out backup repository at any time.
  • Supports any backup target that Veeam supports: Windows or Linux servers with block volumes, File Storage as a network share. All the features of any storage device or system are preserved.
  • Let you set up a granular performance policy.
  • Provides nearly unlimited cloud-based storage capacity by using Object Storage to offload data from extents to the cloud for long-term storage.

To deploy a scale-out backup repository, you need to first configure several backup repositories. As explained in the earlier topics, you can use the Block Volume and File Storage services as performance extents (in the performance tier), and you can use Object Storage as a capacity tier.

  • The performance tier of a scale-out backup repository is used for fast access to data and can contain one or more performance extents. For more information, see "Performance Tier" in the Veeam Backup & Replication User Guide for VMware vSphere.
  • The capacity tier is an extra tier of storage that can be attached to a scale-out backup repository. Data can be transported from the scale-out backup repository performance extents to the capacity tier for long-term storage. For more information, see "Capacity Tier" in the Veeam Backup & Replication User Guide for VMware vSphere.
Userlevel 6
Badge +1

I recently ran in to an issue where we had a few of our SOBRs offloading to OBS which was added using a Gateway server for access.  There was an issue where it told me that we were out of disk space but thinking to myself that cannot be as we have Petabytes of storage for Object.

I investigated the logs and even worked with support where we found out the issue was on the GW server which uses the C:\Windows\Temp folder by default to store data before offloading to the Object Store.

Well it is nice to see there is now a KB article that shows how this can be fixed in three ways.  If you have this issue check it out here - KB4283: Scale-Out Backup Repository Offload task fails with "There is not enough space on the disk" (veeam.com)

The options are -

  1. Increase the space on the GW server - so your C drive
  2. Move to another GW server with more space for C drive
  3. Change the VeeamBackupTemp location to another drive with lots of space

 

I’ve now a ticket open for 4 weeks about errors during offload like “Error: There is not enough space on the disk.”. But there is no space issue. There are GB’s free, I don’t see any increase in disk usage at that time. This is a Linux GW, not Windows. Support was then suspecting its not a space issue but a CPU/MEM problem because of this in the logs:

 

This is indicated by this message:[05.09.2022 06:56:16.942] alg      | WARN|Traffic history size is dangerously high.

 

But I can’ see any bottleneck. No monitoring system shows any high load. sar on cli also shows nothing.

Really strange, it started beginning of September and only affected a few offload tasks, others were running without issues. An no errors for the last 8 days.

Did anyone see this error with some other root cause than low disk space?

 

We have different problem here. 

We’ve started to use Linux Immutable repos recently, our proxies and gateway servers are still on Windows. For some reason we are facing troubles with this cache. Even thought we setup explicitly gateway for Azure blob which is pointing to GW server (running windows), the cache is still being created to Linux Immutable repo ! :( does anyone knows why is this happening? 

Userlevel 7
Badge +8

 

I add this tips related to SOBR

This reg helps keep the space between extents balanced.

How to enable advanced space control for Scale-out Backup Repository

KB2282: How to enable advanced space control for Scale-out Backup Repository (veeam.com)

Veeam Backup & Replication introduces new space update logic. When extent is assigned to a new task, service cache updates free space information with the real one and subtracts it with estimated sizes of all the tasks currently going to this extent.

You can switch to new logic by creating the following registry key:

Key: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\SobrForceExtentSpaceUpdate
Type: DWORD
Default Value: 0 (Disabled)

and setting value to 1 (Enabled).

Yes this is a great key to add as well.  Thanks for sharing.

Userlevel 7
Badge +4

 

I add this tips related to SOBR

This reg helps keep the space between extents balanced.

How to enable advanced space control for Scale-out Backup Repository

KB2282: How to enable advanced space control for Scale-out Backup Repository (veeam.com)

Veeam Backup & Replication introduces new space update logic. When extent is assigned to a new task, service cache updates free space information with the real one and subtracts it with estimated sizes of all the tasks currently going to this extent.

You can switch to new logic by creating the following registry key:

Key: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\SobrForceExtentSpaceUpdate
Type: DWORD
Default Value: 0 (Disabled)

and setting value to 1 (Enabled).

Comment