Storage full but does not match VEEAM used.....??

  • 6 September 2021
  • 8 comments
  • 490 views

Userlevel 1

Hi All

 

Had a case open with VEEAM and NetApp neither of which is going anywhere pretty quickly, so will throw it out there and see if anyone has any ideas.

We have a large estate broken down in to areas which mirror the configuration of each other so a nice lot of reference points.

NetApp 16Tb LUN and 16TB volume mapped to a Veeam backup server.

The client wrote a lot of data to this back up location and took it offline. So we cleared snapshots down and brought it back online.

 When the mapped drive was browsed it could be seen there was a lot (TBs + TBs) of “Dead” VM backup files VEEAM advised these could be deleted which they were.

However the adjustment hasnt carried through to the storage end 

VEEAM UI thinks there is 11.9 Tb free NetApp only 2 Tb ** After Storage Snapshot removal 

Have stopped the jobs 

reset the storage password

Re-scanned the storage

Re-scanned the appropriate repository.

Any thoughts at all ..?

 

P


8 comments

Userlevel 7
Badge +20

Hi All

 

Sorry for delay here

Yes it is Mounted as ReFS

@ Chris You say it takes some time to balance it out and “more accurately” reflect what is actually going on. Any idea how long I.E should it be Weeks Days or Hours ..?

 

The volume associated to the LUN is Thin provisioned but Dedupe and Compression are not enabled at a Storage level.

All visible storage Snaps were cleared down to bring the LUN \ Volume back online and the Massive \ unexpected backup data removed from the job. When the mapped drive pointed at the storage was physically browsed some remnants of this data were seen to remain these were physically deleted.

I'm trying to see if NetApp is keeping anything locked in the background or which is hidden with their support.

The overall backup jobs should be not exceed the LUN\Volume capacity.

I can understand the numbers not tallying exactly but its Tbs out...

At what Netapp-Layer do you check the free-capacity of the volumes?

We’ve got two different conversations happening here about ReFS & NetApp.

 

My recommendation is to focus on the NetApp pieces. ReFS can be used to fast clone which is where block cloning will come into play, this would mean that the amount of backups you have could look larger than the repository shows within Windows. More information on that here.

 

I personally don’t believe that this is the problem you’re having, as your discrepancy is between the Free/Used/Total space that Windows is seeing and what NetApp is reporting, correct?

 

Can you answer these questions for me:

  1. Is your NetApp exclusively used for this Veeam LUN you’re reporting?
  2. Have you configured space reclamation on your LUN? A handy tutorial explaining the storage concepts here.

In essence this issue looks like either you’ve got multiple LUNs consuming more space, leaving you with less free space on the underlying SAN. OR your SAN isn’t freeing up space as it becomes available. Provided your SAN is powerful enough to handle the additional workload I would be tempted to enable Dedupe & Compression, you won’t get much space savings if your workloads have already been Deduped and Compressed by Veeam but as Veeam handles this on either a per backup job or per VM basis (depending on your Veeam configuration), you may be able to increase storage efficiency still by handling this at the array level.

 

I hope this helps!

Yes I agree with this.  ReFS if used takes no more than a day to release space but Veeam Console will “never” report the correct numbers for you.  This is where you need to check the file system and storage itself as noted by many.

Userlevel 7
Badge +20

Hi All

 

Sorry for delay here

Yes it is Mounted as ReFS

@ Chris You say it takes some time to balance it out and “more accurately” reflect what is actually going on. Any idea how long I.E should it be Weeks Days or Hours ..?

 

The volume associated to the LUN is Thin provisioned but Dedupe and Compression are not enabled at a Storage level.

All visible storage Snaps were cleared down to bring the LUN \ Volume back online and the Massive \ unexpected backup data removed from the job. When the mapped drive pointed at the storage was physically browsed some remnants of this data were seen to remain these were physically deleted.

I'm trying to see if NetApp is keeping anything locked in the background or which is hidden with their support.

The overall backup jobs should be not exceed the LUN\Volume capacity.

I can understand the numbers not tallying exactly but its Tbs out...

At what Netapp-Layer do you check the free-capacity of the volumes?

We’ve got two different conversations happening here about ReFS & NetApp.

 

My recommendation is to focus on the NetApp pieces. ReFS can be used to fast clone which is where block cloning will come into play, this would mean that the amount of backups you have could look larger than the repository shows within Windows. More information on that here.

 

I personally don’t believe that this is the problem you’re having, as your discrepancy is between the Free/Used/Total space that Windows is seeing and what NetApp is reporting, correct?

 

Can you answer these questions for me:

  1. Is your NetApp exclusively used for this Veeam LUN you’re reporting?
  2. Have you configured space reclamation on your LUN? A handy tutorial explaining the storage concepts here.

In essence this issue looks like either you’ve got multiple LUNs consuming more space, leaving you with less free space on the underlying SAN. OR your SAN isn’t freeing up space as it becomes available. Provided your SAN is powerful enough to handle the additional workload I would be tempted to enable Dedupe & Compression, you won’t get much space savings if your workloads have already been Deduped and Compressed by Veeam but as Veeam handles this on either a per backup job or per VM basis (depending on your Veeam configuration), you may be able to increase storage efficiency still by handling this at the array level.

 

I hope this helps!

Userlevel 7
Badge +13

Hi All

 

Sorry for delay here

Yes it is Mounted as ReFS

@ Chris You say it takes some time to balance it out and “more accurately” reflect what is actually going on. Any idea how long I.E should it be Weeks Days or Hours ..?

 

The volume associated to the LUN is Thin provisioned but Dedupe and Compression are not enabled at a Storage level.

All visible storage Snaps were cleared down to bring the LUN \ Volume back online and the Massive \ unexpected backup data removed from the job. When the mapped drive pointed at the storage was physically browsed some remnants of this data were seen to remain these were physically deleted.

I'm trying to see if NetApp is keeping anything locked in the background or which is hidden with their support.

The overall backup jobs should be not exceed the LUN\Volume capacity.

I can understand the numbers not tallying exactly but its Tbs out...

At what Netapp-Layer do you check the free-capacity of the volumes?

Userlevel 1

Hi All

 

Sorry for delay here

Yes it is Mounted as ReFS

@ Chris You say it takes some time to balance it out and “more accurately” reflect what is actually going on. Any idea how long I.E should it be Weeks Days or Hours ..?

 

The volume associated to the LUN is Thin provisioned but Dedupe and Compression are not enabled at a Storage level.

All visible storage Snaps were cleared down to bring the LUN \ Volume back online and the Massive \ unexpected backup data removed from the job. When the mapped drive pointed at the storage was physically browsed some remnants of this data were seen to remain these were physically deleted.

I'm trying to see if NetApp is keeping anything locked in the background or which is hidden with their support.

The overall backup jobs should be not exceed the LUN\Volume capacity.

I can understand the numbers not tallying exactly but its Tbs out...

Userlevel 7
Badge +13

Sound interesting @pk101 ! Could you describe how NetApp Volume is mounted to the proxy host? What filesystem is used?

Maybe is has to do with storage thin-provisioning? So maybe you promised your proxy to provide - for example - 100TB but NetApp had at this point in time just 50TB left. When Veeam allocated about 40TB, Veeam sees 60TB free, NetApp just 10TB - not dedup, compression or whatever in this example.

Userlevel 7
Badge +17

ReFS is my first thought, too.

Or you have some kind of thin provisioning enabled on your NetApp and exceeded the physical storage now.

 

Userlevel 7
Badge +20

Hi,

 

This is likely because your LUN being presented to Veeam (assuming this is a repository) isn’t aware of NetApp level raw device usage. For example, space saving techniques such as compression and deduplication. Your Veeam server cares that the block is written but is unaware if the block was deduplicated with a different block. Such scenarios cause discrepancies between the physical consumption and logical provisioning of space.

 

You said you cleared snapshots down but did you remove them all? If the LUN has snapshots still then whilst blocks are freed “live”, the data is being retained in a snapshot.

 

Another consideration: is the NetApp only being used for this LUN? As these LUNs are thin provisioned it’s possible that you had space provisioned that you didn’t have spare in the first place.

 

Most SANs expect a data deduplication ratio of some sort so I could have 10TB RAW capacity. But as my data compresses or deduplicates well I could create multiple LUNs of 10TB in size. However once I breach the amount of physical space (or alternatively a reservation), my LUN is taken offline.

 

Does any of what I’ve said align with what you’re seeing?

Userlevel 7
Badge +20

What is the file system of the mapped drive?  Is it ReFS by chance?  If so it takes time to free up space and show the proper amount. But Veeam never reports the proper space usage with this file system.

Comment