Question

Backup fails because no space left on device with 3.2 TB free on NFS repository


Userlevel 3

Hi, 

I’m running a backup job of fairly big VM (HDD1 = 1 TB and HDD2 = 3.9 TB) using VEEAM 11.0.1.1261.

After Deduplication and compression, the backup size ends up around 2.4 TB.

The backup storage is a NAS mounted using an NFS connection. In the console, I can check that it still has 3.2 TB left.

It has been working fine until last week. 

Error message : 28/04/2023 16:12:33 :: Processing ubuntu Error: No space left on device.
NFS status code: 28
Cannot get count from WRITE3res.
Failed to download disk 'EvaBuntu_2 - Copy.vhdx'.
Shared memory connection was closed.
Failed to upload disk. Skipped arguments: [shadowSpec>];
Agent failed to process method {DataTransfer.SyncDisk}.


 

Why is my backup job complaining that there is no more space left while I have 3.2 TB left on my repository?

Any advice much appreciated :)


15 comments

Userlevel 7
Badge +17

Is there a synthetic full created with this backup? This needs some additional temporary space….

Userlevel 3

No I have not been using using synthetic full so far.

Although, I’m seriously considering trying this solution to reduce backup time...

Userlevel 7
Badge +13

What kind of storage are you using? Could it be possible, that it ran out of space and the NFS store is backed by a thin provisioned volume?

Userlevel 3

What kind of storage are you using? Could it be possible, that it ran out of space and the NFS store is backed by a thin provisioned volume?

I’m using a raid 5 volume on a NAS. It is not configured with thin provisioning. 

Userlevel 7
Badge +20

I would check the logs to see if there is something else wrong which are located here - C:\ProgramData\Veeam\Backup and then the job folder under here.

I believe but need to verify that you need enough space for the entire VM even though Veeam doesn't write that much but again I will need to verify this as I may be wrong.

Also what type of backup are you running Incremental Forever or Reverse Incremental?  This can play a role too.

Userlevel 7
Badge +16

To use NFS, you have to have a Gateway Server for the Transport service as noted in the Guide. I wonder if there is a space issue with that device? Are you using Linux or Windows as your Gateway?

Userlevel 7
Badge +20

To use NFS, you have to have a Gateway Server for the Transport service as noted in the Guide. I wonder if there is a space issue with that device? Are you using Linux or Windows as your Gateway?

Yes, this is a good catch from Shane which I would also check.

Userlevel 7
Badge +10

Hi @syntax-error have you disabled “Network recycle bin” on your NAS? 

Userlevel 7
Badge +6

Hi @syntax-error have you disabled “Network recycle bin” on your NAS? 

Yes, if it’s a Synology or QNAP, I’ve run into that a few times where the share/mount has free space, but on the back-end of the storage, it’s run out of space because the old .vbk’s and .vib’s are being stored in a Recycle Bin on the NAS that’s not visible to Veeam.

Userlevel 3

To use NFS, you have to have a Gateway Server for the Transport service as noted in the Guide. I wonder if there is a space issue with that device? Are you using Linux or Windows as your Gateway?

My Veeam B&R runs on a windows server. It’s all local. So I assume the gateway is on the same machine. Here is a screenshot of my disks and it doesn’t seem to be running out of space (a backup is running at the time the screenshot - 95% completed)

 

Userlevel 3

Hi @syntax-error have you disabled “Network recycle bin” on your NAS? 

Yes, if it’s a Synology or QNAP, I’ve run into that a few times where the share/mount has free space, but on the back-end of the storage, it’s run out of space because the old .vbk’s and .vib’s are being stored in a Recycle Bin on the NAS that’s not visible to Veeam.

I had the recycle bin enabled on the shared volume on my Synology. I have disabled and emptied it.

Userlevel 3

I have noticed that up to april 1st, the backup time was around 11 hours. 

Since that date, it takes more than 55 hours… I also have to figure out why...

Userlevel 7
Badge +10

Hi @syntax-error have you disabled “Network recycle bin” on your NAS? 

Yes, if it’s a Synology or QNAP, I’ve run into that a few times where the share/mount has free space, but on the back-end of the storage, it’s run out of space because the old .vbk’s and .vib’s are being stored in a Recycle Bin on the NAS that’s not visible to Veeam.

I had the recycle bin enabled on the shared volume on my Synology. I have disabled and emptied it.

have you done a rescan of your repository to check free space? 

As wrote @dloseke 

it’s run out of space because the old .vbk’s and .vib’s are being stored in a Recycle Bin on the NAS that’s not visible to Veeam.

This was my experience too. 

Userlevel 3

have you done a rescan of your repository to check free space? 

 

I can’t because the repository is currently locked by a running backup. 

I’ll do it as soon as it’s finished.

Userlevel 7
Badge +10

have you done a rescan of your repository to check free space? 

 

I can’t because the repository is currently locked by a running backup. 

I’ll do it as soon as it’s finished.

this is incorrect…. you can rescan repository receiving some warning for used files due to backup job running. 

Comment