Skip to main content
Question

Backup fails because no space left on device with 3.2 TB free on NFS repository


Hi, 

I’m running a backup job of fairly big VM (HDD1 = 1 TB and HDD2 = 3.9 TB) using VEEAM 11.0.1.1261.

After Deduplication and compression, the backup size ends up around 2.4 TB.

The backup storage is a NAS mounted using an NFS connection. In the console, I can check that it still has 3.2 TB left.

It has been working fine until last week. 

Error message : 28/04/2023 16:12:33 :: Processing ubuntu Error: No space left on device.
NFS status code: 28
Cannot get count from WRITE3res.
Failed to download disk 'EvaBuntu_2 - Copy.vhdx'.
Shared memory connection was closed.
Failed to upload disk. Skipped arguments: [shadowSpec>];
Agent failed to process method {DataTransfer.SyncDisk}.


 

Why is my backup job complaining that there is no more space left while I have 3.2 TB left on my repository?

Any advice much appreciated :)

15 comments

JMeixner
Forum|alt.badge.img+17
  • On the path to Greatness
  • 2650 comments
  • May 3, 2023

Is there a synthetic full created with this backup? This needs some additional temporary space….


No I have not been using using synthetic full so far.

Although, I’m seriously considering trying this solution to reduce backup time...


regnor
Forum|alt.badge.img+14
  • Veeam MVP
  • 1352 comments
  • May 3, 2023

What kind of storage are you using? Could it be possible, that it ran out of space and the NFS store is backed by a thin provisioned volume?


regnor wrote:

What kind of storage are you using? Could it be possible, that it ran out of space and the NFS store is backed by a thin provisioned volume?

I’m using a raid 5 volume on a NAS. It is not configured with thin provisioning. 


Chris.Childerhose
Forum|alt.badge.img+21

I would check the logs to see if there is something else wrong which are located here - C:\ProgramData\Veeam\Backup and then the job folder under here.

I believe but need to verify that you need enough space for the entire VM even though Veeam doesn't write that much but again I will need to verify this as I may be wrong.

Also what type of backup are you running Incremental Forever or Reverse Incremental?  This can play a role too.


coolsport00
Forum|alt.badge.img+20
  • Veeam Legend
  • 4133 comments
  • May 3, 2023

To use NFS, you have to have a Gateway Server for the Transport service as noted in the Guide. I wonder if there is a space issue with that device? Are you using Linux or Windows as your Gateway?


Chris.Childerhose
Forum|alt.badge.img+21
coolsport00 wrote:

To use NFS, you have to have a Gateway Server for the Transport service as noted in the Guide. I wonder if there is a space issue with that device? Are you using Linux or Windows as your Gateway?

Yes, this is a good catch from Shane which I would also check.


Andanet
Forum|alt.badge.img+11
  • Veeam Legend
  • 352 comments
  • May 3, 2023

Hi @syntax-error have you disabled “Network recycle bin” on your NAS? 


dloseke
Forum|alt.badge.img+8
  • Veeam Vanguard
  • 1447 comments
  • May 3, 2023
Andanet wrote:

Hi @syntax-error have you disabled “Network recycle bin” on your NAS? 

Yes, if it’s a Synology or QNAP, I’ve run into that a few times where the share/mount has free space, but on the back-end of the storage, it’s run out of space because the old .vbk’s and .vib’s are being stored in a Recycle Bin on the NAS that’s not visible to Veeam.


coolsport00 wrote:

To use NFS, you have to have a Gateway Server for the Transport service as noted in the Guide. I wonder if there is a space issue with that device? Are you using Linux or Windows as your Gateway?

My Veeam B&R runs on a windows server. It’s all local. So I assume the gateway is on the same machine. Here is a screenshot of my disks and it doesn’t seem to be running out of space (a backup is running at the time the screenshot - 95% completed)

 


dloseke wrote:
Andanet wrote:

Hi @syntax-error have you disabled “Network recycle bin” on your NAS? 

Yes, if it’s a Synology or QNAP, I’ve run into that a few times where the share/mount has free space, but on the back-end of the storage, it’s run out of space because the old .vbk’s and .vib’s are being stored in a Recycle Bin on the NAS that’s not visible to Veeam.

I had the recycle bin enabled on the shared volume on my Synology. I have disabled and emptied it.


I have noticed that up to april 1st, the backup time was around 11 hours. 

Since that date, it takes more than 55 hours… I also have to figure out why...


Andanet
Forum|alt.badge.img+11
  • Veeam Legend
  • 352 comments
  • May 4, 2023
syntax-error wrote:
dloseke wrote:
Andanet wrote:

Hi @syntax-error have you disabled “Network recycle bin” on your NAS? 

Yes, if it’s a Synology or QNAP, I’ve run into that a few times where the share/mount has free space, but on the back-end of the storage, it’s run out of space because the old .vbk’s and .vib’s are being stored in a Recycle Bin on the NAS that’s not visible to Veeam.

I had the recycle bin enabled on the shared volume on my Synology. I have disabled and emptied it.

have you done a rescan of your repository to check free space? 

As wrote @dloseke 

it’s run out of space because the old .vbk’s and .vib’s are being stored in a Recycle Bin on the NAS that’s not visible to Veeam.

This was my experience too. 


Andanet wrote:

have you done a rescan of your repository to check free space? 

 

I can’t because the repository is currently locked by a running backup. 

I’ll do it as soon as it’s finished.


Andanet
Forum|alt.badge.img+11
  • Veeam Legend
  • 352 comments
  • May 4, 2023
syntax-error wrote:
Andanet wrote:

have you done a rescan of your repository to check free space? 

 

I can’t because the repository is currently locked by a running backup. 

I’ll do it as soon as it’s finished.

this is incorrect…. you can rescan repository receiving some warning for used files due to backup job running.