vCLS appliance showed up in one of our backup jobs. This caused the backup job to show as failed.

Interesting....never heard of this happening before @spider32 

strange issue, a normal running vCLS system should never be part of the backup.

Is it possible that you recently had discovered a problem in your vSphere Cluster? Modified HA settings, added/removed hosts? Or maybe changed settings regarding the retread mode (where you can also deprovision the vCLS systems and also deploy them from scratch)? 

edit: If so, then *this* vCLS machine could potentially be an orphaned vCLS system - you should be able to delete this (option not greyed out) in the context menu. But if this is true, your vSphere team, should also check, what causes this behaviour.

Best regards Markus

Hi @spider32 , I was searching around the Hub here to see if I could find more info related to vCLS VMs being processed and came across the following post:

It appears since Veeam v10a, Veeam automatically excludes vCLS VMs from being processed/backed up, although I couldn’t find definitive documentation stating such (maybe Release Notes for that version no longer Internet accessible).

How do you configure VMs to be backed up in your Jobs?...by Datastore container maybe? If so, maybe Veeam somehow backed up a vCLS VM in one of your Datastores you use as a container to backup from? You can control what Datastore vCLS VMs get placed on...maybe try this as well?

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-6C11D7F9-4E92-4EA8-AA63-AABAD4B299E7.html

Still, if Veeam does have automatic exclusions for vCLS VMs (and makes sense to me they should), you should probably ping Veeam Support to see why, at least for this 1 Job of yours and this one time, Veeam did backup a vCLS VM. I do assume you’re on a Veeam version later than 10a?

A few years ago I has a similar issue (not backup related) and I remember my action was to disable and enable the vcls VM. 

Our backup jobs are volume based. So, it backups all VMs in a particular volume. It so happened that vSphere decided to put all the vCLS appliances in this volume which is totally fine. Like what @coolsport00 mentioned, Veeam automatically excluded all the vCLS appliances from the backup except this one. I think Veeam does the exclusion by using the vCLS folder. However, this particular vCLS appliance for some weird reason is not in the vCLS folder and we are unable to move this to the vCLS folder thus the backup job tries to backup this appliance. Workaround was manual exclusion but really curious how this happened and what would be the fix. Powering off/on the appliance is not working. We tried to move the vCLS appliance to the vCLS folder but getting Operation failed permission denied (as a vCenter admin). I have relayed the added info to our vSphere admin. Let you know any updates. Thanks.

hi @spider32, as mentioned, check retreat mode in vSphere. Your vSphere Admin could simply deprovision these vCLS systems with a single click, and automatically redeploy them.
There must something happened in your vSphere Cluster regarding the vCLS to get orphaned/corrupted or something like this. 
best regards Markus

Yes, retreat mode, @Dynamic,  was suggested by our vsphere admin but since this is a production environment and the issue is not really critical, they would like to schedule when to do this one. I will update if this resolves the issue. Thanks.

@spider32 - ok..keep us posted how things go.

Best.

This began happening to me recently after I upgraded vCenter/VCSA and Veeam to the latest version of each. VCSA 8.0.3.0000 and Veeam 12.1.2.172. I have excluded the offending VCLs VMs but I’m not sure why they’re being included in my backup jobs at all. I forced VCSA to recreate the VMs but that hasn’t resolved the problem.

@bbrown57 - I recommend reaching out to Veeam Support to see why Veeam isn’t excluding those vCLS VMs by default.

Hi @bbrown57, i agree with @coolsport00 . But be aware that 8.0U3 is currently not supported by Veeam (KB2443), even though it’s working. I think you will see also a warning within your jobs.

But your problem is indeed usefull for the support team, because it’s maybe related to U3.
Regarding U3, Veeam will offer support normally within 90 days since GA.

Greeting Markus

Hi @bbrown57 - did you reach out to the support?

...i can confirm that this is an issue regarding 8U3 (which as mentioned is not yet supported). A colleague of me, is running in the exact same issue.

So you have to wait for the next VBR, when 8U3 will be officially supported, then imo this would be adressed. In the meantime you have to exclude these vCLS systems.

@Dynamic Yes, I reached out to support. They also confirmed to me that it is a vCenter 8U3 issue. They said they should have a patch in August to resolve the problem. For now I just excluded the VCL VMs from the backup job and it’s been working fine.

glad to hear :)

Also you could exclude the whole vCLS folder (or work with a vSphere Tag like “nobackup” and those tag to the folder and define an exclusion within VBR for this tag).

We are still on 7.0.3. We are still working on scheduling doing the Retreat mode. It’s currently low priority for the Virtualization Team due to the workaround. 

I will open a ticket to support since we are in a holding pattern due to the Retreat mode implementation.

Thanks for the update @spider32 

It is indicated as a known issue by Broadcom when you are on 8U3, and will be fixed in future versions of vCenter. For now I advice to raise a support ticket with Broadcom and they can assist in solving the issue. The vCLS in the Discovered virtual machine folder needs to be removed based on KB article: https://knowledge.broadcom.com/external/article/370286/manually-destroy-embedded-vcls-on-an-80.html The vCLS in the folder will becomes orphaned. After removing it from the vCenter database (by Broadcom Support), remove the folder Discovered virtual machine. Put the cluster in retreat mode and after that into embedded vCLS mode. This fixed the issue, and remember: always make a snapshot and/or a valid backup of your database 😉