Perform import backups. Refer this link

https://helpcenter.veeam.com/docs/backup/vsphere/importing_backups.html?ver=110

Does this work for a granular CDP Failover without having vbk Backups?

Most important is to have configuration backups of the Veeam Database. With this you can recreate the VBR server.

After restoring the configuration you have all job and server definitions on your new server.

And if the repositories are present with the same paths on the new server you don’t have to import your backups.

The CDP is a replication system, in extreme cases you could log the offsite Vmware\HyperV infrastructure and manually start the replicated VMs in the order needed .

I often use Veeam just for the replication system. 
For example for a DR test I don't use the failover features etc. I manually turn on the replicated VMs in a bubble and redirect via public Ip the users who need to test the DR.
So to keep the production infrastructure up & running,

But you don’t replicate the VBR server itself, do you? As far as I know this is not possible.

It works for me if I use a Single Replica Job for the VBR Server itself, probably this could be a solution.
The Proxy method should be NBD and not hotadd.

Finally, I assume a granular failover to specific RPO is without VBR not possible:
https://helpcenter.veeam.com/docs/backup/vsphere/cdp_failover_point.html?ver=110

By Design the VBR should run at the DR Site.

The next question should be: can I failback to an older VBR Configuration Backup or Replica of the VBR Server and start a CDP Failover to most recent state after a global failure?

The replication is not transaction consistent for the database, as far as I know.

So, I would have a configuration backup at hand….

Having your backups offsite is key in this scenario. A backup of your configuration file would allow you to get VBR back up and going in the event that you have to recreate your backup server. Once you have VBR back up, you should be able to “re-map\reseed” your backups into the jobs, in order to maintain your backup chain.

in this specific cae , no.

But certainly it is always useful to replicate VBR if you have resources available!  

To me trying to use CDP is too complex for the VBR server. Doing a configuration backup and if you use templates in your environment you can get the server back in under an hour.  Deploy template, install Veeam and then restore the configuration.  I have done this many times and to even upgrade the OS on most of our servers.