TL;DR: The first comment I’ve made, after this post, is details of specific problems, and doesn’t really need to be read unless you want to give advice on the specifics. The second comment is more significant, as is this post.
As I seem to be continually fighting against Veeam's overall design/features/capabilities to make things work for over a year now I thought I'd ask a general planning question to determine if maybe there's just a better way to be using Veeam that isn't what we're doing now. Since I imagine we can't possibly the only company that provides a managed cloud-based backup service using Veeam, but isn't a full MSP.
Fair warning, this is going to be a little ranty, but I've been dealing with this on a daily basis for over a year now and so my frustration with trying to make it work is rather high.
My main goal here is to not really "start over" but essentially act like I'm starting over, and determine if maybe the entire way I'm using Veeam should be done differently. Should I be using different Veeam software in some places, should I not be trying to do some things with Veeam, etc... I don't expect a Veeam sales-person to ever say "Veeam really isn't good for what you're trying to do" but basically I came to the conclusion months ago that Veeam isn't good for what I'm trying to do with it, and so I'm looking to see if other people have found some ways around the issues I'm having so that they can make Veeam work for them. Though, personally, every "workaround" or "tweaking" needed to make Veeam work still seems like it shouldn't be a thing, since other software is much simpler and doesn't require so many workarounds or tweaks to make it work.
So quick overview, we have servers of our own, in a datacenter. These servers run Veeam cloud infrastructure and all the individual Veeam software components are split up, each component/application is installed in a separate virtual machine. Including the SQL server for the VCC server being separate from the VCC server itself. Note that "VCC server" here refers to the VBR server software acting as the Cloud Connect server.
Our customers are mostly small businesses, there are some exceptions, but most of them have fewer than 5 devices and lack any dedicated IT staff. So what we do is provide the backup service, and monitor things ourselves for errors or problems with the backups. Some customers choose to get status reports, but even of the ones that do, they typically don't do anything when there's a problem, they just leave it to us, which is what they pay us for so that makes sense. We also have some customers managed by actual MSPs but that's not a high percentage of our overall customer base.
Personally I have used other backup software before, and literally everything I've used has been less problematic. I figure it can't just be me though, so I'm assuming either other people do have the same sorts of issues and have come up with solutions, or somehow I'm the only one who actually knows things don't have to be this problematic, that there are plenty of better products out there. And full disclosure, if it were up to me, I wouldn't be using Veeam. But the people who actually make that decision still think Veeam is the best choice despite the ongoing plethora of daily problems. The simple fact that we have a team of people to maintain the software is a red flag for me. In the past all the other software I've used has been more "Set and forget" sort of things. Install it, pick what to backup, where to backup to, and when to do it, and you're done. But actually getting that to happen with Veeam seems to be much more complicated for so many of our customers that we do (apparently like many companies that use Veeam) actually have a team of people to try and make it work on a daily basis.
Veeam Only Service Provider Planning
So, my main issues I've been having lately are:
Remote management is sorely lacking. I know the marketing guys who said out the email saying the VSPC software enables an MSP to monitor and manage everything remotely across all of their companies are just marketing people, they don't actually use the software, but it is far from that. On a daily basis I have to use the VSPC, VBR console, and a third-party remote desktop application to monitor and manage everything.
Partially this is because support is always asking for things that Veeam doesn't do. Like collecting data about Windows Events or the contents of folders, changing registry keys, editing files, all sorts of things that are simply not part of Veeam. Noting that "support asking" wouldn't be an issue if it "just worked" like the marketing guys say it does, like other software I've used does. I have recently determined, but yet to actually do it, that apparently I can install the VBR server for free on an otherwise Agent-only computer, so then I can use the remote powershell functionality to do more. And that's fine, I don't mind PowerShell. However the process to install the VBR server takes hours. And is really complicated. So it's very inconvenient to do and redo and undo regularly if that's going to be the solution for me going forward. I don't understand why Veeam is so complicated, but it is. Somehow the software takes more space on the computer than Windows itself does in some cases, and takes hours to install where I can literally wipe a drive and reinstall the entire operating system in 20-30 minutes. Windows used to always be that thing that I'd be like "well nothing's as complicated as Windows" but since it was the operating system and not some application it was more acceptable to me that it was complicated, took a lot of space, and took sooo long to install, where "sooo long" is 20-30 minutes. Veeam is just an application. I understand that the VBR server does some more complex things, but at its core, it's a backup software. It literally needs to just copy data from one place, to another place, on a schedule. But getting it to actually do that seems to be really challenging. Also the Agent software takes an annoyingly long time to install too, considering what it is, but realistically that only takes 20-30 minutes most of the time.
I've known for a while but it's recently become a more significant issue that Veeam deletes partially transferred backup files. Which results in it needing to start over again if a backup fails part way through. So if I have a computer that's backing up a few TBs of data, but it goes offline or fails for some reason before it finishes, it needs to start all over again. Which seems to make Veeam unsuitable for any computer with an unstable internet connection or that isn't online for whatever the needed period of time is. Other software I've used will retain partially transferred backup data, so the backup process doesn't have to start over from the beginning each time. Which makes backing up on an unstable connection or without staying online long enough to complete the backup in one session feasible. However Veeam doesn't have that ability, which is made extra annoying to me because my thought is that Veeam transfers data, partially, so it's incomplete, but it's there. At this point that's better than what it does now. But someone at Veeam spent time to add the functionality to check for a partial backup and delete it. So from my perspective, every other software I've used has a useful feature. Veeam spent time to intentionally remove that useful feature from the software. It's not entirely impossible that I'm wrong, maybe that's not exactly the case, but the end result is still the same. Everything else I've used has a useful feature, which seems like a very basic feature to me, but Veeam doesn't have it.
On the topic of basic features, accounts. Account permissions, they can not be changed. This isn't really something that would affect my decision to use Veeam, but it's just another annoyance whenever I have to do it. I'll admit to not having used every software/service out there, but I literally don't think I've ever seen something where accounts have configurable permissions that can't be changed after the account is created. I honestly can't comprehend who thought to have permissions but not make them changeable. It seems like another really basic feature that everything else has that Veeam doesn't have.
I'm not going to go into too much whining about what Veeam chooses to restrict to certain license types, because that's the sort of way they're supposed to be making money. (Note I say "supposed to be" because I'm convinced the software is poorly designed on purpose and they spend development time blocking certain features like the above mentioned retention of partial backups, just because they want it to be complicated and buggy so that people need to pay for support. I can't understand how else so many things seem to be broken or intentionally designed in a way that makes no sense to me.)
But, some features that I've not seen restricted to a license type on other software are restricted by Veeam, like how often a backup can be scheduled. Things like SQL backups make sense, but things like backing up more often than once a day, which I think is already bordering on unreasonably long to go without a backup, doesn't make sense to me to restrict to a particular license. I also don't understand why it seems someone spent development time (again) to remove the fully developed and supported features of "backup when target is connected" and "backup when powered on if the backup was missed" from the more expensive license. Typically more expensive software editions add features, not take features out. As-is there's no way to actually have all the desired functionality. I can have some with the cheaper license, some with the more expensive license, but I have no ability to actually have all the features I want. I have to pick one or the other, and neither is complete.
Those things being more significant issues lately because it's come to my attention that most of the problems I've been spending the past months, countless hours, at least a dozen forum threads and support cases, to try and fix are actually not issues, but rather just the software on the computer restarting. Whether because it crashed, was manually stopped and started, or the computer restarted and so the software was restarted too. I'm rather bothered that it took as long as it did before someone determined that to be the case, but it seems like a plausible explanation for a lot of the problems I'm having. Note that it wouldn't be a problem if I could set the computer to backup more often than once per day, but that costs more money. And I can't realistically tell customers they need to change how they use their computers just so the backups can run. It's supposed to "just work" without requiring significant changes to the environment. It's really not realistic to design an IT environment of any size around the software that backs it up, I would expect the backup software to work in any environment.
For comparison, what I'm used to not needing to do to make backup software work is things like changing registry keys, configuration files, having SQL databases at all (but I can accept that's how Veeam works if it were to manage everything itself), having custom user accounts for the backup software, I would expect it to just be installed and running as a service. But we have to maintain lists of assorted different customer login credentials, which to me is a significant security problem that I've never had before. But for all of our customers we have to have them share with us at least 1 user account's login credentials. More than 1 in most cases. And typically that then means asking them to have accounts on their computers or Active Directory environments that they don't change the passwords to, because we may need to login at any time. Other software (as in, every other application I've used of any sort anywhere) can be installed with administrative access initially, and uses standard built-in service accounts so it doesn't require adding user accounts and maintaining login credentials. Veeam can't even install it's own software updates without a human providing login credentials.
Also Veeam's collection of applications that are required on the customer's infrastructure to work is similarly problematic, there's a lot of separate things to maintain that all seem to have their own issues and in many cases seem to only barely work together. Needing to create virtual machines in the customer's environment is definitely not "just works" and I would consider that to be a more major modification to the customer's environment. I understand that people have gotten VBR to work without that, and I don't see any reason why it wouldn't, but Veeam continues to insist that it's needed, so we do that, however it adds another layer of complexity, another modification, and another time consuming thing that needs to be set up and maintained that other software with the same functionality doesn't have. What I'm used to is installing the application, one application, on the computer with the data being backed up, just one, then that's it. No custom user accounts, no virtual machines, no SQL databases, and certainly no registry modifications.
Again the highly complicated setup process would be less of an issue if it were set up automatically and we didn't have the potential security issue of all the user accounts that are involved and their associated login credentials that need to be maintained and shared between us and the customer. These things being less of an issue of course if we were a full MSP managing all of the customer's environment, because while it is a modification, it's a modification to an environment that we maintain, and login credentials that the customer doesn't ever actually need to have. However we are not, which gets back to my thoughts that this whole thing isn't designed for use by a "backup only service provider".
Veeam also lacks the ability to do remote volume level recoveries, which I would think is the most common type of recovery, but admittedly I may just be thinking from a more "disaster recovery" type of viewpoint rather than "user doesn't know what they're doing and they deleted something they now need" sort of viewpoint. Also apparently Veeam stores data in different formats depending on the operating system it comes from, which contributes to the feeling that the different Veeam components are made by completely separate teams and they have only the bare minimum amount of effort to make it all work together, not that "Veeam" is one product that works across multiple operating systems, rather "Veeam Agent for Linux" is one product and "Veeam Agent for Windows" is another product, and they just happen to both be made by the same company. Also, side note, these different formats have the same file extension, the purpose of the file extension is to tell the operating system and humans what format a file is in in a quick and easy to understand manner, so using the same extension for different incompatible file formats is confusing. Same goes for the log bundles by the way, totally different format, for which I don't understand, but it makes more sense if we don't think of "Veeam" as one product, but rather a collection of products developed by different teams that all share a name.
My overall impression of Veeam is that it feels like everything is a couple of years old, still very much in development, lacking of a lot of basic functionality, and the different components don't work well together and have different features. There are things that the Windows Agent has that are missing from the Linux and Mac Agents for instance, which I can accept because despite feeling like it's a very new and "rough around the edges" software, the Windows Agent is actually several years old. However, the Linux Agent has features already that the Windows Agent is missing, the Mac Agent may as well, but I'm not entirely sure. The VSPC server doesn't actually provide any sort of "management" capability for VBR servers comparable to the VBR console, which makes it feel incomplete. The separate VSPC Management Agent software being a separate software at all adds to the overall feeling that the VSPC software is another separate product from the VBR server and the Agent software and that it's all just got the bare minimum effort to make it all work together. It feels very far from a polished product where the entire functionality is all in one product, it feels more like a bunch of separate applications that are all new and "rough around the edges" and have only the bare minimum of effort put in to make them work together. Which confuses me greatly since Veeam as a company is nearly 20 years old now and employs thousands of people across multiple countries. It feels like the development team is more focused all the time on adding new functionality which probably won't work right until a couple years after the release, instead of fixing problems that have been around for years and just generally polishing things, making what's already there work better or more efficiently. Most of my "support" cases seem to end after days of back and forth with "it's supposed to do that, we won't be changing it" or "we'll consider changing it or fixing it in a future release, but that release could be years away since no one else has this issue". So, while I want to do my best to understand if the development team is being told to work on certain things, it seems no matter what my problem is, everything just comes back around to being on the development team to do something with. Which gets back to me thinking Veeam just isn't suitable for what I'm trying to do here, maybe people want it to be, but if it needs to be "developed" before it works, then it's not yet suitable.
My understanding is that Veeam was originally made to backup VMWare virtual machines in an isolated, self-contained, one company, local network environment. And its ability to do anything else seems to just be the result of various features that were "slapped on with bubblegum" over time (to quote another forum user) and it doesn't work very well in an environment like ours where we want to do backups over the internet, at numerous companies, with numerous different types of environments, and lack the full control of an company's internal IT staff or an MSP.
So I'm curious if anyone has suggestions or advice overall on how they've handled using Veeam in a similar environment, anything that maybe I'm missing, or maybe if someone actually thinks that Veeam is just not meant to be used like this, let me know. I really am trying to make Veeam work, but it seems like every week I just get back to more of the same, either Veeam doesn't do what I'm trying to do, or what I believe to be a problem isn't actually a problem, or I can make it work but I have to do a bunch of stuff to work around the problem or tweak it just right to make it work, none of which has been my experience with other software that I've used. So thinking I can't possibly the only one trying to use Veeam in such an environment, so maybe I'm just doing it wrong. Admittedly I have determined numerous places where I would do things differently if I were setting up Veeam from scratch, but I wasn't with the company when they started using Veeam, so I wasn't involved in the initial planning phase. Though, I must admit to not feeling there should be a "planning" phase, because again, it's just a backup software. It should be a simple, install, tell it what, where, and when to backup, then you just leave it and forget about it and it "just works" on it's own. But again, if I were forced to use Veeam I would've done things differently, mainly in that I would be using VBR servers literally everywhere, never having only Agents, and so I'm wondering if other people have things set up differently.
If any other service providers, especially any that are providing only Veeam backup services, not a full MSP service, want to share what your environment is like, and that of any customers, that would be potentially quite helpful information.
+20
Wow that is a long post for sure. Let me read it next week after vacation when I am back and give my input. I work at an MSP and Veeam is a huge part of our services.
+22
I think the thought is in the right place with VSPC but there are definitely challenges. One area that I have been playing with is Migration. Migrating to a new server is not straight forward, at least my tests seem to behave differently at different times (could be my lack of full understanding), so I would have hoped for “backup configuration” feature for it just like with VBR. Mind you we are still waiting for that with VBM365.
To be fair Service Providers do demand features asap and Veeam are trying to keep them happy. I think feedback is essential to help them get things up to speed.
+6
If your a propartner you should be able to speak to your Veeam account manager and get some help reviewing the design / deployment and they will be able to provide some valuable advice and guidance.
+6
There’s a lot (and overwhelming large amount really) to unpack here. That said, the Service Provider Console has gone though a lot. It does a lot more now that previous iterations and they keep expanding it’s capabilities on each version. A lot of the last version I feel like was focused on integrating with Veeam Backup for Microsoft 365. That said, there were improvements made in the VBR support. I’ve been managing a lot of VBR deployments, but as a MSP mostly (a few clients we don’t really manage their IT and just provide backups). I really started into the Cloud Connect backups in the past month or two as most clients have full blown VBR deployments which add much more flexibility. The Veeam Agent I feel is pretty limited in what it can do in comparison. For this reason, larger clients have a VBR server on-site, either physical with local storage (our preference) or virtual with a NAS (our older deployment model). This is much better for me than using the Agent direct to cloud or a VCC target.
I have to get on a call, but I’d circle back around on this again when I get some time because of the length of the post and how many points you’ve made.
Thanks for the feedback, I am aware of a lot of people who have indicated improvements being made, but for our environment (which is almost entirely VSPC managed agents) it seems like there are just as many new problems with each update as things that get fixed, so it’s still a full time job just trying to make it all work.
And support keeps getting back to me on nearly everything with essentially “we’ll work on changing it in a future release”, noting that “change” is not “fix” because it’s not “broken” and I’ve learned “future release” is essentially never because apparently I’m the only one who wants things like backups that don’t get deleted if only part of the data gets transferred, common sense things to me that every other product I’ve used does, but apparently Veeam considers them uncommon enough that it’s just indefinitely “a future release”. And while I’ve only been using Veeam for 18 months I’ve seen enough other people on the forums also saying they’ve been asking for features for years and still haven’t gotten them.
It just adds to the impression that the development team is all focused on adding new functionality without fixing what they already have. And while I can recognize my request for “resumable backups” isn’t really a bug that’s broken in the software, it is a new feature (though it seems to me that the feature is just “remove the code that deletes the partial backup”) but I do consider it broken, because until it’s implemented a computer without a consistent internet connection good enough to complete the backup in one session just can’t be backed up. Support’s suggestions are things like, let’s try a different a router, or let’s try a different internet service provider. Which, I’m not even going to suggest to my customers. No other software needs those things and I am certainly not going to try to tell customers that they need to have particular network hardware or a particular ISP in order to back up their computers, they would just go find someone else, and I would probably encourage it. It’s certainly far from the “just works” type of thing that Veeam’s marketing team seems to say all the time. And it all seems like a hold-over from Veeam’s original design of backing up VMWare systems in an isolated network. It may have been acceptable to require a perfect network connection when we’re connecting one server to another server sitting on the same rack, but connecting computers over the internet, especially some on cellular hotspots, laptops via wifi, customers with slow DSL connections, any number of variables, every other software I’ve used can handle poor connections, it might just take longer. But Veeam seems to just not try.
I do understand to an extent that sort of thing is management’s decision, so I’m not personally assigning blame to the development team, just Veeam as a whole. And I can also legitimately accept the “we’ll work on it for a future release” answer as a legitimate answer. However, without any solution in the meantime it’s just another scenario (and this is just one example) where Veeam doesn’t work for us. It’s not an overly complicated situation, if it doesn’t work then it doesn’t work, but I spend hours each week trying to make things work and most of what support comes back to me with “we passed it to the development team” and then support just sort of leaves it at that. Which, okay, if they can’t do anything because it’s a problem in the code, then okay. But if there’s no ETA for a fix, which seems to be the norm, then it’s just more and more things piling up that make Veeam unsuitable for our situation. After all, I can’t really tell customers “there’s a problem with your backups, it’ll be fixed sometime in the future”, especially when I know for a fact that simply using different software would both fix their immediate problem, but also be more reliable and easier to manage going forward.
And I could go on for a long time about everything that seems to just be poorly designed like it had no common sense involved, but those are mostly things we work around, so they don’t completely hinder our ability to use Veeam, though with other software we wouldn’t be “working around” so many things either.
Again, if for no other reason than because of the enormous amount of effort it takes to set up Veeam, and the fact that it’s basically set up for us, I do want it to work, it just seems like I have more interest in making it work than Veeam as a company does, at least for our specific scenario.
Full disclosure, I formerly used Acronis, and managed 10x the number of computers as I have now, that was at a different company, when I came to the company I’m with they already had Veeam set up, it’s always had problems and that’s been my main thing on a daily basis is making Veeam work as best as possible. So virtually all my comparisons are to Acronis, but I’ve used some other software in smaller environments as well, and Veeam has a lot of things that are just designed completely differently than other software, and it’s all taken a lot of adjusting to how Veeam works (or in a lot of cases, doesn’t work). However I have legitimately been trying to make Veeam work for us, it just is a daily thing for me. Whereas with Acronis, I had a smaller team, 10x the computers, and there was maybe 0.1% or 0.2% of the computers that had actual problems related to the backup software on a daily basis. (Things like the computer was offline or the backup storage destination was full are not being counted, and I don’t count those as issues with Veeam either.) Of which things were always easy fixes like, reinstall it, or some setting to change. I never once contacted Acronis’ support for anything. But with Veeam we have backup software related problems affecting 15-20% of our computers on a daily basis. And each thing takes several days at least of going back and forth with support to troubleshoot, test, isolate an issue, and then just get back “we’ll fix it in a future release” or on a small number of things we do get fixed, hopefully it doesn’t just break again tomorrow. (Noting I do consider it a “backup software related” problem if it’s a stupid design thing like deleting partially transferred backups or not allowing backups when the backup target is connected on a “Server” licensed Agent.)
I suppose ultimately my experience with other software is that backups really should be “set and forget”. I should be able to just install the software, set what to backup, where to backup, and when to backup, and that’s basically it, perhaps some other settings I can tweak if really desired. But then it should just work. By itself. Reliably. Not require multiple full-time employees working to keep things backing up like we have now.
And that’s the biggest change to coming to Veeam, that companies using Veeam have large teams, whole departments dedicated to managing and maintaining backups. Whereas what I’m used to is something that more accurately “just works” and so from my perspective, even the amount of work I do in a single week to make Veeam work for us is more than I would expect to do in a whole year back when I was working with Acronis full time. Excluding of course any time spent helping users with recovering data, but that’s actually a very rare occurrence in my routine.
+6
I’ve been reading your comments, and it seems one of your biggest frustrations is remote management.
I honestly never use it, I’ll be accessing that system some other way.
That said, I don’t think its fair to expect a backup software to be a full remote management tool. Can VSPC manage policies and jobs for a Veeam agent? Yes. Can it manage the remote machine itself? No, and I’m not sure its a good idea for it to have this capability even if someone wanted to make it do that.
Most MSPs I know have different tools for remote monitoring/management of machines - either VMs or physical systems. If a customer wanted me to manage their backups, I would be adding on a remote monitoring/management agent, even if the ability within the backup agent existed to do most tasks. Otherwise you will find yourself in a situation (no matter what software you are using to back up a system), when that software fails. Even remote monitoring/management software can fail to start or crash at times. Its always good to have a plan B.
I do understand and accept the “plan B” idea, and to be honest I do agree, but only to the extent that I would want a secondary application to use in the (should be rare, not daily like it is with Veeam) scenario where the main application stopped working. However because we are not a full MSP, and provide exclusively backup services, nothing else. Just backups. We don’t have that (which is ultimately a decision made by people at the company before I was around, I would’ve advised against it, but we’re here now).
However, I do believe the application should be inherently remotely manageable if it’s sold as a “for service-providers” application. Which Veeam is. And, yes, I do agree it doesn’t need to be able to manage the whole computer, but if it could manage the whole computer (say, with PowerShell, remote desktop access, something) that would solve all of the missing features in the software itself in one quick fix. Virtually all of the support cases I open with Veeam’s support team regarding the agent, they all want me to run a log collection script to get things like Windows Events. Veeam can’t do this with the agent software, but they all want it, as though I’m expected to be using different software to begin with for remote management. Similarly there’s a lot of “change the registry key” or “change the configuration file” or things like that.
I’m aware the VBR server could do these things (with remote PowerShell functionality) so as far as I’m concerned, the code exists, but in a different Veeam product. The agent seems to have completely different functionality entirely. It lacks the PowerShell capability VBR has, where VBR lacks any useful remote management capability via the VSPC server, and it’s probably a network configuration issue in most cases, but I’ve literally never managed to make the “remote desktop” capability of the VBR server work, and the “remote console” feature rarely works because of having so many different versions installed all over the place.
But, support cases aside, there are still some big holes in the VSPC server’s management capability (of just agents) such as there seems to be no ability to actually manage backup files, to move or delete them for instance. Which makes backing up to a location I don’t have other access to useless, such as a local drive connected to the computer. It’s just not realistic for that sort of scenario.
For comparison, while I am stuck with Veeam right now, my past experience was with Acronis, which provided all the features I listed, such as full management of all remote software (especially because Acronis is literally one product, one application, which works with one management interface, not several applications, with different management interfaces depending on what you want to do) and had things like management of remotely stored backups. But it also has, even in the most basic, cheapest, entry-level versions, full remote desktop support and the ability to run commands (PowerShell or otherwise, depending on the OS). So, yes, I know Veeam isn’t Acronis, and it doesn’t need to have the same level of functionality, but I don’t understand how some of these basic features still aren’t a thing. And I don’t accept for a moment the explanation that (like I so often get told) I’m the only one who wants it and no one at Veeam thought it would be useful.
+6
You’re probably not the only person who wants it, but as said above, the majority of people already have that remote management capability with other software. Thus there really aren’t many who ask.
Sounds like you can make a solid case with your leadership team. You are being directed to use Veeam, so you need XYZ remote management software to go along with it. Seems pretty clear cut to me.
Comment
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.