VeeamON 2024 - Use Code "COMMUNITY10" for 10% Off!
I’ve seen this multiple times now with VMware. You cancel a task via vCenter or the ESXi UI and it just stays there hanging and nothing further happens. Normally this situation resolves itself after some time, but just last week I had a case where the task never stopped. While the Webclients weren’t really helpful in that case, we were able to cancel this task with the ESXi commandline.Warning: Interupting certain tasks, for example snapshot consolidations or Storage vMotions, can lead to issues or corrupted VMs. So only do this if you don’t have any alternatives and need the task to be canceled. And better have a backup at your hand.SolutionStart the SSH service on the ESXi host where your VM is registered Connect via SSH Query the VM ID: vim-cmd vmsvc/getallvms Query all tasks for this VM: vim-cmd vmsvc/get.tasklist “VM ID”Check the Task information: vim-cmd vimsvc/task_info “haTask….” Is the task still state still active? Note: The screenshot is just an example; the task state is s
You might or might not know this, but Wasabi does have a minimum storage duration policy in place.What's that?Depending on your pricing model, each uploaded object will be billed for a minimum amount of time.Pay-as-you-go customers have 90 days, reserved capacity customers (pre-paid) will have 30 days.If you delete an object before this minimum duration is over, the remaining time will still be charged.You can find more information on this website:https://wasabi-support.zendesk.com/hc/en-us/articles/360058734492-How-does-Wasabi-s-minimum-storage-duration-policy-work-ExampleYou upload 1TB of objects to Wasabi and have the 90 days minimum storage duration policy in place. After a week you decide to delete the whole 1TB.This means you pay for:7 days timed active storage 83 days timed deleted storageCan I recover deleted objects?You might think, that as you pay for deleted objects, you may be able to recover those. Unfortunately this is not the case, so you can’t use this as a recycle bin.
After changing the hostname of your VBR server, you might have issues accessing the integrated PostgreSQL database. For example when creating a new database or restoring configuration, you see the following error: “SSPI authentication failed for User XYZ”While the database connection itself uses “localhost”, the pg_ident.conf references the hostname itself for administrative access.So after changing the hostname, you need to replace the old hostname with the new one inside the pg_ident.conf file. Also you can add additional or different users to access the PostgreSQL instance. The default path to this files is “C:\Program Files\PostgreSQL\15\data\”.There’s a KB article from Veeam which mentions this error. This also includes how to check the PostegreSQL logs: https://www.veeam.com/kb1471
With the newest release V12, Veeam Backup & Replication is now capable of implementing immutability with HPE StoreOnce Catalyst stores. HPE calls this ‘Independent Software Vendor (ISV) Controlled Data Immutability (ISV-DI)’. With this post I want to quickly describe the requirements and configuration steps.RequirementsThere are some requirements, that need to be met in order to use immutability with StoreOnce systems.you need to have a Gen4 StoreOnce system with at least firmware 4.3.2 you need to configure Dual Authorization Maximum ISV Controlled Data Retention needs to be set to 365000 in the StoreOnce Catalyst Store you need to use forward incremental backup jobs and/or enable GFS for Backup Copy Jobs → For more information check the helpcenter Enable Dual AuthorizationWith Dual Authorization certain tasks on a StoreOnce require a second factor before they’re executed. This second factor isn’t a OTP token or something similar but rather a second user/password which you need to
As this was a hot topic on many plattforms, I thought it was worth sharing here.Today Dell and Veeam have released the new storage plugin for PowerStore OS 3.x.In general the plugin will only work with Veeam Backup & Replication build P20220302 or later. But what’s important to note, if you’re on P20220302 you need to install a private hotfix before installing the plugin itself; the details can be found in KB4243. The plugin is available for download in the Storage Plugin Section or directly here.If you want to join the discussion in the R&D forums or give feedback, here’s the corresponding topic.
The new year starts very well...with a new problem in regards to Office 365 backup.Since 01/01/2022 teams backup could fail with the following error:Failed to process team: XYZ Organization (ID: XYZ) which the team (ID: XYZ) belongs to not found.Veeam is aware of this problem and currently preparing a hotfix for this issue. If you want to be among the first to receive the fix, then open a support case. If you can wait, I will update this post as soon as the fix is publicly available.For further information see the following R&D forums post:https://forums.veeam.com/veeam-backup-for-office-365-f47/o365-backups-fail-since-1-1-2022-01-00-t78479.html#p439853 Update #1:The Hotfix is now available via SupportThe hotfix for the latest 5d CP2 (build 5.0.3.1051) has been provided to support.Update #2:The fix is now available through this KB; only for the latest VBO build at the moment (Thanks @JMeixner):https://www.veeam.com/kb4268Update #3:The hotfix has been replaced with a cumulative patc
Great news for anyone who still uses public folders in Exchange Online. Today Veeam has released patch P20230704 for Veeam Backup for Microsoft 365. And this patch brings back public folder backup! 🥳Added support for backup and restore of public folder and discovery search mailboxes for organizations using modern app-only authentication.Many organizations still use public folders and store valuable data in them. So now finally they will able to protect them again with modern authentication.Besides that there are some smaller enhancements, like the possibility to identify shared mailboxes. This was no longer possible with modern authentication only.And, just like in VBR, VB365 now also doubles the license grace limit if you enable ‘Update license automatically’. This will make licensing even more flexible as you have more time to order additional licenses.Subscription: 10 user accounts or 10% of the license count >> 20 user accounts or 20% of the license count Rental: 20 user acc
In my last post I covered the requirements and necessary steps to configure immutability for HPE StoreOnce with Veeam V12. In this post I want to focus on how immutability behaves in different scenarios.Enable immutability for existing Catalyst storesIf you have existing Catalyst stores, the good news is that it’s possible to enable immutability for them. Let’s take a look at the Veeam helpcenter:Note that if you enable immutability and Veeam Backup & Replication does not start a new backup chain and still continues the chain, the whole backup chain is marked as immutable.https://helpcenter.veeam.com/docs/backup/vsphere/storeonce_supported_features.html?ver=120#immutabilitySo, after you configure immutability, the current or new backup chain will be made immutable. All other existing backup chains won’t be touched and therefore stay mutable.This is also exactly what I could reproduce in my lab environment. After enabling immutability, the current chain is made immutable, while all
Just a quick info for the community, as the deadline is today at 08:00PM/20:00 PT.VMware is doing a challenge on their Hands-On Lab Plattform via VMware Odyssey. There are 4 different labs which prepare participants for the VCP DCV (Datacenter Virtualization Professional) exam. In each lab one can collect points and the top 10 receive a price.At the moment there are less then 10 people in the leaderboard, which means, if I get it right, chances are high to win a prize 😉VMware Hands-on Labs Request
My Sysadmin story is rather old, from my own beginnings in IT. It’s not directly Veeam related but the message is valuable for every backup administrator.The outageThere’s been a certain Linux VM which was running a very critical software for that company. Software updates were done by the vendor itself so there wasn’t much to take care of. Of course, besides including it in the backup process.At some point there was an outage in the datacenter which shutdown all virtual machines. After the power had been restored, all virtual machines were started again without any issues; at least almost without any issues.The Linux VM didn’t come up and stayed at the boot screen without any progress. After a bit of debugging without a solution, we decided to restore the VM as it was in the morning and no changes occurred since the nightly backup.Unfortunately, the restored VM had just the same problem and refused from booting up. And also the 2 day old restore point behaved the same.At that point we
Part 3 of the ‘HPE StoreOnce: Immutability with v12’ series is about a small, but very useful new feature. Before going into it, here are the links to Part 1 and Part 2, in case you’ve missed them.Thanks to Federico Venier from HPE I noticed that the latest StoreOnce release 4.3.6 did introduce two-factor authentication!StoreOnce 4.3.6 supports Two-factor authentication (2FA) which provides additional security for users logging into the StoreOnce system, granting clients access to the system only when they present at least two factors of authentication.https://support.hpe.com/hpesc/public/docDisplay?docId=sd00002919en_us&page=GUID-D9CE533F-9920-4656-A77D-4210E4EC56DE.htmlUntil now, an attacker would have been able to cause damage if he got access to the Admin and Security officer credentials. I described this process in Part 2 of the series: Veeam V12: Immutability with HPE StoreOnce - Part 2Now, with 2FA, the security of HPE StoreOnce backup repositories gets a huge boost.Enable T
Inspired by @Geoff Burke‘s post about immutability for Kasten with Wasabi, I decided to post the same for Veeam Backup & Replication. If you want to check out Geoff’s post, then follow this link:What/who is Wasabi?Probably Wasabi doesn’t need to be introduced. But if you can only think about the paste you get with Sushi, then you should check out their cloud object storage. Besides AWS and Microsoft, Wasabi is one of the Vendors, who often gets recommended for usage with Veeam Backup. The reason for that is their really competitive pricing, combined with great functionality/quality and low complexity.How to create an immutable bucketAfter registering an account with Wasabi, the creation of an immutable bucket isn’t complicated at all.Just make sure that you select the right region for your location, and also enable bucket versioning and object locking.Afterwards you need to create an access key. For testing purposes you can use the root account key, but for production you should cr
As you may know, with vSphere 7 Update 1 VMware introduced the vSphere Clustering Service.The plan is to get more independent of vCenter Server for certain cluster features like for example DRS.Where you certainly notice the difference, compared to older releases, are the vCLS agent VMs shown in vCenter.https://blogs.vmware.com/vsphere/2020/09/vsphere-7-update-1-vsphere-clustering-service-vcls.htmlThere's no need to backup them and if you have a current Veeam version (v10a Patch 20201202 or higher), they're are automatically excluded from your jobs.The agent VMs are manged by vCenter and normally you should not need to look after them.Normally…yesterday we've had the case were some of the vCLS VMs were shown as disconnected; like in this screenshot:Checking the datastore we have noticed that those agents VM had been deployed to the Veeam vPower NFS datastore.As this datastore is managed by the Veeam server and can connect/disconnect at any time, it's a very bad location for the vCLS VM
In nicht einmal einer Woche, am 23.05.2023, startet die VeeamON!Die VeeamON ist Veeam's jährliche Konferenz zum Thema Backup, Wiederherstellung und Datensicherheit.Während dem Event gibt es zu verschiedenen Themen Präsentationen und Vorträge, zum Teil mit Bezug auf das jeweilige Produkt von Veeam, zum Teil aber auch allgemein gehalten oder von Veeam Partnern durchgeführt.Anbei eine Auflistung der Themengebiete (Details siehe: https://www.veeam.com/veeamon/):Auch in diesem Jahr wird das Event hybrid veranstaltet. Zum einen kann man vor Ort in Miami teilnehmen, und zum anderen aber auch virtuell. Nachdem gerade für unsere Region Miami nicht unbedingt in der nächsten Nähe liegt, bietet sich die virtuelle Teilnahme natürlich an.Der Umfang ist hier gegenüber dem in-person Event etwas reduziert. Neben den live Keynotes kann man an pre-recorded sessions teilnehmen. Hier bestand in der Vergangenheit die Möglichkeit interaktiv Fragen zu stellen oder mit den Teilnehmern zu chatten.Eine Liste an
A few weeks ago I started to build my agenda for VeeamON 2023 in Miami. As there’s lots of interesting content covering various topics, it wasn’t easy at all to decide which sessions to attend. With over 130 sessions, I had conflicting schedules and so had to leave out some of those. This was especially hard as all sessions offer valuable knowledge, so I hope that recordings of most of them will be available after VeeamON.With this post I want to highlight ‘just some’ of the sessions I’ve picked for my agenda. There are much more great sessions, more that I could list here, so be sure you check the complete agenda.In-Person10 Years Working with Veeam: How it Helped Shape Our Careers All Demo Session for Veeam Hardened Repository Architecting Veeam Backup for Microsoft 365 at Scale A-Z Automation Journey with Veeam’s Automation Desk QuizMania – Veeam 100 against the World Ransomware Recovery Unplugged Tape Your Data Now! Veeam Backup & Replication v12 Hidden Features Veeam Repositor
Update 12.04.2023: Microsoft hat das Problem behoben und der Fehler sollte nicht mehr auftreten (https://forums.veeam.com/post483606.html#p483606)Hallo zusammen,eventuell kann es derzeit in unserer Region zu Fehlern bei der Sicherung von Sharepoint mit Veeam Backup for Microsoft 365 kommen:Could not establish trust relationship for the SSL/TLS secure channelIm Veeam Forum haben das Problem einige User gemeldet, und Veeam arbeitet derzeit aktiv daran. Vermutlich liegt die Ursache, wie so oft, auf Seiten von Microsoft:Quelle im R&D Forum: https://forums.veeam.com/post483281.html
Veeam has just informed it’s customers about an existing vulnerability in Veeam Backup & Replication. Unauthorized users may be able to request encrypted credentials from the VBR service, and therefore get access to the backup infrastructure. The KB articles haven’t received updates so far, but the vulnerability did get a CVSSv3 score of 7.5.EDIT: The KB article has just been published: KB4424It’s recommended to patch VBR as soon as possible!SolutionPatches have been released for VBR V11 and V12. Please keep in mind that older releases are also affected, but no longer get fixes. So you need to upgrade your installation to a supported release.Patch for Veeam V11a KB article for V11 Patch for Veeam V12 KB article for V12WorkaroundAs a temporary workaround you can block access to TCP port 9401 on your Veeam Backup & Replication server. This will affect the connection of mount servers to the VBR server, so only use this if you don’t have a distributed Veeam environment. And still a
Yesterday Microsoft published a patch for Outlook which fixes an Elevation of Privilege Vulnerability with a CVSSv3 Score of 9.8. Attackers can gain access to users credentials or NTLM hashes via a prepared email which contains a UNC path to a SMB share. The victim doesn’t even have to open/view the email; the login attempt happens as soon as Outlook receives the email.FixPatches have been published for Microsoft Outlook 2013 and above. As this vulnerability is very critical and does have a low complexity, you should patch as soon as possible. Also it seems that threat actors are using this vulnerability since over a year.Patches are linked in the corresponding article from Microsoft: https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-23397Please be aware that possibly all other Outlook versions which are already End of Life (2010,2007,...) are also affected, but don’t receive any patches anymore.WorkaroundThere are 2 workarounds available:Add users to “Protected Users Secu
Just a quick tip regarding the database used for Veeam Backup & Replication and Enterprise Manager. You need to keep both on the same database engine, so either on Microsoft SQL or PostgreSQL. If you mix both database types, you will be presented with a warning when you try to add a VBR server to EM:If you’re planing to migrate from Microsoft SQL to PostgreSQL, then keep in mind to also migrate the Enterprise Manager database.You can also run into this problem when setting up a new Enterprise Manager as it will by default install and use PostgreSQL. In that case either select a different database during the setup or alter the database connection with the “Configuration Database Connection Settings” utility. I haven’t read about this requirement so far, but the helpcenter mentions it:Note that configuration databases of the Enterprise Manager server and backup servers added to the Enterprise Manager infrastructure must use the same database engine.https://helpcenter.veeam.com/docs/b
I have just discovered an interesting session in the VeeamON agenda: “Object First Makes Veeam Backups Better”“Object First” is the new company from the Veeam founders Andrei Baronov and Ratmir Timashev!So far there’s not much information available, but according to an interview on vmblog.com (https://vmblog.com/archive/2022/05/06/veeamon-2022-q-a-object-first-will-showcase-how-it-provides-primary-target-object-storage-for-veeam.aspx) we’re going to see an object storage based primary backup repository specifically designed for Veeam Backup.I personally didn’t hear anything about this company before and would have probably missed the session, as it’s already tomorrow and not found in the main agenda.The session will start on Monday at 12:00PM PT or 9:00PM GMT+2. In case you’re also interested, here’s the link to the session: https://web.cvent.com/hub/events/659c08fe-e178-47e3-b354-f251151abefe/sessions/d9ce1628-a7d1-44fb-8684-cb1525ec061eAlso here’s the webpage of Object First: https:/
This week I was able to attend to a Veeam VMCA training (Veeam Backup & Replication V11: Architecture and Design) held by @poulpreben.VMCA IntroductionSome words on the VMCA: The VMCA is the Veeam Certificied Architect, the second and highest Veeam certification after the VMCE. While the VMCE is more focused on technical topics, the VMCA is all about designing the backup solution and learning the "Veeam Architecture Methodology".We already have great posts about the VMCA, both the course and the exam, here in the community, so I'll won't go too deep and link to the corresponding posts:VMCA2022 anouncment from @MicoolPaul:VMCA2022 review from @Chris.Childerhose:VMCA2022 EXAM review from @Michael Melter : VMCA CourseDirectly from the beginning of the course, we could notice how different this training is going to be compared to others.Preben explained to us, that there's no lab part and that we won't work with a Veeam console. Instead there were team exercises, where we work together
After the initial setup of Veeam Backup for Microsoft Azure I’ve had two different issues.The backup jobs finished with following warning:Crash-consistent snapshot of XYZ has been created. Cannot download VSS agent binaries from the storage account: The subscription is not registered to use namespace 'Microsoft.ServiceBus'. See https://aka.ms/rps-not-found for how to register subscriptions. Also FLR restores failed:Preparing worker for restore failed. Error: Cannot find any worker. Worker deployment failed, see logs for details Both of these messages had the same root cause and especially the first one already points to the right direction, the “Microsoft.ServiceBus” Resource Provider wasn’t registered in Microsoft Azure.The solution is really simple, just register this Resource Provider via the Azure Portal or use the Azure Powershell. Register-AzResourceProvider -ProviderNamespace Microsoft.ServiceBus I’m not sure if this Resource Provider should be registered by default, and if so,
Just a quick post on an interesting issue with the Guest Credentials test in Veeam Backup & Replication we’ve seen last week. Instantly after clicking “Test Now” in the Guest Processing section of a certain backup job, an error was displayed."Building list of machines to process Error: (Child group-vXYZ (Folder) for object XYZ was not found)"Normally you see such errors with vSphere when a certain object has been deleted or recreated. While Veeam displays names of objects, internally it works with MoRef IDs. So we’ve checked the job selection, custom credentials and the application aware processing settings. But all listed objects were still valid, especially no folders where used anywhere.We’ve contacted Veeam Support and had a remote session. The support engineer discovered some orphaned VM folders in the exclusions. After removing those, the error disappeared and the credential check worked again. From a logical point I thought those weren’t relevant for the check, so I didn’t c
This has been a hot topic since some time: With administrative access to the VBR server, credentials stored in the configuration database can be decrypted.That may sound critical at first, but one shouldn't forget the important detail 'administrative access'.Veeam stores its credentials in an encrypted state using the Microsoft Data Protection API and a Machine key specific to the VBR server. This is a security standard and a safe way of storing passwords, while enabling the Veeam services to use those for authentication.And just like the services, an administrator can access and decrypt the credentials.This is no secret and Veeam has even published a KB article on how to recover credentials from the configuration database: https://www.veeam.com/kb4349And again, please remember that this requires administrative access to the backup infrastructure.In general an administrator or 'root' will always have the highest available permissions and therefore can do anything on a system.This is th
I'm sure everyone has heard a story about ransomware attacking virtual machines from within ESXi. While I thought that those were rare, special cases, I was quite surprised when reading VMware's security blog.One post mentions 15 different ransomware families, which 'support' VMware ESXi.AvosLocker Babuk Black Basta BlackCat Cheerscrypt Conti DarkSide/BlackMatter Defray777/RansomEXX GwisinLocker HelloKitty Hive Lockbit Luna RedAlert REvilSome of them were designed for ESXi, others are more generic and don't even shutdown the VMs before starting the encryption.At the end they cause catastrophic damage and are hard to detect or stop.Protect against ESXi ransomwareIt seems that ransomware on ESXi hosts often will need to be executed manually.So the primary goal will be to secure access to the ESXi hosts:separate management and production network lockdown ESXi shell/SSH regularly patch ESXi hosts (some attacks were possible via unpatched vulnerabilites) keep ESXi credentials secure monitor
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
