July 27, 2023

DRaaS, Veeam

Veeam replication disks stuck on Linux proxies

We have recently encountered issues with Veeam replication jobs where they would fail with the following errors: Error: Failed to open VDDK disk [[DATASTORENAME_0] PATH_0] ( is read-only mode – [true] ) –tr:{ –tr:VDDK error: 1 (Unknown error). Value: 0x0000000000000001 –tr:Failed to open virtual disk [DATASTORENAME_0] PATH_0 (flags: 4) –tr:Failed to execute VDDK command ‘Open’ In this scenario, the replications were using Linux Veeam virtual hot-add appliance backup proxies. When we checked the Veeam backup proxies there were hot-add snapshot disks still attached from the replicas after the jobs had completed. This was locking those disks as read-only and preventing Veeam from being able to take additional snapshots and perform the replications. Here are two possible solutions: SSHD.conf file configurations These issues can be caused by timeouts and disconnects that can be resolved by tuning the SSH configuration on the proxy. To make these adjustments modify the /etc/ssh/sshd_config file and add the following lines: You may need to restart the sshd service or restart the machine for these changes to take effect. Disabling multipathing In the Session.log log file there may be errors similar to the following: Error Failed to get guest OS path for newly attached disk [Datastore] VM_replicaNew/VM_1-000003.vmdk: there are 2 new disks with uuid 00000000-0000-0000-0000-000000000000 (System.Exception) These point to issues that can happen on linux systems where multipath is enabled and hot-add is used. These commands will disable multipathing, they will also reboot the system so do not attempt this while jobs are running: Veeam has more information on this issue here: KB4460: Failed to get guest OS path for newly attached disk there are 2 new disks with uuid (veeam.com) In our case we made both changes and our issues so far have not returned.

Office 365 Backup, Veeam

Connection Failed During Veeam Backup for Microsoft 365 Service Provider Restores

When performing Veeam Explorer restores connecting to a service provider, sometimes these restores can fail with a Connection Failed message. Looking in the Veeam explorer logs there will typically be references to a version mismatch. Ex: 4/26/2023 1:10:47 PM 20 (11320) Connection failed4/26/2023 1:10:47 PM 20 (11320) Error: Incompatible Veeam Backup for Office 365 server version (received: 11.1.0.600, expected: 10.0.3). This is due to a different version of the Veeam explorer being installed on the tenant’s Veeam Backup and Replication server than the service provider’s Veeam for Microsoft 365 server. The solution to this is to download the same version of the Explorers currently installed on the service provider’s Veeam for Microsoft 365 server. This can either be downloaded from Veeam in the form of the Veeam for Microsoft 365 installer .iso or by contacting the service provider and asking for the individual Explorer installer .msi files. Once downloaded and installed. Try the service provider explorer again and the “Connection Failed” error should be gone.

Office 365 Backup, Veeam

Migrating Veeam for Microsoft 365 Repository to a New Object Storage Bucket

Veeam does not support moving data between object storage buckets. Veeam has guides for: For the scenario where data needs to be moved from one object storage provider to another Veeam support will suggest performing a new full backup to the new repository. However, this will cause a loss of retention if the old bucket is not retained. Note: While we have tested the following method up to VB365 v7 and confirmed it has worked for our data, please be aware that this is an unsupported method for migrating Veeam for Microsoft 365 data. The following method will allow data to be transferred between object storage buckets. We will be essentially following the steps outlined in the first KB article noted above, KB2649, but instead of migrating to a new server the server will not change. You may get an error when trying to add the repository that the repository already belongs to another backup server. In that case, navigate to the RepositoryLock folder in the new bucket and delete the lock file located there. Finally, you can edit the backup job and point it to the new repository. The initial backup pointing to the new repository will take a while to complete depending on the size of your data. This first backup will be marked as a full, because it will be synchronizing the new repository with the production data, however it will still only transfer the changed data.

Office 365 Backup, Veeam

Permissions Error Performing Exchange Restore to M365

During an M365 Exchange mailbox restore using Veeam, the following errors can occur: Failed to open mailbox: user.name@domain.com Failed to access mailbox. Mailbox does not exist. The specified object was not found in the store. Default folder Root not found. You can find these additional events in the Exchange explorer logs: Validating if any of the required roles (Global Administrator, Exchange Administrator) is assigned to the current user: admin@domain.com… Error: The account does not have permission to impersonate the requested user. This is caused by missing permissions on the account performing the restore, specifically the ApplicationImpersonation role. To add this permission to the user account we have to connect to Exchange Online using the PowerShell module. The following commands will connect and add the proper role: Make sure to replace admin@domain.com with a global administrator account to run these commands. Then retry the restore and it should go through as expected.

Office 365 Backup, Veeam

Veeam for Microsoft 365 v7 – Adding Buckets with Versioning Enabled

Veeam for Microsoft 365 v7, by default, prevents adding buckets with versioning enabled. More information on that can be found in the Veeam KB article here. When trying to add these buckets this message appears: The selected bucket supports blob versioning.Select a bucket with disabled blob versioning. Some S3-compatible storage providers do not offer the ability to disable versioning. To proceed with adding buckets from these storage providers a configuration change is needed on the Veeam for Microsoft 365 server. To make that change: Note that after adding a bucket with versioning enabled all backups targeting this repository will finish with the following warning: Object versioning is enabled on the bucket/container. Some backup data cannot be removed by retention and will remain in the repository There is no way to disable these warnings.

Scroll to Top