Time again for a new episode in the “in 5 minutes” series. Today I’ll be discussing Veeam Cloud Connect.
Swedish only!
In 5 minutes: Veeam ONE
Next up: Veeam ONE. A solution for many things, such as:
Monitoring
Reporting
Capacity planning
Let’s have a short discussion.
Swedish only.
In 5 minutes: Veeam Backup & Replication
The series continues with Veeam’s flagship product for backing up and restoring virtual machines, files and objects from applications. And replicating VMs.
Swedish only.
How and why series – Veeam Backup & Replication SureBackup
After the first episode of my “How and why”-series in which I talked about VMware VSAN I thought it’d be fun to show you why I love Veeam so much and in particular the function SureBackup. SureBackup is all about verifying your backup data (your VMs) in a super smart way. The bottom line is, if you need to restore anything you KNOW that the restore will work.
Again, the video is in swedish only.
(Re)claim your space!
In one of my previous posts, Is Bitlooker from Veeam a game-changer?, I wrote about the benefits of using Bitlooker for backup jobs when using Veeam Backup & Replication v9.x however Bitlooker is a feature that is not only available for backup jobs – you can use it for replication jobs as well.
So I thought it’d be fun to see what difference, if any, it makes. The goal of my tests is to figure out the most effective way of copying/replicating a VM from one host to another.
The set up for the test:
A virtual machine is installed with Windows Server 2016 standard edition. 100 GB disk assigned to the VM, thin provisioned. The disk is then filled with files (a bunch of iso-files of different sizes). That’s the baseline, then roughly 85 GB will be deleted (all the added iso-files) – then trashcan will be emptied. So we’ll have some blocks containing stale/old data, the blocks are marked as available to be reused from the operating system point of view but they haven’t been zeroed out so from any hypervisor (outside the VM) it just looks as any other block containing data.
Operating system installed (Windows Server 2016) and updated. The VM now consumes 13,5 GB worth of storage.
Then a bunch of files were added (almost) filling the entire disk.
From the vSphere side of it:
At this point the just added files were removed and trashcan emptied.
And from vSphere:
Now the command ‘ls’ will not show the actual size, so ‘du’ can be used instead to see the actual size of the vmdk file:
I’m going to test 4 different scenarios:
- Migrate the VM from one host to another offline (VM will be shutdown).
- Replicating the virtual machine with VMware vSphere Replication 6.5.
- Replicating the virtual machine with Veeam Backup & Replication without Bitlooker enabled.
- Replicating the virtual machine with Veeam Bitlooker enabled.
The thesis, or point to prove, is that test 1-3 will have no or little impact on the size of the vmdk file however – magic will happen on test 4. So lets perform the tests and find out for real!
Test 1:
VM moved to another host while offline and now let’s explore what can be seen using different methods.
Inside the VM:
From the host:
So no change in vmdk file size as expected.
Test 2:
The virtual machine will be replicated to another host using VMware vSphere Replication 6.5.
VMware vSphere has been configured using the following settings:
Not alot of settings, in fact, the above settings will have no impact on the vmdk size. They will only have control how the snapshot on the VM will be generated (crash consistent vs consistent backup) and the impact the replication job will have on the network.
Inside the VM:
From the host:
Since ‘ls’ doesn’t show the actual size on a thin disk, disk usage ‘du’ is used instead:
So no change in vmdk file size as expected.
Test 3:
The virtual machine is replicated to another host using Veeam Backup & Replication v9.5.
Replication from a Veeam perspective has been set up, to make a fair comparison to the VMware replication test (test 2), the Veeam job will not use exclude swap file blocks:
Processed and read data in the picture below tells us that Veeam doesn’t know the difference between blocks in use and blocks marked as deleted (the same applies for almost all backup vendors):
Inside the VM:
From the host:
And using ‘du’:
So no change in vmdk file size as expected.
Test 4:
Know time for the fun stuff. The virtual machine will be replicated to another host using Veeam Backup & Replication v9.5. We will use both space saving techniques we can enable on the job (with application-aware processing we can also exclude specific files, folders, file extensions but we’re not using that feature in this test)
Now, this is the magic we were looking for! The proxy server has processed all of the data but it has only read data that contain used blocks!
From the VM:
From the host:
Now the vSphere web client combine the .vmdk and -flat.vmdk file into one (like it’s done forever):
And the disk usage utility shows:
Yikes! That’s cool stuff!
Conclusion:
Bitlooker is a feature you should have enabled on any relevant job. It certainly can be used to reclaim that precious storage space you so desperately need. Heck, why no use it as part of your normal failover testing, cause you’ll already doing that right? Once a month (or how often you feel appropriate) do a planned failover using Veeam Backup & Replication, verify that you DR plan works and as an added bonus you reclaim disk space in the process!
And yet another benefit is the spent replicating the virtual machine, without Bitlooker it took 30 minutes to replicate the VM from one host to another but it was just shy of 7 minutes with Bitlooker enabled.
So seriously, why are you not using this magic thing? There’s only one drawback, Bitlooker supports only NTFS file system (=Windows VMs).
My top 5 features of Veeam Availability Suite 9.5
5. Direct restore to Azure
Restore backup from any Veeam backup product to Microsoft Azure: From Backup & Replication, Backup Free edition or Endpoint backup free.
P2V or V2V? Not a problem! This means VMs from any hypervisor or physical machines, even VMs running in any cloud where you can install Endpoint Backup Free or Windows/Linux agents from Veeam are eligible to restore. And it doesn’t matter if you’re running Windows or Linux.
4. Enhanced VMware vCloud Director support for service providers
Let your cloud tenants use native vCloud authentication to access the new self-service backup and restore portal in Enterprise manager. Let them use predefined backup jobs to protect their vApps and restore VMs, vApps or guest files.
3. Proxy affinity
With proxy affinity you can control what proxy servers are allowed to use specific repositories. It’s an easier way with less administration – no need to select individual proxy servers in each and every backup job (if needed) to keep backup traffic local to the site.
2. Cloud Connect for service providers
There are a few functions added to Cloud Connect providers that are not new per se but introduced to Cloud Connect in version 9.5. Such as:
Per-VM backup file chains support
Scale-out Backup repository support
Advanced ReFS integration support
1. Advanced ReFS Integration
Hands down my favorite new function is the ReFS integration. This is a fantastic technology from Microsoft and the integration Veeam made with it in Backup & Replication 9.5 is nothing short of a amazing. Fast clone and spaceless full backup, sounds good right? Who needs dedupe storage anyway? And no need to worry (?) about data integrity with the use of integrity streams that is storage-level corruption guard on steriods.
Windows 2016 ReFS as a Veeam backup repository?
Do we really need dedupe storage for our long term backups? Well, dedupe storage certainly has it’s merits in the toolbox when you design your backup environment. But we do need to be aware of some of the drawbacks with dedupe, for instance performance for random I/O and rehydration of data if doing synthetic full backups. However, there’s been a great disturbance in the Force as of recently.
Microsoft made a huge release earlier this year with the “2016”-suite, I thought it’d be a good idea to go through how to set up a server to be used as a repository for Veeam Backup & replication 9.5 and show some of the results. More specifically, let’s use the amazing Resilient File System (ReFS) v3 found in Windows Server 2016.
Windows Server 2016 can be installed in two modes, with or without a GUI. Now, if no GUI is installed you can use powershell to do the configuration but for simplicity we’re going to use the GUI which is called “Desktop experience” in the installer.
So boot your server from the dvd/iso and select Windows Server 2016 (Desktop Experience) of your flavor.
In the next step, select “Custom: Install windows only (advanced)”
Select the drive to install Windows on:
Now the installation will start copying files to the drive and commence the installation of Windows.
Click on Tools and select Computer Management in the server manager window:
Open up the Disk management tool, this is where you find the attached drives on the server. In my case I have one drive where I installed windows and two additional drives. I’ll make some benchmarks later so I’m going to format one drive with ReFS and the other one with NTFS for comparison, but first let’s get them online:
Then initialize the disks:
Next up, we create a new simple volume on one of the disks:
Select file system and give the volume a name:
Next disk we format with NTFS instead:
The result looks like this:
So let’s test it out and see if there’s any noticeable difference when using these as target for our backups.
I’ve created a folder on each of the disks and made two Veeam Backup Repositories, I’ve created two jobs backing up the same VM running Windows Server 2016(with different targets) and set up synthetic full backup, this is where Windows ReFS should really shine if you believe the hype.
When the backup has run a few iterations it was time for the Synthetic Full backup using the NTFS backup repository as target:
So with NTFS, a full backup, a few incremental backups and at the end a Synthetic full backup used roughly 12 GB of disk space. Good deduplication and compression and not a lot of changes in the VM of course. But what about the ReFS repository?
Same backup intervall with a Synthetic full backup at the end generated roughly 7 GB go disk usage, pretty significant reduction in usage if you ask me. Now disk usage is when of the key features on why to use ReFS with it’s “spaceless full backup technology” but there’s another benefit the has the same groundbreaking impact: Time.
Time it takes to make the full backup. Let’s take a look at the time it took to create the synthetic full backup on the NTFS repo.
4 minutes and 15 seconds is not the bad for my VM, but imagine it being dozens of VMs…How does the ReFS repo compare then?
11 seconds! Must be a typo? NO, it’s really not. Quite impressive. Now you might rethink the dedupe storage approach, you have all the space savings (almost) of the dedupe storage but performance of a traditional disk with no need to rehydrate data.
If you want to make it highly resilient you can leverage another Microsoft technology called Storage Spaces Direct where you’d get fault tolerance and “auto-healing” as well.
A fantastic job from Microsoft with ReFS/Storage Spaces Direct and Veeam with the integration of the technology into Backup & Replication!
Veeam, PowerShell and SAN snapshot
So a week or so ago I wrote a post about using PowerShell to add snapshots to a SAN found in Veeam Backup & Replication. It was a quick test to see if it worked, now I’ve slightly improved the script.
The Get-HP4* cmdlets is specifically for HPE StoreVirtual VSA/P4000/LeftHand line. If you have another supported SAN Storage system use the correct cmdlets:
NetAPP Storage Systems
HPE 3Par StoreServ Storage Systems
HPE StoreVirtual Storage Systems
EMC VNX Storage Systems
You can accomplish the same thing using the management tool for the SAN, taking recurring snapshots. But in the case of HPE StoreVirtual it’s a licensed feature and it can only occur every 30 minutes so if you need it more often or you’re lacking the license you can use the PowerShell script instead.
Add-PSSnapin VeeamPSSnapIn -ErrorAction SilentlyContinue
$source_storage = ‘Veeam-VSA-MGMTG’
$source_cluster = ‘veeam-vsa-cluster’
$source_vol_name = ‘datastore1’
$snapshot_name = ‘Oh_snap_’+(Get-Date -Format MMddhhmm)#Create a new snapshot
try {
‘Trying:’
$getvolume = (Get-HP4Storage -Name $source_storage | Get-HP4cluster -Name $source_cluster | Get-HP4Volume -name $source_vol_name)
‘getvolume-Name: ‘ + $getvolume.Name
‘getvolume-InternalId: ‘ + $getvolume.InternalId
‘getvolume-IsThin: ‘ + $getvolume.IsThinProvision
‘getvolume-Size: ‘ + $getvolume.Size$getvolume | Add-HP4Snapshot -name $snapshot_name
} catch {
‘Failed to find storage, cluster or datastore’
‘Unable to create snapshot’
break
}#Remove the oldest snapshot if more than 4 are available
$getsnapshot = ($getvolume | Get-HP4Snapshot | Where-Object {$_.Name -like ‘Oh_snap_*’})
$snapshot_count = @($getsnapshot).Countif ($snapshot_count -ge 4) {
$getsnapshot | Sort-Object creationtimeutc | Select-Object -First 1 | Remove-HP4Snapshot
}
Populate your Veeam lab with PowerShell
If you, like me, have the need to constantly rebuild a lab environment where the servers are installed already but it lacks any configuration you probably realized that PowerShell is you friend. I have a lab environment that I tear down and build up again really often using templates in my VMware environment. In this environment I have all the infrastructure components installed but not configured in Veeam Backup & Replication so whenever I want to show-and-tell I first need to configure stuff. It might take a while to do, so why not automate with PowerShell?
The script below adds a few managed servers, adds backup proxies, creates a Scale-Out Backup Repository with 2 extents, adds 2 WAN accelerators. On top of that it adds a Tape proxy, connects to a HP VSA and takes a snapshot.
Add-PSSnapin VeeamPSSnapIn -ErrorAction SilentlyContinue
$Infra_Administrator = "Domain1\Administrator"
$Infra_Password = 'Password1'
$Lab_Administrator = "Domain2\Administrator"
$Lab_Password = 'Password2'
$ESXi_root = "root"
$ESXi_Password = 'Password3'
$Oracle_User = "oracle"
$Oracle_Password = 'Password4'
$HPE_User = 'HpeUser'
$HPE_Password = 'Password5'
$VBRserver = Get-VBRServer -Name "VEEAM-VBR.domain1.local"
Add-VBRCredentials -Type Windows -User $Infra_Administrator -Password $Infra_Password -Description $Infra_Administrator
Add-VBRCredentials -Type Windows -User $Lab_Administrator -Password $Lab_Password -Description $Lab_Administrator
Add-VBRCredentials -Type Linux -User $Oracle_User -Password $Oracle_Password -SshPort 23 -ElevateToRoot -AddToSudoers -RootPassword $Oracle_Password -Description "oracle"
#Add servers as ”managed servers”
Add-VBRESXi –Name "VEEAM-ESX" -user root -password 'Password3'
Add-VBRWinServer -Name "VEEAM-HYPERV" -credentials $Infra_Administrator
Add-VBRWinServer -Name "VEEAM-Remote" -credentials $Infra_Administrator
Add-VBRHvHost -Name "VEEAM-HYPERV" -credentials $Infra_Administrator
#Remove/Add proxy with 1 concurrent task limit
Get-VBRViProxy -Name "VMware Backup Proxy" | Remove-VBRViProxy -Confirm
Add-VBRViProxy -Server $VBRserver -Description "VMware Backup Proxy" -MaxTasks 1
#Add Backup Repositories and Scale-Out Backup Repository
Add-VBRBackupRepository -Server $VBRserver -Name "Remote Repository" -Folder "X:\Backups" -Type WinLocal -MaxConcurrentJobs 4 -Credentials $Infra_Administrator
Add-VBRBackupRepository -Name "Local Backup Repository" -Server $VBRserver -Folder "E:\Backups" -Type WinLocal -MountServer $VBRserver -VPowerNFSFolder "C:\ProgramData\Veeam\Backup\NfsDatastore" -MaxConcurrentJobs 4 -Credentials $Infra_Administrator
Set-VBRConfigurationBackupJob -Repository "Remote Repository"
Add-VBRScaleOutBackupRepository -Name "Main Backup Repository" –PolicyType DataLocality –Extent “Default Backup Repository”, “Local Backup Repository”
#Add WAN accelerators
Add-VBRWANAccelerator -Server "VEEAM-Remote" -Description "Remote WAN Accelerator" -CachePath "X:\VeeamWAN" -CacheSize 10 -CacheSizeUnit GB
Get-VBRLocalhost | Add-VBRWANAccelerator -Description "Local WAN Accelerator" -CachePath "X:\VeeamWAN" -CacheSize 10 -CacheSizeUnit GB
#Add a Virtual Lab and Application group
$VLABhost = Get-VBRServer -Type ESXi
$VLABdatastore = Find-VBRViDatastore -Name "datastore1" -Server $VLABhost
Add-VSBVirtualLab -Name "VEEAM-ESX VLAB1" -Server $VLABhost -Datastore $VLABdatastore
Find-VBRViEntity -Name "VEEAM-DC01", "VEEAM-EX01" | Add-VSBViApplicationGroup -Name "Exchange"
#Add SAN and Tape and make a snapshot on the SAN
Add-HP4Storage -DnsOrIpAddress "10.20.30.40" -User $HPE_User -Password $HPE_Password -Description "HPE Storage"
Get-VBRServer -Name "VEEAM-Remote" | Add-VBRTapeServer
Get-HP4Volume -name "datastore1" | Add-HP4Snapshot -name "datastore1_SS_1"
Veeam and PowerShell: A perfect match!
I read a really good and useful blog post a while ago from Preben Berg from Veeam describing how to use PowerShell to restore a database from backup to a dev environment. This made me think on another scenario that would be fun to script that might come in handy someday.
What if you have a SAN that you would want do snapshots on once an hour and save some of those historical snapshots rotating the oldest one.
Now for the disclaimer part, this is merely meant to showcase how you might accomplish this. There are no safety features built in. Do not use it in production and use it at your own risk. However if you’d like to do some further testing of your own, you can download a virtual SAN from HP – free of charge up to 1 TB. Nice!
First things first. We need to create a PowerShell script.
Let’s define which volume to use:
$source_vol_name = "datastore1"
Then we add a naming convention to use for the snapshots with a timestamp at the end:
$snapshot_name = "Oh_snap_"+(Get-Date -Format MMddhhmm)
Let’s count how many snapshots exist on the volume:
$snapshot_count = @(Get-HP4Volume -name $source_vol_name | Get-HP4Snapshot).Count
Now let’s create a snapshot:
$snapshot_create_session = Get-HP4Volume -name $source_vol_name | Add-HP4Snapshot -name $snapshot_name -description "Automated snapshot"
I would like to save the 4 latest snapshots on the volume and delete the oldest:
if ($snapshot_count -ge 4 {
$snapshot_remove_session = Get-HP4Volume -name $source_vol_name | Get-HP4Snapshot | Sort-Object creationtimeutc | Select-Object -First 1 | Remove-HP4Snapshot
}
If I would leave out “Get-HP4Volume -name $source_vol_name” in the above statement then I would delete to oldest snapshot available on any of the volumes (not just datastore1) which is not our intent.
Let’s put it all together:
Add-PSSnapin VeeamPSSnapIn -ErrorAction SilentlyContinue
$source_vol_name = “datastore1”
$snapshot_name = “Oh_snap_”+(Get-Date -Format MMddhhmm)
$snapshot_count = @(Get-HP4Volume -name $source_vol_name | Get-HP4Snapshot).Count$snapshot_create_session = Get-HP4Volume -name $source_vol_name | Add-HP4Snapshot -name $snapshot_name -description “Automated snapshot”
if ($snapshot_count -ge 4 {
$snapshot_remove_session = Get-HP4Volume -name $source_vol_name | Get-HP4Snapshot | Sort-Object creationtimeutc | Select-Object -First 1 | Remove-HP4Snapshot
}
That’s all, just a few lines of code and we can accomplish cool things.
Then all we have to do is add the script as a scheduled task on the VBR server and run it once an hour or what ever intervall we’d like.
But let’s see how the infrastructure looks prior to running the script:
Open Task Scheduler and create a new task:
Let’s configure it to run once an hour starting at 10 PM:
We add and action, starting the PowerShell script:
Looks ok once it’s added:
First time the script ran a snapshot was created as expected:
(I changed the script to run every 5 minutes to speed up the process), now we have 4 snapshots:
When the fifth snapshot has been taken the oldest snapshot is deleted (the snapshot created at 10:07 PM in the previous picture):
You must be logged in to post a comment.