VMworld day 1

So this is it, now the future begins. Before Pat Gelsinger took the stage a painter created a piece of work using VR, interesting to see:

At stage Pat Gelsinger today annonunced new cloud services but amongst others but the most interesting announcement made was AppDefense a technology that will help us stop chasing bad and instead ensuring good as Pat Gelsinger put it.

AppDefense will basically learn how the applications work in your virtual machines and create a manifest that will be used to protect the applications from doing things it’s not supposed to do.

And  today VMware Cloud on AWS as an offering is available. Will be really interesting to see the adoption rate.

VMware Cloud Foundation is update to a new version.

All announcemnts can be found here.

VMworld Partnerday

A short summary of the day: Sunday, the day before the big event, is dedicated to partners. I spent the day in really interesting sessions ranging from vSAN Networking design to NSX selling. A all sessions were informative and exciting,  the one that had the biggest impact on me was a session called NSX assessment led selling. 

Now I’m really looking forward to the main event starting tomorrow.

Pre-VMworld #PAR4398BCU

I arrived yesterday and VMworld started today for me as I’m working for a VMware distributor I attended a bootcamp focusing on vSAN. The core of the session is to enable partners to achieve the VTSP certification (a pre sales certification) but a was more of an open discussion with some of the greatest minds on vSAN. james LeFort, Jeff Hunter, Pete Koehler, John Nicholson amongst others. 

It was a really great session, a lot of topics were discussed – high and low. A huge boost of confidence in vSAN.

Summing it all upp, it was a great start to VMworld and I’m now as excited as ever for the main event, but tomorrow will be some additional partner training before VMworld starts on monday – should also be fun!

VMware vSAN 2017 Specialist exam experience

A few days ago VMware Education released a brand new series of exams with the badge name ”Specialist”. Currently there are two exams available, vSAN 2017 (called 2VB-601 in official certification lingo) and vRealize Operations 2017 (2VB-602). I do like a good challenge so I scheduled the exam basically as soon as I heard about the exam and started preparing for it immediately – I basically only had a days worth of preparation due to the fact I wanted it done before heading out to Las Vegas and VMworld 2017 (but I’ve been using, talking and writing about vSAN for quite a while) . You can read more on what’s expected of you before scheduling the exam here.  But in short, if you hold a valid VCP6 for any solution track you can just schedule the exam.

So how did I prepare for the exam?  Well, as I already mentioned I’ve been using vSAN for a long time but more recently I attended the instructor lead vSAN class in April without knowing that there would be a certification introduced just a few months later. The class I attended was called VMware Virtual SAN: Deploy and Manage [V6.2] and as you probably guessed already based on the name was focused on vSAN version 6.2. So now I have access to the official ”Student Lecture Manual” that goes along with the class, obviously that’s a good starting point as far as studying goes.

To get up to speed on the latest version of vSAN I really recommend heading over to #vBrownBag and catch the episode called VMware vSAN 6.6 Deep Dive with Jase McCarty.

And for some hands on experience I’d recommend the VMware Hands On Labs (free of charge – thats just fantastic, thanks VMware!) where you can  find lots of labs and a few vSAN related labs as well. Or if you have a home lab I would really recommend William Lams automated vSphere Lab Deployment – I use it all the time and it’s nothing short of amazing!

Another great resource is of course the VMware SABU (Storage and Availability Business Unit) team’s resource page for technical documentation called storage hub. Two documents I found useful were called ”VMware vSAN 6.5 Technical Overview” and also the latest version ”VMware vSAN 6.6 Technical Overview”.

Don’t forget to read through the exam prep guide either. It’s the exam blue print stating all the topics that will be covered on the exam. Quite important.

The exam itself consist of 60 multiple choice type of questions. You’re allotted 105 minutes to complete the exam with an automatic 30 minute language extension, and that should be more than enough time to complete the exam. I tend to do exams rather fast, but I spent roughly 45-50 minutes on the test including reviews.

And as always, when you click to end the test it takes a few seconds to understand and have it sink in if you passed the test or not. And I did, so now I’m officially not only a vExpert vSAN but also a VMware vSAN 2017 Specialist. Woho me!

How and why series – Veeam Backup & Replication SureBackup

After the first episode of my ”How and why”-series in which I talked about VMware VSAN I thought it’d be fun to show you why I love Veeam so much and in particular the function SureBackup. SureBackup is all about verifying your backup data (your VMs) in a super smart way. The bottom line is, if you need to restore anything you KNOW that the restore will work.

Again, the video is in swedish only.

VMware vExpert VSAN 2017

Through out the years I’ve always believed that blogging is a great way of sharing information. I’ve followed lots of blogs for my own learning process (too many to mention but a huge thanks to all my fellow bloggers) and enjoy reading high level posts as well as deep dives. That’s the beauty when bloggning, the posts can be about anything, high or low, 10 000 feet or up close an personal. When I started bloggning it was with that same feeling of letting other people know about this great new thing I discovered or how easy it was to set up a specific function, basically sharing is caring.

So sharing comes first but when it leads to some sort of recognition it really makes blogging even more exciting. I’ve had the great fortune of being appointed as a VMware vExpert 8 times, I have to pinch myself to really understand and let that sink in. Wow, that’s amazing, all I ever wanted to do was to share some bits and pieces that was running through my mind. So I’m really blessed with the recognition I’ve already received but today is yet another good day.

This morning when I woke up I saw an email from Corey Romero, responsible for the vExpert program over at VMware. Corey usually sends a few emails a week so nothing out of the ordinary at that point, but the subject got me really excited. It read ”Welcome to the 2017 vExpert VSAN Program”. Jumping up and down with joy. It, at first, didn’t really sink in so I had to visit the announcement page to verify that my name was there…And yes it was. So I’m humbled, proud as can be and now I’m even more excited to leave for VMworld in just over a week.


A great big ”Thanks” to all the bloggers I ever followed, to VMware for the great products, again to VMware for having such a great recognition program in vExpert and to anyone that reads and promote blogs.

A new series: How and why – Starting with VMware vSAN

I’ve been thinking about creating and sharing short (no more than say 30 minutes) videos of how certain functions or features works and how to set it up. I’m going to create a series of video content that might be useful over the next few week. I’ve decided to name the series ”how and why” since this is the angle I’m taking – how do you setup the feature and whats the benefit of actually using the feature.

So the inaugural video will be based on VMware vSAN. Feel free to drop me tweet on future videos you’d like to see.

The video is in swedish only at the moment.

(Re)claim your space!

In one of my previous posts, Is Bitlooker from Veeam a game-changer?, I wrote about the benefits of using Bitlooker for backup jobs when using Veeam Backup & Replication v9.x however Bitlooker is a feature that is not only available for backup jobs – you can use it for replication jobs as well.

So I thought it’d be fun to see what difference, if any, it makes. The goal of my tests is to figure out the most effective way of copying/replicating a VM from one host to another.

The set up for the test:

A virtual machine is installed with Windows Server 2016 standard edition. 100 GB disk assigned to the VM, thin provisioned. The disk is then filled with files (a bunch of iso-files of different sizes). That’s the baseline, then roughly 85 GB will be deleted (all the added iso-files) – then trashcan will be emptied. So we’ll have some blocks containing stale/old data, the blocks are marked as available to be reused from the operating system point of view but they haven’t been zeroed out so from any hypervisor (outside the VM) it just looks as any other block containing data.

Operating system installed (Windows Server 2016) and updated. The VM now consumes 13,5 GB worth of storage.

Then a bunch of files were added (almost) filling the entire disk.

From the vSphere side of it:

At this point the just added files were removed and trashcan emptied.

And from vSphere:

Now the command ‘ls’ will not show the actual size, so ‘du’ can be used instead to see the actual size of the vmdk file:

I’m going to test 4 different scenarios:

  1.  Migrate the VM from one host to another offline (VM will be shutdown).
  2. Replicating the virtual machine with VMware vSphere Replication 6.5.
  3. Replicating the virtual machine with Veeam Backup & Replication without Bitlooker enabled.
  4. Replicating the virtual machine with Veeam Bitlooker enabled.

The thesis, or point to prove, is that test 1-3 will have no or little impact on the size of the vmdk file however – magic will happen on test 4. So lets perform the tests and find out for real!

Test 1:

VM moved to another host while offline and now let’s explore what can be seen using different methods.

Inside the VM:

From the host:

So no change in vmdk file size as expected.

Test 2:

The virtual machine will be replicated to another host using VMware vSphere Replication 6.5.

VMware vSphere has been configured using the following settings:

Not alot of settings, in fact, the above settings will have no impact on the vmdk size. They will only have control how the snapshot on the VM will be generated (crash consistent vs consistent backup) and the impact the replication job will have on the network.

Inside the VM:

From the host:

Since ‘ls’ doesn’t show the actual size on a thin disk, disk usage ‘du’ is used instead:

So no change in vmdk file size as expected.

Test 3:

The virtual machine is replicated to another host using Veeam Backup & Replication v9.5.

Replication from a Veeam perspective has been set up, to make a fair comparison to the VMware replication test (test 2), the Veeam job will not use exclude swap file blocks:

Processed and read data in the picture below tells us that Veeam doesn’t know the difference between blocks in use and blocks marked as deleted (the same applies for almost all backup vendors):

Inside the VM:

From the host:

And using ‘du’:

So no change in vmdk file size as expected.


Test 4:

Know time for the fun stuff. The virtual machine will be replicated to another host using Veeam Backup & Replication v9.5. We will use both space saving techniques we can enable on the job (with application-aware processing we can also exclude specific files, folders, file extensions but we’re not using that feature in this test)

Now, this is the magic we were looking for! The  proxy server has processed all of the data but it has only read data that contain used blocks!

From the VM:

From the host:

Now the vSphere web client combine the .vmdk and -flat.vmdk file into one (like it’s done forever):

And the disk usage utility shows:

Yikes! That’s cool stuff!


Bitlooker is a feature you should have enabled on any relevant job. It certainly can be used to reclaim that precious storage space you so desperately need.  Heck, why no use it as part of your normal failover testing, cause you’ll already doing that right? Once a month (or how often you feel appropriate) do a planned failover using Veeam Backup & Replication, verify that you DR plan works and as an added bonus you reclaim disk space in the process!

And yet another benefit is the spent replicating the virtual machine, without Bitlooker it took 30 minutes to replicate the VM from one host to another but it was just shy of 7 minutes with Bitlooker enabled.

So seriously, why are you not using this magic thing? There’s only one drawback, Bitlooker supports only NTFS file system (=Windows VMs).

The unsung heroes of DRS in vSphere 6.5

VMware vSphere 6.5 has been out for a while now. Some features didn’t get as much publicity as it should, at least in my mind. A few of the really cool features that were introduced in vSphere 6.5 related to DRS, opening up for additional configuration and control of your cluster and virtual machines. It’s found under DRS settings and is called ”Additional Options” where you’ll see 3 new settings:

  • VM distribution

Under normal circumstances DRS will load balance VMs so each virtual machine has the resources it requires, taking into account the cost of vMotioning workloads around to new hosts (as a side note, CPU and RAM has been used since inception of DRS but now it’s also network aware) as one decision-making parameter. This will result in a cluster where you normally won’t see an exact split/distribution/load balancing of the virtual machines cross the hosts, some hosts may be running more VMs than the others. That’s not a problem since the VMs have the resources they need, they are not starved for resources, but the user perception of the DRS functionality is that VMs will be split equally across all hosts. One concern that might require some rethinking of that approach is availability, what if the host running the majority of VMs crash? HA will certainly restart the VMs but a lot of services will be impacted in the environment – Enter VM distribution!

The purpose of VM distribution is to have a fair distribution of VMs on your hosts. Or as it says on one of the official VMware blogs: ”This will cause DRS to spread the count of the VMs evenly across the hosts.”

However, DRS will always prioritize load balancing over the VM spread, so even distribution of VMs is done on a best-effort basis.


  • Memory Metric for Load Balancing

DRS will mainly consider active memory when load balancing, as opposed to consumed or granted memory to VMs. DRS will also take some overhead memory into consideration, so it will use active memory + a 25 % overhead as its primary metric. Active memory is a ”mathematical model that was created in which the hypervisor statistically estimates what Active memory is for a virtual machine by use of a random sampling and some very smart math.  Due diligence was done at the time this was designed to prove the estimation model did represent real life.  So, while it is an estimate, we should not be concerned about its accuracy.”

A really good read on the subject, and where the quote above came from, is ”Understanding vSphere Active Memory”.

Now it’s possible to change what memory metric that is used when load balancing, instead of active memory we can use consumed memory. And by the way, the definition of consumed memory: ”Amount of guest physical memory consumed by the virtual machine for guest memory. Consumed memory does not include overhead memory. It includes shared memory and memory that might be reserved, but not actually used.

Virtual machine consumed memory = memory granted – memory saved due to memory sharing” – Quote from ”Memory Performance Counters – An Evolved Look at Memory Management”.

This option is equivalent to setting the existing cluster advanced option PercentIdleMBInMemDemand with a value of 100.


  • CPU Over-Commitment

It’s now possible to enforce a specific level of overcommit of vCPU to pCPU resources in the cluster, if you only want to allow 4 virtual CPUs per physical core in your cluster you can now have it configured and enforced by the cluster. When the cluster runs out of pCPU resources you won’t be able to power on any more virtual machines. You define a percentage of over commitment of the entire cluster, setting it from 0 % (= no over commitment allowed or put in another way: 1 vCPU = 1 pCPU) up to 500 %.

vSphere Data Protection

VMware has for a pretty long time integrated (at least license wise) a backup engine with vSphere. It’s available as a stand alone OVA download but once deployed it integrates with the vSphere web client. The current release, version 6.1.3, was brought online on the 15:th of November 2016.

The new release of VMware Data Protection, VDP for short, supports VMware ESXi 5.1 or later, vSAN 5.5 or later.

Since it’s been a while since I last had my hands on VDP,  I thought it be a good idea to see how the product has evolved. How to install, configure, backup and so on.

Full disclosure: Amongst others, I’m a Veeam Certified Trainer but I’ll try my best not to be too biased.

Installation is straight forward, get the VDP OVA from VMware bundled with vSphere and deploy the appliance to a host in your environment. I had some issues with my small home lab, DNS needs to be set up and configured correctly with both forward and reverse lookups. Since I don’t have DNS in my lab I was not able to get passed the initial configuration wizard in the VDP, merely altering the hosts file was not enough. But as usual, whenever I need to google a problem I always wind up on William Lam’s webpage – Thanks William! (note to self: Set the web browser start page to virtuallyghetto.com and save valuable time).

VMware has some best practices when deploying VDP:

    • For a better performance of vSphere Data Protection in a high-scaled environment, where Data Domain is the destination or the target, use the following reference to deploy the number of virtual machines according to the capacity of vSphere Data Protection appliance:
vSphere Data Protection Appliance Capacity Number of Virtual Machines to Deploy
0.5 TB 20
1 TB 30
2 TB 40
4 TB 80
8 TB 100
  • For effective load balancing, deploy a maximum of 10 vSphere Data Protection appliances with 100 virtual machines per vSphere Data Protection appliance, in a single vCenter Server domain.
  • In a large environment, deploy a maximum of 8 proxies per vSphere Data Protection appliance regardless the size of the vSphere Data Protection appliances.
  • For better performance of backups and restores, limit the size of each virtual machine to a maximum of 2 TB.

Installing and configuring the VDP:

Once deployed and configured management of jobs are done using the vSphere web client, first thing to do is to connect to the VDP from the web client:

When connected you have a few options, creating a new backup job, verifying a backup job or restoring a backup. You can also download ”Application Backup clients” which can be useful when managing VMs running SQL, Exchange or Sharepoint. The purpose of the agents are to get application consistent backup of VM and have the logs being truncated afterwards.

When you click on the Application Backup clients you end up on the configuration page of the VDP, at the bottom there are links to the individual clients. And on this page you can configure email settings or check log files as well if needed.

Creating a backup job is pretty easy, go through the wizard a change settings, select either ”Guest images” or ”Applications”. If you select Applications you can select what application and specific database you need to back up.

Select what to backup, entire VM or individual disks.

Now you select which VMs to backup, either select individual VMs or a parent container such as a resource pool or a host (or indeed the entire vCenter server). This is where I find myself questioning VMware’s approach a bit. Why can’t you use folders or tags? Grouping VMs in resource pools just for the sake of backup is a bad thing, resource pools are for dividing up resources and should never be used as a ”backup grouping mechanism”. No Tags, folders and so on? Really VMware? Eat your own dog food.

Next you set up the schedule for the backup, you can only set up one backup/restore point per day at most. So you can’t really have advanced scheduling, is backing up a VM once a day good enough? If so then it’s not a problem.

Next, select how long you’d like to keep the restore point. You can either select a simple set up where you set how many days you store the backup or to a specific date but you also have the option of creating a GFS set up. A Grandfather, Father, Son backup is used if you’d like to save weekly, monthly or yearly backups. Nice!

Give the backup job a meaningful name.

Verify that everything is ok.

There you have it, quick and easy to set up. The GUI shows all your available backup jobs on the specific VDP.

Once the backup has been performed you might want to verify that the backup is actually useful, that it can be restored if need be.

Create a new backup verification job.

Select the virtual machines you want to verify.

A heartbeat test will be performed, i.e. the virtual machine will be powered on and VMware will verify that it has a heartbeat to the host (meaning that the operating system of the virtual machine actually started and that the VMware tools service started as well, which will indicated that we have a running OS). But if you provide a script of your own, and have it placed inside the VM you can basically test anything you want. You can perform a much more comprehensive and advanced test for your VMs with a script.

Select a host where you’d like to restore and power on the VM, make sure it has enough resources to actually perform the verification test.

Select the datastore to be used when restoring the VM. Again, the host has to have enough CPU and RAM available to power on the VM but more importantly it has to have enough diskspace as well. Since we have to restore the entire VM the host would need to have free disk space on a datastore. Might not be a problem if we’re verifying a small VM but what about your fileserver or that huge mail server? Do you have X TB available to actually be able to perform the test?

Set a schedule for the verification job, once a week perhaps?

And we give the job a meaningful name.

Verify that everything is ok.

Now we have a summary of all verification jobs available on this VDP server.

It’s possible to either schedule a verification job or to run it manually.

Files of the virtual machine will be restored to the designated datastore and to a folder named VDP_VERIFICATION_<vm-name> -<unique number>.

And there you have it, a successful backup verification performed. Now I have the confidence to restore this particular VM if it would break down for any reason.



If you are serious about your backup data you would probably want to design the backup environment in a way that it is protected from a complete site failure. That’s not a problem in VDP, there’s a built in function to replicate your backup information making it easy to have a secondary copy of your data on a remote site.

Backups created with VDP 6.0 or later can be replicated to another VDP appliance, to an EMC Avamar server, or to a Data Domain system.


If you need to restore virtual machines, vCenter and VDP needs to be available however there is a way to restore directly to a host if needed. If for instance you need to restore vCenter itself. The processes is called ”Emergency restore” and entails disassociating the VDP from vCenter prior to restoring.

File level restore:

File level restore is possible using the ”VDP Restore Client”, no need for an image based backup and a separate file level backup. The VDP Restore Client is accessed using a web browser from the VM to which you want to restore a file. So it has to be running, no redirected restores unless you’re using ”Advanced login” then it’s possible to mount any backup file and restore it to the current VM you are running the VDP Restore Client on. And since you have to connect using a web browser to the VDP server from the VM, you’re required to have network connection (and any potential firewall ports opened up for traffic). VMware VIX anyone?

The Good:

  • It’s available on almost all vSphere editions (only missing in the smallest, vSphere Essentials kit).
  • Easy to set up, easy to configure.
  • Verify the backup, always a good thing.
  • You can create GFS retention policies.
  • Since you can replicate backups/VMs to a secondary VDP appliance you can protect your workload from a complete site failure, nice!
  • Makes use of CBT to minimizes the impact of the backup.
  • It has some  basic backup functionality which would go a long way for some environments.

The Bad:

  • The VDP appliance version 6.1 does not support backups and restores of virtual machines on Virtual Volumes (VVOLs).
  • An emergency restore operation restores a VM directly to the host that is running the VDP appliance. And the host running VDP can’t be part of the vCenter inventory, if it does – the host also has to be disassociated with vCenter prior to the emergency restore.
  • The Restore Client service is only available to virtual machines that have backups that are managed by VDP. This requires you to be logged in, either through the vCenter console or some other remote connection, to one of the virtual machines backed up by VDP.
  • If you need advanced scheduling of backups, multiple backups a day of a VM.


If I try to summarize, VDP has it’s merits. It’s not the full blown availability solution that for instance Veeam represents but then again VDP is free of charge for all licenses of vSphere (it’s available for vSphere Essentials + and upwards) and is not intended to be used for larger organizations I imagine. It’s very easy to set up and if you don’t have any advanced requirements on your backup it should fill your needs. You do need to be aware of and plan for things like resource consumption for backup verification and how to handle file level restores. And it’s a bit disappointing to find a blog post from 2013 talking about tags and folders as ”containers” when creating backup jobs, but nothing has happened yet. It would make a huge difference in terms of management of your jobs.

So, for some organizations, it will be just what the doctor ordered but for others it may be….well, to basic.