Virtual VMUG – Upgrading and Mastering vSphere 5.5

Today I presented a session at the Virtual VMUG conference titled “Upgrading and Mastering VMware vSphere 5.5″. It was a session based around the upgrade process to vSphere 5.5 and a few little tricks for playing with vFRC and VSAN inside AutoLab. Thanks for all those of you that attended, I was inundated with questions during and after the presentation. I’ll be sure to get your contact details and answer those I didn’t get to within the allotted timeframe.

As promised, below is the (higher quality) recording from the demo section of the presentation. The slide deck was pretty light on, but if people want it just comment below and I’ll upload that too.

Holiday Book Giveaway

There’s a number of great giveaways around this time every year. A couple of them are giving away copies of my book Mastering VMware vSphere 5.5. If you want to score a copy, I suggest you head to the following links:

I’m also going to give away a signed or electronic copy of my book to one lucky recipient. Simply leave a comment below and I’ll pick a winner at random on Christmas Eve.


Congratulations to AlexC – the winner after I randomly generated a number (20) between 1 and 61 (total number of comments).

Rightsizing vFlash Read Cache (vFRC)

Last week I presented a couple of sessions at the annual Sydney vForum, people sure are hungry for information on vSphere 5.5. There were around 400 people in my session on Architecting vSphere 5.5, it was standing room only! You can find my presentations here and hereGiven this thirst for this information, I thought I would dive a little deeper into one of my favourite features in vSphere 5.5, vFRC (vFlash Read Cache). I would like to thank Mark Achtemichuk and Sankaran Sivathanu for their indirect assistance with this. I’d also HIGHLY recommend reading the vFRC Performance Whitepaper.

One of the most important considerations when implementing vFRC is the block size chosen for each VMDK cache. Since vFRC is configured on a per VMDK basis, each cache can have a different block size, ranging from 4KB to 1024KB with 8KB being the default. Why do we have an option for this configuration? Efficiency.

All applications are different, even the same app could have a different storage IO profile depending on the application use. There are some general rules that you could follow, such as Oracle using IO of 8KB or MS SQL using 64KB but unless you do some proper monitoring, there’s no real way to tell what your particular application’s IO pattern predominantly uses.

Why does it matter? Well as I mentioned earlier, it’s around efficiency. When we configure vFRC for a certain block size, the application data block that is cached resides inside a vFRC block. If the cache block size does not match the application’s IO block size we get one of two things:

  1. If the cache block size is too small this increases cache misses.
  2. If the cache block size is too large there is internal fragmentation – wasted space in each cache block. This leads to a reduced overall cache capacity.

One more thing we need to consider is memory overhead. As we decrease the block size of the cache, the number of blocks increases. Keeping track of more blocks, takes more memory. You can find the details one the amount of overhead required in the whitepaper mentioned earlier.

So how do you ensure that the block size you set for each VMDK cache is correct? A handily little tool built into ESXi called vscsiStats. Now I’m not going to go over what this tool is in great detail, Duncan and Cormac have already done that. However I would like to explain the steps required to configure vFRC with the correct block size for your workloads.

  1. Log into the host that runs the vFRC candidate via SSH or via the console command line.
  2. List the currently running VMs and their VMDKs: vscsiStats -l
  3. Start capturing I/O for a particular candidate: vscsiStats -s -w YYYYY -i XXXX ( Where YYYYY is the Virtual Machine wouldGroupID and XXXX is the Virtual SCSI Disk handleID)
  4. Run the typical workload within the candidate VM
  5. After the workload has run for a bit of time, display the histogram for “IO Length”: vscsiStats -p ioLength -c -w XXXXX -i YYYY

You output will be similar to the list below. Each line in the following output shows a different IO (block) size from 512-bytes to 524288-bytes (512-Kilobytes) The first number (before the comma) shows how many IOs, the second number (after the comma) shows the IO size

Histogram: IO lengths of commands,virtual machine worldGroupID,128782,virtual disk handleID,8192 (scsi0:0)

  • min,512
  • max,1052672
  • mean,18434
  • count,21150
  • Frequency,Histogram Bucket Limit
  • 1576,512
  • 1073,1024
  • 539,2048
  • 375,4095
  • 5219,4096
  • 428,8191
  • 4246,8192
  • 787,16383
  • 1858,16384
  • 3877,32768
  • 62,49152
  • 405,65535
  • 155,65536
  • 32,81920
  • 324,131072
  • 138,262144
  • 9,524288
  • 47,524288

Finally, stop vscsiStats from collecting data: vscsiStats -x

As you can see from the captured data above, I/O size of 4KB has the largest count. Does this mean we should set the vFRC to 4KB? Not necessarily. If it was a higher count by a large margin, then yes this would be the correct setting. However we need to consider the entire I/O profile. For this example, I would start by setting the vFRC block size to 8KB or 16KB. This will ensure there is not too much cache fragmentation, but also covers most of the I/O range fairly well.

Once the block size is set, monitor your cache! Keep an eye out for cache hit percentage using:

esxcli storage vflash cache stats get –m <module > – c <cache file> 

I’m going to be running some benchmarks to try and explain what kind of performance difference there is between a default vFRC configuration and one that has been rightsized. I’ll share my results here when it’s done. In the mean time, make sure you size your vFRC correctly!

vBrownBag – VMware VSAN

I’m going to be presenting a vBrownBag session this Wednesday night (US) / Thursday morning (AUS) on how to set up VSAN in your lab. There’s been a few blog posts around already, Duncan, William and Cormac (to name a few) all have something for those wanting to read up before the session.

But for those that can’t attend in person, I thought I would answer a question I got from a customer today “How is this VSAN software defined storage?”

Not wanting to repeat too much of my vBrownBag session here, my answer revolved around the “defined” part. We “define” how we want to storage to operate on a per VM basis. Now, until we get vVOLs this isn’t going to become as simple on larger arrays, but when we control everything from the hypervisor like we do with VSAN this is a reality today.

So what can we control? Well once VSAN is set up (and boy is it simple!), you need to use the Virtual Machine Storage Policy section of the vSphere Web Client to tell VSAN how to treat your VMDKs.

There’s more detail in Cormac’s whitepaper here, but you’ll find the following are options for defining your storage policies on VSAN:

Number of disk stripes per object

The number of HDDs across which each replica of a storage object is striped. A value higher than 1 may result in better performance (for e.g. when flash read cache misses need to get services from HDD), but also results in higher use of system resources. Default value: 1, Maximum value: 12.

Flash read cache reservation (%)

Flash capacity reserved as read cache for the storage object. Specified as a percentage of the logical size of the object. To be used only for addressing read performance issues. Reserved flash capacity cannot be used by other objects. Unreserved flash is shares fairly among all object. Default value: 0%, Maximum value: 100%.

Number of failures to tolerate

Defines the number of host, disk or network failures a storage object can tolerate. For n failures tolerated, “n+1” copies of the object are created and “2n+1” hosts contributing storage are required. Default value: 1, Maximum value: 3.

Force provisioning

If this option is enabled, the object will be provisioned even if the policy specified in the storage policy is not satisfiable with the resources currently available in the cluster. VSAN will try to bring the object into compliance if and when resources become available. Default value: Disabled.

Object space reservation (%)

Percentage of the logical size of the storage object that will be reserved (thick provisioned) upon VM provisioning. The rest of the storage object is thin provisioned. Default value: 0%, Maximum value: 100%

vSphere 5.5 – ESXi minimum RAM

With vSphere 5.5 about to go GA, something that people may not know is that there’s an uplift in the host minimum memory requirement. I noticed early in the beta process that if I configured a host with only 2GB of RAM (the previous maximum) while it would install and run fine, as soon as I joined it to a cluster the host would PSOD. I soon learned that the memory footprint of ESXi has increased by a little bit, as soon as the FDM service was started, it would tip over the edge.

So, with that in mind I had to increase my nested hosts to 3GB to get them to run properly… when you add in VSAN and any other services needed, you may find that even that is not enough. ESXi 5.5 “officially” requires a minimum of 4GB of RAM.

Time to upgrade those lab hosts! 

Announcing Mastering VMware vSphere 5.5

After months of learning, writing and endless lab rebuilds I am very excited to announce the project I have been working on:

Mastering VMware vSphere 5.5

When Scott Lowe first approached me in late 2012 about a possible writing opportunity I couldn’t begin to understand what the next 9 months would entail. The project has certainly been mammoth, but one that I’ve thoroughly enjoyed along the way. And the best bit is yet to come, I get to share it with the rest of the VMware community!

This new revision has been revised for the newly announced vSphere 5.5 and covers all the features introduced in both 5.1 and 5.5. Getting up to speed with the vSphere Web Client has never been easier, as every example both new and old has been written with the Web Client in mind. At a high level, some the new coverage includes:

  • Single Sign-On
  • VSAN
  • vFlash
  • LACP
  • vC Ops

As Scott mentioned when he announced the 5.0 edition “This book won’t go as deep as some other books on the market, but that’s not really its purpose.” – and the same goes for this edition. The idea is real world, practical information to help you get the most out of vSphere 5.5.

I would like to put a big thank you out there to the contributing authors, Forbes Guthrie, Matt Liebowitz and Josh Atwell. These guys each submitted a quality chapter towards this book and I am ever grateful for their contributions. And of course, thank you to Scott for this opportunity. I hope that I have lived up to your expectations with this work and you have no regrets handing “your baby” over to my care. I certainly will pay it forward when the time is right as you so graciously did for me.

So, I guess the question is, when is it available? It’s available for pre-order right now from Amazon, and should be generally available late October / early November.

Don’t forget the Study Guides

I’ve had a number of people talk to me about community based VMware study guides over the past few months. It’s been a little while since their creation, but the vBrownBag study guides are still relevant to VCP5-DCV, VCAP5-DCA and VCAP5-DCD. There’s also the VCDX Boot Camp we did late in 2012.

Along with these community guides, VMware has more recently added a heap of great (free) content on including some very handy VCDX Mock Panels.


The vCenter Inventory Service

The vCenter Inventory Service is used for 2 main things. First of all it’s the place that stores all of the custom tags used within the vCenter Web Client. But what’s not as well known is that the vCenter Inventory Service is also proxy (or cache) for the Web Client. If you’re using the traditional vSphere Client, the Inventory Service is not used, it’s simply bypassed… hence no tags in the older client. Prior to vSphere 5.1, while the service did exist, it wasn’t separated out as an individual component. From vSphere 5.1 onwards this is a separate installable component that can co-exist on the same, or be split out onto a separate, windows server. Currently, separating out components on the linux based vCenter Server Appliance is not possible (unless you like to hack things like William does)

Why we need the Inventory Service

As I stated before, the Inventory Service is designed to reduce load on the vCenter server itself (VPXD). Traditionally only 10% of traffic to a vCenter Server consists of writes, and obviously the other 90% are reads. So if you can cache those reads closer to the Web Client and don’t have to ask vCenter to retrieve from its database each time, you can see a significant improvement in response times; while also reducing the load on the more critical vCenter Server itself.


So how much improvement in terms? Well if you look at the #vBrownBag from earlier in the year Justin King (VMware’s resident vCenter guru) outlines the numbers.

As you can see from the table, if you have a large environment with many administrators the Inventory Service can really be of benefit. The thing to keep in mind here is, if you have a large scale environment where should I deploy this service to see the benefit?


table.tftable {font-size:12px;color:#333333;width:100%;border-width: 1px;border-color: #a9a9a9;border-collapse: collapse;}
table.tftable th {font-size:12px;background-color:#b8b8b8;border-width: 1px;padding: 8px;border-style: solid;border-color: #a9a9a9;text-align:left;}
table.tftable tr {background-color:#ffffff;}
table.tftable td {font-size:12px;border-width: 1px;padding: 8px;border-style: solid;border-color: #a9a9a9;}

Client # Sessions vCenter CPU
VI Client 100 50%
Web Client 180 25%

Where the Inventory Service should be installed

While it makes sense to use the vCenter Web Client when scaling out to a large environment, you need to ensure that the Inventory Service is installed in the right location to be of most benefit.

I was recently onsite with a customer who were planning to install the Inventory Service on the same VM as their vCenter Server, but there were splitting out the Web Client to a separate server. I advised them that although this will reduce the load on the vCenter Server, the best place for this service is not on the vCenter Server itself, but to locate the Inventory Service on either its own server, or alongside the vSphere Web Client.


Wrapping Up

Remember the KISS principle, you do not need to split out your components unless you have specific requirements and/or need to scale your environment. Always keep in mind the hardware requirements for each component and how those change if you co-locate services on the same server (see here). If you do split out the components, think about what the component is, and where it best fits. Along with storing tags, the idea of the Inventory Service is to reduce the number of queries to the vCenter Server… place it accordingly!

If you’re running your vCenter Server components as a VM (if not, why not?) think about your options. What about configuring DRS affinity rules to keep the Web Client and the Inventory Service VMs together, but your vCenter server separate? Everything depends on your specific requirements, but understanding the architecture and how different components work will ensure you can scale your environment effectively.

I’d be interested to know if anyone else has similar (or different) stories on where they locate the vCenter Inventory Service. Feel free to comment below.

“Random” sound in OSX

I changed over to using a Mac as my primary machine (again) around 12 months ago. Its a great workhorse for a laptop, the SSD and 16GB of RAM make it perfect for AutoLab.  However, every now and then I notice an annoying sound that I couldn’t work out where it was coming from. Admittedly I didn’t try too hard to find the source, but it wore me down today.

The culprit? iMessage. 

Every time someone signed in or out of Google Chat, Jabber etc, I got an annoying swoosh sound.  When I found it, I certainly felt like a noob.

Simple fix, head into iMessage -> Preferences -> Alerts. Change these two events to have no sound: Buddy Becomes Available and Buddy Becomes Unavailable

Problem solved. 


What it takes to become a vExpert

​What does it take to become a vExpert? Here’s what I did over the past 12 months.

There’s been a lot of chatter over the last few days with the recent announcement of the VMware vExpert 2013 group. I was fortunate enough to be given this honour again this year, an award that I’m surprised not more people know about!

I’ve had numerous conversations with customers and other VMware employees about “what it takes to be a vExpert”. Even within VMware, there’s people who are not aware of this program, so I do everything within my power to promote it both to employees and customers alike.

So, what does it take to be awarded a vExpert? Well, there are three available paths, the Evangelist Path, the Customer Path and the VPN (VMware Partner Network) Path. Without going into too much detail on each (as you can read about each here), I’ll explain what was in my application.

I applied under the Evangelist Path, with the following points against my name for 2012:

My list of things may be bigger or smaller than other vExpert award winners, but if you’re doing any of these things within the VMware community make sure you apply next year so you can get the recognition you deserve! It’s a great program, and I thank John Troyer and the rest of the VMware Social team for running it.

Edit: If anyone wants help obtaining vExpert, let me know as I have far too many VMware community ideas than I have time for!