Managing vCenter 6 and the PSC Services

Introduced with vSphere 6 is the Platform Services Controller, or PSC for short. Simply put, the PSC is a bunch of services that can reside embedded within a vCenter server, or can be external to the vCenter server. William outlined how to monitor vCenter and PSC services using VIMTOP on a vCSA, but in this article I’ll outline what you can configure using the Web Client.

 System Configuration”/> Administration -> System Configuration

To manage the vCenter Server or PSC services, on the vSphere Web Client home screen navigate to Administration -> System Configuration. From here you have 2 options, Nodes or Services.


From the Nodes selection, you have the ability to select any one of your vCenter or PSC instances that are joined to a single SSO domain. When installing either a vCenter or a PSC server you have the option to create a new or join an existing SSO domain. Once you join this domain you are enabling this instance to participate in “enhanced linked mode”. Remember, linked mode now works in either the Linux based vCSA or Windows based installable vCenter server.

Node Summary
Node Summary
Node Service Health
Node Service Health

A node can either be a vCenter server, an external PSC or a vCenter Server with an Embedded PSC (the case for these screenshots). Once you have a node selected, the right hand section of the Web Client will show you the information about that Node.

The selected Node’s summary page has some general information, such as IP, hostname, type (vCenter, PSC or both), health, uptime and the virtual machine on which the Node resides.

Node Workload
Node Workload

Also on the summary page is a Workload section. This outlines some of the virtual machine in-guest statistics, such as Storage, Memory, Swap and Load.

Finally on the summary tab is the Services Health and Health Messages. It buckets the services into the categories of Critical, Warning, Unknown, Good and Not Applicable. Clicking on each will list the services under each category.

On the Node’s Monitor tab, you can start to see a little more detail about the Workload. Within this tab are for sub-tabs for Networking, Storage, Memory and CPU. Clicking on the workload statistics on the summary tab will bring you into the relevant sub-tab here for more details.

Getting to the Manage tab -> Settings sub-tab of a selected node, you will find a number of helpful tools. Firstly, if you’re using the linux based appliance (and you should be!) there are Access settings that you can also configure on the VM’s DCUI. From here you can enable Local login, SSH login or the Bash shell. Going one more step, you can now change the appliance (vCenter or PSC) name, DNS, IPv4 or IPv6 settings. Within this tab you can also configure the in-guest firewall rules to block or allow IPs per ethernet interface. Finally, this is also where you can join and leave an Active Directory domain. I will leave the the Certificate Authority sub-tab for another post.


Going back to the top and selecting Services instead of Nodes will give you the visibility and settings for all services within your SSO domain. The following table outlines the available services with their default startup type for a vCenter Server with an embedded PSC and embedded Postgres database (AKA everything on one node):

Service Description Startup-type
Auto Deploy Supports network-based deployment of ESXi hosts. Manual
Content Library Service Enables sharing and management of VM templates and ISO images across vCenter instances Automatic
Data Service Universal query API to VMware CIS data Automatic
Hardware Health Service Collection and analysis of IPMI sensor metrics from hardware running ESXi Automatic
Inventory Service Enables search, list, query and extension of vCenter inventory information Automatic
License Service Provides licensing support for the vSphere environment Automatic
Transfer Service Enables movement of content like VM templates, scripts, ISO images across sites and vCenter instances Automatic
VMware ESX Agent Manager ESX Agent Manager (EAM) is the simple and fully-integrated way to deploy and monitor ESX Agent VMs and VIBs on ESX hosts. Automatic
VMware Message Bus Configuration Service VMware Message Bus Configuration Service Manual
VMware Open Virtualization Format Service Enables open virtualization format based provisioning of virtual machines via Content Library Automatic
VMware Performance Charts Service Provides Overview Performance Charts support for vSphere Web Client. Automatic
VMware Postgres Embedded VMware Postgres Relational Database Automatic
VMware Syslog Service Provides syslog support for VMware CIS services Automatic
VMware vCenter Server VMware vCenter Server Automatic
VMware vService Manager VMware vService Manager Automatic
VMware vSphere ESXi Dump Collector VMware vSphere ESXi Dump Collector enables support for collecting core dumps from remote hosts. Manual
VMware vSphere Profile-Driven Storage Service VMware vSphere Profile-Driven Storage Service Automatic
VMware vSphere Profile-Driven Storage Service vSphere Virtual Infrastructure Management Client Automatic
vAPI Endpoint Provides single point of access to vAPI services Automatic

Some of the services have a number of configurable settings behind them on the Manage tab, but others do not. Regardless, I would recommend that you only change these if absolutely necessary (advised to by VMware). The other thing shown is whether changing the setting will require a restart of that particular service.

Node Services: Start, Stop, Re-Start, Start-Up Type
Node Services: Start, Stop, Re-Start, Start-Up Type

Starting, stopping, restarting or changing a service’s startup type is configured on a per-node basis. Selecting the Node you wish to configure the service on, then going to the Related-Objects tab will allow you to set these.

You should find managing the Linux (and Windows) vCenter instances a little easier now. There’s certainly a lot more options to configure and a lot more visibility into what’s going on.

Announcing Mastering VMware vSphere 6

Over the last 10 months, with my contributing authors, Grant Orchard and Josh Atwell, I have been working on an update to the Mastering VMware vSphere series. What does the updated book cover? Well, I tried to stick to the very well received “Scott Lowe Formula” from the previous Mastering vSphere 4 and 5 but at the same time continue to put my own take on things after the vSphere 5.5 revision. Scott Lowe has once again graciously written the forward for this book and I think his words summarize it well.

Mastering VMware vSphere 6 is more than just a new version of a well-respected tome, because Nick has done more than just refresh the material found within these pages. Yes, you’ll find in-depth coverage of new features in vSphere 6; that includes features like long-distance vMotion and cross-vCenter vMotion, a new version of VSAN, and the long-awaited SMP Fault Tolerance that will allow you to use VMware Fault Tolerance with up to 4 virtual CPUs. Yes, you’ll find coverage of NFS 4.1 support, Virtual Volumes, and a new version of Network I/O Control. Of course, there is so much more in vSphere 6 than just what I’ve mentioned here, but you’ll have to read the rest of the book to find out what else is included!

— Scott Lowe

Yesterday VMware announced that vSphere 6.0 will be made available by the end of March and I’d like to advise that Mastering VMware vSphere 6 is available for pre-order today and will be shipping or downloadable within days of the software being made available.

Rightsizing vFlash Read Cache (vFRC)

Last week I presented a couple of sessions at the annual Sydney vForum, people sure are hungry for information on vSphere 5.5. There were around 400 people in my session on Architecting vSphere 5.5, it was standing room only! You can find my presentations here and hereGiven this thirst for this information, I thought I would dive a little deeper into one of my favourite features in vSphere 5.5, vFRC (vFlash Read Cache). I would like to thank Mark Achtemichuk and Sankaran Sivathanu for their indirect assistance with this. I’d also HIGHLY recommend reading the vFRC Performance Whitepaper.

One of the most important considerations when implementing vFRC is the block size chosen for each VMDK cache. Since vFRC is configured on a per VMDK basis, each cache can have a different block size, ranging from 4KB to 1024KB with 8KB being the default. Why do we have an option for this configuration? Efficiency.

All applications are different, even the same app could have a different storage IO profile depending on the application use. There are some general rules that you could follow, such as Oracle using IO of 8KB or MS SQL using 64KB but unless you do some proper monitoring, there’s no real way to tell what your particular application’s IO pattern predominantly uses.

Why does it matter? Well as I mentioned earlier, it’s around efficiency. When we configure vFRC for a certain block size, the application data block that is cached resides inside a vFRC block. If the cache block size does not match the application’s IO block size we get one of two things:

  1. If the cache block size is too small this increases cache misses.
  2. If the cache block size is too large there is internal fragmentation – wasted space in each cache block. This leads to a reduced overall cache capacity.

One more thing we need to consider is memory overhead. As we decrease the block size of the cache, the number of blocks increases. Keeping track of more blocks, takes more memory. You can find the details one the amount of overhead required in the whitepaper mentioned earlier.

So how do you ensure that the block size you set for each VMDK cache is correct? A handily little tool built into ESXi called vscsiStats. Now I’m not going to go over what this tool is in great detail, Duncan and Cormac have already done that. However I would like to explain the steps required to configure vFRC with the correct block size for your workloads.

  1. Log into the host that runs the vFRC candidate via SSH or via the console command line.
  2. List the currently running VMs and their VMDKs: vscsiStats -l
  3. Start capturing I/O for a particular candidate: vscsiStats -s -w YYYYY -i XXXX ( Where YYYYY is the Virtual Machine wouldGroupID and XXXX is the Virtual SCSI Disk handleID)
  4. Run the typical workload within the candidate VM
  5. After the workload has run for a bit of time, display the histogram for “IO Length”: vscsiStats -p ioLength -c -w XXXXX -i YYYY

You output will be similar to the list below. Each line in the following output shows a different IO (block) size from 512-bytes to 524288-bytes (512-Kilobytes) The first number (before the comma) shows how many IOs, the second number (after the comma) shows the IO size

Histogram: IO lengths of commands,virtual machine worldGroupID,128782,virtual disk handleID,8192 (scsi0:0)

  • min,512
  • max,1052672
  • mean,18434
  • count,21150
  • Frequency,Histogram Bucket Limit
  • 1576,512
  • 1073,1024
  • 539,2048
  • 375,4095
  • 5219,4096
  • 428,8191
  • 4246,8192
  • 787,16383
  • 1858,16384
  • 3877,32768
  • 62,49152
  • 405,65535
  • 155,65536
  • 32,81920
  • 324,131072
  • 138,262144
  • 9,524288
  • 47,524288

Finally, stop vscsiStats from collecting data: vscsiStats -x

As you can see from the captured data above, I/O size of 4KB has the largest count. Does this mean we should set the vFRC to 4KB? Not necessarily. If it was a higher count by a large margin, then yes this would be the correct setting. However we need to consider the entire I/O profile. For this example, I would start by setting the vFRC block size to 8KB or 16KB. This will ensure there is not too much cache fragmentation, but also covers most of the I/O range fairly well.

Once the block size is set, monitor your cache! Keep an eye out for cache hit percentage using:

esxcli storage vflash cache stats get –m <module > – c <cache file> 

I’m going to be running some benchmarks to try and explain what kind of performance difference there is between a default vFRC configuration and one that has been rightsized. I’ll share my results here when it’s done. In the mean time, make sure you size your vFRC correctly!

vSphere 5.5 – ESXi minimum RAM

With vSphere 5.5 about to go GA, something that people may not know is that there’s an uplift in the host minimum memory requirement. I noticed early in the beta process that if I configured a host with only 2GB of RAM (the previous maximum) while it would install and run fine, as soon as I joined it to a cluster the host would PSOD. I soon learned that the memory footprint of ESXi has increased by a little bit, as soon as the FDM service was started, it would tip over the edge.

So, with that in mind I had to increase my nested hosts to 3GB to get them to run properly… when you add in VSAN and any other services needed, you may find that even that is not enough. ESXi 5.5 “officially” requires a minimum of 4GB of RAM.

Time to upgrade those lab hosts! 

Announcing Mastering VMware vSphere 5.5

After months of learning, writing and endless lab rebuilds I am very excited to announce the project I have been working on:

Mastering VMware vSphere 5.5

When Scott Lowe first approached me in late 2012 about a possible writing opportunity I couldn’t begin to understand what the next 9 months would entail. The project has certainly been mammoth, but one that I’ve thoroughly enjoyed along the way. And the best bit is yet to come, I get to share it with the rest of the VMware community!

This new revision has been revised for the newly announced vSphere 5.5 and covers all the features introduced in both 5.1 and 5.5. Getting up to speed with the vSphere Web Client has never been easier, as every example both new and old has been written with the Web Client in mind. At a high level, some the new coverage includes:

  • Single Sign-On
  • VSAN
  • vFlash
  • LACP
  • vC Ops

As Scott mentioned when he announced the 5.0 edition “This book won’t go as deep as some other books on the market, but that’s not really its purpose.” – and the same goes for this edition. The idea is real world, practical information to help you get the most out of vSphere 5.5.

I would like to put a big thank you out there to the contributing authors, Forbes Guthrie, Matt Liebowitz and Josh Atwell. These guys each submitted a quality chapter towards this book and I am ever grateful for their contributions. And of course, thank you to Scott for this opportunity. I hope that I have lived up to your expectations with this work and you have no regrets handing “your baby” over to my care. I certainly will pay it forward when the time is right as you so graciously did for me.

So, I guess the question is, when is it available? It’s available for pre-order right now from Amazon, and should be generally available late October / early November.

Renaming the vSphere SSO Database

Renaming the vSphere SSO database is simpler than you’d think.

The other day while working with a customer, we needed to rename the vSphere 5.1 Single Sign On database to ensure it aligned to naming conventions. As I’d never had to do this before, I read up on a few different KB articles: 204552820335162045528

Based on the information presented I did the following:

  • Backed up the SSO configuration using the “Generate vCenter Single Sign-On backup bundle” link in the Start -> Programs menu from the SSO server.
  • Performed a backup of the original SSO DB within the SQL Management Studio from the DB VM.
  • Finally, I took a snapshot of both the DB and SSO VMs as a safeguard.

Once the fallback plan was in place, I stopping the “vCenter Single Sign On” service on the SSO VM, then renamed the RSA database on the DB server to the conforming name.

Then I thought it was a matter of using the following command on the SSO VM:

ssocli configure-riat -a configure-db –database-host new_database_server –database-port new_database_port -m master_password

However, you may notice with the SSOCLI utility while there’s a Host, Port, Instance, Username and Password option… there is NOT a database name option. (I mistook the “instance” option allowing me to do what I needed at first). This perplexed me at first, how do I reconfigure SSO to point to a renamed database? After a little digging I worked out that the database name is configured not through the SSOCLI utility, but through a configuration file:

It’s as simple as changing the following line:


And that’s it! Restart the SSO service on your SSO VM and then fire up the vSphere Web Client. Also, thanks to Gabe for pointing me in the right direction for the config file.

VXLAN with vSphere

There’s an interesting article recently published by VMware’s Technical Marketing team around using VXLAN with vSphere. Typically most information around VXLAN that I have seen is geared towards integration with vCloud Director to stretch organisation and DC networks over multiple segments. This article however specifically deals with just the vSphere potion.

So for those of you that want to delve into VXLAN with a more traditional vSphere environment but without the added complexity of vCloud, I recommend you check it out:

VMware Port Requirements

I’m currently working on a project in a very secure network. Sometimes it almost feels like every other server is in it’s own DMZ and I’m constantly looking up what the network ports need opening beteeen them.

The following document outlines port requirements. The reason I like it so much is because it’s not just one product, it’s for most of VMware’s product portfolio! Very handy.