Getting Started with Kubernetes and VMware Integrated Openstack 4.0

VMware Integrated Openstack 4.0 is available with a great new feature for users wanting to explore container orchestration.  Kubernetes provides enterprise grade container orchestration, enabling coveted technology such as auto-scaling and auto-healing.

Pairing VIO with Kubernetes makes sense as it augments the out of the box feature set with that provided by the vSphere Hypervisor and NSX, providing a much higher level of centralized management and augmented networking capabilities over the core mainline Openstack product.

The Kubernetes + OpenStack bits are available in VMware’s download portal under the VIO 4.0 download page. A 60 day trial license is included out of the box to get you started:

VIO 4.0 and Kubernetes Download Page

After downloading, go thru the normal OVA installation and deployment.  There are some great blogs out there on deploying VMware Integrated Openstack such as this blog post by Sean Whitney.


Once you’ve deployed the Kubernetes appliance, log in with the root credentials you created during OVA deployment:

Kubernetes Login Page


First, we will need to deploy a new provider.  We will target the previously deployed VIO 4.0 as our new provider.  You are allowed to have multiple providers, so you could target multiple different instances of VIO or an Openstack provider hosted in the public cloud as well.

Deploying New Cloud Provider


Name your provider appropriately and set the provider type.

Set Provider Type


In the next step, we are going to need the SSL Certificate from our VIO instance.  In my lab, the cert I’m using is the default cert and is self signed.  By providing Kubernetes the cert out of band and double checking the thumbprint, we can validate that we have the correct certificate and our K8S cluster is trusting the correct provider.

Exporting SSL Cert 1

Make sure to add the .crt extension to the exported file.  I find it easiest to use the Firefox web browser to do this as the latest versions of Chrome have moved the certificate viewer and exporter into the developer tools.  Whichever is easiest for you is fine.

Exporting SSL Cert 2

Fill out the Openstack authentication information in the next step of the wizard.  Note the Keystone public URL for authentication in VIO 4.0 is the base URL of your openstack instance with on port 5000 with /v3 added to the end of the URL.  This information is available under the info popup as well.

OpenStack Authentication for K8S


Make sure to use the cloud image version of Ubuntu, it is correctly prepared with the necessary packages to make everything work out of the box. It is available in VMDK format (as well as many others) here.

You’ll need to upload the image to your Openstack instance in the project you are using for Kubernetes before proceeding.

Kubernetes Image Selection


Choose an appropriate external network to provide network access to your K8S cluster.

External Network Configuration

Choose the internal network you want your Kubernetes deployed containers to utilize.

Internal Network Configuration


Once this information is added, you are ready to deploy your cloud provider!  This should only take a few minutes.  Once completed, it will look similar to this.

Completed Cloud Provider


Once you have a provider added, you are ready to create your first Kubernetes Cluster.  First, select the infrastructure provider we just created above.

Select Provider


Next, choose the security group you want to use to secure network traffic for this cluster.  This will vary depending on the environment and applications running in the cluster.

Cluster SG


Choose the users and groups who will be able to access and utilize the cluster.

User and Group Access Selection

Cluster creation can take awhile.  You’ll see the cluster being created in the VIO with Kubernetes UI.

Cluster Creation

During cluster creation, you should see the infrastructure for the cluster being created in the vSphere client.

vSphere tasks executing

Next Steps Deploying Your First Kubernetes Pod:

Once the cluster deploys, you should be able to access and authenticate against the kubernetes API.  From this point on, the process is identical for developers and users as consuming any other Kubernetes cluster.  This is a well documented and often written about process.  For a great starter tutorial, check out this post by Raziel Tabib at CodeFresh.

vRealize Automation 7: Adding NSX Integration

vRealize Automation 7 introduces some new big features for Networking and Security integrations like support for On Demand Security Groups, on demand load balancers, and Security Tags from the blueprint layout.

In order to use these features, you’ll need a functioning NSX installation, and then need to do the following configuration tasks to get it all working-

1. Add an NSX Manager in your vRealize Orchestrator client

vRealize Automation 7 uses vRealize Orchestrator to execute operations against NSX.  You will need to navigate the vRA 7 landing page and click the Orchestrator Client link (if you are using the embedded vRO, otherwise, navigate to the landing page for your external vRO appliance)

vRealize Automation Landing Page

Use your administrator@vsphere.local (or vRO admin credentials) to log in, and navigate to /Library/NSX/Configuration/Create NSX Endpoint to start the endpoint creation:

vRealize Orchestrator: Add a NSX Endpoint

After the endpoint is added, verify you are able to browse the NSX inventory on the inventory tab:

Browse NSX Inventory from vRO

This completes the addition of NSX to vRO.add_new_vRO_endpoint

2. Add Orchestrator as an endpoint to vRealize Automation

To use Orchestrator to manipulate NSX objects, it must be added as a new endpoint.  Do not confuse this with the Orchestrator configuration options under the Administration tab- those are used from XaaS blueprints rather than VM provisioning.  The endpoint configuration options you need are located under Infrastructure / Endpoints.

Add Orchestrator Endpoint

One important change to note for the embedded vRealize Orchestrator 7 is the API interface is NOT running on port 8281 anymore, I’ve posted my example URL below:


Make sure to use credentials that are tested by using them to log in to the vRO instance.  Also, you will need to add a custom property for vRO priority- VMware.VCenterOrchestrator.Priority = 1 (or another number if you are ordering multiple vRO instances).

vRO Endpoint Config

Once you’ve added the endpoint, you will want to make sure data collection completes successfully.  If Data Collection fails, go back thru all your endpoint configuration and make sure it is correct.

vRO Data Collection


3. Specify manager for network and security platform

Once the orchestrator has been configured to connect to the NSX Manager, you need to specify and bind the NSX manager to the vSphere Endpoint- This is done under Infrastructure/Endpoints (if you do not see this tab, try logging in as configurationadmin@vsphere.local or administrator@vsphere.local)-

Networking and Security Config

Once configured, you should be able to initiate Data Collection on the Compute Resource (not located on properties of the endpoint)- The Compute Resource should be available at Infrastucture / Compute Resources / Compute Resources. You should your relevant compute resources by cluster name:

Compute Resources Data Collection

When in the data collection screen, scroll to the bottom and check the Networking and Security Data Collection.  If everything is configured successfully, you should see a successful data collection after it completes-



Once data collection completes successfully, you should be able to include NSX constructs in your blueprint design-  below, you can see a on-demand security group being added to a blueprint-

Networking and Security Blueprint Components


The on demand and blueprint layout features really make configuration and deployment of complex multi tier applications and custom firewall rulesets in NSX significantly easier to deploy- less than 6 months ago this sort of feature set would require extensive custom vRO code.  It is great to see it in the core product now.

vRA 6.2.1 Remote Console with Load Balancer

I was working with a client today to troubleshoot and resolve issues with vRA (vCAC) 6.2.1 Remote Console through a F5 load balancer.  Here are the key takeaways for getting it to work:

  • You will need a new pool or port service for Port 8444.  This is the port used from the client system to the vRA Web Appliance.
  • vRA Web must have connectivity to the vCenter on port 443 and to the ESXi server where the VM resides on port 902.
  • IF using F5 BIG IP with a version earlier than 11.4.0. there is a bug where the Load Balancer drops WebSocket traffic.  WebSocket traffic is used for remote console in vRA 6.2.1.  Here is the kbase article

The workaround is documented in the article, but is essentially to not use an HTTP profile for the 8444 load balancing pool and to configure it to pass raw TCP traffic.  I’ve included a screenshot below:

F5 Web Sockets workaround
F5 Web Sockets workaround


Hope this helps! -Justin

vCAC CPU Memory Hot Add Disable

Today while working with a client we encountered an issue where CPU and Memory Hot Add were causing failures when a VM was edited.  vCAC showed the VM stuck in the mode On (Reconfigure.WaitingforRetry)-

Here is what it looks like in vCAC 6.1: (click for zoom)


In vCenter, the task to reconfigure CPU and memory fails, with the failure looking like this:

vCenter hotadd fail
vCenter hotadd fail

To fix this, there are 2 options:

  1. Configure vCenter templates to Support Hot Add (if the OS supports it.
  2. Disable CPU/Memory Hot add by adding a custom property to the blueprint, or a build profile.

Lets look at option 1 first.  Edit the VM in vCenter- (if it is a template, convert to VM first then edit):

Navigate to the Options tab and choose Memory/CPU Hotplug

vCenter Enable Hotadd
vCenter Enable Hotadd

For the second solution (Disabling CPU Hotadd), you can add the following custom properties:

  • VirtualMachine.Reconfigure.DisableHotCpu = true
  • VirtualMachine.Reconfigure.DisableHotMemory = true

Lastly, you for VMs stuck in the Reconfigure.WaitingforRetry state, make sure you have entitlements for Execute Reconfigure and Cancel Reconfigure, which should allow you to shut down the VM and then execute the reconfig, or cancel it.

Note: The custom properties added for disabling hot add will only apply to NEW VMs deployed, existing VMs will have to have the custom properties added manually.

Puppet Camp Portland 2015

I was recently invited to speak at Puppet Camp Portland 2015- The event was extremely informative, and I had a chance to interact with a large group of Puppet Labs employees and customers.  The event was held at the Oregon Museum of Science and Industry (OMSI) in downtown Portland- I arrived around 8AM and was impressed with the aesthetics of the building-

Oregon Museum of Science and industry
Oregon Museum of Science and industry

After checking in, I visited the theatre space where the speakers would give their presentations, which happened to be a full IMAX theatre, complete with 4 story screen, dual 4K Christie projectors, and Dolby Atmos sound system.  The screen was intimidatingly enormous, but after awhile I think all the presenters got used to it, and found it pretty beneficial to be able for the audience to be read small print and code on slides while projecting.

Puppet Camp Students
Puppet Camp Students

My presentation topic was Using Puppet with Self Service Provisioning – I went through my base material in about 30 minutes and had an extremely productive Q&A session for about another 15 minutes- I think it is clear that this is an extremely hot topic with respect to configuration management-  provisioning a Virtual Machine in 30 minutes doesn’t do any good if there is a stack of software that has to be installed and configured by specialists that takes another 2 weeks.

My full slide deck and a video of the presentation is available on the Puppet Labs website:

Standing Desk

Standing desks.  They are all the rage these days, and for quite a few good reasons.  If you aren’t privy to the “Standing Desk Revolution” the tl;dr is sitting for 8-10 hours is really bad for you- standing is quite a bit better for your health.

Since I work from home the majority of the time, I felt it was worth the investment to buy a sit/stand desk.  After doing the obligatory internet research, I decided on the Fundamentals EX Electric Desk.

After assembling, attaching the desktop computer holder, adding a new monitor mount to the rear of the desk, and some cabling work, here is what the finished product looks like:

Here are some additional assembly photos if you are interested:

The legs are surprisingly heavy and all parts are made of powdercoated steel.  The entire desk is quite heavy but has an extremely durable feel.

Standing Desk Parts
Standing Desk Parts

The work surface is predrilled- flanges bolt on to the sides of the legs which are then screwed into the underside of the desk.  The legs could be pretty much used which any surface, but the kit I ordered came with a nice work surface in a slightly off white color which is great for contrast with small parts that may end up on the desktop.

Mounting a desk leg
Mounting a desk leg

Hooking up the electronics- One of the two legs is the ‘control leg’ and links to both the controller and the other leg, as well as the power supply.  The first time the desk is plugged in, it must be ‘calibrated’ so the legs can determine their relative position and synchronize their height.

Desk with legs and electronics mounted
Desk with legs and electronics mounted

Overall, I’m quite happy with the result.  Currently, I’m standing about 20% of my workday, but I hope to increase that number continually until I’m up around 100% standing while working.

Tips for Working from Home

When the topic of profession comes up in friendly conversation, I often try to avoid disclosing that I work from home until all the other high level details of what I do on a day to day basis have been covered.  A large portion of the population seems to still believe that “Real Jobs” require you to load up into a metal box with wheels and transport yourself to an arbitrary location to sit in a cube in front of a computer all day.

An increasing number of employers are realizing that requiring employees to burn fuel and time to sit in a computer in an office that costs the company money to heat, cool, and clean is a waste of money.  If the evaluation of your employee’s contribution is based on the actual work they do rather than measuring the amount of time they warm a chair per day, working from home makes a lot of sense.  As a Senior Consultant for VMware, I write code to interconnect and automate customer systems with VMware products.  If a piece of VMware software doesn’t do what you want out of the box, we code functionality to add the features requested by customers.

The majority of this work is done in my house from my home workstation.  This blog post is a collection of tips and tricks I’ve accumulated that have drastically boosted my productivity and effectiveness in my home office- I hope you find them useful as well.

 The Internet Connection: More Important Than a Reliable Car


When working from home, your internet connection is critical.  If your connection is down,  you might not be able to work at all.  A fast, stable and reliable internet connection is a must, and in my opinion, you should also have a backup plan for outages.  Options for  a backup plan might include:

  • Buy a secondary internet connection (DSL, etc.) for use during outages
  • Work from an alternate location during an outage (Starbucks, etc.)
  • Work off cell phone tether, etc.
Secondary Internet Connection is a necessity

In my case, I use a USB 4G LTE modem attached directly to my router which enables internet connection failover.  The connection speed is less than ideal, but works in the rare event of an outage on my primary internet connection.



Synology DS 1813+

I primarily use my Synology DS1813+ as Network Attached Storage for my VMware vSphere home lab.  That said, it’s role in my home office has grown to include:

  • Syslog Server
  • CIFS (Windows File Share) Server
  • SMTP (Email) Relay Server
  • Media (DLNA) Server

So for use as a VMware storage target, what sets the Synology DS1813+ apart from your other run of the mill home NAS?  Here are the top features that have made it a killer device for my lab:

  • x3 1GB NICs, bonded via LACP for 3Gbps bandwidth
  • Support for SSD TRIM
  • Support for MTU 9000 (Jumbo Frames)
  • Support for Cloud Backup (Amazon Glacier)
  • Both iSCSI and NFS capabilities
  • Support for Active Directory / LDAP user accounts

As seen below, I’ve taken 3 of 4 NICs and created a LACP bond:


Synology DS1813+
Synology DS1813+

Another Home Lab

Hello and welcome to  My name is Justin Jones- I’m currently employed as a Senior Consultant in Integration and Automation at VMware.  This post documents my home lab build- however, I’ve decided to include a little twist on the standard home lab build.  As a remote employee, I work from home a great deal of the time.  This post includes tips and tricks I’ve found helpful in being a home office worker to make my work life easier and more productive.

Priorities (Use Case)

  1. Low Power (includes bonuses such as quiet, low heat emission, and lower electricity bill)
  2. High Capacity (Specifically Memory and Storage)
  3. High Performance

My home lab resides in my home office, where I spend a good deal of my workday on the phone.  Standard rackmount servers that sound like a hair dryer on full blast and put out an equivalent amount of heat were out of the question.  One popular solution I’ve seen is Mac Minis- I believe these are a pretty solid choice, but with a maximum of 16GB of RAM (from my observations Home Labs are typically memory constrained), I would need 6 Mac Minis to obtain ~96GB RAM capacity.

The ESXi Hosts

  • x3 Shuttle SH67H3 ($250 each)
    • Low Power i5 2400S CPUs ($200)
    • 32GB Memory ($325)
    • Dual Port PCI-E GB NIC (HP NC360T) ($40)
    • Cost per unit: about $800, mostly in RAM


  • Synology DS1813+ (8 bay, $1,000)
    • x4 Crucial M500 CT 960 SSDs – 960 GB -($500 each)
    • x4 Samsung HD204UI – 2 TB – ($125 each)

My lab predates the availability of vSAN.  If I was building a 3 node lab today, I’d give vSAN serious consideration-  for those that don’t have licenses available or for those who want a storage system for more than just VMs, I’d give a nice NAS like the Synology a try.  I’ll planning a separate post detailing its use.

During my day to day job, I provision probably 5 VMs a day on average,  and as many as 20-30+ on a heavy day.  This is because I write and test software integrations that modify VM pre and post build processes,  so part of debugging my code is frequently building a VM.  Yes, I do use linked clones in some cases, but sometimes code needs to be tested in ways that exactly reproduce client configurations, and linked clones cannot be used.  If you do the math, an average of 5 VMs per day is 25 a week, or 1300 per year.

That means shaving 1 minute off provisioning time equates to over 21 hours of time I get back not waiting to see if a code change fixed a bug over the course of a year- With that kind of time, 4 1TB SSDs in RAID 5 make a lot of sense 🙂


Low end commercial switches like PowerConnect and ProCurve can be had for less than $200 each- Due to my goals of low noise and low power consumption, I chose to go with the 2816/2808 due to them being fanless, low power, and compact.

  • Dell PowerConnect 2816
  • Dell PowerConnect 2808
  • Asus RT-N66U Router
  • Aris Cable Modem

Host Utilization


Think 96GB of RAM for a home lab is a lot?  It goes pretty quick:

Home Lab Host Utilization
Home Lab Host Utilization – click for zoom

And the Virtual Machines:

Home Lab VMs
Home Lab VMs – click for zoom

So, revisiting the originally stated goals, let’s take a look at power consumption.  I’m using a Belkin Conserve Insight to measure my power consumption.


Belkin Insight
Belkin Insight – click to zoom


261W Total Power Consumption, for a home lab with:

  • 29 GHz of CPU
  • 96 GB of RAM
  • 8TB of NAS Storage (2.5 TB of which is SSD)
  • 3 ESXi Hosts with x3 1Gb links each
  • 2 Gigabit switches Switches (16 port, 8 port)
  • Router
  • Cable Modem

All using less power than a 27″ iMac under full load.


Home Lab Photo
Home Lab Photo – click to zoom

Hope this post gives you some ideas, feel free to contact me if you have any questions!