The one about how "Nothing is Virtual"

Nothing is Virtual because everything eventually makes the journey down to what I call bits on metal. At some point every virtual machine, every hypervisor, and every network becomes real bits represented by real signals traveling on real fiber and real wires.

Anyone who's worked with me for a while knows that I am never content with merely a working theory of how a system works. I have an implicit desire to look at the system's fundamentals and what it is that under lies an application. I've had the good fortune to have been given a solid grounding in computing theory and practical application early in my career and this lets me get beneath the surface issues most developers focus on.

This is one of the reasons I ended up hacking the Linux kernel, working on drivers, and understanding virtualization technologies even though I was a "web developer", "database administrator", or some other stripe of application developer. Knowing what happened beneath the application made me better at the game of creating working applications that could scale and troubleshooting applications in the wild. It's one of the reasons I was able to work as a consultant, architect, and troubleshooter while also able to work as a coder.

When I look at OpenStack's current state of affairs I see a system that could truly benefit from some deeper insight into the assumptions the technologies used in its development make. For example, OpenStack makes assumptions about Python which lead to a set of assumptions about its supporting C libraries. These two tiers of assumptions relate to real world operational behaviors that only cloud administrators would ever really have to come to terms with in real practice.

One example is the SSL certificate validation behaviors I wrote about previously. OpenStack assumes that Python will use a trusting model when accepting SSL certificates by default. This is built on assumptions about the C libraries that Python sits on top of. Some of these C libraries in turn have assumptions about the environment variables on the OS they sit on top of. Those assumptions might include hidden variables (hidden from the Python scripts' perspective) that toggle behavior in ways that the cloud administrator might be surprised at.

Another example of OpenStack making assumptions that may not hold is here in a patch (see link) I've submitted for Nova. The Python makes assumptions about how the OS will expose it's mount points. These are specific assumptions made for Linux which currently exposes mount points via a file system interface. The technique used in Linux is nice matched to the Unix Philosophy but it is a specific assumption that limits OpenStack Nova to only functioning on Linux distributions that honor this convention (I know of none that do not honor this right now, but some of us remember the 1990's and how divergent some Linux distros could get.)

As a Java developer and Java architect understanding the JVM and how it interacted with Hypervisors made me better at understanding how my PaaS hosted applications would scale on top of the IaaS infrastructure they were hosted on. In OpenStack, I'm looking to find the same insights to make Nova better. That's because Nothing is Virtual.

So what I'm looking for now in my OpenStack work are the ways in which the hypothetical and virtual components interact with what's really real. This includes the details of how the various Nova hypervisors interact with their real hardware, what assumptions they make about the real infrastructure and how those assumptions affect scale and fitness of the platform.

This is important because nothing is virtual and IaaS suffers from a lack of understanding this principle as much as anything else.

The one about SSL and signed certificates

Greetings Stackers!

Here's an example of a tiny thing that can ruin your day if you aren't paying close attention. 

• in nova.conf the option ssl.ca_file switches on strict SSL certificate checking

Suppose you have signed certs everywhere except for one place. Well, you're gonna have a bad day, 'cuz you can only pick *all* certs need to be signed by a CA or only *no* certs need to be signed with a CA in Nova. So, just be aware.

I've also found this example https://gist.github.com/ssbarnea/8007689 (backup here) that shows how setting global environment variables can change the behavior of libraries underneath Python without touching anything in Python itself. That means you can change Python's strictness about SSL globally without necessarily exposing any visible control anywhere within OpenStack's configurations.

You've been warned.

Keyword spam to see if this post can get into a search engine:

Verify return code: 21 (unable to verify the first certificate)
nova-compute
vCenter unsigned SSL certificate

clouds bleeding context

In OpenStack havana we had a bug that did not affect functionality, but it bothered me a great deal. In OpenStack grizzly when the Nova Compute driver for VMware's management API would create a new instance (aka virtual machine) it would name the VM in vSphere's API based on the pattern instance-000001 where the number portion of the instance's name was the row number that the instance record was represented by in the Nova database.

There's several really glaring problems here. First, it violates Nova rather blatantly. We were taking a value that you could never utilize in the exposed Nova API and flagrantly exposing it to the world. That's a major violation of Nova's internal implementation details.

Fortunately in Havana we managed to solve this problem and the vSphere name is based on a value that is exposed by Nova. The UUID for the instance is now the vSphere name. We kept the vestigial code that did lookups based on the pattern /instance-\d+/ around just for people doing Nova upgrades. The result is that at least Nova's internals don't bleed into vSphere.

The bug I've filed today is related to the fact that in vSphere the name attribute of a guest virtual machine is not immutable. If a VI Admin edits the instance name with the initially released Havana version of the driver that admin can break their OpenStack automation. Ideally, this should never be a problem since the Nova Compute manager should be able to accomplish all the tasks that a VM administrator or tenant need, but in practice this is likely to be a sore spot with hybrid cloud administrators working in cloud environments that are both OpenStack and vSphere and managed simultaneously through tools like Power CLI or RVC which will allow modification of the name.

Hopefully we'll get a fix that can be applied that will make the linkage between Nova Compute and vSphere's admin tools more robust but until then, there are cloud details that bleed through each layer and that's a sign of possible trouble in design, implementation, and administration.

I'm just trying to watch out for that admin with too many hats to wear.