The one about how "Nothing is Virtual"

Nothing is Virtual because everything eventually makes the journey down to what I call bits on metal. At some point every virtual machine, every hypervisor, and every network becomes real bits represented by real signals traveling on real fiber and real wires.

Anyone who's worked with me for a while knows that I am never content with merely a working theory of how a system works. I have an implicit desire to look at the system's fundamentals and what it is that under lies an application. I've had the good fortune to have been given a solid grounding in computing theory and practical application early in my career and this lets me get beneath the surface issues most developers focus on.

This is one of the reasons I ended up hacking the Linux kernel, working on drivers, and understanding virtualization technologies even though I was a "web developer", "database administrator", or some other stripe of application developer. Knowing what happened beneath the application made me better at the game of creating working applications that could scale and troubleshooting applications in the wild. It's one of the reasons I was able to work as a consultant, architect, and troubleshooter while also able to work as a coder.

When I look at OpenStack's current state of affairs I see a system that could truly benefit from some deeper insight into the assumptions the technologies used in its development make. For example, OpenStack makes assumptions about Python which lead to a set of assumptions about its supporting C libraries. These two tiers of assumptions relate to real world operational behaviors that only cloud administrators would ever really have to come to terms with in real practice.

One example is the SSL certificate validation behaviors I wrote about previously. OpenStack assumes that Python will use a trusting model when accepting SSL certificates by default. This is built on assumptions about the C libraries that Python sits on top of. Some of these C libraries in turn have assumptions about the environment variables on the OS they sit on top of. Those assumptions might include hidden variables (hidden from the Python scripts' perspective) that toggle behavior in ways that the cloud administrator might be surprised at.

Another example of OpenStack making assumptions that may not hold is here in a patch (see link) I've submitted for Nova. The Python makes assumptions about how the OS will expose it's mount points. These are specific assumptions made for Linux which currently exposes mount points via a file system interface. The technique used in Linux is nice matched to the Unix Philosophy but it is a specific assumption that limits OpenStack Nova to only functioning on Linux distributions that honor this convention (I know of none that do not honor this right now, but some of us remember the 1990's and how divergent some Linux distros could get.)

As a Java developer and Java architect understanding the JVM and how it interacted with Hypervisors made me better at understanding how my PaaS hosted applications would scale on top of the IaaS infrastructure they were hosted on. In OpenStack, I'm looking to find the same insights to make Nova better. That's because Nothing is Virtual.

So what I'm looking for now in my OpenStack work are the ways in which the hypothetical and virtual components interact with what's really real. This includes the details of how the various Nova hypervisors interact with their real hardware, what assumptions they make about the real infrastructure and how those assumptions affect scale and fitness of the platform.

This is important because nothing is virtual and IaaS suffers from a lack of understanding this principle as much as anything else.