Tuesday, April 22, 2014

Why Heartbleed Happened

Originally published on Kupeesh!


So what's up with all this HeartBleed nonsense?

What could possibly be behind the greatest crisis in Internet security since the invention of phishing emails?
How could this possibly happen?  What could possibly jeopardize the security of thousands of websites and secure services we take for granted like Google, Tumblr and even banking sites?

I have an easy answer and it points right back to the Achilles heel of Open Source. 

While proponents will argue the merits of solutions that don't come from commercial sources the one inescapable fact of Open Source software is that it's developed under mob rule.

Therein lies the problem. 

While nobody questions the benefits of Open Source software like cost and ease of customization, proponents tend to gloss over the fact that some projects are better managed than others.

Take the case of OpenSSL.  It's the foundation for thousands of web services like Google, Yahoo and even your bank.   Except that somebody wasn't minding the store and for two years the mechanism that was supposed to secure your communications...didn't.

The flaw was inadvertently discovered by Google's Neel Mehta during a routine security sweep but the flaw had been in existence for 2 years.  Overlooked by one of OpenSSL's core developers, Stephen N. Henson, the vulnerability came as the result of additional but apparently untested new functionality known as a Heartbeat for OpenSSL.  The functionality was supposed to function as little more than an "I'm still here!" beacon to whatever service you're connected to.  

The short of it is this...

The problem comes from not bothering to check that what's sent matches what was requested.  A crafty hacker can take advantage by continually sending heartbeat requests claiming to be of a certain size but not actually being that size.  The server dutifully responds by sending back a response of the claimed size to the client and inadvertently dumping the contents of its memory to fill the otherwise empty space of the response.  The contents of which have been shown to contain user credentials among other compromised information.

It's apparently a simple fix but it's taken two years for anyone to notice. 

Meanwhile, nobody knows how long the bad guys have been aware of the flaw.  How can something like this get by the supposed vigilance of security gurus and major corporations alike? 
I can tell you how, it's endemic, it's cultural and it's arrogance...

It's a misguided belief that oversight of a product is best left to a community regardless of its qualifications to do so.  A community that frequently finds itself more concerned with the technical wizardry of its products than the users who deploy them

It's the same mindset that's kept other Open Source offerings like Linux in the shadows of Windows.  Let's be honest here.  You can only stomach so many unintelligible whitepapers or narcissistic support forum posts before you just give up.  The inmates are indeed running the asylum...

Heartbleed shines a light on the failure of the Open Source community in that it lays open the lack of even the most basic oversight of a critical and widely used service.  It's not so much about the failure of OpenSSL but rather that nobody including its chief stewards noticed the problem for two years.


This is nothing less than a reality check on the entire Open Source community.  One that should be raising questions in anyone that relies on their wares.

Wednesday, February 5, 2014

An ESXi host, a NAS and NFS

When you're an IT guy tinkering is part of your lifestyle.  There has to be at least a modicum of curiosity about how things work.   We drive department heads crazy because just keeping stuff working isn't good enough for us.  We want that extra Megabit of throughput or another free Gigabyte out of the SAN.

Sadly, most of us can't afford to set up an server farm in a spare bedroom just to satisfy our need to tinker; but with virtualization you can come very close.

That is of course the promise.  Being able to harness the same capabilities of a small enterprise with a lot less hardware is undeniably a good thing.  Letting us run wild in our own little enterprise is even better.

VirtualBox, VMWare Workstation and their kind is fine for taking a new OS out for a spin but they fall a bit short for giving you real world skills.

ESXi, however, is another story.  Maybe more than any other platform, it's probably the most useful and relevant virtualization lab platform you can experiment with.  Don't confuse this with your grandpa's ESXi, though. 

Starting with version 5, VMWare decided that ESXi is the one hypervisor to rule them all instead of just being ESX's little brother.  That means anything you do with ESXi translates to what you can do in the enterprise.

It's one thing to play with virtual servers in VMWare but it's quite another to play with the platform itself.   After all, the more you tinker with it the better it works right?  Well, at least till we blow something up...

So this time around I decided I wasn't satisfied just locking myself out of the VSphere Client because I forgot the password.  I wanted to get some external storage online but I didn't have a spare ISCSI array laying around.   So I decided to venture into the wonderful world of NFS.

In ESXi you've basically got 2 options for storage.

1. Local - meaning it's either physically attached to the host or on a dedicated backbone via ISCSI

2. NFS - Which is pretty much "other"

Local's easy, if your storage controller can see it so can VMWare.  ISCSi  adds a wrinkle but so long as your target's on the network it's not a big deal.

NFS, ah, that's a different story.  A lot of Sys Admins can go an entire career without having to deal with it.   Like SMB, NFS is designed to offer up access to files on a network.  Those shares are usually hosted on Unix servers but unlike SMB, NFS is designed to fool your local PC into thinking a network resource is local.  An impressive feat compared to the clunky "Map Network Drive" or CLI "Net Use" commands in Windows.

Ok, so we know what NFS is but why do we care about it for VMWare?  Simple, most NAS storage devices will have support for the protocol to allow UNIX clients to access their shares.  Set up NFS on your NAS and you've given ESXi another potential datastore to play with.

NFS has it's quirks but most NAS management interfaces make it a relatively painless process to set up.  Once that happens just be sure you have a solid network connection between your virtual host(s) and the NAS.


Follow along with the video as I set up an NFS share for ESXi 5.5, play with a VM that lives on it and even break it!