Why Heartbleed Happened

Originally published on Kupeesh!

So what's up with all this HeartBleed nonsense?

What could possibly be behind the greatest crisis in Internet security since the invention of phishing emails?

How could this possibly happen?  What could possibly jeopardize the security of thousands of websites and secure services we take for granted like Google, Tumblr and even banking sites?

I have an easy answer and it points right back to the Achilles heel of Open Source. 

While proponents will argue the merits of solutions that don't come from commercial sources the one inescapable fact of Open Source software is that it's developed under mob rule.

Therein lies the problem. 

While nobody questions the benefits of Open Source software like cost and ease of customization, proponents tend to gloss over the fact that some projects are better managed than others.

Take the case of OpenSSL.  It's the foundation for thousands of web services like Google, Yahoo and even your bank.   Except that somebody wasn't minding the store and for two years the mechanism that was supposed to secure your communications...didn't.

The flaw was inadvertently discovered by Google's Neel Mehta during a routine security sweep but the flaw had been in existence for 2 years.  Overlooked by one of OpenSSL's core developers, Stephen N. Henson, the vulnerability came as the result of additional but apparently untested new functionality known as a Heartbeat for OpenSSL.  The functionality was supposed to function as little more than an "I'm still here!" beacon to whatever service you're connected to.  

The short of it is this...

The problem comes from not bothering to check that what's sent matches what was requested.  A crafty hacker can take advantage by continually sending heartbeat requests claiming to be of a certain size but not actually being that size.  The server dutifully responds by sending back a response of the claimed size to the client and inadvertently dumping the contents of its memory to fill the otherwise empty space of the response.  The contents of which have been shown to contain user credentials among other compromised information.

It's apparently a simple fix but it's taken two years for anyone to notice. 

Meanwhile, nobody knows how long the bad guys have been aware of the flaw.  How can something like this get by the supposed vigilance of security gurus and major corporations alike? 

I can tell you how, it's endemic, it's cultural and it's arrogance...

It's a misguided belief that oversight of a product is best left to a community regardless of its qualifications to do so.  A community that frequently finds itself more concerned with the technical wizardry of its products than the users who deploy them

It's the same mindset that's kept other Open Source offerings like Linux in the shadows of Windows.  Let's be honest here.  You can only stomach so many unintelligible whitepapers or narcissistic support forum posts before you just give up.  The inmates are indeed running the asylum...

Heartbleed shines a light on the failure of the Open Source community in that it lays open the lack of even the most basic oversight of a critical and widely used service.  It's not so much about the failure of OpenSSL but rather that nobody including its chief stewards noticed the problem for two years.

This is nothing less than a reality check on the entire Open Source community.  One that should be raising questions in anyone that relies on their wares.

Microsoft's pot calls out Google's kettle

Article first published as Microsoft's Pot Calls Out Google's Kettle on Technorati.

A friend of mine sent me a few links to a Microsoft blog where the company's legal department apparently has issues with the recent FTC ruling on Google's alleged anti-competitive practices. I could write paragraphs of babble but I think a direct quote sums up Microsoft's position best.

"Unfortunately, this agreement appears to be less demanding than the pledge the U.S. Department of Justice received from Apple and Microsoft nearly a year ago."

Microsoft's ruffled feathers come from Google's apparent blocking of the Microsoft YouTube app from Windows phones going back as far as 2010.  A service available to Apple and Android users by the way.  Again we'll let a direct quote tell the story...

"Google often says that the antitrust offenses with which it has been charged cause no harm to consumers. Google is wrong about that. In this instance, for example, Google’s refusal deprives consumers who use competing platforms of a comparable experience in accessing content that is generally available on the Web"

A more classic case of a jealous pot bashing the upstart kettle there has never been.  Replace "Google" with "Microsoft" in the preceding quote and it's 1996 all over again.  And there's the rub.  Microsoft doesn't think Google is getting punished enough and has somehow equated an inferior YouTube experience on Windows phones with Google's apparent monopoly on Internet search.

" We are concerned that the FTC may not have obtained adequate relief even on the few subjects that Google has agreed to address."

Microsoft appears to want to link two largely unrelated topics in their complaint which in spite of their claims to the contrary equate to little more than sour grapes. Microsoft's assertions bank on the short memory of the Internet and consumers in general.  Ironically a Google search can correct that.

Remember that the FTC went after Microsoft not because they had the leading operating system but because they abused their market position with it.  The tight integration of Internet Explorer with Windows operating systems starting with Windows 95 made competing browsers inferior on the platform.  It also allowed Microsoft to claim that the Browser and the Operating System were too closely interrelated to expose the details to competing browsers.  To do so would allegedly expose their trade secrets.  The FTC didn't buy it.

As for search, if Microsoft Bing is damaged by Google's dominant position in search and Google's related products it has less to do with anti-competitive business practices than it does with poor execution.   Let's be honest, nobody goes around saying, "I'll Bing that." Google's about search and has managed to leverage that simple fact in all of its products.  Bing isn't a poor search engine because of Google, It's just a poor search engine.

Consumer preference can be influenced but not controlled.  That's why Bing has been and always will be an also ran search engine just like Yahoo and the bygones like altavista or askjeeves.  Windows Phone still holds less than 5% of the worldwide Smartphone market and will likely continue to do so until it offers the features that can lure consumers away from Apple or Android.

Neither Google nor Apple have any obligation to help them improve that number.  A fact driven home by Google's impending removal of support for syncing Microsoft Exchange accounts (via Exchange Activesync) to its Gmail service as of January 30th.  Luckily Microsoft has it's hotmail service to fall back on.  Yes, that's a joke.

The bottom line is that right now Google's services are the consumer's choice for web applications.  Apple learned that the hard way with the Apple Maps fiasco but to their credit at least they tried.  Instead of complaining about being excluded from Google's services Microsoft should be trying to create a better competing service.  After all, software and servers are their thing, right?

Then again what else would you expect from a Microsoft legal blog?

The Pitfalls of Virtualization - Part 1 A little history

At the start let me quiet your fears, I'm not going to bash virtualization or the cloud, they're great options.  I'd even go so far as to say they're fast becoming the de facto standard for dealing with your data.

First a little history because contrary to popular belief virtualization was not present at the Big Bang.

When virtualization was in its infancy the promise was great but the future not so certain.  There was more chance of your office PC running a Linux distro than I.T. trusting their infrastructure to a server that didn't have a physical off button.   Virtualization was immature and more often than not when something bad happened there was little chance of recovery.  Worse, compatibility problems with operating systems left many deployments relegated to a corner running a few instances of UBUNTU.  I.T. was still suffering from the implosion of the tech bubble and didn't need another reason to worry about their jobs. 

Sometime around 2007 virtualization became acceptable.  Corporate bean counters liked the idea of doing more with less.  Hardware and storage costs were falling and Virtualization gained credibility when large companies coming up on hardware refresh cycles decided to make the move. 

VMware, Microsoft and Citrix responded with their own offerings promising ease of administration, lower energy costs  and better use of hardware.  It wasn't uncommon, for example, for the average windows server to only be using 40% of its capacity at any time.  Virtualization promised to fix that.

How times have changed.  Now you're hard pressed to find server hardware dedicated to something other than virtualization in any organization of size.   Gone are the days of hovering around the lobby waiting for your RAID controller to show up so you can get your new Exchange server up and running.  Now, a few clicks, an ISO image and you can have a new server online in minutes. 

Unfortunately, virtualization is a victim of its own success.  The bean counters have become addicted to the whole concept of more with less resulting in more downward pressure on  I.T. budgets. 

As a result, more often than not when I run into an organization heavily dependent on virtualization the hardware is at least 5 years old and probably repurposed from something else.  I recently walked into a multimillion dollar company, for example,  that was relying on second hand hardware sourced from EBay to run their virtual server farm!

I'm all for recycling but no server is immortal regardless of the operating system it's running and after awhile hardware will start to fail.  With the rapid pace of Moore's law it's not uncommon to find parts availability for servers relegated to the secondary market within 2 years.   Server hardware still tends to be proprietary and unlike your home computer isn't available at Newegg.  If it's obsolete you roll the dice and hope that hot deal on EBay isn't for something worse than what you already have.

The promise of cheap or free frequently guts reason, however, forcing I.T. departments into less than best practices.

Continued in Part 2

The Pitfalls of Virtualization - Part 2 Virtual Realities!

So it seems that virtualization's benefits can be quickly negated by an overly zealous accounting department.

Still the benefits are considerable. 

The aforementioned leveraging of hardware resources, reduced power consumption and the ability to allocate resources on the fly are undeniable benefits.  We're not quite at plug and play, however, and virtualized environments introduce their own caveats. 

Take hardware compatibility for example.  It's actually more of an issue with virtual environments than physical.  Remember we're dealing with layers of abstraction between your operating system and the hardware. Since virtualization vendors know their product can end up on everything from a re-purposed desktop to server class hardware they know better than to t try to support every configuration.  That means you're likely to be on your own if your chosen platform isn't on their compatibility list.

If your chosen virtual platform doesn't know how to talk to your SAN adapter, for example,  you're at a standstill if it's not on the compatibility list.  Nothing like trolling forums for support while your Fortune 500 company waits.   The same can be said for physical servers but a virtual host usually serves more than one virtual machine which just added an unwanted exponent to your headache. 

It takes some time to figure out the nuances of managing a virtualized environment as well.  Keeping in mind that everything you're seeing is largely an artificial construct and not necessarily reality has found more than one administrator scratching his head.

Ignore that fact at your own peril as It's far too easy to over commit a virtual resource and suddenly find alarms because you've overtaxed your processor and evaporated your storage.   Oh yeah, and all those angry voicemails on your phone.

That brings up another annoyance, licensing.  

While VMWARE, for example, will allow you to have a fully functional virtual host ready to accept as many virtual machines as you can throw at it for free, scaling that up to enterprise level can be an exercise in futility. 

Just like Microsoft, figuring out what you need is never straightforward and usually involves engaging a consultant unless you like to pay for things you don't need.  I've yet to walk into a VMware shop that had the right licensing mostly because the IT director decided to just wing it.  Unfortunately that route usually means the loss of much of the functionality virtualization offers. 

Just for fun, I went online searching for licensing packs for VMware and found a dozen vendors selling 100 concurrent user licenses for $25000.  They all had the exact same description which told me nothing about the product aside from how much better my life would be should I make the purchase.  It makes me miss the days of shrink-wrapped software.  Back then, I didn't need a 5 figure consultant just to figure out how to spend my money!

It seems the more user friendly things get the more money I have to pay someone to explain it to me.

We wrap it up in Part 3

The Pitfalls of Virtualization - Part 3 The Cloud

So if you really don't want to deal with the pitfalls of your own virtual infrastructure you have the option to use someone else's.  

Yes, I'm talking about the cloud which promises unlimited potential so long as your internet connection is working.

Bean counters like the cloud too.  After all to them it's almost free.  No hardware costs, no support overhead and virtually no downtime just a monthly invoice.

That's the promise at least...

Far beyond simple cloud storage from services like Dropbox, software as a service and hosted services via the cloud offered cost savings over the traditional model of keeping it all onsite. 

The highest profile players in the space currently are Microsoft (of course) and Google.  Both are more than happy to rent you their infrastructure for a "nominal" fee. 

The early days of this kind of service tended to over promise and under deliver.  Outages, bankruptcy, vague Service Level Agreements (SLA's) and questionable security hindered adoption.  

Imagine a law firm storing its confidential client files with a cloud provider who suddenly goes out of business. 

A good System Admin knows better than to put all their eggs in one basket but the question of who owned the data in the cloud still remained.  Could they trust that the data would be returned or destroyed if the unlucky provider went under? 

Around the same time Software as a Service(SAS) vendors came along promising universal access to business applications via the cloud.  Data protection showed up via something called software escrow.  Software Escrow promised your data would be safe with a third party should something go wrong.  

Salesforce and Google docs were the first examples but because of the vagaries of their SLA's most businesses decided to stick with their local office suites from vendors like Microsoft. 

Speaking of Microsoft...

Seeing an opportunity to appease the bean counters in the face of resistance to  their ever increasing software licensing costs they came up with Office 365 and Windows Azure.  Moving responsibility for messaging and data to Microsoft's cloud not only reduced infrastructure costs but in some cases headcount.  Why have a legion of IT professionals when any problem could be solved with a phone call?

There's nothing wrong with the logic but sometimes the execution can leave something to be desired.  Salesforce seems to have an outage at least once a year  and Microsoft Office 365 users have found themselves relying on smartphone messaging when hosted exchange servers go MIA.  Lest we forget the troubles with Google's cloud services. 

The hidden costs of cloud services have to come into play at some point.  Nothing's free as many a surprised supervisor has found when faced with a bill for his users going over their mailbox limit.  The purported cost savings in the server room can quickly be offset by the subscription model employed by cloud providers.

Just because I.T. services have moved out of the office and into the datacenter doesn't mean you don't have to pay for them. 

Cloud providers usually have tiered SLA offerings which means how fast they deal with an issue is directly related to how much you're paying.  If it's a system wide issue the SLA goes out the window.

Of course nothing's perfect and highlighting the flaws is no condemnation.  For the most part cloud services have lived up to their claims.  Like anything else the wise IT Pro knows not to rely on anything  exclusively.   Google docs offline? Work with a local copy.   Hosted Exchange services down?  Chances are you have more than one email account available to you elsewhere.

At this point the bloom is off the rose.  Virtualization is approaching the ubiquitous and there's no turning back.  Chances are at least some of the applications you work with every day have at least some portion living in the cloud.

That's not a bad thing just know that it's not the only thing.  As the old saying goes don't put all your eggs in one basket.  

SOPA Gives Rise to the Voice of the Internet Community

Article first published as SOPA Gives Rise to the Voice of the Internet Community on Technorati.

January 18th, 2012 found an Internet community that may have finally shook off its timid boy in the basement image.  This was the day when browsing to Google.com found the logo covered with black duct tape and Wikipedia showing an anti-SOPA legislation page instead of that article about Carrie Nation that you were looking for.

Countless Internet websites and content providers took up arms, so to speak, and expressed unity with the movement.  Google offered the opportunity to sign a petition against SOPA/PIPA and had 4.5 million signatures in 24 hours. 

Twit.tv broadcasted all of their programming in black and white, Wired.com censored their own content with black bars reminiscent of declassified documents like those seen on investigative news shows.

Information from proponents of the legislation cite the need to stop piracy and preserve intellectual property rights.  Those against claim dangerous ambiguity in the wording and technical issues that would essentially put the U.S. government in the role of a proxy to your Internet browsing and break a number of current and future security measures in the process.  Arguments against also highlight the ineffectiveness the measures would have on stopping piracy as well as degradation of the user experience.

Former Senator Chris Dodd and current MPAA CEO  has called the actions of participating web content providers a "gimmick" with further commentary reminiscent of the former President Bush's "You're either with us or you're with the terrorists" rhetoric.

Sweeping measures to combat illicit activity aren't new to U.S. politics.  With ax-handle diplomacy Carrie nation set the stage for the 18th Amendment to the U.S. constitution.  It took the 21st amendment to attempt to reverse the resulting rise in organized crime and alcohol related deaths related to dubious sources of bootleg alcohol. 

It's an issue not well suited to the 30 second sound bite regardless of how traditional media chooses to frame it.   The devil is in the details and it is the details that have opponents to SOPA/PIPA up in arms.

More information is available from a number of sources too numerous to list here but a simple google search, a viewing of any of Twit.tv's programs from January 18th or a visit to Tim O'reilly's google plus page are good places to start.