The Geek Closet - Netfinity Project

I was rummaging around my geek closet....

You know you have one.  That dark little room crammed with discarded tech that you haven't touched in years but you "may have a use for someday."

In there I have a tapestry of my career in IT.  From bins of cables, cards and their associated connecting tissue to stacks of hardware long since retired to "someday" status.

Someday has come for you....

About 10 years ago while working for one of my clients during an upgrade project I purchased their soon to be unused and obsolete server hardware. 

I knew what I was getting into having actually installed these servers years before.  I had some vague ideas about what I'd do with them but nothing definitive.  

For all I knew they'd never be more than impromptu jack stands but I saw an opportunity even if I didn't know what fruit that would bear.

Within a few months I did manage to find a purpose for a few of them when I went into business with a friend of mine.  But after a year the business passed away into obscurity like so many others and they returned to the geek closet.

Until the other day.

I opened the geek closet and gazed upon these once mighty hunks of iron sadly sitting idle under stacks of similarly situated techno-cruft.

No, I needed, wanted, to do something with these servers and as luck would have it I was desperately searching for new content for my IT channel on YouTube.

Thus the Netfinity Project was born.  

It's a new video series that covers my attempt to re-purpose a couple of these old servers and gives you some insight into what I consider to be the golden age of hardware and IT.  From the late 90's to the early 2000's IT was all about the hardware and the only talk of "clouds" and "as a service" had to do with rain and valet parking. 

Hardware got better because the software demanded it.  Hardware is unquestionably better now but it lives in bland, clinical warehouses far from view.  An abstraction out of sight and out of mind.  

There's something sad about that.  I remember the pride of IT Manager's throwing open the doors of the server room.  This was where the magic happened, a tangible representation of the greatness of his enterprise.  Whirring, beeping, lights flashing and disks spinning.  You could almost feel the heartbeat of business within those cold server cases.

It's worth documenting and to some extent resurrecting if I can.

Thus the video series where I go through the highs and lows of making some old IBM Netfinity 5000's relevant again.

Check out this blog for regular updates on the video series.  The first few are below.  Enjoy!

Breaching your Security Britches

I'm still basking in the glow of the holiday season as I write this.  It's the day before New Year's eve and things are pretty quiet in the IT world.  On the IT jobs front there's a few listings all searching for the "impossible candidate" but most of them are just duplicates from agencies trying in vain to snap up those last few contracts before the Calendar ages another year.

IT budgets are still tight and salaries still aren't where they should be.  Of course if anything's increased,  it's the strain on IT staff.  It's a perfect recipe for disaster as expectations of the impossible become the norm. 

Case in point, the botched rollout of the Obamacare website.  Political motivations aside I knew it would fail.  Not because it's a bad idea but because like many corporate IT projects, nobody bothered to ask the IT guys.  It's a government venture after all, rife with bureaucratic red tape and too many layers of management.  None of which have any  clue about managing a successful IT project. 

To a public convinced that YouTube "just works" and the Internet requires nothing more than a WiFi connection  there's no further deliberation necessary.  The chant goes, "I want this, make it happen next week!"

It's a sad but common state of affairs.  IT departments are far too often under the purview of senior management with a ready, fire, aim philosophy and bad information.

All end users know is that they want more..."something" and increasingly, IT is in no position to say no.  There's even an accepted accreditation, the ITIL, that embraces the premise.  Give them what they want and to hell with the consequences even if you have to undermine the infrastructure to do it.

I can guarantee this is the root cause of most security breaches like those we saw with Target this year and Barnes and Noble in 2012.  They all stem from somebody giving in.  I can just see the exchange now...

IT Guy: You know, we really haven't updated the servers in 5 years and I'm worried about securing our customer data. BigBoxCo just got hacked last week and they've got the same stuff we do.

Accounting Supervisor (his boss): What? did the server's stop working? I got my email today and I was able to get to Ebay...

IT Guy:  No no, they're working fine but we're doing a lot of transactions and there's known vulnerabilities in our encryption algorithms.  We need to address this.

Accounting Supervisor (his boss):  Ok, but the servers are working right?

IT Guy: Yeah but that's not the point...

Accounting Supervisor (his boss): Well, do what you can with that, maybe you can fix it on your lunch break.  Just don't spend any money and for god's sake don't take down the servers for more than 10 minutes. Customers hate that!

IT Guy: Uh, ok but we don't have any failover so that's kind of impossible...

Accounting Supervisor (his boss): Oh, and lets relax those password requirements, I don't like changing it all the time and like to use my dog's name instead.  Maybe you can do that with the customer sites as well.

IT Guy: <sigh> Yeah....

I've had these conversations and they're more common than you think.  So guess who gets the blame when bad things happen. 

You can't have it both ways.  In our example above, the IT guy is right but that has to be balanced against the so-called "business case." 

Problem is the "business case" is often one-sided and incomplete.  That leaves plenty of opportunity for disaster.  It shows up in unexpected service outages, poor performance and workarounds that leave the door wide open for social engineering.

And that's the rub...

Look deep into the root cause of these high profile security breaches and you find out that somebody cut a corner.  It's human nature to want to make others happy.  So when faced with a painful and unnecessarily complex procedure that violates that desire, social engineering takes over.

"Ok, we'll relax the password policy for you"

 or

"Well we don't know much about how secure their servers are but this hosting provider is cheaper.  Oh yeah and they host porn sites so they must know what they're doing with all that traffic!"

Users let down their guard because the bad guys know your process and take advantage of its flaws.  IT guys let down their guard because they don't have the power to say no.  It's the same failing just expressed in different contexts. 

We have IT security vulnerabilities precisely because the way we interact with technology doesn't match up with our nature.  Human nature says to take the path of least resistance and 23 character passwords with mixed case, numbers and special symbols don't cut it.  Yeah, I know there's LastPass but that's a band-aid to the core problem.

So how do we secure anything in the face of all this opposition?

 It's simple, we stop thinking about "enforcing" anything.  Nobody likes to be under anyone's "enforcement."  Instead we start taking into account how people use technology instead of getting in the way of it with some clunky authentication mechanism.

While we're at it, why are we still using payment methods directly connected to bank and credit accounts?  Why aren't pre-paid instruments more popular?  I know the reason, they're a pain to use and like our 23 character password, nobody likes that much "resistance."

You now, it wasn't so long ago you could go to a store and buy things on credit. 

Not Visa or American Express credit, I'm talking about store credit.  You paid your bill every month directly to the store.  No personal information or bank accounts involved.  You just plopped down your money and you were done.

Of course we live in a world where we're forced to live beyond our means which has parlayed itself into ever increased complexity.  Banks and credit card companies have made millions based on the fact that nobody in business is willing to handle their own credit accounts if they even bother to have them.

So there's another feat of social engineering.  Another layer of abstraction between what we're trying to accomplish and what we ultimately DO accomplish. 

Somehow, we've managed to accept the ridiculous as a reasonable premise. 

That's exactly why nothing will change and security will ultimately fail simply because it's based on building a mountain of complexity where a bit of positive social engineering would do far better.

Of course there's too much money in that mountain of nonsense we keep adding to.  Entire industries owe their existence to it and nothing will change because of it.

So you have two choices, live like a hermit and pay cash for everything or accept that till somebody gets a clue there is no security or privacy.

If you need an example, try this...

Imagine you and your 5 year old daughter are at a restaurant for lunch.  A strange man approaches her and offers her candy.  Completely ignoring all your admonitions to the contrary she reaches for it.

What do you do?

It's likely you immediately intervene.  Depending on the threat it can range from tackling the guy to a dirty look.  Either way you took control of your own security concerns and it was a pretty simple process.  Nothing was going to happen without your direct involvement.

Put that in the context of how security works now, however, and you'd spend 10 minutes trying to remember your mother's maiden name and date of birth before you lifted a finger. 

Yeah, it's really that bad and exactly why security concerns in IT or otherwise need to be reframed.  All of these heaped on layers of band-aids and bailing wire are all for naught.  In the end we're not really securing anything.  How can we? We're never allowed to participate in the process. 

It's like the old joke where the man goes to the doctor and says, "Doc, it hurts when I do this!" and the only advice from the doctor is to say, " Then don't do that..." 

The more we remove human nature from the equation, the less meaning security has.

The Pitfalls of Virtualization - Part 1 A little history

At the start let me quiet your fears, I'm not going to bash virtualization or the cloud, they're great options.  I'd even go so far as to say they're fast becoming the de facto standard for dealing with your data.

First a little history because contrary to popular belief virtualization was not present at the Big Bang.

When virtualization was in its infancy the promise was great but the future not so certain.  There was more chance of your office PC running a Linux distro than I.T. trusting their infrastructure to a server that didn't have a physical off button.   Virtualization was immature and more often than not when something bad happened there was little chance of recovery.  Worse, compatibility problems with operating systems left many deployments relegated to a corner running a few instances of UBUNTU.  I.T. was still suffering from the implosion of the tech bubble and didn't need another reason to worry about their jobs. 

Sometime around 2007 virtualization became acceptable.  Corporate bean counters liked the idea of doing more with less.  Hardware and storage costs were falling and Virtualization gained credibility when large companies coming up on hardware refresh cycles decided to make the move. 

VMware, Microsoft and Citrix responded with their own offerings promising ease of administration, lower energy costs  and better use of hardware.  It wasn't uncommon, for example, for the average windows server to only be using 40% of its capacity at any time.  Virtualization promised to fix that.

How times have changed.  Now you're hard pressed to find server hardware dedicated to something other than virtualization in any organization of size.   Gone are the days of hovering around the lobby waiting for your RAID controller to show up so you can get your new Exchange server up and running.  Now, a few clicks, an ISO image and you can have a new server online in minutes. 

Unfortunately, virtualization is a victim of its own success.  The bean counters have become addicted to the whole concept of more with less resulting in more downward pressure on  I.T. budgets. 

As a result, more often than not when I run into an organization heavily dependent on virtualization the hardware is at least 5 years old and probably repurposed from something else.  I recently walked into a multimillion dollar company, for example,  that was relying on second hand hardware sourced from EBay to run their virtual server farm!

I'm all for recycling but no server is immortal regardless of the operating system it's running and after awhile hardware will start to fail.  With the rapid pace of Moore's law it's not uncommon to find parts availability for servers relegated to the secondary market within 2 years.   Server hardware still tends to be proprietary and unlike your home computer isn't available at Newegg.  If it's obsolete you roll the dice and hope that hot deal on EBay isn't for something worse than what you already have.

The promise of cheap or free frequently guts reason, however, forcing I.T. departments into less than best practices.

Continued in Part 2

The Pitfalls of Virtualization - Part 2 Virtual Realities!

So it seems that virtualization's benefits can be quickly negated by an overly zealous accounting department.

Still the benefits are considerable. 

The aforementioned leveraging of hardware resources, reduced power consumption and the ability to allocate resources on the fly are undeniable benefits.  We're not quite at plug and play, however, and virtualized environments introduce their own caveats. 

Take hardware compatibility for example.  It's actually more of an issue with virtual environments than physical.  Remember we're dealing with layers of abstraction between your operating system and the hardware. Since virtualization vendors know their product can end up on everything from a re-purposed desktop to server class hardware they know better than to t try to support every configuration.  That means you're likely to be on your own if your chosen platform isn't on their compatibility list.

If your chosen virtual platform doesn't know how to talk to your SAN adapter, for example,  you're at a standstill if it's not on the compatibility list.  Nothing like trolling forums for support while your Fortune 500 company waits.   The same can be said for physical servers but a virtual host usually serves more than one virtual machine which just added an unwanted exponent to your headache. 

It takes some time to figure out the nuances of managing a virtualized environment as well.  Keeping in mind that everything you're seeing is largely an artificial construct and not necessarily reality has found more than one administrator scratching his head.

Ignore that fact at your own peril as It's far too easy to over commit a virtual resource and suddenly find alarms because you've overtaxed your processor and evaporated your storage.   Oh yeah, and all those angry voicemails on your phone.

That brings up another annoyance, licensing.  

While VMWARE, for example, will allow you to have a fully functional virtual host ready to accept as many virtual machines as you can throw at it for free, scaling that up to enterprise level can be an exercise in futility. 

Just like Microsoft, figuring out what you need is never straightforward and usually involves engaging a consultant unless you like to pay for things you don't need.  I've yet to walk into a VMware shop that had the right licensing mostly because the IT director decided to just wing it.  Unfortunately that route usually means the loss of much of the functionality virtualization offers. 

Just for fun, I went online searching for licensing packs for VMware and found a dozen vendors selling 100 concurrent user licenses for $25000.  They all had the exact same description which told me nothing about the product aside from how much better my life would be should I make the purchase.  It makes me miss the days of shrink-wrapped software.  Back then, I didn't need a 5 figure consultant just to figure out how to spend my money!

It seems the more user friendly things get the more money I have to pay someone to explain it to me.

We wrap it up in Part 3

The Pitfalls of Virtualization - Part 3 The Cloud

So if you really don't want to deal with the pitfalls of your own virtual infrastructure you have the option to use someone else's.  

Yes, I'm talking about the cloud which promises unlimited potential so long as your internet connection is working.

Bean counters like the cloud too.  After all to them it's almost free.  No hardware costs, no support overhead and virtually no downtime just a monthly invoice.

That's the promise at least...

Far beyond simple cloud storage from services like Dropbox, software as a service and hosted services via the cloud offered cost savings over the traditional model of keeping it all onsite. 

The highest profile players in the space currently are Microsoft (of course) and Google.  Both are more than happy to rent you their infrastructure for a "nominal" fee. 

The early days of this kind of service tended to over promise and under deliver.  Outages, bankruptcy, vague Service Level Agreements (SLA's) and questionable security hindered adoption.  

Imagine a law firm storing its confidential client files with a cloud provider who suddenly goes out of business. 

A good System Admin knows better than to put all their eggs in one basket but the question of who owned the data in the cloud still remained.  Could they trust that the data would be returned or destroyed if the unlucky provider went under? 

Around the same time Software as a Service(SAS) vendors came along promising universal access to business applications via the cloud.  Data protection showed up via something called software escrow.  Software Escrow promised your data would be safe with a third party should something go wrong.  

Salesforce and Google docs were the first examples but because of the vagaries of their SLA's most businesses decided to stick with their local office suites from vendors like Microsoft. 

Speaking of Microsoft...

Seeing an opportunity to appease the bean counters in the face of resistance to  their ever increasing software licensing costs they came up with Office 365 and Windows Azure.  Moving responsibility for messaging and data to Microsoft's cloud not only reduced infrastructure costs but in some cases headcount.  Why have a legion of IT professionals when any problem could be solved with a phone call?

There's nothing wrong with the logic but sometimes the execution can leave something to be desired.  Salesforce seems to have an outage at least once a year  and Microsoft Office 365 users have found themselves relying on smartphone messaging when hosted exchange servers go MIA.  Lest we forget the troubles with Google's cloud services. 

The hidden costs of cloud services have to come into play at some point.  Nothing's free as many a surprised supervisor has found when faced with a bill for his users going over their mailbox limit.  The purported cost savings in the server room can quickly be offset by the subscription model employed by cloud providers.

Just because I.T. services have moved out of the office and into the datacenter doesn't mean you don't have to pay for them. 

Cloud providers usually have tiered SLA offerings which means how fast they deal with an issue is directly related to how much you're paying.  If it's a system wide issue the SLA goes out the window.

Of course nothing's perfect and highlighting the flaws is no condemnation.  For the most part cloud services have lived up to their claims.  Like anything else the wise IT Pro knows not to rely on anything  exclusively.   Google docs offline? Work with a local copy.   Hosted Exchange services down?  Chances are you have more than one email account available to you elsewhere.

At this point the bloom is off the rose.  Virtualization is approaching the ubiquitous and there's no turning back.  Chances are at least some of the applications you work with every day have at least some portion living in the cloud.

That's not a bad thing just know that it's not the only thing.  As the old saying goes don't put all your eggs in one basket.  

Maintenance Windows

Subtle admonitions, Strict adherence to SLA's or boldface demands...

Ever try to convince an entire enterprise that you need to take down their servers for a weekend?  There's always resistance and even if you get your time window somebody's going to complain that they can't get to their stuff. 

Maybe somebody higher up in the organization will make you postpone your maintenance window just because they can.  "No, we can't send our satellite office of 3 people home an hour early.  It will impact our performance!"

Ah, office politics.  The great monkey wrench...

It's understandable in this day and age of 24/7 everything that users expect zero downtime.  That's reasonable given ideal circumstances. My experience has yet to show me an enterprise where that ideal exists.

In fact, it's impossible unless the enterprise is based on IT.  Think online universities or Large software companies.  Unless IT is at the core of the business it's not a priority.

Strangely enough, IT is at the core of most businesses whether the business knows it or not.  Your users just take it for granted.  "It worked yesterday so it'll work tomorrow so there's no need to inconvenience me."

There's a few approaches to deal with this.

You can just ignore the necessary maintenance and wait for something to blow up.
Then you get all the time you need to take care of things.  The downside is you're probably going to lose a weekend, the department will be blamed for being incompetent and somebody's going to get shown the exit.

You can force extensions to maintenance window by ignoring the predetermined time limits but you won't make too many friends and that exit door is likely to be in your future.

You can make the argument that 24/7 availability is unrealistic without putting resources in place to make it possible.  That's reasonable but will likely fall on deaf ears.

Seems like a no-win scenario. 

Unless you can get support from someone other than your IT director it is.

Unreasonable maintenance windows and a lack of proper resources is a systematic problem outside of your ability as an IT professional to fix. 

The fact is, if you're in an organization that won't allocate resources to meet demands then you need to get out.  It's really that simple so don't overthink it.

Discovery Channel's Mythbusters may have been able to successfully polish cow patties but in the end they were still cow patties.  Take a lesson from that...