Monday, December 30, 2013

Breaching your Security Britches

I'm still basking in the glow of the holiday season as I write this.  It's the day before New Year's eve and things are pretty quiet in the IT world.  On the IT jobs front there's a few listings all searching for the "impossible candidate" but most of them are just duplicates from agencies trying in vain to snap up those last few contracts before the Calendar ages another year.

IT budgets are still tight and salaries still aren't where they should be.  Of course if anything's increased,  it's the strain on IT staff.  It's a perfect recipe for disaster as expectations of the impossible become the norm. 

Case in point, the botched rollout of the Obamacare website.  Political motivations aside I knew it would fail.  Not because it's a bad idea but because like many corporate IT projects, nobody bothered to ask the IT guys.  It's a government venture after all, rife with bureaucratic red tape and too many layers of management.  None of which have any  clue about managing a successful IT project. 

To a public convinced that YouTube "just works" and the Internet requires nothing more than a WiFi connection  there's no further deliberation necessary.  The chant goes, "I want this, make it happen next week!"

It's a sad but common state of affairs.  IT departments are far too often under the purview of senior management with a ready, fire, aim philosophy and bad information.

All end users know is that they want more..."something" and increasingly, IT is in no position to say no.  There's even an accepted accreditation, the ITIL, that embraces the premise.  Give them what they want and to hell with the consequences even if you have to undermine the infrastructure to do it.

I can guarantee this is the root cause of most security breaches like those we saw with Target this year and Barnes and Noble in 2012.  They all stem from somebody giving in.  I can just see the exchange now...

IT Guy: You know, we really haven't updated the servers in 5 years and I'm worried about securing our customer data. BigBoxCo just got hacked last week and they've got the same stuff we do.
Accounting Supervisor (his boss): What? did the server's stop working? I got my email today and I was able to get to Ebay...
IT Guy:  No no, they're working fine but we're doing a lot of transactions and there's known vulnerabilities in our encryption algorithms.  We need to address this.
Accounting Supervisor (his boss):  Ok, but the servers are working right?
IT Guy: Yeah but that's not the point...
Accounting Supervisor (his boss): Well, do what you can with that, maybe you can fix it on your lunch break.  Just don't spend any money and for god's sake don't take down the servers for more than 10 minutes. Customers hate that!
IT Guy: Uh, ok but we don't have any failover so that's kind of impossible...
Accounting Supervisor (his boss): Oh, and lets relax those password requirements, I don't like changing it all the time and like to use my dog's name instead.  Maybe you can do that with the customer sites as well.
IT Guy: <sigh> Yeah....
I've had these conversations and they're more common than you think.  So guess who gets the blame when bad things happen. 

You can't have it both ways.  In our example above, the IT guy is right but that has to be balanced against the so-called "business case." 

Problem is the "business case" is often one-sided and incomplete.  That leaves plenty of opportunity for disaster.  It shows up in unexpected service outages, poor performance and workarounds that leave the door wide open for social engineering.

And that's the rub...

Look deep into the root cause of these high profile security breaches and you find out that somebody cut a corner.  It's human nature to want to make others happy.  So when faced with a painful and unnecessarily complex procedure that violates that desire, social engineering takes over.

"Ok, we'll relax the password policy for you"


"Well we don't know much about how secure their servers are but this hosting provider is cheaper.  Oh yeah and they host porn sites so they must know what they're doing with all that traffic!"

Users let down their guard because the bad guys know your process and take advantage of its flaws.  IT guys let down their guard because they don't have the power to say no.  It's the same failing just expressed in different contexts. 

We have IT security vulnerabilities precisely because the way we interact with technology doesn't match up with our nature.  Human nature says to take the path of least resistance and 23 character passwords with mixed case, numbers and special symbols don't cut it.  Yeah, I know there's LastPass but that's a band-aid to the core problem.

So how do we secure anything in the face of all this opposition?

 It's simple, we stop thinking about "enforcing" anything.  Nobody likes to be under anyone's "enforcement."  Instead we start taking into account how people use technology instead of getting in the way of it with some clunky authentication mechanism.

While we're at it, why are we still using payment methods directly connected to bank and credit accounts?  Why aren't pre-paid instruments more popular?  I know the reason, they're a pain to use and like our 23 character password, nobody likes that much "resistance."

You now, it wasn't so long ago you could go to a store and buy things on credit. 
Not Visa or American Express credit, I'm talking about store credit.  You paid your bill every month directly to the store.  No personal information or bank accounts involved.  You just plopped down your money and you were done.

Of course we live in a world where we're forced to live beyond our means which has parlayed itself into ever increased complexity.  Banks and credit card companies have made millions based on the fact that nobody in business is willing to handle their own credit accounts if they even bother to have them.

So there's another feat of social engineering.  Another layer of abstraction between what we're trying to accomplish and what we ultimately DO accomplish. 

Somehow, we've managed to accept the ridiculous as a reasonable premise. 

That's exactly why nothing will change and security will ultimately fail simply because it's based on building a mountain of complexity where a bit of positive social engineering would do far better.

Of course there's too much money in that mountain of nonsense we keep adding to.  Entire industries owe their existence to it and nothing will change because of it.

So you have two choices, live like a hermit and pay cash for everything or accept that till somebody gets a clue there is no security or privacy.

If you need an example, try this...

Imagine you and your 5 year old daughter are at a restaurant for lunch.  A strange man approaches her and offers her candy.  Completely ignoring all your admonitions to the contrary she reaches for it.
What do you do?

It's likely you immediately intervene.  Depending on the threat it can range from tackling the guy to a dirty look.  Either way you took control of your own security concerns and it was a pretty simple process.  Nothing was going to happen without your direct involvement.

Put that in the context of how security works now, however, and you'd spend 10 minutes trying to remember your mother's maiden name and date of birth before you lifted a finger. 

Yeah, it's really that bad and exactly why security concerns in IT or otherwise need to be reframed.  All of these heaped on layers of band-aids and bailing wire are all for naught.  In the end we're not really securing anything.  How can we? We're never allowed to participate in the process. 

It's like the old joke where the man goes to the doctor and says, "Doc, it hurts when I do this!" and the only advice from the doctor is to say, " Then don't do that..." 

The more we remove human nature from the equation, the less meaning security has.

Monday, September 30, 2013

IT jobs and out of state recruiters

It would be completely legitimate to mistake my adventures in IT job hunting for little more than thinly veiled hit pieces.  In a way they are but only to illustrate the antics of the bad players. 

Reality check!  You really need to take off the rose colored glasses and stop taking job-seeking advice from 1963.

Make no mistake, the people you're interviewing with are not your, "Friends, Romans, Countrymen.." or anyone else with your best interests at heart.  

They're in it for them so you'd better know what's in it for you because giving your 2 weeks notice 3 weeks into the job is too late to do your due diligence.

It's not like I'm actually endeavoring to end up on American HR's Least Wanted...

I've just been (un)fortunate enough to collect the evidence from the worst offenders.  I'm not saying that everyone you'll ever interview with is out to screw you but everyone has an agenda and whether or not you want to satisfy it is a personal choice.  It's only fair that you have all the information before you sign on the dotted line.

Thing is, you rarely get the dossier before you get to the conference room so try to pick up your hints from clues in the job posting.  An overly aggressive tone is obvious with words like "Professionals Only" or "Must have" before every sentence.  What may not be are terms like "24/7 environment"  That's not boasting about their uptime, they're expecting anyone they hire to submit to that availability.  At least you know up front but better run organizations don't require indentured servitude. 

Only James Bond is on call 24/7 and I can guarantee you're not going to have near as much fun as he does...

It gets trickier if it's a blind phone screen.  You have to listen for the clues.  Are they reading questions out of a book? Is there more than one person on the call?  Is the demeanor friendly or have you felt more warmth from your last visit to the DMV.  Is every question answered with a question or a string of rehearsed responses.  What if you go off their script?  Do they seem annoyed, combative or deflect the query?  In that kind of scenario I wouldn't ask about covered parking.

Of course defenders of the HR and recruiting professions will tell you every story has 2 sides but when it comes to job interviews it's not about them it's about you.  The key attribute any hiring manager wants to see is submission.  They're the boss and it's their game but it seems more often than not they're not playing fair.

All the HR pundits will give you sage advice about being the most attractive candidate but in the end it may as well be admonitions from your mother....

"Sit up straight!, mind your manners! Eat your vegetables.."  Great advice....if you came to adulthood after being raised by wolves but useless otherwise.

Thus we come to my latest, "Hit Piece"

First some background.

I received a phone call from an out of state recruiter for a local position that was to last for 7 weeks.  I usually don't give much weight to out of state recruiters mostly because they rarely have a good relationship with the potential employer.  That's why I don't usually bother with them.  Their relationship with hiring managers is critical to your success.  If they don't have one,you may as well be applying to a plain old Want Ad and throwing your unemployment checks in the trash.

Out of state recruiters are little use to you in prepping for the interview because all they know is what's printed in the job order.  It literally turns into seeing which crap sticks to the wall by sending a parade of hopefuls into the interview process.  It's a roll of the dice based on a punch list of skills and questionable reasoning.

On the other side of the equation (aka: the hiring manager,) it doesn't say much for the company's selection process if they can't be bothered to be a more active participant.  The selection of an agency that has no local presence was likely based on 2 glaring factors that never work in your favor. 

First,  they're probably cheap, as out of state firms can underbid local firms due to low overhead.  All they need is an Internet connection and a phone to do their job.   

Hey! Cheap is great for stuff you buy by the pound but unless you wanted to be treated like a bag of walnuts, it's not going to go well for you.    

Second, it shows the employer has little interest in you as the candidate or any candidate for that matter.  They're just  looking for a warm body that won't make too much trouble (In this case "Trouble" is defined as having a soul).  If that's you, well, you probably stopped reading at the second paragraph and are busily sanitizing your web cache before anyone finds out you came here.  :-)

Against my better judgment, I went ahead with the phone interview.  Fortunately, it gave me a splendid example of how broken the process is.

It began with this email... (names changed)

Hi (candidate),

Thank you for submitting over your resume for the Windows Administrator position in Phoenix, AZ.  I would like to discuss the opportunity with further please call me when you have time to chat.


I called, went through initial screening and received this...

Hi (Candidate),

Thanks for your time today and for your interest in the 7 week contract position with (company).  I am having our Technical Engineer Tom (techguy) give you a call today at 10:00am PST (11:00am MT) to discuss the technical components of the position. 

Please confirm that you have received this email and are set to speak with Tom.  After your conversation with Tom we will get you submitted over for the position and hopefully have next steps very quickly.


Which got me to the tech screener after sending the confirmation.  Apparently I passed muster with "Tom."

Hi (Candidate,)

Could you do a phone interview tomorrow between 1 and 3?


Which of course I made sure I was available for.  I was sent an email with a phone number and conference ID.  The hiring manager couldn't be bothered to call me, instead I had to play with the voicemail system just to talk to him.  Not a good sign, I felt like I was on a treadmill...

Voicemail jail aside, I thought the call went well and lasted about 30 minutes.  He threw a surprise question about working weekends (the first I heard of it and another sign the recruiter had never spoken to him)  but other than that there was nothing out of the ordinary.  What was strange was no technical questions for a very technical job.  It was as though we were just going through the motions.  I tried to keep it light and friendly and the manager seemed to respond well but he wasn't exactly a "touchy-feely" kind of guy.  Oh, and his name was "Hans" and he spoke with a bit of an accent. Achtung baby!

Still, I could have been wrong so I let the recruiter know I thought the interview went well and waited to hear back.  I had no contact information for "Hans" so the whole "thank you" note was impossible.  Although I guess I could have shown some initiative and hacked their phone system to find him...Nah, Sorry HR gurus, I guess I failed you...

So imagine my surprise (not really) when I got this back the following Monday.

They have decided to pass, here was the specifics feedback from our client.  Hope this helps with future interviews!

“Spoke to "applicant" and was not too impressed. He was 5 minutes late to the meeting, no apologies and overall a fairly familiar approach to the conversation. Little too much for an interview, don’t think it will be a good fit to what we need.”

Ok, first off, I wasn't late by the clocks I was looking at and apparently this guy takes being friendly and responsive as "familiar."  

Geez, sorry Kommandant!

It could be a cultural conflict or this guy's just a jerk.  I can tell you that out of hundreds of interviews I've had in my career I can count on one hand the number of people that have responded this way.  In most cases I've found people like this usually end up in a padded room at some point in their lives.

This also shows how detrimental it can be when a recruiter doesn't know their client.  Remember I talked to 2 screeners before getting to the final screening with the client.  There were no issues and I didn't act any differently with "Hans.". 

Here's my email response...
Really?  Didn’t get that read from him during the conversation.
Also didn’t realize that being personable was a character flaw. 

I wasn’t any different with him than I was with you or your tech screener. 

This was supposed to be short term anyway so he shouldn’t be looking for somebody he has to babysit.

Ah well, this is why I generally don’t like large IT organizations. 

Too many insecure managers caught up in their own inflated egos.  

Thanks anyway.

Here's the final proof of why you should never deal with recruiters that don't know their clients.  The email I got back after I sent the above...

Hi (candidate)

Sounds like he is pretty formal guy if that was the situation.  I do not know him personally sorry this one did not work out!


I'd like to say I was sorry too but I can't.  I'm not looking to marry the guy just work for him for 2 months.  Not only did he lie but he let his ego get in front of his judgment. 

In the end it was the perfect confluence of bad recruiting and bad manager.  Learn from it!

You need to stop beating yourself up over things that you have no control over.   IT is rife with screwed up people and lots of them end up in management. 

You can't fix stupid so don't try and just move on to someone who can hold up their end of a conversation. 

Come to think of it, that's good dating advice too!

Sunday, September 15, 2013

Windows 98 SE on VirtualBox

If you're sane the first question should be...."Why????"

Well, because we can and every now and again we need to reacquaint ourselves with just how good we have it these days.

It's Windows 98 on VirtualBox!  Follow along as I suffer the pain of trying to get a 15 year old operating system working in a VM.  

No, it didn't go quietly or softly into the night...

BTW, if you want to see some real pain check out the Windows 3.11 video below it.


Thursday, August 29, 2013

Virtualization on the cheap with VirtualBox

Recently, I've been working on virtualization projects and one of them just didn't rise to the level of a VMWare rollout.  Even ESXi was a bit much for the deployment so instead we went with virtualbox which was much cheaper (free) and easier to administer.

Follow along with the following videos as I show you how I "kick it old school" with DOS and Windows for Workgroups in the latest VirtualBox platform.  Why such an old OS? Hey, anyone can load up Ubuntu or XP but a 20 year old OS takes some real effort and forces you to get up to your elbows in the virtualization platform.


Monday, August 26, 2013

JTT 5 Ballmer on the way out but is it really all his fault?

The latest episode of Just Talkin' Tech features a conversation about the impending departure of Steve Ballmer from Microsoft and how the company needs to change to survive.  I argue that Ballmer can't be held entirely responsible for Microsoft's missteps and include Microsoft's board of directors including Chairman Bill Gates partially responsible.

My friend lays the blame squarely on Ballmer alone suggesting that the board essentially deferred to his will.
By the way the video features Windows Vista, one of Microsoft's biggest failures under his tenure.


Friday, August 23, 2013

Ballmer's on the way what

Amidst the cheers on the trading room floor that sent Microsoft shares 9% higher on Friday, the tech world seems to have gotten its wish.  Steve Ballmer will leave Microsoft's CEO chair within the next 12 months.

As one of the tech world's longest sitting  CEO's and target for all things evil about Microsoft it's easy to misread today's news. 

Yes, he's leaving and yes he's been the guiding hand of Microsoft through 3 major versions of windows, two game consoles and a cloud strategy.  But Microsoft CEO's don't operate in a vacuum  anymore and the company's reigns are firmly held in check by a board of which Bill Gates is still the chairman.

There have been calls for a younger, more hip CEO who will drive Microsoft's new devices and services strategy and put Microsoft on par with the Facebook's of the world. 

The problem is, nobody relies on Facebook to run their business applications.  Social networks are a just a tool in a larger business strategy.  One that Microsoft has admittedly been slow to adopt just like its haphazard cloud strategy that was late to the game and confusing to its market.

Ballmer's evangelism for Microsoft often bordered on the fanatical even in the face of opposition from the market but he never acted alone.  The departure of a number of Microsoft execs and the recent reorganization of the company's business units didn't happen without the board's blessing.

Microsoft may not be the lean, hip company it was back in the days of Gate's tenure but it really can't be.  Most business runs at least in part on Microsoft products.   That the company has been able to  maintain itself as the standard by which all others (including open source) are judged is no accident.

If Ballmer can be blamed for anything it's an ever changing mixed message and desperate bid to monetize anything even remotely connected with Microsoft.  In the grand scheme of things it doesn't matter who sits in the CEO chair. 

Microsoft now is what IBM was in the 1980's.  Decent products but not very sexy. 

In a recent conversation with a friend of mine, he suggested a possible solution for Microsoft's stodgy, un-hip image.

Be like Toyota...

Toyota's Scion brand is specifically targeted at young hipsters with a lineup of products no Camry owner would be caught dead in.  That's smart.  The brand has successfully preserved the parent company's traditional market share while attracting a younger demographic they wouldn't have otherwise.

That's the direction Microsoft needs to move in.  Let's face it, You aren't going to sell many Cadillac's to the twenty something demographic.  Microsoft needs to separate itself from the consumer markets by placing its consumer strategy in an abstraction.   

Only then can Microsoft concentrate on its core business  customers while successfully engaging consumer markets.  You can't have the same message for both but if you separate the businesses you can have your cake and eat it too.

Saturday, July 6, 2013

The Truth about IT

I'm going to let you in on a secret...

If you're in IT nothing you do really matters.  At least not in the sense  of doing anything the world cares about.  I've said it before, IT exists to service people who actually create something.  It's a classic service industry job no matter what your title and that puts you right up there with the plumbers and auto mechanics. 

Plumbers don't get their names on monuments.  Neither do cab drivers, doormen or IT pros.  The world becomes more inconvenient without them but in the end they only exist to make someone else's life easier. 

I suppose that's why there's so many certifications for the profession now, assuming you could call it a profession.  It's not good enough to just be competent, now we need a governing authority to validate us.  A governing authority that's built a multi-billion dollar business out of our own insecurities.  As though we needed another reason to doubt our own abilities.

Even worse, validation from these governing authorities is suspect.  They exist less for the advancement of knowledge than their own revenue.  Why for instance would I need to not only carry a certification for Network knowledge but also specific product knowledge?

If I've been building networks for 20 years who cares if I have a Network + , A+ or ITIL anyway?  Those are supposed to be vendor neutral.  Surprise! network + was primarily developed by Cisco and A+ was written by hardware manufactures like HP, Dell and IBM.

 ITIL was written by masochists.

I've actually been disqualified for entry level tech jobs with (at the time) a decade of experience because I didn't have the right certification.  Never mind that on top of the experience  I had an electronics engineering degree, actually know what to do with a logic probe and was trained to troubleshoot PC's at board level. 

Because I refuse to spend $110 on an exam to see if I know which pin of a Molex connector is ground I can be denied a job I could do in my sleep because somebody thinks managing IT is all about the right certification.   If I've been working with networking  and computer equipment for most of my career why is it important what brand names were on the faceplates?

I know why and so do you, the people you're talking to don't have a clue about the department they're managing or worse don't really care. 

What's really sad is that your choice to work in the field means you have an 8 in 10 chance for having to work for them.

Here's a newsflash for the uninitiated, In the IT space everybody has to play by the same rules or nothing works. 

Do you really believe that the whole world runs exclusively on Cisco, Juniper, HP or 3Com equipment? 
Then it might surprise you to know that two brand agnostic organizations (IEEE, IETF) make the  Internet and networking in general possible.  They could care less whose brand is on your router so long as it plays by the rules they make. 

It's a perfect example of how useless vendor certifications are and an equally perfect indicator of a broken IT organization should it subscribe to another useless certification, the ITIL.

If you see a job description that demands a specific brand certification or worse the dreaded "familiar with ITIL methodology" you're likely dealing with a hiring manager that doesn't understand their own department.  A hallmark of an ITIL organization shows itself when ability is defined only by plug and play solutions no matter how ineffective.  They're not looking for ability, they're looking for the status quo even if it's dysfunctional.

I'm not against education or training, I'm just against the commoditization of it.  I'm also against lazy management practices based on buzzwords and fads.  Effective IT managers know at least something about the nuts and bolts of their department and can lead from experience.   

Ineffective IT managers are just waiting around till the next seminar on their way to their next job.  They rule by edicts lifted straight out of the framework of the dogma they've embraced and then slam the door on your way out. 

Good managers lead from experience not seminars.  You have to know what's right to know when things have really gone wrong.  You also have to give a damn or you're useless. 

That can be a tall order considering how neutered the IT function has become in the last decade.  IT budgets are strictly administered (usually by someone other than IT) and organizations don't have the latitude they once did.

If you're unlucky enough to be working in an organization that's fully embraced the ITIL construct you can expect to be constantly swimming upstream.  ITIL demands that technology and technical concerns take a back seat to the every whim of the user base.  

Yes, IT is a service profession but constant pandering leads to sloppy IT organizations and eventually to another tenet of ITIL, workarounds.  In other words, band-aids and bailing wire take precedence over actually fixing the problem.

Everyone demands zero downtime but refuses to do what it takes to achieve it.  IT professionals are expected to be on-call 24/7 for a wage that doesn't compensate them for that commitment.   Lest we forget the hardware to make it happen.

You're a fool if you accept that but most fools in IT do. 

If you care about what you do you then you don't mind the crazy hours but I can count on one hand the number of people I've met who were doing more than going through the motions.  Here's another newsflash, they weren't working 9 to 5.

We've come full circle.  The cold hard reality is that IT is a service profession and doesn't lend itself to rigid schedules and the meaningless busywork. 

So what can you look forward to if you happen to land that "dream?" IT job.  Maybe you're the new System Admin or IT manager.  Good for you.  Now that the pleasantries are over it's time to figure out if you'll be around long enough to wear out your office chair.

Let's play a little point/counterpoint framed in the context of questions asked in the average IT interview...

Why do you want to work in IT?
I love technology and I want to help people make the best use of it.
Technology makes the world go round and it's better than flipping burgers for a living.  But not by much.

Where do you see yourself in 5 years?
Hopefully in a management position so I can help further business goals using IT.
Probably in management because the last 2 IT managers quit and I managed to not get fired.  However, I know I'll be burned out because nobody is willing to do what it takes so I'll just bide my time.  I'll probably attend a lot of seminars looking for a way to make my life easier and do the bare minimum.  I'll hire lots of people with letters behind their names to make the department look good till I move on to my next job.

What is IT's function?
To further the organization's goals with the services I provide.
To keep the executive suite from complaining too much and maintaining the status quo because nobody really gives a damn what I do so long as they get their email and can play Farmville on their IPAD.

What's the value of IT certification?
It shows a commitment to continuous improvement and allows me to keep up to date with the latest technology.
Nobody cares about my MBA so I better have some more letters behind my name if I want to keep my job. 

What's more important, Technology or customer service?
IT is a service industry, our users, customers or whatever the label, come first.  They need to feel confident in the resources we provide
IT IS a service industry and you shouldn't be in it unless you understand that.  However, if you don't have the resources to do your job properly and decisions are made based on price instead of value (there's a difference) then you're never going to achieve customer satisfaction.  That is unless they're satisfied with you constantly saying "I'm sorry"

Ok, so the counterpoint looks like someone you wouldn't want within 100 miles of an IT department.  Here's the rub, the Point column is largely BS and everyone in the field knows it.  It looks good on glossy brochures but you can't get blood from a stone. 

Most IT departments are lucky to get new patch cables let alone the resources they need to meet user  demands.  The "Counterpoint" side is sadly closer to reality than anyone wants to admit to.  It's why I personally prefer contract work because I'd rather not waste time spinning my wheels and going nowhere while the rest of the department struggles for legitimacy that the field just can't provide.

IT can be rewarding but not when it's treated like the accounting department.  Truth be told if technology was forced to evolve in an average IT department, we'd still be using Motorola Dynatacs and dialing up to AOL for email on 28.8K modems.

The problem with corporate IT management methodologies is that you have a lot of people sitting around just waiting for something to happen.  When you are busy it's generally because one of the band-aids fell off the server and the CEO is screaming about how he can't get to EBay because of it.

Lets' face it, we're the plumbers, electricians and architects of technology.  We know what to do and how to do it (Hopefully?)  The best measure of success is that things work when they need to and that's the only goal that matters.

 We do what we do because we care about our life's work. (Well, at least first before we get jaded.)  Thing is,  that much ambition doesn't get you too far in most corporate IT departments.  Too much passion scares corporate types which means you either get shown the door or you just trudge on and abandon your soul.

Where a real tradesman is looked upon as the authority in their profession, IT is frequently seen as a necessary evil.  It's hard if not impossible to meet the demands of an organization when your hands are tied by corporate dogma. 

It's not that I'd advocate IT having no oversight but when you have to pass every decision through the  corporate litmus test, the result is going to be mediocre at best. 

Thus we have the "standard of mediocrity" ultimately resulting in IT pros clutching at straws to appear relevant.  That feeds my earlier assertion of how IT certifications prey on your own insecurities.  If you really care about the field and really want to effect change then you have to be brave enough to not just suffer in silence or load your resume up with worthless certifications.

The best known figures in technology didn't hide behind their cubicles hoping for someday.  Bill Gates of Microsoft and Steve Jobs  of Apple would have never survived in the average IT department.  Hell, they couldn't even stand college where at least there was beer and girls!

So why do we expect people who thrive on technology to function like accountants?  Would it be reasonable to expect an auto mechanic to fix your car in your bathroom? 

Well, I'm sorry to tell you, that's the reality of IT more often than not.  Unless you can make a go of it with the feast or famine cycle of consulting (as in independent not Robert Half) you're stuck with it. 

Of course, If you're more into the management track or the cubicle lifestyle I suppose it can work for you but I can guarantee I won't at least not as your employee.

Saturday, June 29, 2013

IT Legacies or IT Curse: Epilogue

IT legacies: Epilogue

So what happens when you're the guy tasked with fixing the mess someone else left behind?  In my case the best course of action was to take it one step at a time.

In the precursor to this post, IT Legacies, IT curse, I described an IT organization focused heavily on data but blind to the fundamentals of networks and Windows servers.  In the succeeding weeks since I wrote that article I found a multitude of evils compounded by daily demands that only served to highlight the tasks before me.

There was no area of IT that I didn't have to address but  for now I'll focus on what I found to be one of the most serious issues.

One of my first tasks was to straighten out the Windows domain.  Any admin worth his salt knows that the decision to use Active Directory(AD) automatically invokes the prerequisite of having a second Domain Controller (DC) somewhere in the organization.   Reason being, a second DC allows authentication tasks to be balanced across two servers and ensures you're not relying on one copy of the AD database for your entire Windows network. 

It's common sense if you understand how AD works.  In short, if your deployment doesn't merit a second DC then you don't really need AD.  It's that simple.  Think of it as a the law of AD regardless of the version of Windows Server you're running.

That said, I knew a lone domain controller in an organization of any size was asking for trouble.  Worse the one I found  was seemingly deployed as an afterthought.   On reflection, it probably was.

If you're going to run a Windows AD DC (I know a lot of acronyms) there's going to have to be a DNS server somewhere and with one DC on the network it wasn't a stretch to figure out where it would live. 

Remember that Active Directory is heavily dependent on the information contained within DNS.  If we accept that DNS functions as a kind of phonebook to allow easy lookup of the information contained within the AD database then we begin to see how critical it is for it to be functioning correctly.

Imagine my surprise when my investigation into the DNS information living on this solitary DC found that it was not being integrated into AD at all.  It's a basic configuration step requiring little more than a checkmark on a properties page of the DNS zone's configuration. 

I was mystified at this which led me to investigate why someone would choose to depend on an easily corrupted  text file to store critical information when it could be safely stored within the AD database and replicated at the same time as other AD data. 

Well ok, that's assuming there's something to replicate to.  I began to doubt my own procedures for a moment.  Perhaps there was something special going on that I just didn't understand.  Perhaps some weird 'Nix server had a problem with AD integrated DNS zones.  Perhaps the moon was in the 7th house and Jupiter was aligned with Mars...

I found nothing, so if for no other reason than to comfort my own ego I turned to Technet for validation of my own beliefs. 

In the end my convictions were vindicated.  There's nothing that precludes an AD integrated zone from interacting with other non AD DNS servers or clients.  I really do try to keep an open mind but when we stray into the ridiculous my verbalizations immediately become colorful and it's best not to be around me.

By the way, at one point there actually was another DC in the domain but it disappeared into the ether well before I arrived.  I never did figure out where it went.  Perhaps this was why the DNS zone wasn't integrated into DNS although that would solidify my belief that the previous IT team didn't know what the !&@*! they were doing. 

Considering the existing non-Ad DNS zone was still referencing a now absent DC it's apparent the removal wasn't graceful.  In fact the event log was filled with failed messages about replication.  Perhaps the DNS server complaints in the event logs caught the attention of someone on the former IT staff and their solution was to rip DNS out of AD. 

Now remember, the second DC should never have gone AWOL in the first place but if you're going to take it out at least do it right.  They didn't...

What I was left with was a server that had been searching in vain for a companion that hadn't existed for at least a year or more.  Considering this lone DC had a full office 2007 installation and two different resident user applications running that no longer served any purpose it was no wonder that this server would sit at 99% CPU load for hours on end. 

It was a DC that was treated like a workstation managed by an IT team that didn't have a clue.  It's really that simple.

The net effect were logon times that could take minutes in an organization where there would never be more than 10 simultaneous logons ever.  Logon scripts frequently failed, mapped drives disappeared and resources would remain inaccessible to users for up to 15 minutes after logon.

So, in my second week I began scrounging for parts and built a second DC out of cast off hardware.  That was the easy part.  Getting an operating system on it, however, was no easy task.  I can build Windows servers in my sleep but only if I have access to the necessary resources.  Of course with this client, I didn't.  Nobody knew where the server licenses were let alone the installation media yet I knew they had to exist because at some point there was at least one more DC. 

That led to a weeklong treasure hunt that was only partially successful but highlighted another issue with the IT organization.  I ultimately came up with a legal but kludgy solution that I won't go into here.  Suffice it to say that Peter was left the poorer for Paul's needs.

That was pretty much the order of the day during my tenure and was a necessary modus operandi if I wanted to get anything done.  As bad as it was, however, every upturned rock was an opportunity to change things for the better and in reality that was my role.  It's one thing to complain but another to do something about it. 

I was fairly well clued in on my first day when I found a backup job stuck in error status for 6 months and had to get my supervisor to call someone outside the company for the Administrator password.

By the way, that thing about the backup isn't an exaggeration.  There had literally been no data backup for almost half the server resources in 6 months.  The first request for file restoration I received took the better part of a day to satisfy and only after intense digging within old backup catalogs and a bit of luck. 

That's one thing Backup Exec is good for, endless catalogs.  At least Backup Exec's Continuous Protection server was good for something for a change because I didn't have as much as a shadow copy to work with otherwise.

Instead of going into any more lengthy diatribe on the specifics of what else was wrong with this organization, for brevity's sake I'm just going to list the issues.  See if any of them look familiar...

  1. One domain controller
  2. Messed up DNS configuration
  3. 192.168.1.x  ( Yep, just like at your house)
  4. Broken backups (Running on Backup Exec 11D - one of the buggiest versions BTW)
  5. Paying  support fees for software no longer used
  6. No documentation or obsolete documentation
  7. Slow logons
  8. Bad password policy
  9. Using Domain administrative credentials to run application services
  10. Outdated desktops (average 5 years old)
  11. VMWare ESXi deployed in an enterprise environment
  12. Lack of an enterprise management application suite
  13. No anything
  14. Backup tapes stored onsite
  15. No standard workstation model
  16. No accountability for outside vendors
  17. Obsolete Server and Network resources
  18. No or obsolete IT inventory information
  19. No update policy
  20. No control of IT budget (Not really that big of surprise these days but this was REALLY bad)
  21. Inconsistent licensing for all IT resources
  22. Reliance on outside vendors for critical IT functions (Proprietary DB systems, What' s the Admin Password?, etc)
  23. No network diagram
  24. Critical offsite servers that IT had no access to
  25. Inadequate broadband connection (5Mbits to handle everything including 2 busy DB's and a web server)
  26. New IT resources deployed without testing
  27. IT projects planned and scheduled without notifying IT
  28. Business phone system on its last legs (was purchased for $300 on EBay after the last one blew, really)
  29. Haphazard IT planning (or NO IT planning)

I know I probably forgot something but that list is pretty damning.  If you've been in the field for awhile you probably have a few of those items on your list too maybe even a few more.  Hopefully not all at the same time!

The point is that the key to fixing problems in IT is to identify them in the first place.  This particular organization was a victim of ad-hoc management.   In short, IT was repeatedly blindsided by issues that wouldn't have existed if someone was paying attention. 

You know, stuff like what the administrator passwords are and where the server room key is...

As the demands of the business were shoehorned into a dysfunctional IT methodology more and more time was spent putting out fires and less on ensuring a reliable environment.  Users eventually got used to doing less with less simply because there wasn't any alternative. 

But hey, at least the databases worked, if you could access them that is...

It's a common problem in IT but that doesn't make it acceptable.  I'm a firm believer in the KISS (Keep IT Simple Stupid) principle.  Things only get complex when you stop paying attention.  If you've got a mess, just break it down in to it's most basic parts instead of trying to do everything at once.  You're only human and the fact is, everyone's suffered this long so they can wait a bit longer for things to be done right. 

My immediate predecessor apparently couldn't embrace that philosophy as he left after only 2 days.  Admittedly it was one of the worst IT shops I'd ever walked into but nothing's impossible if you put the problem in the right context.  I was allowed to do that so when I left I felt good that I'd not only addressed the most egregious issues but laid out a framework to  build on.  That'll keep you from spending all your weekends babysitting servers and crossing your fingers Monday morning. 

I know I keep talking about "The Basics" and "Foundations."  You may be wondering what I mean by that.  Time for another list but don't worry it's a short one this time.


  1. Document EVERYTHING and make sure everyone knows where it is
  2. Critical IT procedures need to be codified (you know, like water on a burning server is a bad idea...)
  3. Keep track of your resources
  4. Don't be cheap! Get what you need to get the job done consistently and reliably
  5. Remember your place, IT is a service job and your users are your customers so keep them in mind in all you do
  6. Don't let anything or anyone interfere with Rule 5

You wouldn't blame your car for running out of gas if you never looked at the gas gauge so why would anyone think that ignoring the basics of your IT organization would have any better result? 

A week before my departure the business hired on a full-time IT manager and I was glad my efforts were able to provide him with more than horror stories.  Having laid out the issues, current configuration and procedures put in place he had a better starting point than I did.

I spent my last few days composing a document outlining everything I'd learned about the organization as well as common procedures and recommendations for improvement.   He's got a long road ahead of him but at least he's got an idea of where he's going.

I was just glad to provide the roadmap...