Thursday, May 30, 2013

IT Legacies or IT curse

Legacy, whenever the word comes up I think of characters like Don Corleone, Michael Jordan and Bill Gates.  Legacy means at some point you've left something that will endure long after you've moved on.
In IT, legacies are usually synonymous with curses and lately I've been doing a lot of cursing.  I've inherited a legacy of sorts with the start of a new contract. 

It's with an industrial electronics company with an IT history that stretches back at least as far as Novell Netware from the piles of old software I've found.

From what I've been able to gather from sources inside the company as well as former IT employees (now acting as fair weather consultants) there was a time when IT was ruled with an iron fist.  The head of the gang of six, as I now refer to them, was an unpleasant database administrator.   His rule was absolute because databases and specifically the data within them are this company's lifeblood. 

In case you haven't gotten the picture yet, he who holds the keys to the data holds the company by the short hairs.  A legacy that's persisted to the present day.  A legacy that now haunts my every billable hour.
To say that the previous IT manager abused his position would be an understatement.  The only force that could displace him and eventually the entire gang of six was when technology began to make it possible to bypass him. 

The tipping point? When someone made the decision to move a critical resource, messaging, out of his control and into the cloud.  Not that it was ever of any real concern to him but this sudden defiance of his rule was taken as a bad omen of things to come.  For him it was time to abdicate the throne.

Remember, up to that point, his rule was absolute like some tyrannical overlord.  User concerns were of no concern with unbridled shouting matches the norm if any dared question the authority of the gang of six.  By the way, the primary duty of the gang of six was to massage the data.   Everything else was secondary and it's still evident in the present day.

Not long after the overlord's departure the remaining gang members found themselves in an uncomfortable position.  Instead of tightly regimented tasks assigned by their overlord they now had to support an unpredictable user base.  A role they seemed unsuited for from the evidence I've uncovered so far.

There was even one who did nothing all day except "cleanse" the data.  Which for all intents and purposes involved little more than editing spreadsheets produced by the overlords many data mines.

Say what you want about Microsoft's Office 365 but I for one have a new found respect for anything that would get rid of that kind of IT organization. 

Shortly after the overlord exited the company in a huff due to the peasant revolt and subsequent embracing of cloud democracy other members of the gang of six soon fell away. 

Fast forward to the present day and open on me sifting through the remains of an abandoned IT department.

Surprisingly, instead of boxes or assorted refuse piled up in some kind of dilbertesque cubicle nightmare,  I find dedicated office space.  There's actual doors, desks, rooms and even a workshop with the remains of long abandoned projects still waiting for attention on the workbenches along the wall.

It's a little spooky, like someone let loose with a neutron bomb.  I keep looking for those outlined shadows on the walls that look like the photo negative of somebody's shadow.  Thankfully, I haven't found any yet.

Somebody was trying to be organized at some point but like the old saying goes, the proof is in the pudding.  I'll put it to you this way, I don't think Bill Cosby will be doing any commercials for these guys.

As I start my third week I already have bad memories.  One of the worst was working on what should have been a relatively simple project with one of the former gang of six (now a consultant).  It was a simple PC swap but the wrinkles became evident all too soon.  The now "consultant" refused to come onsite turning what should have been a day's work into three.   When things went wrong it wasn't the disembodied voice on the speakerphone that had to bear the brunt of a fuming department head.

 Unforeseen complications that should have been addressed in a lab instead of on the production floor almost cost the company thousands in lost revenue.  Quick and dirty fixes instead of solid solutions ruled the day. 
It's not that he didn't mean well but the lack of planning and laissez faire attitude toward the critical nature of this aspect of the business bothered me.  When I discussed it with the site supervisor he recognized the attitude immediately.  It was no different under the rule of the overlord.

IT is a dynamic affair and no one day is like the next but there is always ample opportunity for good planning that includes a fallback position if something goes wrong.  At this company it seems things going wrong are the order of the day. 

As I traverse the lonely halls and rows of cubicles I see evidence of the same kind of haphazard deployment and lack of planning that have crippled IT in the organization.  In short, the tyrants didn't need to burn the castle as they fled, it was already crumbling.

By now it's become obvious to me that IT people that massage databases for a living aren't always the best qualified to make decisions for an entire organization.  Some things are better left to those of us who'd rather not spend our time memorizing SQL queries and worrying about primary keys.

Let me be blunt.  I respect someone who has the ability to manage huge volumes of data and understand the intricacies of how it all relates to each other.  In the end, however, these people are power users not network or system administrators.  There's simply too much to keep track of in both disciplines for anyone to perform both functions competently.   Defy that logic and you end up with an organization like the one I'm trying to piece back together.

Shortly before I arrived the company suffered something largely unheard of in IT these days.  A rampant virus infection.  Caused by ineffective security policies, lack of management tools and outdated security software, it was an inevitable event just waiting for an opportunity.

Token gestures of security like a recently implemented mandatory password policy are good but trivial in the context of a broken IT organization. Make no mistake, this is a broken IT organization.  Its most poignant  symbol,  a four foot high pile of discarded rack servers numbering in the dozens just inside the door of the abandoned IT workshop.  With the relative dearth of IT services available it seems more like evidence of a scorched earth policy than evidence of a virtualization project. 

Truth be told many likely became unnecessary with the move to cloud services and a "sort of" consolidation of physical servers into virtual.  Ultimately, however, it seems more like evidence of retaliation than consolidation.  Ask any system admin, for example,  how many Domain controllers should exist in ANY Windows network and the answer you'll get is 2. 

We currently have one and it lives on a virtual server with no backup and no failover.  That's one of my priority projects by the way.  I'm utilizing a discarded hulk and parts scavenged from the carcasses of similarly afflicted hardware.

Questionable licensing, inadequate resources and a lack of documentation are all symptoms of an IT organization in disarray.  When the answer to "What's the admin password" requires a phone call to an outside party for the answer you know you have issues. 

In short, my job is to try to rebuild an IT organization while trying to convince a wary executive suite that I seek no term as the next IT overlord

The most significant hurdle has little to do with a new server or software, however.  Years of abuse from a bad IT organization has made every purchase and every policy change, no matter how insignificant, an exercise in bureaucracy.  A trait that has seen others (not of the gang of six) leave after only hours.
There's such distaste for the way things were that I'm relegated to cubeville, far away from those cozy IT digs.

My story is ongoing and luckily much of my remedial work can take place after hours where I'm free to curse the names of my predecessors without concern for delicate ears. 

The lesson here can be summed up in one of the first meetings I had with my site supervisor.  We were discussing how the IT organization should be structured in the future.  Remember, right now I'm the entire IT department apart from a few specialized "consultants" still massaging the data.

He thought the organization should be headed up by yet another maestro of data to which I replied, " So you're ok with the way things were?"

He replied, " Well, no, it was awful" 

Don't get me wrong, data's important and this company lives and dies by it.  Thing is, I see a future with at least two people to keep this place humming along.  One a data specialist, the other concentrating on the network and server infrastructure.  Both can offer support and with a properly running IT shop, both will have plenty of time to support the user community instead of putting out fires and making excuses.


...and both will be equals.

No comments: