Showing posts with label computers. Show all posts
Showing posts with label computers. Show all posts

Friday, February 24, 2017

The Geek Closet - Netfinity Project



I was rummaging around my geek closet....

You know you have one.  That dark little room crammed with discarded tech that you haven't touched in years but you "may have a use for someday."

In there I have a tapestry of my career in IT.  From bins of cables, cards and their associated connecting tissue to stacks of hardware long since retired to "someday" status.

Someday has come for you....

About 10 years ago while working for one of my clients during an upgrade project I purchased their soon to be unused and obsolete server hardware. 

I knew what I was getting into having actually installed these servers years before.  I had some vague ideas about what I'd do with them but nothing definitive.  

For all I knew they'd never be more than impromptu jack stands but I saw an opportunity even if I didn't know what fruit that would bear.

Within a few months I did manage to find a purpose for a few of them when I went into business with a friend of mine.  But after a year the business passed away into obscurity like so many others and they returned to the geek closet.

Until the other day.

I opened the geek closet and gazed upon these once mighty hunks of iron sadly sitting idle under stacks of similarly situated techno-cruft.

No, I needed, wanted, to do something with these servers and as luck would have it I was desperately searching for new content for my IT channel on YouTube.

Thus the Netfinity Project was born.  

It's a new video series that covers my attempt to re-purpose a couple of these old servers and gives you some insight into what I consider to be the golden age of hardware and IT.  From the late 90's to the early 2000's IT was all about the hardware and the only talk of "clouds" and "as a service" had to do with rain and valet parking. 

Hardware got better because the software demanded it.  Hardware is unquestionably better now but it lives in bland, clinical warehouses far from view.  An abstraction out of sight and out of mind.  

There's something sad about that.  I remember the pride of IT Manager's throwing open the doors of the server room.  This was where the magic happened, a tangible representation of the greatness of his enterprise.  Whirring, beeping, lights flashing and disks spinning.  You could almost feel the heartbeat of business within those cold server cases.

It's worth documenting and to some extent resurrecting if I can.

Thus the video series where I go through the highs and lows of making some old IBM Netfinity 5000's relevant again.

Check out this blog for regular updates on the video series.  The first few are below.  Enjoy!







Thursday, May 19, 2016

Take control of those nightmare technical interviews


I had a recent experience that was a perfect example of what it's like to go through a bad technical interview.

I say "bad" because the whole time I was there it was less about what I knew and more about trying to make me look like an idiot.

Yeah, I know there's such a thing as the hot seat and technical interviews are designed to be tough.  But we strayed from the technical into the psychological for no good reason other than one of the guys across the table from me just wasn't going to ever be a fan.  It became a game of minefields.

Thing is, I wasn't playing which just made the inquisitor across the table make more of an ass out of himself the longer it went on.

I've often said that the interview process is adversarial.  The premise being that you're either lying, unqualified or unworthy of being in the same room with a "guru."

In the video below I give you some pointers on how to get the best outcome you can without sacrificing your dignity in the process.


If you can, make a positive out of a negative.


Thursday, March 12, 2015

The Case against Open Source


Every now and then I'm up pretty early on a Wednesday morning and if my Squeezbox radio happens to be on I'm probably going to hear at least part of Randal Schwartz's weekly window into all things Open Source, Floss Weekly.

Randal's a nice enough guy and if we're honest one of a scarce few real geeks left on TWIT...

So I listen for awhile.  That is, up until the content ends and the propaganda starts...

The premise of Open Source is sound enough.  It's community driven often filling a need that's either not being adequately addressed by more traditional offerings or breaks new ground.  It also gives budding tech types somewhere to try out their ideas without fear of running afoul of someone else's copyright.  Of course it also has the frequent advantage of being free of charge in hopes of continuing development and maximizing distribution.

It's how Linux, Apache Web Server and Wordpress came to be. 

Considering much of what you see on the web depends on at least one of those open source projects there's a strong case for community driven alternatives.

Which would normally be the end of the story but for the past few years where a number of projects have been taking aim at the enterprise.  Everything from telephony to CRM is in the mix.

Which is fine so long as you've got support for them.

And there's the rub....

In the landscape of current technology solutions you really have two options.  You can pay a lot of money now for somebody else's pre-packaged whatever or try an Open Source alternative and pay someone to make it work later.

That is the dichotomy of so-called "Open" and "closed source" projects. 


Open source sprang from the belief that software development should not be a dark art kept in bowels of some mega corporation who controls its every permutation.  Anyone who's dealt with botched Microsoft updates bringing their business to a standstill can identify with that.

The Microsoft's of the world may be more user friendly and better supported but they're by no means perfect.  

Customization is limited and new features often only come with a new version which starts a whole new round of checkbook bleeding.

That's supposedly one of the advantages of Open Source.  Being community driven, changes happen more quickly and development is more responsive to the user base.  But what is perceived as a strength becomes a weakness when you realize that the word "community" can easily be replaced by "mob rule."

Just because updates come along more frequently doesn't mean the problem you're having gets resolved or the feature you want will show up. The squeaky wheel gets the grease as they say and if your problem isn't at the top of the community's list of priorities you're pretty much out of luck.

There's also the possibility that an update actually makes a problem worse or breaks unrelated services.  Something very common especially in the Linux world.

Of course you could always try to fix it yourself. There's plenty of White Papers, community forums and support avenues available.  Or at least that's the sales pitch.

The real story is that White papers, those tomes of wisdom, are written by developers... for developers.   If you don't speak the language they're little more than insomnia cures.  Ever read a phone book?  It's like that.

Riveting...

Community forums?  Those are fun too.  Populated by the equally afflicted and rarely served.  You may get lucky and get an answer but most of the time it's just a lot of wailing followed up by arrogant guru types belittling hapless victims for not reading the white paper more closely.

So much for the "community"

How about support directly from the development team? 

See above...

Even if there are thousands of contributors to a project, development usually ends up being controlled by a select few.  Infighting is frequent and is the primary reason you see so many variants of the same core project.  It splinters the community and makes support even more difficult.

Established projects aren't immune from the chaos and code rot either. Take the example of the popular open source web hosting control panel Zpanel.  Zpanel is a free alternative to the commercial Cpanel product offering a similar experience for far less cost (as in free).  

Unfortunately, it hasn't been updated in a year and much of the functionality is broken leaving users flailing while the "official" support team remains silent.

It's gotten so bad that the dev team actually shut down the public support forum shortly after a user reported a security issue to them which even when proven was subsequently denied.  In short a promising stable project has become broken due to ego and neglect.  A post-mortem that's all too common.

Still, If you want to sign up with Zpanel's official maintainer, Hostwinds, you may get some support, if you pay for it.  They call it "Premium" support and require a paid Hostwinds account.

Let's also remember that Open Source devotees often cite superior security of their wares.  That can be true but only so long as somebody's paying attention.  Apache has had numerous security flaws for example so too has OpenSSL and lest we forget the granddaddy of them all a BASH shell vulnerability that went unchecked for 20 years.  Yes, technically BASH isn't Open Source but its code is and it's maintained the same way.

Don't get me wrong, there's nothing wrong with paying for support.  It's been the foundation of many Independent consultants for years. 

What is wrong is foisting an unstable product on a hapless user base and then charging them to fix your own mistakes.  

Even Microsoft will refund a support charge if they find out it's their problem.

In the case of Zpanel their only response to the charge is that it's a product created on their own free time and thus isn't a priority.

So much for pushing the state of the art...

Read the next excerpt I took straight from Opensource.com, a leading Open Source publication...

Doesn't "open source" just mean something is free of charge?

No. This is a common misconception about what "open source" implies. Programmers can charge money for the open source software they create or to which they contribute. But because most open source licenses require them to release their source code when they sell software to others, many open source software programmers find that charging users money for software services and support (rather than for the software itself) is more lucrative. This way, their software remains free of charge and they make money helping others install, use, and troubleshoot it.

In other words, if you expect the same kind of experience you get from closed sources you're going to pay for it either in time or money.  Nothing is free.

There's a common quip when describing the "Free" nature of Open Source.  They say it's "Free" as in speech not "Free Beer."

Cute but oversimplified.

In a world built on consumerism, free speech doesn't hold a candle to free beer.   Besides, If you accept the Open Source view of freedom then "free speech" ends up unintelligible gibberish.

Which coincidentally is a lot like your support options.

There are just far too many projects out there that are the very antithesis of usability unless you're the type that likes to write Apache modules for fun.  Many are bleeding edge offering promise but in any other realm they'd be considered an "Alpha" release.

Do they really want me to put my neck on the line for an ideology?

I'll put it this way.  If you're ok with rolling out a "Developer Preview" of the Windows operating system (aka: Beta) to your entire enterprise then you're probably ok trusting that same enterprise to poorly supported open source software.

There's a history in Open Source that goes beyond just sticking it to the establishment.  It hearkens back the days when computer guys had all the answers.

Open source is where the gurus go.  You can trace its roots to the custom applications that literally held business hostage in the early days of enterprise computing.  Back then business wanted computerization but there were very few who knew how to make it work.  There were no Microsoft's just hardware and a few people that knew how to press the right buttons and work the magic.  Those people held the keys to the kingdom tightly.

The inroads of Windows and Mac operating systems in the early 90's eliminated the need for such exalted wizardry.  Any bright kid with a couple of exam cram books could run an enterprise.  The wizard gurus were none too pleased to see their grip on power loosening.

Ok so that's a bit melodramatic but there was definitely a lot of ego bruising going on when the PDP-11's got kicked to the curb.  I'm in danger of flying off on a tangent that sounds like something found in a Tolkien trilogy so I'll just wrap up this thought with this. 

There's a reason there's so much bile hurled at the likes of Microsoft by the Open Source community.  Contrary to the marketing, It's not about some David vs. Goliath battle.  It's simpler than that.  It really comes down to ego and wanting back the days when the Uber Geek held all the cards.

Control the Information and you control the world. 

They eschew anything "packaged" instead touting the virtue of getting one's hands dirty.  To hell with those "lazy" users wanting everything "handed" to them.  Every child should know C-Sharp by the age of 3!

They don't get it.  They just can't understand why everyone doesn't want to be a part-time software engineer.  Which is the root of the attitude and the reason why Open Source tends to have a narcissistic vibe even while proclaiming the democracy of a community.

If the masses will not be turned they will be ruled...

Phew!

Who knew it was so political!

It's not all bad, however, and there are good ideas and good projects out there but there's no guarantee they'll stay that way.  Projects can start out with lofty aspirations but most are just some poor Joe looking to fix his own issues.  Once the problem is solved the project is abandoned.  

As such, the world of Open Source is a wonderful laboratory but little more.  A place to try new approaches and work out the bugs but not to trust an entire enterprise to unless someone has taken it to the next level as in the example of Red Hat Enterprise Linux (RHEL. )

Even then a competent talent pool to administer it will be much shallower than its competition and more expensive since it's still a niche skill set.

The bottom line is this.  No business should be held hostage by what is all too often the product of a hobbyist's whim who got inspiration from an Internet forum.  

Yes there are serious Open Source initiatives out there but most of them aren't ready for prime time and if their devs are honest with themselves, never will be. 

Open source is great for advancing the art but artists make bad businessmen.  



Saturday, June 29, 2013

IT Legacies or IT Curse: Epilogue

IT legacies: Epilogue

So what happens when you're the guy tasked with fixing the mess someone else left behind?  In my case the best course of action was to take it one step at a time.

In the precursor to this post, IT Legacies, IT curse, I described an IT organization focused heavily on data but blind to the fundamentals of networks and Windows servers.  In the succeeding weeks since I wrote that article I found a multitude of evils compounded by daily demands that only served to highlight the tasks before me.

There was no area of IT that I didn't have to address but  for now I'll focus on what I found to be one of the most serious issues.

One of my first tasks was to straighten out the Windows domain.  Any admin worth his salt knows that the decision to use Active Directory(AD) automatically invokes the prerequisite of having a second Domain Controller (DC) somewhere in the organization.   Reason being, a second DC allows authentication tasks to be balanced across two servers and ensures you're not relying on one copy of the AD database for your entire Windows network. 

It's common sense if you understand how AD works.  In short, if your deployment doesn't merit a second DC then you don't really need AD.  It's that simple.  Think of it as a the law of AD regardless of the version of Windows Server you're running.

That said, I knew a lone domain controller in an organization of any size was asking for trouble.  Worse the one I found  was seemingly deployed as an afterthought.   On reflection, it probably was.

If you're going to run a Windows AD DC (I know a lot of acronyms) there's going to have to be a DNS server somewhere and with one DC on the network it wasn't a stretch to figure out where it would live. 

Remember that Active Directory is heavily dependent on the information contained within DNS.  If we accept that DNS functions as a kind of phonebook to allow easy lookup of the information contained within the AD database then we begin to see how critical it is for it to be functioning correctly.

Imagine my surprise when my investigation into the DNS information living on this solitary DC found that it was not being integrated into AD at all.  It's a basic configuration step requiring little more than a checkmark on a properties page of the DNS zone's configuration. 

I was mystified at this which led me to investigate why someone would choose to depend on an easily corrupted  text file to store critical information when it could be safely stored within the AD database and replicated at the same time as other AD data. 

Well ok, that's assuming there's something to replicate to.  I began to doubt my own procedures for a moment.  Perhaps there was something special going on that I just didn't understand.  Perhaps some weird 'Nix server had a problem with AD integrated DNS zones.  Perhaps the moon was in the 7th house and Jupiter was aligned with Mars...

I found nothing, so if for no other reason than to comfort my own ego I turned to Technet for validation of my own beliefs. 

In the end my convictions were vindicated.  There's nothing that precludes an AD integrated zone from interacting with other non AD DNS servers or clients.  I really do try to keep an open mind but when we stray into the ridiculous my verbalizations immediately become colorful and it's best not to be around me.

By the way, at one point there actually was another DC in the domain but it disappeared into the ether well before I arrived.  I never did figure out where it went.  Perhaps this was why the DNS zone wasn't integrated into DNS although that would solidify my belief that the previous IT team didn't know what the !&@*! they were doing. 

Considering the existing non-Ad DNS zone was still referencing a now absent DC it's apparent the removal wasn't graceful.  In fact the event log was filled with failed messages about replication.  Perhaps the DNS server complaints in the event logs caught the attention of someone on the former IT staff and their solution was to rip DNS out of AD. 

Now remember, the second DC should never have gone AWOL in the first place but if you're going to take it out at least do it right.  They didn't...

What I was left with was a server that had been searching in vain for a companion that hadn't existed for at least a year or more.  Considering this lone DC had a full office 2007 installation and two different resident user applications running that no longer served any purpose it was no wonder that this server would sit at 99% CPU load for hours on end. 

It was a DC that was treated like a workstation managed by an IT team that didn't have a clue.  It's really that simple.

The net effect were logon times that could take minutes in an organization where there would never be more than 10 simultaneous logons ever.  Logon scripts frequently failed, mapped drives disappeared and resources would remain inaccessible to users for up to 15 minutes after logon.

So, in my second week I began scrounging for parts and built a second DC out of cast off hardware.  That was the easy part.  Getting an operating system on it, however, was no easy task.  I can build Windows servers in my sleep but only if I have access to the necessary resources.  Of course with this client, I didn't.  Nobody knew where the server licenses were let alone the installation media yet I knew they had to exist because at some point there was at least one more DC. 

That led to a weeklong treasure hunt that was only partially successful but highlighted another issue with the IT organization.  I ultimately came up with a legal but kludgy solution that I won't go into here.  Suffice it to say that Peter was left the poorer for Paul's needs.

That was pretty much the order of the day during my tenure and was a necessary modus operandi if I wanted to get anything done.  As bad as it was, however, every upturned rock was an opportunity to change things for the better and in reality that was my role.  It's one thing to complain but another to do something about it. 

I was fairly well clued in on my first day when I found a backup job stuck in error status for 6 months and had to get my supervisor to call someone outside the company for the Administrator password.

By the way, that thing about the backup isn't an exaggeration.  There had literally been no data backup for almost half the server resources in 6 months.  The first request for file restoration I received took the better part of a day to satisfy and only after intense digging within old backup catalogs and a bit of luck. 

That's one thing Backup Exec is good for, endless catalogs.  At least Backup Exec's Continuous Protection server was good for something for a change because I didn't have as much as a shadow copy to work with otherwise.

Instead of going into any more lengthy diatribe on the specifics of what else was wrong with this organization, for brevity's sake I'm just going to list the issues.  See if any of them look familiar...

  1. One domain controller
  2. Messed up DNS configuration
  3. 192.168.1.x  ( Yep, just like at your house)
  4. Broken backups (Running on Backup Exec 11D - one of the buggiest versions BTW)
  5. Paying  support fees for software no longer used
  6. No documentation or obsolete documentation
  7. Slow logons
  8. Bad password policy
  9. Using Domain administrative credentials to run application services
  10. Outdated desktops (average 5 years old)
  11. VMWare ESXi deployed in an enterprise environment
  12. Lack of an enterprise management application suite
  13. No redundancy...in anything
  14. Backup tapes stored onsite
  15. No standard workstation model
  16. No accountability for outside vendors
  17. Obsolete Server and Network resources
  18. No or obsolete IT inventory information
  19. No update policy
  20. No control of IT budget (Not really that big of surprise these days but this was REALLY bad)
  21. Inconsistent licensing for all IT resources
  22. Reliance on outside vendors for critical IT functions (Proprietary DB systems, What' s the Admin Password?, etc)
  23. No network diagram
  24. Critical offsite servers that IT had no access to
  25. Inadequate broadband connection (5Mbits to handle everything including 2 busy DB's and a web server)
  26. New IT resources deployed without testing
  27. IT projects planned and scheduled without notifying IT
  28. Business phone system on its last legs (was purchased for $300 on EBay after the last one blew up...no, really)
  29. Haphazard IT planning (or NO IT planning)


I know I probably forgot something but that list is pretty damning.  If you've been in the field for awhile you probably have a few of those items on your list too maybe even a few more.  Hopefully not all at the same time!

The point is that the key to fixing problems in IT is to identify them in the first place.  This particular organization was a victim of ad-hoc management.   In short, IT was repeatedly blindsided by issues that wouldn't have existed if someone was paying attention. 

You know, stuff like what the administrator passwords are and where the server room key is...

As the demands of the business were shoehorned into a dysfunctional IT methodology more and more time was spent putting out fires and less on ensuring a reliable environment.  Users eventually got used to doing less with less simply because there wasn't any alternative. 

But hey, at least the databases worked, if you could access them that is...

It's a common problem in IT but that doesn't make it acceptable.  I'm a firm believer in the KISS (Keep IT Simple Stupid) principle.  Things only get complex when you stop paying attention.  If you've got a mess, just break it down in to it's most basic parts instead of trying to do everything at once.  You're only human and the fact is, everyone's suffered this long so they can wait a bit longer for things to be done right. 

My immediate predecessor apparently couldn't embrace that philosophy as he left after only 2 days.  Admittedly it was one of the worst IT shops I'd ever walked into but nothing's impossible if you put the problem in the right context.  I was allowed to do that so when I left I felt good that I'd not only addressed the most egregious issues but laid out a framework to  build on.  That'll keep you from spending all your weekends babysitting servers and crossing your fingers Monday morning. 

I know I keep talking about "The Basics" and "Foundations."  You may be wondering what I mean by that.  Time for another list but don't worry it's a short one this time.

THE BASICS

  1. Document EVERYTHING and make sure everyone knows where it is
  2. Critical IT procedures need to be codified (you know, like water on a burning server is a bad idea...)
  3. Keep track of your resources
  4. Don't be cheap! Get what you need to get the job done consistently and reliably
  5. Remember your place, IT is a service job and your users are your customers so keep them in mind in all you do
  6. Don't let anything or anyone interfere with Rule 5

You wouldn't blame your car for running out of gas if you never looked at the gas gauge so why would anyone think that ignoring the basics of your IT organization would have any better result? 

A week before my departure the business hired on a full-time IT manager and I was glad my efforts were able to provide him with more than horror stories.  Having laid out the issues, current configuration and procedures put in place he had a better starting point than I did.

I spent my last few days composing a document outlining everything I'd learned about the organization as well as common procedures and recommendations for improvement.   He's got a long road ahead of him but at least he's got an idea of where he's going.


I was just glad to provide the roadmap...

Thursday, May 30, 2013

IT Legacies or IT curse

Legacy, whenever the word comes up I think of characters like Don Corleone, Michael Jordan and Bill Gates.  Legacy means at some point you've left something that will endure long after you've moved on.
In IT, legacies are usually synonymous with curses and lately I've been doing a lot of cursing.  I've inherited a legacy of sorts with the start of a new contract. 

It's with an industrial electronics company with an IT history that stretches back at least as far as Novell Netware from the piles of old software I've found.

From what I've been able to gather from sources inside the company as well as former IT employees (now acting as fair weather consultants) there was a time when IT was ruled with an iron fist.  The head of the gang of six, as I now refer to them, was an unpleasant database administrator.   His rule was absolute because databases and specifically the data within them are this company's lifeblood. 

In case you haven't gotten the picture yet, he who holds the keys to the data holds the company by the short hairs.  A legacy that's persisted to the present day.  A legacy that now haunts my every billable hour.
To say that the previous IT manager abused his position would be an understatement.  The only force that could displace him and eventually the entire gang of six was when technology began to make it possible to bypass him. 

The tipping point? When someone made the decision to move a critical resource, messaging, out of his control and into the cloud.  Not that it was ever of any real concern to him but this sudden defiance of his rule was taken as a bad omen of things to come.  For him it was time to abdicate the throne.

Remember, up to that point, his rule was absolute like some tyrannical overlord.  User concerns were of no concern with unbridled shouting matches the norm if any dared question the authority of the gang of six.  By the way, the primary duty of the gang of six was to massage the data.   Everything else was secondary and it's still evident in the present day.

Not long after the overlord's departure the remaining gang members found themselves in an uncomfortable position.  Instead of tightly regimented tasks assigned by their overlord they now had to support an unpredictable user base.  A role they seemed unsuited for from the evidence I've uncovered so far.

There was even one who did nothing all day except "cleanse" the data.  Which for all intents and purposes involved little more than editing spreadsheets produced by the overlords many data mines.

Say what you want about Microsoft's Office 365 but I for one have a new found respect for anything that would get rid of that kind of IT organization. 

Shortly after the overlord exited the company in a huff due to the peasant revolt and subsequent embracing of cloud democracy other members of the gang of six soon fell away. 

Fast forward to the present day and open on me sifting through the remains of an abandoned IT department.

Surprisingly, instead of boxes or assorted refuse piled up in some kind of dilbertesque cubicle nightmare,  I find dedicated office space.  There's actual doors, desks, rooms and even a workshop with the remains of long abandoned projects still waiting for attention on the workbenches along the wall.

It's a little spooky, like someone let loose with a neutron bomb.  I keep looking for those outlined shadows on the walls that look like the photo negative of somebody's shadow.  Thankfully, I haven't found any yet.

Somebody was trying to be organized at some point but like the old saying goes, the proof is in the pudding.  I'll put it to you this way, I don't think Bill Cosby will be doing any commercials for these guys.

As I start my third week I already have bad memories.  One of the worst was working on what should have been a relatively simple project with one of the former gang of six (now a consultant).  It was a simple PC swap but the wrinkles became evident all too soon.  The now "consultant" refused to come onsite turning what should have been a day's work into three.   When things went wrong it wasn't the disembodied voice on the speakerphone that had to bear the brunt of a fuming department head.

 Unforeseen complications that should have been addressed in a lab instead of on the production floor almost cost the company thousands in lost revenue.  Quick and dirty fixes instead of solid solutions ruled the day. 
It's not that he didn't mean well but the lack of planning and laissez faire attitude toward the critical nature of this aspect of the business bothered me.  When I discussed it with the site supervisor he recognized the attitude immediately.  It was no different under the rule of the overlord.

IT is a dynamic affair and no one day is like the next but there is always ample opportunity for good planning that includes a fallback position if something goes wrong.  At this company it seems things going wrong are the order of the day. 

As I traverse the lonely halls and rows of cubicles I see evidence of the same kind of haphazard deployment and lack of planning that have crippled IT in the organization.  In short, the tyrants didn't need to burn the castle as they fled, it was already crumbling.

By now it's become obvious to me that IT people that massage databases for a living aren't always the best qualified to make decisions for an entire organization.  Some things are better left to those of us who'd rather not spend our time memorizing SQL queries and worrying about primary keys.

Let me be blunt.  I respect someone who has the ability to manage huge volumes of data and understand the intricacies of how it all relates to each other.  In the end, however, these people are power users not network or system administrators.  There's simply too much to keep track of in both disciplines for anyone to perform both functions competently.   Defy that logic and you end up with an organization like the one I'm trying to piece back together.

Shortly before I arrived the company suffered something largely unheard of in IT these days.  A rampant virus infection.  Caused by ineffective security policies, lack of management tools and outdated security software, it was an inevitable event just waiting for an opportunity.

Token gestures of security like a recently implemented mandatory password policy are good but trivial in the context of a broken IT organization. Make no mistake, this is a broken IT organization.  Its most poignant  symbol,  a four foot high pile of discarded rack servers numbering in the dozens just inside the door of the abandoned IT workshop.  With the relative dearth of IT services available it seems more like evidence of a scorched earth policy than evidence of a virtualization project. 

Truth be told many likely became unnecessary with the move to cloud services and a "sort of" consolidation of physical servers into virtual.  Ultimately, however, it seems more like evidence of retaliation than consolidation.  Ask any system admin, for example,  how many Domain controllers should exist in ANY Windows network and the answer you'll get is 2. 

We currently have one and it lives on a virtual server with no backup and no failover.  That's one of my priority projects by the way.  I'm utilizing a discarded hulk and parts scavenged from the carcasses of similarly afflicted hardware.

Questionable licensing, inadequate resources and a lack of documentation are all symptoms of an IT organization in disarray.  When the answer to "What's the admin password" requires a phone call to an outside party for the answer you know you have issues. 

In short, my job is to try to rebuild an IT organization while trying to convince a wary executive suite that I seek no term as the next IT overlord

The most significant hurdle has little to do with a new server or software, however.  Years of abuse from a bad IT organization has made every purchase and every policy change, no matter how insignificant, an exercise in bureaucracy.  A trait that has seen others (not of the gang of six) leave after only hours.
There's such distaste for the way things were that I'm relegated to cubeville, far away from those cozy IT digs.

My story is ongoing and luckily much of my remedial work can take place after hours where I'm free to curse the names of my predecessors without concern for delicate ears. 

The lesson here can be summed up in one of the first meetings I had with my site supervisor.  We were discussing how the IT organization should be structured in the future.  Remember, right now I'm the entire IT department apart from a few specialized "consultants" still massaging the data.

He thought the organization should be headed up by yet another maestro of data to which I replied, " So you're ok with the way things were?"

He replied, " Well, no, it was awful" 

Don't get me wrong, data's important and this company lives and dies by it.  Thing is, I see a future with at least two people to keep this place humming along.  One a data specialist, the other concentrating on the network and server infrastructure.  Both can offer support and with a properly running IT shop, both will have plenty of time to support the user community instead of putting out fires and making excuses.


...and both will be equals.

Monday, April 22, 2013

Know your place


I've been in the IT game for awhile but unlike most of my peers I haven't spent any great amount of time in any one place.  It's not that I don't believe in long-term relationships, on the contrary my average client has been with me at least 5 years.  I'd just rather be as productive as I can be instead of treading water with busywork.

To be successful in consulting you have to learn to be attentive to your client's needs.  That means doing what they need you to do in a timely fashion and then get out of their way.  Often times that means getting out your comfort zone when they throw something at you from left field. 

You have to be adaptable and keep up with current technology but that can be difficult if you're not working in large organizations with big IT budgets. 

Still, you have to realize that whether you're dealing with 15 or 5000 users, at its core IT is always the same.  Everything scales.  The only real difference is the people providing the IT services. 

At some point most IT organizations grow beyond the capabilities of one person.  Maybe it's a specialized application that needs a dedicated person or just plain old growth.  It doesn't matter so long as everyone understands the fundamentals.

I'm not talking about acing your IT exams or memorizing all the Active Directory FSMO roles either.  No, the fundamentals I'm talking about have very little to do with technical buzzwords and everything to do with IT's role in any organization.

In short, know your place. 

That's actually a brick wall I've been running into lately especially in a profession with declining wages and a bad job market.  It seems that IT managers are more concerned about the skill of the day or how many letters follow your name than whether or not you understand IT's role.

Hot skills come and go and to be an expert in anything in IT ultimately has about as much importance as winning first place in a snowman building contest.  Nobody's going to care after tomorrow. 

It's not about the skills, it's about your ability to use them to serve your users.   So long as you have the capabilities to adapt and a point of reference it's not a big deal if you don't match up to someone's skill punch list. 

That's what I attribute whatever success I've enjoyed in my own career to.  My role has been one of service; no more, no less.  Anyone who thinks that IT is anything more than that is quite simply an egomaniac.    
Yes, IT provides the medium that powers a connected world but in the grand scheme of things it's not important for its own sake.  

We in IT simply provide the means for other people to accomplish their goals.  
I'm perfectly ok with that but many in IT aren't and they refuse to hear anything that doesn't glorify the profession.  They inflate their technical accomplishments, create needless workflows (busywork) and body block anything that threatens their fragile egos. 

I've been in the field for quite some time now and while the phrase is tired I literally have forgotten more than most IT managers know at this point.  Familiarity with a specific IT platform is only valuable so long as it remains viable to the organization.  Once it's outlived its usefulness you need to move on but the lessons learned continue to have value.  They are the true definition of skill.  



Whether you're an admin or a CIO you have to realize the value of IT has nothing to do with buzzwords or brands.  It's got everything to do with ability and attitude, however, and they aren't defined by fads.

I actually find it amusing that anyone in IT attributes the word "skill" to anything that has a brand name attached to it.  It's probably the only profession that discriminates based on marketing jargon.  When you consider that the non-IT equivalent to a tech job is an auto mechanic you start to realize how ridiculous it is to be passed over because of familiarity with one brand name over another. 

I mean, does anyone actually believe that a Chevy mechanic is incapable of working on Fords?  
Generally we don't label auto mechanics by their brand affiliation, they're just mechanics.  The skill is in being able to understand automotive systems no matter who made them.  That's because at their core they're designed the same way regardless of whose label is on that grill.

Yet as an IT worker you're led to believe that managing Cisco branded switches has taught you nothing about managing one from HP or Dell.

It's a poor interviewer that doesn't realize that I've spent my career going the extra mile and continually learning new skills to fit my client's needs.  I tend to be more practical and don't spend my nights pouring over the latest database or scripting languages. I'm too practical for that.  I'm only interested in what makes my users happy because I know my value to them depends on it.

I had an opportunity to speak with just such a misinformed IT manager recently concerning an IT support position.  When he asked  the, "Tell me about yourself" question I obliged by giving him a short synopsis of my career and my commitment to serving my users.  In fact I actually told him my view of the value of IT in an organization. 

His response? " Where do you see yourself in five years" 

In other words, he wasn't listening in fact I knew he hadn't even looked at the resume that was forwarded to him from the pleasant HR guy I'd talked to a week before.  

I could excuse the fact that he was 20 minutes late in calling me ( a time he chose by the way) or that he was interviewing me while obviously doing something else. 

What I couldn't excuse was the attitude.

 I knew I'd encountered yet another IT egomaniac who felt threatened by the truth.  At the end he asked if I had any questions and of course I asked him what his ideal candidate looked like. 

By the way hiring managers, it's a great question and people like me only ask it to see if you've been listening to us.  If I don't ask it, I don't care.

He responded with a parade of meaningless buzzwords and brand names (most of which I was familiar with by the way) and nothing about serving the customer.  That told me he was just looking for a mindless automaton and in retrospect I should have ended the call right there.  Unlike him, however, I try not to make snap judgments.

Considering this position was customer facing the number one priority should have been my attitude toward service.  That goes double when you consider how heavily customer facing my career has been to date.  Instead, he chose to focus on buzzwords.  When asked if I had any other questions I gave him the opportunity to come clean. 

I asked, "So what how do you feel about me as a candidate so far?"  His response, " Not too good"
I ended the phone call. 

I've dealt with hundreds just like him and knew we were never going to be on the same page.  Being in the field as long as I have been I've had the opportunity to be on the other side of the desk.  That means I have my own criteria in mind whenever I'm in the interview process.

 For example; If I'd feel comfortable hiring my potential boss in my own organization then I know it's going to be a good fit.  If, however, I know I'd be kicking them out the door faster than they came in...

Look IT Managers, If you're passing over dedicated, motivated and experienced candidates because their qualifications don't stroke your ego you really need to get out of the field.  Somehow, somewhere along the line you've forgotten your place and now...

You're just in the way.

Thursday, August 18, 2011

Maintenance Windows

Subtle admonitions, Strict adherence to SLA's or boldface demands...

Ever try to convince an entire enterprise that you need to take down their servers for a weekend?  There's always resistance and even if you get your time window somebody's going to complain that they can't get to their stuff. 

Maybe somebody higher up in the organization will make you postpone your maintenance window just because they can.  "No, we can't send our satellite office of 3 people home an hour early.  It will impact our performance!"

Ah, office politics.  The great monkey wrench...

It's understandable in this day and age of 24/7 everything that users expect zero downtime.  That's reasonable given ideal circumstances. My experience has yet to show me an enterprise where that ideal exists.

In fact, it's impossible unless the enterprise is based on IT.  Think online universities or Large software companies.  Unless IT is at the core of the business it's not a priority.

Strangely enough, IT is at the core of most businesses whether the business knows it or not.  Your users just take it for granted.  "It worked yesterday so it'll work tomorrow so there's no need to inconvenience me."

There's a few approaches to deal with this.

You can just ignore the necessary maintenance and wait for something to blow up.
Then you get all the time you need to take care of things.  The downside is you're probably going to lose a weekend, the department will be blamed for being incompetent and somebody's going to get shown the exit.

You can force extensions to maintenance window by ignoring the predetermined time limits but you won't make too many friends and that exit door is likely to be in your future.

You can make the argument that 24/7 availability is unrealistic without putting resources in place to make it possible.  That's reasonable but will likely fall on deaf ears.

Seems like a no-win scenario. 

Unless you can get support from someone other than your IT director it is.

Unreasonable maintenance windows and a lack of proper resources is a systematic problem outside of your ability as an IT professional to fix. 

The fact is, if you're in an organization that won't allocate resources to meet demands then you need to get out.  It's really that simple so don't overthink it.

Discovery Channel's Mythbusters may have been able to successfully polish cow patties but in the end they were still cow patties.  Take a lesson from that...

Wednesday, August 17, 2011

The 2 faces of an IT pro.

So after my first post you've probably concluded I'm an arrogant SOB with an attitude problem.

Well, I'll fight you tooth and nail about the arrogant part but the attitude problem I'll fully embrace. :)

Over my career thus far I've seen two primary types of IT people; I like to call them the Fundamentalists and the Frauds.

Wow, that sounds like some kind of profound observation there!  Actually, I'm just pleased that I found another word that starts with "F" to go along with "Fraud".

Now I'm not saying that there's a bunch of IT people who sacrifice Xbox consoles to some giant, flame encircled Proliant server somewhere.  Nor do I suggest the other group is running bot nets and stealing PIN codes from your local ATM.

No, what I'm getting at has more to do with an IT person's motivation for doing the job.

Fundamentalists (from my point of view) are those whose motivation stems from a deeply rooted desire to leverage technology for the benefit of their organization.  There's no room for BS in this definition and the status quo is nothing more than a starting point.  Force a fundamentalist into a badly managed IT organization and you'll soon have a full scale revolt (of 1) on your hands.

We'd all like to believe we fit that definition wouldn't we...

Maybe, maybe not.  A devout fundamentalist strives for the ideal to exclusion of all else.  That can be a problem.  I've met brilliant IT people who couldn't hold up their end of a conversation to the point of almost social retardation.  Unless you're writing code for Face book your career opportunities will be few and far between. 

Now, the Frauds (again from my point of view).  Frauds aren't necessarily bad IT people.  They're not the laziest or least dedicated.  In fact the very thing that makes them frauds is the elaborate construct they painstakingly maintain just to appear valuable to their organization.  At some point the Fraud settles for the status quo and tries not to be the squeaky wheel.  Only high profile projects that support their construct are given priority and all available resources are marshaled to support it.   Frauds aren't born, they're made and we've all been one or will be at some point in our career.  Here's why...

At some point many IT people tire of running headlong into the brick wall that is senior IT management that many enterprises employ.  Instead of constant frustration they learn to game the system by only involving themselves in projects where their supposed herculean efforts can be easily seen. 

If there isn't a high profile project available they'll often create one.  They'll work innumerable hours, sacrifice personal life and family just to maintain the construct.  Unfortunately, all this time and effort maintaining their image leaves little time for dealing with...wait for it...Yes! the fundamentals of their job.  Maintenance of the infrastructure and upgrading of skills fall by the wayside with more work being assigned to lower level IT people and heavier reliance on outside consultants to perform tasks that should be basic to their job.  "Ah, but if senior management sees my great deeds and supposed dedication I'll be just fine!"  Yes, for awhile you will...

I've run into both types in varying degrees.  Maybe it's the former clerk who happened to be the techie person in the office and is suddenly the network administrator.  Being an expert in Excel and clearing paper jams in the printer is a poor foundation for dealing with network issues.  Still it was probably a bump in pay and how hard could this computer stuff be anyway.

It could be the grizzled veteran who just got sick of hearing the word "No".  So he/she gets the lay of the land and figures out what it takes to make the executive suites happy with the minimal amount of effort. 

The key difference here is the divergence of motivations.  The fundamentalist motivation is rooted in accomplishment with or without the laurels.  The Fraud is more concerned with self preservation.

Stay in a badly managed organization long enough, however, and even a fundamentalist can become a fraud.  The lesson here: Get out when you don't care anymore.

There is no black and white steadfast rule.  Life is about shades or grey.  We're IT and we have to deal with people so a degree of self-preservation is of course a good thing from a financial point of view at least.  Still the motivation has to be (or should be) more fundamentalist than Fraud.  Let's give it a ratio of 80/20 nodding toward the fundamentalist. 

Anything less and your just wasting time and making yourself miserable.  How much fun can it be to constantly be watching your back anyway?