I've just returned from a late night session at one of my clients.
It all started with an email from a user at the site informing me that another user was having issues with their PC running slowly and not allowing their email client to function.
Most of my client sites are small offices with less than 10 users so reporting issues to me is an informal process. So while I got the report about the other user I also had a few requests from the user that sent the email.
Turns out the other user's problems were related to a rather nasty piece of malware (TDS4 rootkit) that did all those nasty things that rootkits tend to do.
It was polymorphic so it evaded the virus scanner...
It denied access to task manager and loaded the CPU to 100% with constant attempts
to download more malware...
And finally it tried to open random nefarious web pages.
I've been dealing with this kind of issue a lot lately but usually it's the XP 2012 Fake AV that fools users into installing the malware then digs itself in, destroys the user profile and in some cases downloads more malware allowing the infected pc to become part of a torrent serving botnet.
I had just cleaned up both the reporting user and the other user's pc 3 weeks before. They knew what happened and why. They were given admonition against trusting anything they didn't already use on a regular basis and shown what it was that caused the problem I had to fix.
So I got the obligatory nodding of the head and promise that they'd be more vigilant because after all security is everyone's responsibility right?
Well, I guess I should just accept that it's just my responsibility. You think I'd learn after almost 20 years...
The sad fact is that users can care less about the damage a malicious trojan or entrenched rootkit can do to their PC. After all, that's what you're there for and they expect you'll fix it before they're back from lunch. The next killer app that promises endless coupons or installs a cute dancing cow on their desktop will quickly nullify every attempt to counter such social engineering. It's not unlike a speeding driver who when caught blames the car for his actions because it goes too fast.
Not the best analogy, I know...
So the battle for social engineering is lost. It must be because we've been droning on about responsible use of computers for decades now and our advice is still largely ignored or at least quickly forgotten.
So now we have to take preemptive action. That usually involves the installation of a layered protection system consisting of not just Anti-Virus but also anti-malware software to save the user from themselves.
At my client sites I currently use Sophos for Anti-Virus and Malwarebytes for malware protection. I find it a good combination of security software that keep a small footprint and don't fight with each other when doing their jobs. This part is important. Avoid bloated packages that get in the way of workflow and perform only marginally.
I'm not afraid to say openly that I find Most Symantec and McAfee products to be absolutely useless when it comes to malware and rootkits. Worse, if the consumer versions end up in a business setting. They become almost completely ineffective and are sure to cause user complaints as these lumbering giants steal system resources and get in the way of every mouse click needlessly.
The KISS principle is very relevant here. Stick with products that do the one thing they do well and don't try to be anything else. I used to recommend AVAST! until it contracted the Symantec bloat disease and became an ineffective security solution.
If you find yourself at an infected user's PC searching the Internet for another software package to do what you thought you had already paid for it's a good indicator that it's time for a change. Sounds obvious but it's surprising how much an IT department will put up with just because they have a history with one vendor.
In some cases a user will figure out how to shut off the security software if they feel it's too intrusive. Try to avoid that scenario if possible. Unfortunately if your clients are still using Windows XP and have legacy software then you'll have a hard time keeping them out of the settings since there are still far too many applications that require administrator privileges. Since an Administrator account trumps all else, any pc with user running as local administrators is at risk. Expect some type of security issue at some point in this case.
In spite of all your efforts to deploy the perfect security suite, you're bound to get complaints from users that they can't get their favorite site to work anymore. What they don't tell you is that it's the same site that almost destroyed their PC on their last visit. They'll scowl and complain regardless of the evidence or they'll claim the security software interferes with their work.
Unless their work is collecting coupons or evaluating dancing cow version 2.3.2 I can honestly care less. I'm not draconian, I just don't want to bankrupt my client fixing the same problem over and over again. It's boring and hurts your credibility in the long run.
This is where communication comes in. You have to let your clients (the ones who sign your check) know what's going on and why you're doing it. Explain to them the implications to their business and stress the costs involved including: lost productivity, lost data and of course the cost of having you waste more time fixing the same problem.
The only way to fight entrenched bad habits is irrefutable evidence that it's costing your client/business money. Nobody in their right mind is going to argue your logic especially if they sign your checks
Tuesday, August 23, 2011
Thursday, August 18, 2011
Maintenance Windows
Subtle admonitions, Strict adherence to SLA's or boldface demands...
Ever try to convince an entire enterprise that you need to take down their servers for a weekend? There's always resistance and even if you get your time window somebody's going to complain that they can't get to their stuff.
Maybe somebody higher up in the organization will make you postpone your maintenance window just because they can. "No, we can't send our satellite office of 3 people home an hour early. It will impact our performance!"
Ah, office politics. The great monkey wrench...
It's understandable in this day and age of 24/7 everything that users expect zero downtime. That's reasonable given ideal circumstances. My experience has yet to show me an enterprise where that ideal exists.
In fact, it's impossible unless the enterprise is based on IT. Think online universities or Large software companies. Unless IT is at the core of the business it's not a priority.
Strangely enough, IT is at the core of most businesses whether the business knows it or not. Your users just take it for granted. "It worked yesterday so it'll work tomorrow so there's no need to inconvenience me."
There's a few approaches to deal with this.
You can just ignore the necessary maintenance and wait for something to blow up.
Then you get all the time you need to take care of things. The downside is you're probably going to lose a weekend, the department will be blamed for being incompetent and somebody's going to get shown the exit.
You can force extensions to maintenance window by ignoring the predetermined time limits but you won't make too many friends and that exit door is likely to be in your future.
You can make the argument that 24/7 availability is unrealistic without putting resources in place to make it possible. That's reasonable but will likely fall on deaf ears.
Seems like a no-win scenario.
Unless you can get support from someone other than your IT director it is.
Unreasonable maintenance windows and a lack of proper resources is a systematic problem outside of your ability as an IT professional to fix.
The fact is, if you're in an organization that won't allocate resources to meet demands then you need to get out. It's really that simple so don't overthink it.
Discovery Channel's Mythbusters may have been able to successfully polish cow patties but in the end they were still cow patties. Take a lesson from that...
Ever try to convince an entire enterprise that you need to take down their servers for a weekend? There's always resistance and even if you get your time window somebody's going to complain that they can't get to their stuff.
Maybe somebody higher up in the organization will make you postpone your maintenance window just because they can. "No, we can't send our satellite office of 3 people home an hour early. It will impact our performance!"
Ah, office politics. The great monkey wrench...
It's understandable in this day and age of 24/7 everything that users expect zero downtime. That's reasonable given ideal circumstances. My experience has yet to show me an enterprise where that ideal exists.
In fact, it's impossible unless the enterprise is based on IT. Think online universities or Large software companies. Unless IT is at the core of the business it's not a priority.
Strangely enough, IT is at the core of most businesses whether the business knows it or not. Your users just take it for granted. "It worked yesterday so it'll work tomorrow so there's no need to inconvenience me."
There's a few approaches to deal with this.
You can just ignore the necessary maintenance and wait for something to blow up.
Then you get all the time you need to take care of things. The downside is you're probably going to lose a weekend, the department will be blamed for being incompetent and somebody's going to get shown the exit.
You can force extensions to maintenance window by ignoring the predetermined time limits but you won't make too many friends and that exit door is likely to be in your future.
You can make the argument that 24/7 availability is unrealistic without putting resources in place to make it possible. That's reasonable but will likely fall on deaf ears.
Seems like a no-win scenario.
Unless you can get support from someone other than your IT director it is.
Unreasonable maintenance windows and a lack of proper resources is a systematic problem outside of your ability as an IT professional to fix.
The fact is, if you're in an organization that won't allocate resources to meet demands then you need to get out. It's really that simple so don't overthink it.
Discovery Channel's Mythbusters may have been able to successfully polish cow patties but in the end they were still cow patties. Take a lesson from that...
Labels:
24,
7,
afterhours,
computers,
down,
downtime,
I.T.,
Maintenance,
network admin,
servers,
service,
users,
work
Wednesday, August 17, 2011
The 2 faces of an IT pro.
So after my first post you've probably concluded I'm an arrogant SOB with an attitude problem.
Well, I'll fight you tooth and nail about the arrogant part but the attitude problem I'll fully embrace. :)
Over my career thus far I've seen two primary types of IT people; I like to call them the Fundamentalists and the Frauds.
Wow, that sounds like some kind of profound observation there! Actually, I'm just pleased that I found another word that starts with "F" to go along with "Fraud".
Now I'm not saying that there's a bunch of IT people who sacrifice Xbox consoles to some giant, flame encircled Proliant server somewhere. Nor do I suggest the other group is running bot nets and stealing PIN codes from your local ATM.
No, what I'm getting at has more to do with an IT person's motivation for doing the job.
Fundamentalists (from my point of view) are those whose motivation stems from a deeply rooted desire to leverage technology for the benefit of their organization. There's no room for BS in this definition and the status quo is nothing more than a starting point. Force a fundamentalist into a badly managed IT organization and you'll soon have a full scale revolt (of 1) on your hands.
We'd all like to believe we fit that definition wouldn't we...
Maybe, maybe not. A devout fundamentalist strives for the ideal to exclusion of all else. That can be a problem. I've met brilliant IT people who couldn't hold up their end of a conversation to the point of almost social retardation. Unless you're writing code for Face book your career opportunities will be few and far between.
Now, the Frauds (again from my point of view). Frauds aren't necessarily bad IT people. They're not the laziest or least dedicated. In fact the very thing that makes them frauds is the elaborate construct they painstakingly maintain just to appear valuable to their organization. At some point the Fraud settles for the status quo and tries not to be the squeaky wheel. Only high profile projects that support their construct are given priority and all available resources are marshaled to support it. Frauds aren't born, they're made and we've all been one or will be at some point in our career. Here's why...
At some point many IT people tire of running headlong into the brick wall that is senior IT management that many enterprises employ. Instead of constant frustration they learn to game the system by only involving themselves in projects where their supposed herculean efforts can be easily seen.
If there isn't a high profile project available they'll often create one. They'll work innumerable hours, sacrifice personal life and family just to maintain the construct. Unfortunately, all this time and effort maintaining their image leaves little time for dealing with...wait for it...Yes! the fundamentals of their job. Maintenance of the infrastructure and upgrading of skills fall by the wayside with more work being assigned to lower level IT people and heavier reliance on outside consultants to perform tasks that should be basic to their job. "Ah, but if senior management sees my great deeds and supposed dedication I'll be just fine!" Yes, for awhile you will...
I've run into both types in varying degrees. Maybe it's the former clerk who happened to be the techie person in the office and is suddenly the network administrator. Being an expert in Excel and clearing paper jams in the printer is a poor foundation for dealing with network issues. Still it was probably a bump in pay and how hard could this computer stuff be anyway.
It could be the grizzled veteran who just got sick of hearing the word "No". So he/she gets the lay of the land and figures out what it takes to make the executive suites happy with the minimal amount of effort.
The key difference here is the divergence of motivations. The fundamentalist motivation is rooted in accomplishment with or without the laurels. The Fraud is more concerned with self preservation.
Stay in a badly managed organization long enough, however, and even a fundamentalist can become a fraud. The lesson here: Get out when you don't care anymore.
There is no black and white steadfast rule. Life is about shades or grey. We're IT and we have to deal with people so a degree of self-preservation is of course a good thing from a financial point of view at least. Still the motivation has to be (or should be) more fundamentalist than Fraud. Let's give it a ratio of 80/20 nodding toward the fundamentalist.
Anything less and your just wasting time and making yourself miserable. How much fun can it be to constantly be watching your back anyway?
Well, I'll fight you tooth and nail about the arrogant part but the attitude problem I'll fully embrace. :)
Over my career thus far I've seen two primary types of IT people; I like to call them the Fundamentalists and the Frauds.
Wow, that sounds like some kind of profound observation there! Actually, I'm just pleased that I found another word that starts with "F" to go along with "Fraud".
Now I'm not saying that there's a bunch of IT people who sacrifice Xbox consoles to some giant, flame encircled Proliant server somewhere. Nor do I suggest the other group is running bot nets and stealing PIN codes from your local ATM.
No, what I'm getting at has more to do with an IT person's motivation for doing the job.
Fundamentalists (from my point of view) are those whose motivation stems from a deeply rooted desire to leverage technology for the benefit of their organization. There's no room for BS in this definition and the status quo is nothing more than a starting point. Force a fundamentalist into a badly managed IT organization and you'll soon have a full scale revolt (of 1) on your hands.
We'd all like to believe we fit that definition wouldn't we...
Maybe, maybe not. A devout fundamentalist strives for the ideal to exclusion of all else. That can be a problem. I've met brilliant IT people who couldn't hold up their end of a conversation to the point of almost social retardation. Unless you're writing code for Face book your career opportunities will be few and far between.
Now, the Frauds (again from my point of view). Frauds aren't necessarily bad IT people. They're not the laziest or least dedicated. In fact the very thing that makes them frauds is the elaborate construct they painstakingly maintain just to appear valuable to their organization. At some point the Fraud settles for the status quo and tries not to be the squeaky wheel. Only high profile projects that support their construct are given priority and all available resources are marshaled to support it. Frauds aren't born, they're made and we've all been one or will be at some point in our career. Here's why...
At some point many IT people tire of running headlong into the brick wall that is senior IT management that many enterprises employ. Instead of constant frustration they learn to game the system by only involving themselves in projects where their supposed herculean efforts can be easily seen.
If there isn't a high profile project available they'll often create one. They'll work innumerable hours, sacrifice personal life and family just to maintain the construct. Unfortunately, all this time and effort maintaining their image leaves little time for dealing with...wait for it...Yes! the fundamentals of their job. Maintenance of the infrastructure and upgrading of skills fall by the wayside with more work being assigned to lower level IT people and heavier reliance on outside consultants to perform tasks that should be basic to their job. "Ah, but if senior management sees my great deeds and supposed dedication I'll be just fine!" Yes, for awhile you will...
I've run into both types in varying degrees. Maybe it's the former clerk who happened to be the techie person in the office and is suddenly the network administrator. Being an expert in Excel and clearing paper jams in the printer is a poor foundation for dealing with network issues. Still it was probably a bump in pay and how hard could this computer stuff be anyway.
It could be the grizzled veteran who just got sick of hearing the word "No". So he/she gets the lay of the land and figures out what it takes to make the executive suites happy with the minimal amount of effort.
The key difference here is the divergence of motivations. The fundamentalist motivation is rooted in accomplishment with or without the laurels. The Fraud is more concerned with self preservation.
Stay in a badly managed organization long enough, however, and even a fundamentalist can become a fraud. The lesson here: Get out when you don't care anymore.
There is no black and white steadfast rule. Life is about shades or grey. We're IT and we have to deal with people so a degree of self-preservation is of course a good thing from a financial point of view at least. Still the motivation has to be (or should be) more fundamentalist than Fraud. Let's give it a ratio of 80/20 nodding toward the fundamentalist.
Anything less and your just wasting time and making yourself miserable. How much fun can it be to constantly be watching your back anyway?
Subscribe to:
Posts (Atom)