Happened to me today:
"The NFS mounts are all broken, what's changed on the server?"
"Nothing"
"Well, obviously something's changed, or else it would work. What's changed?"
"Nothing"
...
Several more times back and forth...
"Listen, it's pretty [bleeping] obvious that you have [bleeping] changed something. What the [bleep] have you changed so that I can fix my [bleeping] systems and get some [bleeping] work done!"
"Take a chill pill, and relax dude!"
Bull? Meet Mr. Red Flag.
The actual problem? They changed the directory layout of the NFS mount, so everything moved around. Amazingly, he was surprised when everything stopped working.
Lesson? I need to realise that although the guy is lying his ass off, I need to walk away instead of rubbing his nose in it. I'm not going to get a resolution. I need to follow my own advice, and give people back their monkeys.
So, in the future, I'll just send a polite email (CC'ing my supervisor, and their supervisor), explaining why no one in the entire support organisation is able to get any work done. Hopefully they'll get it resolved by the end of the day.
Oh wait, just yesterday I gave someone advice because they had a technical manager doing exactly that! What did I say? Oh yeah, "He's looking for a reason to slip his project and he's looking to blame you. He should know how to manage his own development systems."
So, either I look like a dick because I put the beat down on some poor overworked IT guy who seems to have been here late screwing it up in the first place (and then bald-faced lying about changing anything), or I look like a dick because I'm looking to blame someone for my schedule slippages.
Hmm. I think I'll walk away and talk to someone else in the team next time. That guy has managed to flip my idiot bit.
2 comments:
Couple of key points that you forgot to mention in relation to this specific event:
1. You completely bypassed all procedure by what you did. The procedures that we have in IT are there for a reason. And coming up to someone's desk in the morning with a sore attitude won't get your request handled any faster. So most important lesson to learn is *SUBMIT a TICKET* no matter how important (you think) your problem is.
2. The problem that was solved by this slightly haphazard (and yes rushed) implementation had been around for almost a year, with noone daring to touch it for fear of the type of backlash that you created.
And some things you may not be aware of:
1.IT gets dozens of requests marked "urgent" and a testbed or time for planning/testing at this company is a pipedream. Well unless they get a few more sysadmins which are sorely needed.
2. There was a plan for this specific task, but it went awry because of some complications. And yes as you mention it was done late at night by overworked sysadmins.
3. IT often has to fend for itself against angry users like yourself. And that fell squarely on the sysadmin's shoulders in this case as the IT manager would just sit back and do nothing.
In any case, another crucial lession is control your temper, cause it doesn't have any positive results. And try to empathize sometimes before saying the "f" word!
Not cool at all, and no excuses for it. IMHO you should've received a written warning for your actions.
-T
Dear Anonymous
Fifteen years of working in the IT industry, in systems administration, service desk management, business analysis, and training & implementation management has given me a fairly rounded view of IT operations from all perspectives. After reading your comments I feel compelled to respond.
To address the first two points of the comments...
1. Classic Incident Management scenario. If the organisation can't work it is classed as a Category 1 fault call. That means you stop whatever else you are doing and resolve the Cat1 incident. ...and Yes, that is a NOW thing. Major impact = grumpy users. If the customer comes to you in person with a Cat1 fault, start your investigations immediately, don’t want for the call to be logged. Procedure must be followed – but not to the detriment of the customer. “Thanks – I’ll start looking at that now – can you put all the details you know in a call so I can refer to that as I go?”
2. Effective Change Management will resolve that. There is no excuse for a haphazard and rushed implementation. Understand what you are going to change, understand the impact of the change and understand the risks of the change. Finally, mitigate the risk - communicate with the customer and understand CLEARLY what your fall back and back out position is.
Incident and Change Management are the bread and butter of IT Operations. ...but even before ITIL, they were the common sense foundation of any IT Operations Department. By far the most common cause of service outages is uncontrolled IT changes. It’s dumb, and even dumber because it’s preventable.
With reference to the other points raised;
1. Get over it. Welcome to the real world - IT operations all over the globe work this way. Unless an IT service can prove it's a value-add, it's a cost. Do the best you can with what you have.
2. With effective Change Management this would have been less of an issue - that is why it is important to understand fall-back and back-out plans. Regardless of the progress of the change, whether successful or backed-out ALL SYSTEMS should have been up and running at the end of the Change window - when users were logging back in.
3. If system administrators are providing and supporting the IT services i.e. if there is no formal Service Desk, then they should front the service with the customer. It's not an ideal set up, but many small companies with limited resources operate this model successfully. IT Management should become involved as an escalation point, or as an Incident Manager for Category 1 Faults.
The next few paragraphs seem to be the pain point.
I am a little surprised by the “it was done late at night by overworked sysadmins”. Are you seriously saying this to the service customer? If one of my support staff had taken that approach with a service customer I would have had a quiet conversation with him/her. Regardless of the level of frustration, that is a totally unprofessional response and very likely to escalate the situation. Which I am assuming, is what happened. You’re not selling fish – you’re supporting an IT service and you must understand that in the current day it is critical to any business operation.
I have over 10 years experience in the banking and finance sector and I have seen my fair share of unhappy IT customers. It is beyond naive to expect a customer to tolerate a catastrophic failure of service without some volatile feedback. It goes with the territory. ...and in the commercial world it can incur $$$'s in penalties.
Posting a comment like you have done on this blog is incredible. It illustrates the lack of professionalism and pride in the service provided. You’re arguing with your customer, more than that, you’re arguing with your customer in the public domain. You’re not looking for a resolution. You’re not looking for a compromise. You’re not even looking to see what you can learn from the experience. I hope that other readers realise that some system administrators are very customer focussed and work incredibly hard to provide an excellent standard of service. I know I have worked with some outstanding ones. In fact, I am crossing my fingers as I write this that you don’t work for me. If you do, we need to have some conversations about service delivery. The starting one being why on earth you think this is a suitable conversation to have in the public domain?
I’m hoping this was just a one-off bad day that should have been forgotten about and everyone is now moving on. Although I would suggest you have built yourself some hurdles with your customer that you now need to overcome.
I would leave you with two thoughts:
1. Let it go. Notch the day up to experience – make sure you take what learns you can from it – and move on. You need to find a way forward with your customer. You are providing the service; you need to build the bridges.
2. From a Service Delivery perspective, my rule of thumb has always been that if my customer is going bonkers there is clearly something I have failed to understand, manage or deliver. People don’t go bonkers for no reason.
Post a Comment