Why it's fun

I don't talk about work too much here.  My employer's policy on blogging and other uses of social media is very strongly worded, and I'm under no illusions as to the speed with which my employment would be terminated if I brought up dirt on the company in such a public forum.

The IT organization I work in is pretty mature.  In a field where you expect to move to a new job every two years, our company still expects that if you work hard and stay out of trouble, you will retire with a pretty generous set of benefits after more than 20 years of doing various jobs within the organization.  The trade off for that is you don't tend to work with the bleeding edge of technology, and you do things their way unless told to figure out something new.   Big companies don't stay profitable by chasing the technology dragon and trying every high-speed thing that comes up at conferences.  There's something to be said for using mature, stable technology to turn a profit.

We work under a lot of guidance to use the same procedure to do the same thing every time.  Standardization and lots and lots of planning is the order of the day.  This can lead to a lot of talking followed by short bursts of actual technical work.  My normal rhythm is to spend six to eight weeks of planning and coordination followed by four to six weeks of implementing those plans.  Of course, there's always the day to day work of monitoring my systems and fixing things that break.  Also, not everything is as well planned as it should be, so there is also a lot of work that goes from requirements and planning to implementation and documentation pretty quickly.

Sunday night was supposed to be a pretty boring example of "plan a lot, implement a little".  All I was doing was patching the OS on a few servers and working with a hardware technician to update the firmware on the box itself.  Sounds pretty simple and boring, doesn't it?

Three quarters of the work was exactly that.  I worked from a good project plan, and the techs who worked with me were competent and had done their part before.  Everything went smoothly, at least for a while.

It's that last quarter that made the job interesting.

One of the servers went into a reboot loop.  That means that after applying power to the system, the operating system would try to come up, something would happen, and it would restart.  Repeat ad infinitum.

One of the other servers refused to even get that far.  The system would come up and test itself when plugged in, but would stop before even trying to boot.

For the first box, we did a lot of troubleshooting.  Booting into a maintenance mode didn't show anything wrong with the system.  The hardware logs were useless, and when we'd checked everything and bounced the box, it went right back into its loop.  The technician grabbed a spare box, put the hard drives from my system into it, and it booted as if nothing was wrong.  Well, almost.  All of those reboots had damaged a few things, so I had to fix those too.  Eventually, though, the system came back up as happy as could be.

The other box required some hardware voodoo to fix.  The technician rolled the bones, muttered some profane incantations, and the box came up.  From my perspective, this one was easier than the other one, but still required some recovery work to fix things.

Guess which portion, the easy and sane or the harder and insane, portion of the nights work I enjoyed more.  If you said "You enjoyed reacting to unforeseen problems", there is a box of cookies in the back of the room.  You can help yourself.

There's a fine line between challenging and grinding, and that night I was in the zone.  Three quarters of the night was spent following a project plan.  The other quarter was spent diagnosing and solving problems, and I had a ball doing it.

In any job, you have to strike that balance.  If all you do is follow procedures all day, you'll get bored and lose interest.  If all you do is fight dragons, eventually you'll burn out.  Finding the right balance between boring/easy work and interesting/challenging work keeps your head in the game for the long haul.