Subject: Re: more work in rc.d [was Re: rc, rc.shutdown proposed change]
To: Kevin P. Neal <kpneal@pobox.com>
From: John Nemeth <jnemeth@victoria.tc.ca>
List: tech-userlevel
Date: 03/24/2000 20:10:37
On Aug 3,  9:08pm, "Kevin P. Neal" wrote:
} On Mon, Mar 13, 2000 at 10:52:03PM -0800, John Nemeth wrote:
} > On Jun 29, 10:53am, "Simon J. Gerraty" wrote:
} > } Did I ever mention that the _biggest_ advantage of individual rc scripts has
} > } nothing to do with boot/shutdown managment?  Its being able to tell the 
} > } night watchman at the end of the line to "type '/etc/init.d/ntpd restart'"
} > } or whatever, rather than explain how to run ps, kill etc etc.
} > 
} >      I do not consider this to be an advantage, since there is no way
} > that person should get anywhere near a root prompt.  If the system is
} > so critical that it can't wait for a qualified person to come on site,
} > then the company better be prepared to do whatever is needed to get
} > that qualified person remote access or pay for qualified people to be
} > on site 24x7.  Otherwise, they are just asking for trouble and are
} > likely to have bigger problems.
} 
} Uh, no. What bigger problem do you refer to?
} 
} Say a problem happens at, oh, 5am. Wouldn't it be smart to
} have a 2xMinimumWage person watching for a blinking red light, and
} then typing one command that will get the system through the rest of
} the night? This way the "qualified person" (who gets paid much more
} than 2xMW) won't have to be called, woken up, pissed off, and then 
} compensated?

     You're missing several aspects of the real world here.  First,
night watchmen rarely have computer access at all, and if they do, then
it is limited to one program or a menu with just a few programs.  They
most certainly wouldn't have access to a shell.  Second, blinking red
lights rarely happen, and when they do it usually indicate a serious
problem.  So, it would almost always take a "qualified person" to
diagnose the problem.  If the night watchmen were capable of doing it,
they'd be after your job.  Also, the example of '/etc/init.d/ntpd
restart' is rather silly.  Something that is that easy almost never
happens (unless the sysadmin is incompetant) and there is still the
issue of diagnosis.  Finally, if the "qualified person" gets pissed off
at being woken up, then they are in the wrong job / industry.

} After all, what's the qualified person going to do in the middle of the
} night? Type the same command in to get the system limping along, and
} then go back to sleep. The real problem can be fixed during normal
} business hours (at least typically, in my experience). 

     I don't know what your experience is, but I would say that you are
missing real world experience.  I have spent many hours in machines
rooms in the middle of the night, usually because of hardware problems
(systems that I run rarely go down for any other reason), or because
I'm doing off-hours upgrade/maintenance.  If the machine is important
enough that the qualified person has to come in during the middle of
the night to get it going, then it is important enough that it be fixed
completely.  This is either because the machine is 24x7 mission
critical, or because it would cost major money for the machine to be
down during normal business hours (think of lost wages due to many idle
hands and/or lost sales; these items are why you see really high costs
attributed to downtime, it's not the costs of repairing the system and
bringing it back up).

} You can even use sudo/ru/similar to keep the night watchman from 
} executing any commands other than /etc/init.d/* to minimize the
} damage that said person can do while still allowing for a big
} cost savings from not calling the skilled person.

     You can do a h*ll of a lot of damage with just that, especially
since most of those scripts aren't written in a secure manner.  And,
there is still the issue of diagnosing the problem.  If the person is
capable of diagnosing the problem and choosing appropriate corrective
actions then they might as well be the sysadmin.

}-- End of excerpt from "Kevin P. Neal"