Re: configuring remote headless servers

[steve%prd.co.uk@localhost (Steve Blinkhorn)]

I'm grateful for the sharing of wisdom and experience.    I have
worked out that the servers most likely do have IPMI (they are Fujitsu
Siens Primergy RX100 GSO1), but given their age I suspect it will prove to be
an early version.

I saw something in the BIOS setup that looked related, but given the
urgent need to get them back into service I did not have time to
experiment at base and dare not set them into a novel configuration
(for me).   I have this problem of physical disability which prevents
me working on the machines directly in the machine room.

Perhaps if the ISP who provided them in the first place had thought to
configure IPMI then, my life would have been significantly easier these
past few weeks.

But for now my original question still stands: what about using
/fastboot?

I'm not ignoring the other suggestions, e.g. cross-connecting serial
ports, but at the moment they're not practical.
o

--
Steve Blinkhorn <steve%prd.co.uk@localhost>

> 
> steve%prd.co.uk@localhost (Steve Blinkhorn) writes:
> 
> > Following on from the recent saga of upgrading from 2.0 to 7.0 which
> > assiduous readers may recall, the servers were re-installed in their
> > racks in the data centre.   All was well with one of them but the
> > other apparently failed.   It took three days for an engineer with
> > sufficiently developed skills to become available: He solved the
> > problem by switching the server on.
> >
> > But this led me to wonder how I would cope if, for instance, a server
> > came up in single-user mode requiring an fsck.   Once upon a time I
> > was able to assume that this would be a circumstance familiar to data
> > centre staff, but no longer.   What I would need would be a boot
> > sequence that started the network before any file system checking and
> > allowed remote login.   Alternatively, file system checking could be
> > disabled by default - even if the system went down by power cycling
> > the machine.
> >
> > I can see from the man pages for shutdown(8) and fastboot(8) that
> > there is provision related to this kind of circumstance.   Would it
> > simply be a matter of having an empty file named /fastboot in the root
> > directory?   If it matters, these are i386 machines.
> >
> > Any gotchas with this approach?
> 
> 
> Hello...  There has been several good responses to this, so I doubt that
> I will add much...  but anyway...
> 
> You will really want some sort of remote console, for real and true.
> This means either a serial console or some sort of internal or external
> console redirection.
> 
> For the serial console route, there is support in NetBSD to redirect to
> a serial port all of the console output when the kernel boots.  This
> would take care of your fsck example.  Couple this with a PDU that is
> network connected and can cycle plugs and you can power cycle the system
> and pretty much watch it boot up.  As for the device that is on the
> other end of the serial port, use your other system and cross connect
> them together.  This would require two serial ports per system and will
> work except when BOTH systems are down and nonfunctional.
> 
> Internal console redirection comes in the form of DRAC [Dell], iLO [HP]
> or IPMI [in some cases].  This works well and will provide total console
> redirection even of the BIOS boot process.  There may be an additional
> license required for advanced features, but you may not need those.
> Also, Amazon and ebay often sell the bits and pieces cheaply. This
> arrangement is, by far, the most functional.  DRAC and iLO will allow
> you to power cycle the systems without using an external PDU and you can
> pretty much see everything.
> 
> External console redirection is in the form of a network connected KVM
> box that sits on the video output and keyboard output of the system.  It
> is possible to get very cheap versions of these that MAY just work out
> for you, as long as you keep the arrangement simple [don't chain KVMs to
> KVMs, and the like].  Couple this with a network connected PDU and you
> can hard power cycle the systems pretty simply.
> 
> In a number of these cases it is required that the network connected
> device have Internet access of some form or that there be a jump box /
> VPN arrangement that will allow incoming connections to the PDUs and
> etc..
> 
> Someone mentioned the use of a thumb drive to boot up a minimal kernel
> with openssh running.  That was clever in a number of ways.  It would
> require, probably, someone who can place the thumb drive in the system,
> but they would not have to be any more talented than that.  You could
> probably tie the thumb drive to the system physically such that all
> someone would have to do is place it in a USB port.  Likewise you could
> do something with ANY external booting device such as a CD.  The down
> side is that it wouldn't help with network issues, but most of these
> solutions are dead in the water with that issue anyway.  The real trick
> with this is coming up with the boot, but honestly that wouldn't be very
> hard and could be developed pretty much entirely off site.
> 
> 
> I have been working with data centers and remote sites for a long time
> and I would agree that the "helping hands" support has gone down hill a
> lot and your experience does not supprise me in the least.  It is always
> better to be self reliant as much as possible, if you can manage it, and
> not rely too much on the on-site staff.
> 
> 
> Good luck with your situation.... 
> 
> 
> 
> -- 
> Brad Spencer - brad%anduin.eldar.org@localhost - KC8VKS
> http://anduin.eldar.org  - & -  http://anduin.ipv6.eldar.org [IPv6 only]
>