Re: configuring remote headless servers

[Swift Griggs <swiftgriggs%gmail.com@localhost>]

On Wed, 31 Aug 2016, Steve Blinkhorn wrote:
> It took three days for an engineer with sufficiently developed skills to 
> become available: He solved the problem by switching the server on.

Having found no good way to truly address issues like this without some 
control of my own, I don't deal with an ISP that won't give me power 
control and console. HP ILO's are a good solution since they can be used 
for both a hard power cycle and give you a real remote console. If your 
console stays in text mode, you don't even have to license the iLO. Many 
server BIOSs' have a mode whereby they can provide console support via a 
dedicated serial port (Tyan comes to mind as one of these). If you combine 
that functionality with something than can do remote power control (like a 
Baytech RPC or APC network PDUs) then you've got the same features. 

> But this led me to wonder how I would cope if, for instance, a server 
> came up in single-user mode requiring an fsck.

If you have true console access it wouldn't matter. You'd do the fsck then 
keep truckin'.

> I can see from the man pages for shutdown(8) and fastboot(8) that there 
> is provision related to this kind of circumstance.

I'll just apologize because I doubt my response was what you were looking 
for. I'll simply say this, when it comes to hosted systems, the faster the 
system can bring up the network and ssh with the absolute minimum of 
dependencies, the better. AFAIK, I've never seen an OS that really "gets" 
this, as evidenced that even though OS's *could* use their 
ramdisk/miniroots to launch OpenSSH (and statically link it), they rarely 
do (and there are some reasons, but I usually disagree with their 
importance). For a server without a decent console, having Openssh started 
is a defacto the same thing as having a usable server, thus the strategy 
should place a categorically *premium value* on doing that as soon in the 
boot process as possible with the least number of dependencies.

Also, NetBSD has the ability to redirect the console to a serial port as 
soon as the kernel starts booting. However, you'd need a serial console in 
place first before you can take advantage of that. However, in your 
scenario of the system needing an fsck and stopping the boot process, it'd 
save you from having to call some data-center hands & eyes at your ISP.

-Swift