> > 	i suggested markus to include the reasoning behind the way 3.3 -> 3.4
> > 	upgrade path was annouced.  i think it will help a lot of people to
> > 	understand why it had to be handled this way.
> 
> The reasoning was clear but unreasonable..
> 
> The idea that a significant fraction of end users could just
> immediately drop everything and upgrade to 3.3 is completely
> ridiculous...

Well, I heard from one Fortune 100 company that updated 2300 machines
to 3.3 in 15 hours and accepted the functional problems for the 4 day
window; a very major foreign government agency that filtered protocol
22 on all of their core routers; a large ISP that went through their
sshd_config files and cut down the functionality as much as possible
(and luckily made the right guesses, without any disclosure hints from
me -- they asked "did we limit it enough?" and I said "perhaps.  I
cannot disclose" and they replied "we understand, thanks, we're going
to stick with this stance."), and also from hundreds of individual and
corporate users of unknown size, who disabled or filtered it in some
ways.  SPRINT NOC called me at home and asked some questions,
apparently they realized that our machines are on their network, and
wanted to make sure they would not suffer from this and stop us from
releasing a new version -- that was a real wow moment.  I have
received hundreds and hundreds of mails about this.

That is a very significant community that were fore-warned.

For those who couldn't respond with taking a security stance, I am
sorry.  Their situation is no different than no warning before
disclosure, and yelling at me does not change the way I handled this
or will handle future things like this.

Perhaps it is those user's fault for not being able to respond, and
they should consider how to build infrastructures that are faster at
responding to warnings of future disclosure... or very nasty immediate
disclosures; such as, let's say, resolver issues which we (Joost,
Itojun and I) dislosed in 6 hours instead of via CERT because it
became clear to us that FreeBSD and NetBSD security officers were BUSY
LEAKING THE INFORMATION?!?  (I do not intend to play that blame game
publically at this time).  In that case, we wanted to carefully
disclose the resolved issue but seeing as it was nighttime, there was
no way to contact any vendors, and our hands were forced by
irresponsible "vendor" actions.  I've heard no great shock from the
"security community" at this .... shall we say, user community
tragedy.

As to the community that are thanking me, I kind of wish they would
stop sending me mail -- i've got work to do, but learning about the
places that are using OpenSSH, it is pretty impressive... even when it
is groups like the Israeli army who are being real jerks in the West
Bank...

And as to the bigger picture, both Niels and I have been talking for
quite some time about the worry of SSH usage becoming mono-culture of
OpenSSH only; this is very becoming VERY VERY WORRYING!

	http://openssh.com/usage/graphs.html

But unfortunately every other free SSH server project has been lagging
in relation, as some of their developers sit around and blame us,
rather than finishing their own software.

> particularly since the "privilege separation" features
> necessary to defend against the unmentionable bug were *advertised* as
> having functional regressions!

in 3.3, privsep worked 90% on almost all systems.  5% of that was that
very old solaris or linux 2.2 could not do mmap() with
MAP_ANON|MAP_SHARED, hence compression had to be turned off -- and the
3.3 release documented that to some level (mailing lists did a better
job).  another 5% was that various advanced authentication methods in
linux PAM had to be disabled, which by the way is exactly where the
bug was in any case, and by the way, that bug IS exploitable on some
linux's.  oh yeah, and let us say that .001% is openbsd big-endian 2.9
and 3.0 which has a bug in fd passing... making privsep fail very
badly, perhaps even crash the kernel.

These privsep things were announced to be a "functional regressions"
problem for about 4 days (till monday AM), but then ISS published early,
and guess what... all the Linux vendors during that time (except for
RedHat) jumped, and helped us, and when 3.4 shipped, it was fixed on
all those systems!

The commercial vendors of course were unable to respond, and many have
still to this day NOT release a fixed sshd...

Many people were very happy with the way this was handled.  almost all
the people who yelled at me for how i handled this have sent me
apology letters, once they reached understanding.  The people who have
not done so, I can put in a single basket: Those who already do not
like me.  Oh well.

When ISS decided to give me about 12 night-time hours that they would
be disclosing immediately, there was also no time to contact vendors,
since our vendor community appears to be over 80 players (Like, good
god, HP is apparently thinking of putting OpenSSH running on top of
vxworks onto some of their network laser printers... scary shit).  And
as to CERT, well this is the fastest release of information they have
ever done, according to them it is the first announcement they have
ever sent out within 24 hours.  That was done at my utter insistance;
I said that if an announcement did not go out same day as the ISS
announcement, they would never get help from me again (I assist them
with a certain free service to them every 2nd or 3rd day or so).  I
asked them what the 'E' in their name means if they cannot make a
major internet infrastructure attack notice within 24 hours.

About early bug disclosure to vendors, it was impossible to do so because:

	If I said disable ChallengeResponse, that points that the
	bug is in 500 lines of code

	If I said disable Protocol 2, that points that the bug is in
	5000 lines of code

	If I say it is 2.9.9 and up, that says it is likely in
	ChallengeResponse, since that was the major rewrite between
	2.9 and 2.9.9

	And later we found out the bug actually is also to be found in
	all versions from 2.3.1

	And finally, if I told FreeBSD security-officer then 12 people
	would have known and started leaking the information to NetBSD
	and who knows where else within a 5 hour window, as we saw a few
	days later.

I believe firmly that your supposition that information like this can
be handled carefully is very false.

I think you are being unfair to me :-(