bin/45639: STARTTLS fails sometimes with builtin OpenSSL

[rhialto%falu.nl@localhost]

>Number:         45639
>Category:       bin
>Synopsis:       STARTTLS fails sometimes with builtin OpenSSL
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    bin-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Nov 21 20:00:09 +0000 2011
>Originator:     Rhialto
>Release:        NetBSD 5.1
>Organization:
        
>Environment:
        
        
System: NetBSD radl.falu.nl 5.1 NetBSD 5.1 
(Radl-s_Pervasion_of_the_Incorrect_Chord) #0: Mon Jan 24 20:25:13 CET 2011 
root%vargaz.falu.nl@localhost:/usr/src/sys/arch/amd64/compile/RADL5.1 amd64
Architecture: x86_64
Machine: amd64
>Description:
        I had some mail to a mailbox @tele2.nl. Last weekend they did
        some work on their mail server, and since then, my sendmail
        (from pkgsrc-2011Q3) fails when it tries the STARTTLS command.

$ sudo sendmail -v -q

Running /var/spool/mqueue/pALFphxl017089 (sequence 1 of 1)
<khrjhdsrjkhsdjkf%tele2.nl@localhost>... Connecting to smtp.tele2.nl. via 
esmtp...
220 mailfe06.swip.net ESMTP 5.4.2
>>> EHLO smtp.falu.nl
250-mailfe06.swip.net is pleased to meet you
250-DSN
250-SIZE 314572800
250-STARTTLS
250-AUTH LOGIN PLAIN
250-ETRN
250-TURN
250-ATRN
250-NO-SOLICITING
250-8BITMIME
250-HELP
250-PIPELINING
250 EHLO
>>> STARTTLS
220 please start a TLS connection
<khrjhdsrjkhsdjkf%tele2.nl@localhost>... Deferred: 403 4.7.0 TLS handshake 
failed.
Closing connection to smtp.tele2.nl.
$ 

The mail log file gives a little bit more detail:

Nov 21 16:57:47 radl sendmail[6980]: STARTTLS=client, error: connect failed=-1, 
SSL_error=1, errno=0, retry=-1
Nov 21 16:57:47 radl sendmail[6980]: STARTTLS=client: 6980:error:14092073:SSL 
routines:SSL3_GET_SERVER_HELLO:bad packet 
length:/home/builds/ab/netbsd-5-1-RELEASE/src/crypto/dist/openssl/ssl/s3_clnt.c:906:
Nov 21 16:57:47 radl sendmail[6980]: ruleset=tls_server, arg1=SOFTWARE, 
relay=smtp.tele2.nl, reject=403 4.7.0 TLS handshake failed.
Nov 21 16:57:47 radl sendmail[6980]: pALFphxl017089: 
to=<khrjhdsrjkhsdjkf%tele2.nl@localhost>, 
ctladdr=<rhialto%radl.falu.nl@localhost> (1000/1000), delay=00:06:04, 
xdelay=00:00:03, mailer=esmtp, pri=570349, relay=smtp.tele2.nl. 
[212.247.156.14], dsn=4.0.0, stat=Deferred: 403 4.7.0 TLS handshake failed.

According to the 2nd line, apparently the NetBSD base system SSL library
thinks something is wrong with the data it receives. I don't think I've
seen this before but of course it might be a bug in this version of
OpenSSL that simply wasn't triggered before.

Indeed, when I re-compile sendmail using OpenSSL from pkgsrc (I added
"PREFER.openssl=pkgsrc" to its options.mk file), the above scenario went
fine:

Nov 21 20:39:46 radl sm-mta[17675]: STARTTLS=client, relay=smtp.tele2.nl., versi
on=TLSv1/SSLv3, verify=FAIL, cipher=AES256-SHA, bits=256/256
Nov 21 20:39:48 radl sm-mta[17675]: pALJdhjB026543: to=<kssdkfjdhfjkhfkhkhlsdajk
h%tele2.nl@localhost>, ctladdr=<rhialto%radl.falu.nl@localhost> (1000/1000), 
delay=00:00:05, xdelay=
00:00:05, mailer=esmtp, pri=30367, relay=smtp.tele2.nl. [212.247.156.14], dsn=5.
1.1, stat=User unknown
Nov 21 20:39:50 radl sm-mta[17675]: pALJdhjB026543: pALJdojB017675: DSN: User un
known
Nov 21 20:39:50 radl sm-mta[17675]: pALJdojB017675: 
to=<rhialto%radl.falu.nl@localhost>, d
elay=00:00:00, xdelay=00:00:00, mailer=local, pri=31529, dsn=2.0.0, stat=Sent



>How-To-Repeat:
        Send mail to somebody%tele2.nl@localhost, using STARTTLS, with sendmail
        compiled with the base system (5.1) OpenSSL.
>Fix:
        Workaround: Use the pkgsrc version.
        Fix: should probably be in the base system OpenSSL.

-Olaf.
-- 
___ Olaf 'Rhialto' Seibert  -- There's no point being grown-up if you 
\X/ rhialto/at/xs4all.nl    -- can't be childish sometimes. -The 4th Doctor

>Unformatted: