pkg/39863: firefox+negotiate doesn't play nice with heimdal

[kovert%omniscient.com@localhost]

>Number:         39863
>Category:       pkg
>Synopsis:       firefox+negotiate doesn't play nice with heimdal
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Nov 04 06:40:00 +0000 2008
>Originator:     Todd Kover
>Release:        NetBSD 4.0_STABLE
>Organization:
Omniscient Technologies
>Environment:
System: NetBSD saidin.omniscient.com 4.0_STABLE NetBSD 4.0_STABLE (GENERIC.MP) 
#1: Tue Jun 17 08:22:20 EDT 2008 
kovert%saidin.omniscient.com@localhost:/usr/obj/4.0-stable/i386/amd/guinness/usr/src/os/NetBSD-4.0-branch/src/sys/arch/i386/compile/GENERIC.MP
 i386
Architecture: i386
Machine: i386
>Description:
        I did all my debugging against firefox3, but firefox2 also suffers
        from this behavior, presumably because of the same cause.

        With the negotiate extention turned on, linked against heimdal
        kerberos and gssapi libraries (such as those that ship with
        NetBSD), when going to a web site that uses the negotiate
        extention, firefox segfaults and core dumps.

        firefox actually loads the shared library that provides gssapi
        with dlopen(3) when a page that uses negotiate is opened.

        On startup, firefox is linked against, and the system opens up
        ${PKGBASE}/lib/firefox*/libssl3.so, which defines a SHA1_Update.
        It also opens up ${PKGBASE}/lib/firefox*/libxul.so which
        provides all the gssapi interface functionality.

        When linked against heimdal (such as the one in pkgsrc or the
        system one in NetBSD), and a user goes to a negotiate-friendly
        page, firefox will open libgssapi.so, which opens libcrypto.so
        (openssl), which provides the SHA1_Update this group of shared
        libraries expects.  Unfortunately, there is one already a
        SHA1_Update in core from libssl3.so, that isn't compatible.

        An early call in the gssapi setup causes the wrong SHA1_Update to
        get called, which triggers the core dump.

        As a side note, when running firefox under gdb where the
        segfault happens, gdb actually jumps to the wrong SHA1_Update
        (that is, one that's not run), which can lead to many hours of
        massive confusion.

>How-To-Repeat:
        setup a kerberos infratructure and an apache server with ap-auth-krb5.

        in about:config, setup:
                network.negotiate-auth.trusted-urls to http://,https://

        obtain kerberos tickets, and go to said web page.
>Fix:
        In the workaround dept:

        I was able to set network.negotiate-auth.gsslib to an outside of
        pkgsrc build of MIT's kerberos (/usr/krb5/lib/libgssapi_krb5.so.2.2)
        and if I tweaked the firefox startup wrapper script to have
        /usr/krb5/lib early on LD_LIBRARY_PATH, this caused everything to
        work as expected.

        I experimented with tweaking the dlopen call to use RTLD_LOCAL,
        but that made no difference (presumably because the SHA1_Update
        symbol is already defined).

        The only ideas I was able to come up with were ugly ones, such as
        renaming SHA1_Update everywhere that firefox uses it.

        A less ugly soultion is to tweak firefox to link against
        security/mit-krb5, but that can't be installed at the same time
        as security/heimdal, which makes using other packages when
        kerberos enabled, such as sudo, problematic

        If anyone has suggestions, I can try them.