lib/51452: [patch] bt_conv.c can corrupt btree databases when byte swapping

[Tom Yu <tlyu%mit.edu@localhost>]

>Number:         51452
>Category:       lib
>Synopsis:       [patch] bt_conv.c can corrupt btree databases when byte swapping
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Aug 30 19:40:00 +0000 2016
>Originator:     Tom Yu
>Release:        NetBSD-current as of 20160830
>Organization:

>Environment:
Darwin JACOBS-LADDER.MIT.EDU 14.5.0 Darwin Kernel Version 14.5.0: Thu Jun 16 19:58:21 PDT 2016; root:xnu-2782.50.4~1/RELEASE_X86_64 x86_64
>Description:
Accessing a Berkeley DB btree database of the opposite byte order from
the native byte order can corrupt data.  If there is a record with a
small (non-overflow) key but big (overflow) data, the byte swapping code
in lib/libc/db/bt_conv.c can swap the wrong bytes because it acts as if
there is always an overflow key in the record.

I have not personally confirmed this bug on NetBSD, but it is not
specific to NetBSD.  This bug appears to be original to the Berkeley DB
code as initially imported into NetBSD in 1993 and (at least the btree
part) is largely unchanged in our krb5 tree.  This highly portable bug
is still present in NetBSD-current according to code inspection.  I have
confirmed the bug on at least amd64 Ubuntu 14.04, SPARC Solaris, and Mac
OS X.

I'm reporting this bug here because out of the major open-source BSDs,
NetBSD seems to be the only one that has applied any byte swapping bug
fixes to the btree code.

>How-To-Repeat:
Run a test case such as:
https://github.com/krb5/krb5/pull/527/commits/af3e7a4ac8b49ae8047ae2be7070d32feb69217d

The hex dumps are from actual btree databases created on big-endian and
little-endian hosts using the dbtest program.  The bt_conv.c byte
swapping code is internally consistent, so a round trip through it on a
single platform won't display the bug.

>Fix:
The patch that we applied to krb5 is:
https://github.com/krb5/krb5/pull/527/commits/56d02135d4c77b6aa0cb7136477d426248c60a6f

It might require BSD->POSIX type name fixups.

There are other regression tests in the pull request
https://github.com/krb5/krb5/pull/527

that cover other btree byte swapping issues; you might be interested in
them as well.