lua_Number in the kernel

[Alexander Nasonov <alnsn%yandex.ru@localhost>]

[ Ccing to Justin who seems to be interested in Lua in NetBSD
  but I'm not sure whether he's subscribed to tech-kern@ ].

Like some other people, I beleived that Lua kernel project is dormant
and was just waiting for any activity before starting a discussion here
but Marc replied today to an ongoing discussion on developers@. Hence,
my post.

It's very important to decide on lua_Number type in kernel early because
it affects nearly all Lua code and also because arithmetic operations is
often a cause of security holes.

In Lua, you can use your-own signed type for numbers. De-facto standard
for userspace is double (note that LuaJIT supports _only_ double type)
but ptrdiff_t is also recommented if for some reason double isn't an
option.

So, ptrdiff_t looks like a reasonable choice for the kernel but when I
was developing sljit binding for Lua [1] and I was trying to make it
robust against overflows for two different types (double and ptrdiff_t)
on both 32bit and 64bit platforms, I ended up creating bindings [2] for
arbitrary precision library.

The problem is that there are three different ranges for integer
arithmetic:

1. IEEE 754 double is a always -2^53-1 to 2^53-1 regardless of platform,
2. ptrdiff_t on 2s-complement 32bit platform is -2^31 to 2^31-1,
3. ptrdiff_t on 2s-complement 64bit platform is -2^63 to 2^63-1.

Note also that min values of ptrdiff_t on 2s-complement platforms don't
have a regular math semantic (e.g. negation produces identical value).

It's very challenging to write a code that supports all three options
without resorting to #ifdef (or Lua equivalent) or splitting a number
into low and high halves.

I'd like to propose that lua_Number in the kernel should always be
int64_t (*). This type will guarantee regular arithmetic rules for
the range (-2^53, 2^53) and for 32-bit signed integer range, in
particular.

When possible, Lua code can assume 32-bit arithmetic (without
2s-complement iggerularities). Some Lua libraries (for instance, BitOp)
assume 32bit. Other libraries that we might port to the kernel are often
written in assumption that lua_Number is double and int64_t should be
safe to use too.

When you need a full 64-bit range (signed or unsigned), arbitrary
precision library can be used instead.

[1] https://github.com/alnsn/luaSljit
[2] https://github.com/alnsn/luaBn
(*) I assume that int64_t type is available on all supported platforms
    but it can be emulated by a compiler, of course.

Alex