At a glance at aalib-1.4.0/src/*, the following part seems to cause
binary incompatibility.

  src/aaregister.c:5
> __AA_CONST struct aa_driver * __AA_CONST aa_drivers[] =
> {
> #ifdef DJGPP
>     &dos_d,
> #else
> #ifdef X11_DRIVER
>     &X11_d,
> #endif
^^^^^^^^^^^^^^^^^^^ Here.
> #ifdef LINUX_DRIVER
>     &linux_d,
> #endif
> #ifdef SLANG_DRIVER
>     &slang_d,
> #endif
> #ifdef CURSES_DRIVER
>     &curses_d,
> #endif
> #ifdef OS2_DRIVER
>     &os2vio_d,
> #endif
> #endif
>     &stdout_d,
>     &stderr_d,
>     NULL
> };

Other parts seem to be kind of ABI-compatible between aalib and that
without X.  This need to be examined more, of course.

Regards,
Masao