Subject: Re: Adding TTL information to gethostbyname() and friends
To: NetBSD Networking Technical Discussion List <tech-net@NetBSD.ORG>
From: Greg A. Woods <woods@weird.com>
List: tech-net
Date: 06/02/2003 19:32:02
[ On Monday, June 2, 2003 at 14:53:52 (-0700), Bill Studenmund wrote: ]
> Subject: Re: Adding TTL information to gethostbyname() and friends
>
> No, clue-by-4 the OS vendors/sys admins. The browser authors installed the
> caches as a reaction to perceived (and often real) poor performance of
> their browsers.

Well, OK, then let's clue-by-4 them both for not working together to
find a proper solution to the perceived problem.

Note that some vendors have indeed tried to provide other forms of
solutions for this problem, e.g. the SunOS' nscd.  They're not always
exactly the "right" solution, and the example of 'nscd' is one that still
avoids fixing the TTL problem to such an extent that it often causes
even more problems of its own than it solves.  Even so 'nscd' is still
conceptually better than every individual application instance trying to
do the same thing (even if the applications were to manage to honour DNS
TTLs properly).

> If infrastructure had been around, we would not be where we are now.

The infrastructure has effectively always been around in the sense that
it's a default component of most good OS distributions -- there have
just been too few willing to enable it to be used by default.  :-)

> Then you open yourself up to security issues with bind. Among other
> things. :-|

Who says BIND's named is the only usable DNS cache server?  :-)

If the OS vendor isn't comfortable enough with deploying BIND's named in
a default caching configuration that is enabled by default then it's the
OS vendor's duty to supply an alternative (e.g. as SunOS does).  If the
OS vendor screws up in their choice of alternative (e.g. as Sun did with
nscd), then we can all laugh at them while we deploy better solutions.  :-)

>    If
> embedding a cache in libbind/libresolv/libc/whatever would have worked, we
> would not have this issue now.

I wish to withdraw my suggestion for an in-core cache, but not for that
reason...  :-)

> We aren't going to get the browser authors to remove the caches. I think
> though that we can get them to take TTL info if we give it to them.

Ideally NetBSD's pkgsrc should be removing the application specific
caches for them and feeding them back the patches.  :-)

The most common scenario where DNS lookup latency is perceived is likely
still with running these kinds of applications on non-unix platforms.
Personally I don't feel inclined to provide an API on a unix-like
platform that hacks around problems more specific to non-unix platforms.

I doubt a new TTL-providing API would come anywhere near solving the
problem, even if it were offered by all open source platforms.

However what might actually help convince application authors to avoid
caching DNS information would be to make sure the underlying platforms
their applications run on already do a better job of such caching by
default.  I.e. let's try to eliminate the cause of the problem instead
of providing an ill-conceived work-around.

> Why is it not right? The uses I've seen of caches try to put the cache as
> close to the user as possible. Thus if the application wants it in itself,
> that's as close to the user as you can get.

DNS caching should indeed be as close to the client host as possible,
perhaps even running on the client host if it's the sole similar client
in the vicinity.  However the whole idea of a DNS cache is to share it
not just across time but across as many local clients and application
instances as possible as well.

From RFC 1034:

    A very important goal of the resolver is to eliminate network delay and
    name server load from most requests by answering them from its cache of
    prior results.  It follows that caches which are shared by multiple
    processes, users, machines, etc., are more efficient than non-shared
    caches.

Thus the problem remains that any DNS caching in the application's own
address space is useless for even any other instance of the same
application, let alone any other application even if it runs on the same
client host.  There's no real harm in such per-process caching if it is
done in parallel with proper shared caching, but it is generally useless
as a solution on its own in any environment except maybe a single-user
single-tasking system, but there's no significant benefit either.  More
about this below.

> I agree it's wrong to force applications to have to do the caching
> themselves, but if they want to, why shouldn't they?  The issue we're
> seeing now is that the name lookup routines don't return enough info for
> them to do it right.

As I understood things the reason for not including TTL information in
gethostbyname() and friends in the first place was specifically to
prevent applications from trying to cache it (for too long).  Caching is
a job for the resolver to do transparently to the application, not the
other way around.

In fact the discussion of this issue has been a central part of the
Internet Host Requirements RFC (1123) for over a decade now:

      6.1.3  SPECIFIC ISSUES

         6.1.3.1  Resolver Implementation

            A name resolver SHOULD be able to multiplex concurrent
            requests if the host supports concurrent processes.

            In implementing a DNS resolver, one of two different models
            MAY optionally be chosen: a full-service resolver, or a stub
            resolver.


            (A)  Full-Service Resolver

                 A full-service resolver is a complete implementation of
                 the resolver service, and is capable of dealing with
                 communication failures, failure of individual name
                 servers, location of the proper name server for a given
                 name, etc.  It must satisfy the following requirements:

                 o    The resolver MUST implement a local caching
                      function to avoid repeated remote access for
                      identical requests, and MUST time out information
                      in the cache.

                 o    The resolver SHOULD be configurable with start-up
                      information pointing to multiple root name servers
                      and multiple name servers for the local domain.
                      This insures that the resolver will be able to
                      access the whole name space in normal cases, and
                      will be able to access local domain information
                      should the local network become disconnected from
                      the rest of the Internet.

            (B)  Stub Resolver

                 A "stub resolver" relies on the services of a recursive
                 name server on the connected network or a "nearby"
                 network.  This scheme allows the host to pass on the
                 burden of the resolver function to a name server on
                 another host.  This model is often essential for less
                 capable hosts, such as PCs, and is also recommended
                 when the host is one of several workstations on a local
                 network, because it allows all of the workstations to
                 share the cache of the recursive name server and hence
                 reduce the number of domain requests exported by the
                 local network.

                 At a minimum, the stub resolver MUST be capable of
                 directing its requests to redundant recursive name
                 servers.  Note that recursive name servers are allowed
                 to restrict the sources of requests that they will
                 honor, so the host administrator must verify that the
                 service will be provided.  Stub resolvers MAY implement
                 caching if they choose, but if so, MUST timeout cached
                 information.


Anyone using a stub resolver to contact a recursive cache across a
high-latency link is the one who really needs the clue-by-4, but of
course that includes almost every user stuck with M$-Windoze and a PPP
link.  "Let them eat cake." -- especially since they bought into a
vendor which has consistently failed to meet their real requirements.

Of course there are also deeper issues here to do with the application
design and implementation, especially in the case of WWW browsers.  Who
in their right mind would look up the same hostname multiple times while
rendering one HTML "page"?  Not a human, that's for certain.  However
some browser authors have apparently taken object-oriented design well
beyond its useful limits.  On the other hand even things like a WWW
browsers should not be caching hostname to address translations any
longer than is necessary to render one top-level object.  If they are
going any further, especially if they are going to the length where TTL
becomes an issue, then they are still doing things wrong in a very
fundamental way.  Assuming the BIND suggested default minimum TTL of 300
seconds is probably suicide in today's high-speed Internet, but it
should always be safe to assume that the hostname to address translation
is static enough for at least the time it takes to retrieve and render a
full top-level HTML page (and all its frames and all its images, etc.).

I think it really would be a very large step backwards to invent and
implement a new API to pretend to work around this perceived problem,
and then hope that application authors will universally use this new
API, when much better solutions have effectively always been staring
everyone right in the face.  It would be far less of a botch to
implement a secret in-core cache in the libc resolver (i.e. to add
caching to our stub resolver), but it may still be a waste of time and
effort since it would really only solve the problem in a scenario where
it usually doesn't exist, and/or where better solutions can be far more
easily deployed.

-- 
								Greg A. Woods

+1 416 218-0098;            <g.a.woods@ieee.org>;           <woods@robohack.ca>
Planix, Inc. <woods@planix.com>; VE3TCP; Secrets of the Weird <woods@weird.com>