On Tue, Jul 04, 2000 at 01:06:44PM -0700, Jonathan Stone wrote:
> I am doing lookups of from 100,000 to 1 million unique hostnames, or
> unique IP addresess. For that application,c aching is largely useless
> (since the input queries are unique).  I do lots of lookups in parallel.
[...]
> Batched, parallel, DNS requests of huge numbers of unique hosts or
> huge numbers of unique IP addresses. I'd like to scale up to 1000/sec,
> but I'll probably run out of time to do tuning (have day job, plus
> thesis to write).

Now, maybe this is just me, but if that's the *only* practical
circumstance under which any application needs an FD_SETSIZE so
ridiculously large, why would it be sensible to change it system-wide?

I've maintained more servers running named than I can count on one
hand, though none for huge organizations (but a couple of educational
insitutions are included), on a wide variety of Unix-like operating
systems (and, incidentally, found BIND 8.2.2-p5 on
NetBSD/macppc-current to perform better than either Linux or Solaris,
but not in especially rigorous testing), and seen those kind of
lookups *usually* occur in a state of DoS/attempted compromise, in
which case a bit of throttling wouldn't be a bad idea.

(Incidentally, what are you researching that requires picking on named
like that?)

> The very few people who I know (via personal contacts) doing similar
> things are using linux. their libc already has a FD_SETSIZE of 1024,
> so they just bump "files 1000" in named.conf, and bingo.  Which gets
> us back to square 1...

Yes, well, perhaps using Linux for your research or giving in and
compiling your own BIND every now and then might be appropriate,
considering the change you request is necessary in your specific case
and that of a "very few people" (who are running Linux anyhow) and,
so far as we've seen, no others?

I have four architectures NetBSD supports handy (i386, m68k, powerpc,
and sparc); I'd be glad to do some performance testing of routine
usage (which somebody outside of this flap might want to define) with
an increased FD_SETSIZE. Since my own (parallel computing) research
this summer is sitting atop NetBSD and is wanting excessive FDs as
well, I'd be glad to know if doing so improves performance (thus far
I'm still working my own stuff out, though, I've not really considered
tuning the systems for the application).

       ~ g r @ eclipsed.net