Re: current userland slapd no response

["matthew sporleder" <msporleder%gmail.com@localhost>]

On 9/16/08, matthew sporleder <msporleder%gmail.com@localhost> wrote:
> On 9/16/08, Sarton O'Brien <bsd-xen%roguewrt.org@localhost> wrote:
>  > On Tuesday 16 September 2008 22:51:33 matthew sporleder wrote:
>  >  > > slapd shutdown: waiting for 33 threads to terminate
>  >  > >
>  >  > > I imagine it's a threads issue. I'm updating/building/installing as we
>  >  > > speak but only saw one change to libpthread.
>  >  > >
>  >  > > Anyone know what might be causing this?
>  >  >
>  >  > That's a pretty normal shutdown message (assuming you have a bunch of
>  >  > open connections, anyway).  Can you try starting slapd with -d -1 and
>  >  > seeing if someone obvious comes up?
>  >
>  >
>  > Yeah, I'd imagine it's normal under a heavy load ... these were all stalled
>  >  transactions. Nothing was occurring.
>  >
>  >  Also, I forgot to mention that I could only stop slapd with 'kill -9'.
>  >
>  >  Now there is no load at all ... a simple ldapsearch returns:
>  >
>  >  # /usr/pkg/bin/ldapsearch
>  >  ldap_sasl_interactive_bind_s: No such attribute (16)
>  >
>
>
> Are you using SASL?  If not try ldapsearch -x.
>
>
>  >  daemon: activity on 1 descriptor
>  >  daemon: activity on: 11r
>  >  daemon: read activity on 11
>
> ...
>
> >  ldap_read: want=8 error=Resource temporarily unavailable
>  >  daemon: activity on 1 descriptor
>  >  daemon: waked
>
>
> >  connection_get(11)
>  >  connection_get(11): got connid=0
>  >  connection_read(11): checking for input on id=0
>  >  ber_get_next
>  >  ldap_read: want=8, got=0
>  >
>  >  ber_get_next on fd 11 failed errno=0 (Undefined error: 0)
>  >  connection_read(11): input error=-2 id=0, closing.
>  >
>
> >  conn=0 fd=11 closed (connection lost)
>
>
> These are the keys, I think.  Can you try this again and identify fd
>  11?  I have a feeling it's the bdb file but it would be good to know
>  for sure if it's an index, data file, etc.  (can you make sure db_stat
>  is okay on all of your files?)
>
>
>  >  Undefined error?
>  >
>  >  But now I can actually kill slapd. I caught a dump yesterday but it may 
> not be
>  >  relevant as I hadn't set my stacksize to 4096 as required on amd64.
>  >
>
>
> If it's any help:
>    1443         tag = ber_get_next( conn->c_sb, &len, conn->c_currentber );
>    1444         if ( tag != LDAP_TAG_MESSAGE ) {
>    1445                 int err = sock_errno();
>    1446
>    1447                 if ( err != EWOULDBLOCK && err != EAGAIN ) {
>    1448                         /* log, close and send error */
>    1449                         Debug( LDAP_DEBUG_TRACE,
>    1450                                 "ber_get_next on fd %d failed 
> errno=%d (%s)\n",
>    1451                         conn->c_sd, err, sock_errstr(err) );
>    1452                         ber_free( conn->c_currentber, 1 );
>    1453                         conn->c_currentber = NULL;
>    1454
>    1455                         return -2;
>    1456                 }
>    1457                 return 1;
>    1458         }
>
>  But you really need to figure out why err is getting set to 0 and -2.
>


Okay, I asked in #ldap on freenode and got some good tips.

Basically, I was wrong.  :)  The undefined error is normal because 0
is not an error.  The problem seems to be that ldapsearch closed the
socket and slapd tried to read it (want 8, got 0), which was the cause
of that.  This search, apparently, was not a problem other than you
needing to add -x.  If you can get this to freeze again, try to let it
sit (for like a long time) and see if it eventually does close.  I had
a problem like this on solaris with openldap when I would get into
swap.  If it never closes (after a few hours, for example), then you
can try to get the last info from ktruss, slapd -d -1, etc.

Good luck,
Matt