[ On Saturday, November 10, 2001 at 15:30:50 (+0100), Markus W Kilbinger wrote: ]
> Subject: Re: problem with squid/diskd
>
> >>>>> "Manuel" == Manuel Bouyer <bouyer@antioche.eu.org> writes:
> 
>     Manuel> Being interested by the diskd feature I updated squid to
>     Manuel> the last pkgsrc version. The machine is a i386 running
>     Manuel> NetBSD 1.5.2 Now the problem is that squid doens't work
>     Manuel> properly with diskd: when loading a page with lots of
>     Manuel> small images (like www.redhat.com) some images can't be
>     Manuel> loaded (an error is returned to the browser, and the
>     Manuel> browser display the broken image icon instead). In the log
>     Manuel> I get lots of: storeDiskdSend OPEN: (35) Resource
>     Manuel> temporarily unavailable storeDiskdSend: msgsnd: (35)
>     Manuel> Resource temporarily unavailable
> 
> It's long ago, but adding the following parms to my kernel config
> solved the problem for me:
> 
>   options         MSGMNB=16384    # special stuff for squid diskd
>   options         MSGMNI=41
>   options         MSGSEG=2049
>   options         MSGSSZ=64
>   options         MSGTQL=512
> 
> Standard kernel limits for SYSVMSG etc. are too low.

That's probably part of the problem indeed.  Squid's diskd does need
SYSVMSG and SYSVSHM, though not a lot of each.  This is from FreeBSD,
with four cache_dir's and thus four diskd processes, but should be
indicative of what'll be needed under NetBSD:

$ ipcs -a
Message Queues:
T     ID     KEY        MODE       OWNER    GROUP  CREATOR   CGROUP CBYTES  QNUM QBYTES LSPID LRPID   STIME    RTIME    CTIME
q 1245184   32429056 --rwa------   nobody  nogroup   nobody  nogroup      0      0  16384  31669  3167113:21:01 13:21:01 15:56:02
q 1245185   32429057 --rwa------   nobody  nogroup   nobody  nogroup      0      0  16384  31671  3166913:21:01 13:21:01 15:56:02
q 1245186   32429060 --rwa------   nobody  nogroup   nobody  nogroup      0      0  16384  31669  3167213:21:01 13:21:01 15:56:02
q 1245187   32429061 --rwa------   nobody  nogroup   nobody  nogroup      0      0  16384  31672  3166913:21:01 13:21:01 15:56:02
q 1245188   32429064 --rwa------   nobody  nogroup   nobody  nogroup      0      0  16384  31669  3167313:21:01 13:21:01 15:56:02
q 1245189   32429065 --rwa------   nobody  nogroup   nobody  nogroup      0      0  16384  31673  3166913:21:01 13:21:01 15:56:02
q 1245190   32429068 --rwa------   nobody  nogroup   nobody  nogroup      0      0  16384  31669  3167413:21:01 13:21:01 15:56:02
q 1245191   32429069 --rwa------   nobody  nogroup   nobody  nogroup      0      0  16384  31674  3166913:21:01 13:21:01 15:56:02

Shared Memory:
T     ID     KEY        MODE       OWNER    GROUP  CREATOR   CGROUP NATTCH  SEGSZ  CPID  LPID   ATIME    DTIME    CTIME
m 1245184   32429058 --rw-------   nobody  nogroup   nobody  nogroup      2 393216  31669  3167115:56:02 12:00:04 15:56:02
m 1245185   32429062 --rw-------   nobody  nogroup   nobody  nogroup      2 393216  31669  3167215:56:02 12:00:04 15:56:02
m 1245186   32429066 --rw-------   nobody  nogroup   nobody  nogroup      2 393216  31669  3167315:56:02 12:00:04 15:56:02
m 1245187   32429070 --rw-------   nobody  nogroup   nobody  nogroup      2 393216  31669  3167415:56:02 12:00:04 15:56:02

Semaphores:
T     ID     KEY        MODE       OWNER    GROUP  CREATOR   CGROUP NSEMS   OTIME    CTIME



On these FreeBSD machines I use the following configs:

# For Squid and DISKD, SHMMNI and SHMMNI must be greater than or equal
# to the number of cache_dir's that you have.  SHMMAX must be at least
# 800 kilobytes.  SHMALL must be at least SHMMAX 800 kilobytes
# multiplied by the number of cache_dir's.
#
options         SYSVSHM                 #SYSV-style shared memory
options         SHMMNI=32       # max number of shared memory identifiers (system-wide)
options         SHMSEG=16       # max shared memory segments per process
options         SHMMAXPGS=8192  # max amount of shared memory pages (4k each on i386)
options         SHMMIN=2        # min shared memory segment size (bytes)
options         SHMMAX="(2*1024*1024)"
                                # max shared memory segment size (bytes)
options         SHMALL=SHMMAXPGS # max amount of shared memory (pages)

# The messages between Squid and diskd are 32 bytes.  Thus, MSGSSZ
# should be 32 or greater.  You may want to set it to a larger value,
# just to be safe.
# 
# We'll have two queues for each cache_dir -- one in each direction.
# So, MSGMNI needs to be at least two times the number of cache_dir's.
# 
# MSGMNB and MSGTQL affect how many messages can be in the queues at
# one time.  I've found that 75 messages per queue is about the limit
# of decent performance.  Thus, MSGMNB must be at least 75*MSGSSZ, and
# MSGTQL must be at least 75 times the number of cache_dir's.
#
options         SYSVMSG                 #SYSV-style message queues
options         MSGMNB=16384    # max characters per message queue
options         MSGMNI=40       # max number of message queue identifiers
options         MSGSEG=2048     # max number of message segments in the system
options         MSGSSZ=64       # size of a message segment (MUST be a power of 2)
options         MSGTQL=1024     # max amount of messages in the system

# as far as I know Squid doesn't use any semaphore, but what the heck....
#
options         SYSVSEM                 #SYSV-style semaphores
options         SEMMAP=31       # amount of entries in semaphore map
options         SEMMNI=100      # number of semaphore identifiers in the system
options         SEMMNS=600      # number of semaphores in the system
options         SEMMNU=300      # number of undo structures in the system
options         SEMMSL=SEMMNS   # max number of semaphores per id
options         SEMOPM=101      # max number of operations per semop call
options         SEMUME=100      # max number of undo entries per process




Another thing to watch out for if you've got a large cache machine is
the process size (the above machines, with 4*6.77GB of cache are running
at ~570MB).  I have the following in my kernel too:

# MAXDSIZ is the maximum that the RLIMIT_DATA limit can be set to, and
# the DFLDSIZ is the default value for that limit.  On a multi-user
# general purpose system you might want to set the default lower than
# the max, and explicitly set the maximum with a shell command for
# processes that regularly exceed the limit like INND.
#
options         MAXDSIZ="(1024*1024*1024)"              # maximum RLIMIT_DATA
options         DFLDSIZ="(1024*1024*1024)"              # initial RLIMIT_DATA

# MAXSSIZ is the maximum that the stack limit can be set to.  You might
# want to set the default lower than the max, and explicitly set the
# maximum with a shell command for processes that regularly exceed the
# limit like INND.
#
options         MAXSSIZ="(1024*1024*1024)"              # maximum RLIMIT_STACK

-- 
							Greg A. Woods

+1 416 218-0098      VE3TCP      <gwoods@acm.org>     <woods@robohack.ca>
Planix, Inc. <woods@planix.com>;   Secrets of the Weird <woods@weird.com>