Mark Cullen wrote:
> Stephen Borrill wrote:
> 
>> On Mon, 26 Jun 2006, Mark Cullen wrote:
>>
>>> I am experiencing a memory leak of some sorts related to Samba. I 
>>> strongly believe it is related to the the PR [1] which is still 
>>> 'Open'. I have run the included test program from the said PR, and I 
>>> get the following output:
>>
>>
>> [snip]
>>
>>> Is there any plans to fix this? There is a patch included, but it was 
>>> filed for 1.6 by the looks of it. I'm not sure if it'll work with 3.x?
>>>
>>> As it stands, Samba is fairly unusable for me because of this, or at 
>>> least I think it's this. I never had the problem on FreeBSD, but that 
>>> was with Samba 3.0.21b, I am now using 3.0.22. I've mailed the Samba 
>>> lists, but I am sure it's related to this bug.
>>
>>
>>
>> I had a few email exchanges with Jeremy Allison about this back in 
>> October last year, but I think my emails got swamped by all his other 
>> Samba-related mails. :-)
>>
>> We don't see it at most sites which, at the time, I thought was down 
>> to security = domain vs security = server (and Jeremy suggested there 
>> was a passdb leak too). It was also not clear what triggered it 
>> (NetBSD 1.6.2/Samba 3.0.10 was fine, NetBSD 2.0_STABLE/Samba 3.0.14 
>> was not). It was clear, however, it was a NetBSD-specific problem.
>>
>> I agree that when you hit this memory leak, Samba is unusable 
>> (processes grow pretty rapidly and start running out of data space and 
>> swap). It would be great if this is the cause of the problem.
>>
>> Odd that this has not been fixed - is the patch particularly contentious?
>>
> 
> Hi Stephen,
> 
> It is a rather odd problem indeed, and highly annoying! I'm actually 
> using security = user, so I don't think it's that :-) It could of course 
> be totally unrelated to this NetBSD bug, but the fact it's increasing 
> with directory refreshes makes me want to guess it is. But then it's 
> also very strange that everyone isn't experiencing this behaviour!
> 
> That's exactly what is happening with me. It grows and grows, and 
> eventually it'll reach some kind of limit... and Samba will just plain 
> stop responding to the clients requests filling the logs with:
> 
> [2006/06/25 22:03:48, 0] lib/util.c:Realloc(969)
>   Memory allocation error: failed to expand to 4896 bytes
> [2006/06/25 22:03:48, 0] lib/util.c:Realloc(969)
>   Memory allocation error: failed to expand to 4896 bytes
> [2006/06/25 22:03:48, 0] lib/util.c:Realloc(969)
>   Memory allocation error: failed to expand to 4896 bytes
> [2006/06/25 22:03:58, 0] lib/util.c:Realloc(969)
>   Memory allocation error: failed to expand to 4136 bytes
> [2006/06/25 22:03:58, 0] lib/util.c:Realloc(969)
>   Memory allocation error: failed to expand to 4136 bytes
> [2006/06/25 22:03:58, 0] lib/util.c:Realloc(969)
>   Memory allocation error: failed to expand to 4136 bytes
> [2006/06/25 22:03:58, 0] lib/util.c:Realloc(969)
>   Memory allocation error: failed to expand to 4136 bytes
> [2006/06/25 22:03:58, 0] lib/util.c:Realloc(969)
>   Memory allocation error: failed to expand to 4136 bytes
> [2006/06/25 22:03:59, 0] lib/util.c:Realloc(969)
>   Memory allocation error: failed to expand to 4120 bytes
> [2006/06/25 22:03:59, 0] lib/util.c:Realloc(969)
>   Memory allocation error: failed to expand to 4120 bytes
> 
> Until I kill the offending smbd process (it happens on a sort of 
> 'per-user' basis). But of course, as I am working on the network drive 
> most of the time, killing the smbd process makes Windows go in to a 
> little bit of a frenzy, which then result in me having to either restart 
> or attempt to logoff and log back on again ;)
> 
> I am now setting up a test box again, and I am rebuilding with that 
> patch applied. The patch still applies cleanly apart from the dirent.h 
> patch. Only 2 of 3 'hunks' applied cleanly to that.
> 
> @@ -92,7 +106,7 @@
>  void rewinddir __P((DIR *));
>  #if defined(_XOPEN_SOURCE) || defined(_NETBSD_SOURCE)
>  void seekdir __P((DIR *, long));
> -long telldir __P((const DIR *));
> +long telldir __P((DIR *));
>  #endif /* defined(_NETBSD_SOURCE) || defined(_XOPEN_SOURCE) */
>  #if defined(_NETBSD_SOURCE)
>  DIR *__opendir2 __P((const char *, int));
> 
> That part didn't apply, but that's no biggie really -- I just changed 
> the line manually for now! I shall report back soon on the outcome, 
> hopefully. I have run in to a problem with my ftp-proxy set up which is 
> delaying me from fetching Samba on the test machine. Darn!
> 

Ok, I have patched the test machine with the patches from the open PR. I 
can now hold down refresh in Windows explorer for as long as I like 
without the memory slowly increasing. I strongly believe this may well 
be the solution to my problem, and possibly the same bug others appear 
to be experiencing.

As I mentioned previously, the original patches apply fine to 3.0.1 
except for dirent.h. Attached is (hopefully, forgive me if it's in the 
wrong format or anything, it looks slightly different to the original 
patches but "Works For Me") a patch for dirent.h which will apply 
cleanly to 3.0.1, at least.

I shall build my 'real' box with the patches applied and let you know if 
it does indeed fully solve my problems with Samba's unexpectedly high 
memory usage.

--- include/dirent.h.orig       2005-02-03 04:39:32.000000000 +0000
+++ include/dirent.h    2006-06-26 11:47:51.000000000 +0100
@@ -54,6 +54,19 @@
  /* definitions for library routines operating on directories. */
  #define        DIRBLKSIZ       1024

+/*
+ * One struct _dirpos is malloced to describe the current directory
+ * position each time telldir is called. It records the current magic
+ * cookie returned by getdirentries and the offset within the buffer
+ * associated with that return value.
+ */
+struct _dirpos {
+       struct  _dirpos *_dpnxt;/* next structure in list */
+       long    _dpcookie;      /* key associated with structure */
+       off_t   _dpseek;        /* magic cookie returned by getdirentries */
+       long    _dploc;         /* offset of entry in buffer */
+};
+
  /* structure describing an open directory. */
  struct _dirdesc {
         int     dd_fd;          /* file descriptor associated with 
directory */
@@ -65,6 +78,7 @@
         long    dd_rewind;      /* magic cookie for rewinding */
         int     dd_flags;       /* flags for readdir */
         void    *dd_lock;       /* lock for concurrent access */
+       struct _dirpos *dd_plist; /* linked list of telldir cookies */
  };

  #define        dirfd(dirp)     ((dirp)->dd_fd)
@@ -91,7 +105,7 @@
  void rewinddir(DIR *);
  #if defined(_XOPEN_SOURCE) || defined(_NETBSD_SOURCE)
  void seekdir(DIR *, long);
-long telldir(const DIR *);
+long telldir(DIR *);
  #endif /* defined(_NETBSD_SOURCE) || defined(_XOPEN_SOURCE) */
  #if defined(_NETBSD_SOURCE)
  DIR *__opendir2(const char *, int);