Re: bin/56743: Cannot create more than 1022 files from an AWK program

[mlelstv%serpens.de@localhost (Michael van Elst)]

The following reply was made to PR bin/56743; it has been noted by GNATS.

From: mlelstv%serpens.de@localhost (Michael van Elst)
To: gnats-bugs%netbsd.org@localhost
Cc: 
Subject: Re: bin/56743: Cannot create more than 1022 files from an AWK program
Date: Wed, 9 Mar 2022 18:29:38 -0000 (UTC)

 silas_nbgt22%silas.net.br@localhost writes:
 
 >	It is not possible to create more than 1022 files using using awk '>'
 >	operator. Probably because it keeps files opened and runs out of
 >	available file descriptors.
 
 >	Being able to create thousands of files is a useful feature if one wants
 >	to split a very large file into multiple files based on a regular
 >	expression pattern.
 
 The awk '>' operator works differently than a shell redirect. It opens
 the file and keeps it open, more output to the same file will just
 use the open descriptor. You need to use the close() statement to
 close the descriptor.
 
 Any process has a limited number of file descriptors (1024 by default),
 so after starting a regular program (with stdin,stdout,stderr) you
 can open 1021 files. The limit can be bumped up to a system limit
 (currently 3404). This has nothing to do with awk.
 
 Linux also imposes a process limit of 1024 descriptors, but it can
 usually be bumped to a million or more for programs that need it.
 
 You don't need a huge number of file descriptors in awk to split a
 file into multiple files, a smaller cache of open descriptors can
 be sufficient.