Subject: Re: bin/28492 (more(1) doesn't handle NUL data blocks well)
To: None <gnats-bugs@NetBSD.org>
From: Bob Kemp <bsd@allegory.demon.co.uk>
List: netbsd-bugs
Date: 09/25/2006 23:16:54
--BOKacYhQ+x31HxR3
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Mon, Sep 25, 2006 at 09:20:03PM +0000, Matthias Scheler wrote:

>  > Since I use Java, the memory limits are set higher and the box only has 512M.
>  > If you have more memory, it might not feel so bad to you.
>  [...]
>  > $ ulimit -a
>  > time(cpu-seconds)    unlimited
>  > file(blocks)         unlimited
>  > coredump(blocks)     unlimited
>  > data(kbytes)         3145728		<---
>  > stack(kbytes)        2048
>  > lockedmem(kbytes)    169933
>  > memory(kbytes)       509800
>  > nofiles(descriptors) 500
>  > processes            160
>  > sbsize(bytes)        unlimited
>  
>  [...]
>  
>  Your datasize exceeds the hard limit of my box:
>  
>  $ ulimit -a
>  time(seconds)        unlimited
>  file(blocks)         unlimited
>  data(kbytes)         131072		<---
>  stack(kbytes)        2048
>  coredump(blocks)     0
>  memory(kbytes)       2045980
>  locked memory(kbytes) 681994
>  process(processes)   160
>  nofiles(descriptors) 64
>  sbsize(bytes)        unlimited

In my .tcshrc, I have

    limit datasize      unlimited

but beyond that I don't think there's anything special about my setup.

>  > If you want more info, or for me to try something, just let me know.
>  
>  Can you please provide your kernel configuration and more importantly
>  try with a standard kernel configuration? It looks to me like you
>  extended certain limits above a save boundary.

The config file is attached.

>  And could please also try to run "more" on a very large (e.g. 4GB) file?
>  I bet your problem has nothing to do with a file full of zeros but
>  with the VM limits and the large amount of input data.

Interesting idea but...

     $ ls -ls massive*
     1485872 -rw-r--r--  1 rob  wheel  1520772480 Sep 25 22:37 massiveFile
     922064 -rw-r--r--  1 rob  wheel   943718400 Sep 25 22:39 massiveZeroFile

where they were created by (in effect)

    dd if=/dev/zero bs=100m of=massiveZeroFile

and

    cat /etc/hosts massiveFile >> massiveFile

As you can see the file of zeroes is smaller than the text file but
the text file works just fine and the file of zeroes behaves as reported.

I don't have enough free filespace to create 4Gb files.

GENERIC_LAPTOP doesn't run on my laptop, hangs at RaidFrame init,
so I can't try that.  I'll try supping and updating my system.

Thanks for your help!

Bob

PS: I've just tried 'less' from pkgsrc of a few days ago.
    It behaves differently on the zeroes file.
    It burns 100% of the CPU but uses relatively little memory.
    It is however unresponsive, eg I cannot ^C it or suspend it.
    On the large text file all works fine, just like the NetBSD version.


--BOKacYhQ+x31HxR3
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=TENEBRES


include "arch/i386/conf/std.i386"

options 	INCLUDE_CONFIG_FILE	# embed config file in kernel binary

ident 		"TENEBRES"

maxusers	32		# estimated number of users

# CPU support.  At least one is REQUIRED.
options 	I686_CPU

# CPU-related options.
options 	MATH_EMULATE	# floating point emulation
options 	VM86		# virtual 8086 emulation
options 	USER_LDT	# user-settable LDT; used by WINE

# Enhanced SpeedStep Technology in the Pentium M
options 	ENHANCED_SPEEDSTEP
options 	EST_FREQ_USERWRITE	# any user can set frequency

options 	MTRR		# memory-type range register syscall support

# delay between "rebooting ..." message and hardware reset, in milliseconds
#options 	CPURESET_DELAY=2000

# This option allows you to force a serial console at the specified
# I/O address.   see console(4) for details.
#options 	CONSDEVNAME="\"com\"",CONADDR=0x2f8,CONSPEED=57600
#	you don't want the option below ON iff you are using the
#	serial console option of the new boot strap code.
#options 	CONS_OVERRIDE	# Always use above! independent of boot info

# Standard system options

options 	INSECURE	# disable kernel security levels - X needs this

## options 	RTC_OFFSET=0	# hardware clock is this many mins. west of GMT
options 	NTP		# NTP phase/frequency locked loop

options 	KTRACE		# system call tracing via ktrace(1)
options 	SYSTRACE	# system call vetting via systrace(1)

options 	SYSVMSG		# System V-like message queues
options 	SYSVSEM		# System V-like semaphores
#options 	SEMMNI=10	# number of semaphore identifiers
#options 	SEMMNS=60	# number of semaphores in system
#options 	SEMUME=10	# max number of undo entries per process
#options 	SEMMNU=30	# number of undo structures in system
options 	SYSVSHM		# System V-like memory sharing
options		SHMMAXPGS=32768
options 	P1003_1B_SEMAPHORE # p1003.1b semaphore support

options 	LKM		# loadable kernel modules

options		KLOADER
options		KLOADER_KERNEL_PATH="\"/netbsd\""

options 	USERCONF	# userconf(4) support
options 	SYSCTL_INCLUDE_DESCR	# Include sysctl descriptions in kernel

# Diagnostic/debugging support options
#options 	DIAGNOSTIC	# expensive kernel consistency checks
#options 	DEBUG		# expensive debugging checks/support
#options 	KMEMSTATS	# kernel memory statistics (vmstat -m)
options 	DDB		# in-kernel debugger
options 	DDB_ONPANIC=1	# see also sysctl(8): `ddb.onpanic'
options 	DDB_HISTORY_SIZE=512	# enable history editing in DDB
#options 	KGDB		# remote debugger
#options 	KGDB_DEVNAME="\"com\"",KGDB_DEVADDR=0x3f8,KGDB_DEVRATE=9600
makeoptions	DEBUG="-g"	# compile full symbol table

# Compatibility options
options 	COMPAT_20	# NetBSD 2.0
options 	COMPAT_21	# NetBSD 2.1
options 	COMPAT_30	# NetBSD 3.0
options 	COMPAT_40	# NetBSD 4.0
options 	COMPAT_50	# NetBSD 4.0

options 	COMPAT_OSSAUDIO	# OSS (Voxware) audio driver compatibility
#options 	COMPAT_SVR4	# binary compatibility with SVR4
#options 	COMPAT_IBCS2	# binary compatibility with SCO and ISC
options 	COMPAT_LINUX	# binary compatibility with Linux
#options 	COMPAT_FREEBSD	# binary compatibility with FreeBSD
#options 	COMPAT_PECOFF	# kernel support to run Win32 apps
options		COMPAT_BSDPTY	# /dev/[pt]ty?? ptys.

# File systems
file-system 	FFS		# UFS
#file-system 	EXT2FS		# second extended file system (linux)
#file-system 	LFS		# log-structured file system
file-system 	MFS		# memory file system
file-system 	NFS		# Network File System client
#file-system 	NTFS		# Windows/NT file system (experimental)
file-system 	CD9660		# ISO 9660 + Rock Ridge file system
file-system 	MSDOSFS		# MS-DOS file system
#file-system 	FDESC		# /dev/fd
file-system 	KERNFS		# /kern
file-system 	NULLFS		# loopback file system
#file-system 	OVERLAY		# overlay file system
#file-system 	PORTAL		# portal filesystem (still experimental)
file-system 	PROCFS		# /proc
#file-system 	UMAPFS		# NULLFS + uid and gid remapping
file-system 	UNION		# union file system
#file-system	CODA		# Coda File System; also needs vcoda (below)
file-system	SMBFS		# experimental - CIFS; also needs nsmb (below)
#file-system 	PTYFS		# New pty filesystem -- UNOFFICIAL
file-system	TMPFS		# experimental - Efficient memory file-system

# File system options
#options 	FFS_EI		# FFS Endian Independent support
options 	SOFTDEP		# FFS soft updates support.
options 	NFSSERVER	# Network File System server

# Networking options
#options 	GATEWAY		# packet forwarding
options 	INET		# IP + ICMP + TCP + UDP
options 	INET6		# IPV6
options 	IPSEC		# IP security
options 	IPSEC_ESP	# IP security (encryption part; define w/IPSEC)
options 	IPSEC_DEBUG	# debug for IP security
options 	PPP_BSDCOMP	# BSD-Compress compression support for PPP
options 	PPP_DEFLATE	# Deflate compression support for PPP
options 	PPP_FILTER	# Active filter support for PPP (requires bpf)
options 	PFIL_HOOKS	# pfil(9) packet filter hooks
options 	IPFILTER_LOG	# ipmon(8) log support
options		IPFILTER_DEFAULT_BLOCK	# block all packets by default

#
# wscons options
#
# builtin terminal emulations
#options 	WSEMUL_SUN		# sun terminal emulation
options 	WSEMUL_VT100		# VT100 / VT220 emulation
# different kernel output - see dev/wscons/wsdisplayvar.h
options 	WS_KERNEL_FG=WSCOL_GREEN
#options 	WS_KERNEL_BG=WSCOL_BLACK
# compatibility to other console drivers
options 	WSDISPLAY_COMPAT_PCVT		# emulate some ioctls
options 	WSDISPLAY_COMPAT_SYSCONS	# emulate some ioctls
options 	WSDISPLAY_COMPAT_USL		# VT handling
options 	WSDISPLAY_COMPAT_RAWKBD		# can get raw scancodes
# allocate a number of virtual screens at autoconfiguration time
options 	WSDISPLAY_DEFAULTSCREENS=8
# use a large software cursor that doesn't blink
options 	PCDISPLAY_SOFTCURSOR
# modify the screen type of the console; defaults to "80x25"
#options 	VGA_CONSOLE_SCREENTYPE="\"80x24\""

# Kernel root file system and dump configuration.
config		netbsd	root on ? type ?
#config		netbsd	root on sd0a type ffs
#config		netbsd	root on ? type nfs

#
# Device configuration
#

mainbus0 at root

cpu0 at mainbus0

apm0	at mainbus0			# Advanced power management

# Tuning for power management, see apm(4) for more details.
#options 	APM_NO_IDLE		# Don't call BIOS CPU idle function
#options 	APM_V10_ONLY		# Use only the APM 1.0 calls
#options 	APM_NO_POWEROFF		# Don't power off on halt(8)
#options 	APM_POWER_PRINT		# Print stats on the console

options 	APM_DISABLE_INTERRUPTS=1  # Disable interrupts
options 	APMDEBUG
options 	APMCALLDEBUG


# Basic Bus Support

# PCI bus support
pci0	at mainbus? bus ?
pci*	at mainbus? bus ?
pci*	at pchb? bus ?
pci*	at ppb? bus ?

# Configure PCI using BIOS information
options 	PCIBIOS			# PCI BIOS support
options 	PCIBIOSVERBOSE		# PCI BIOS verbose info
#options 	PCIBIOS_IRQS_HINT=0x0a00 # PCI interrupts hint. IRQ 9 or 11
options 	PCIBIOS_INTR_GUESS	# see pcibios(4)
#options 	PCIINTR_DEBUG		# super-verbose PCI interrupt fixup

options 	PCI_ADDR_FIXUP		# fixup PCI I/O addresses
options 	PCI_BUS_FIXUP		# fixup PCI bus numbering
options 	PCI_INTR_FIXUP		# fixup PCI interrupt routing

# PCI bridges
pchb*	at pci? dev ? function ?	# PCI-Host bridges
pcib*	at pci? dev ? function ?	# PCI-ISA bridges
ppb*	at pci? dev ? function ?	# PCI-PCI bridges
# XXX 'puc's aren't really bridges, but there's no better place for them here
puc*	at pci? dev ? function ?	# PCI "universal" comm. cards

agp* 	at pchb?

# ISA bus support
isa0	at mainbus?
isa0	at pcib?

# PCMCIA bus support
pcmcia*	at pcic? controller ? socket ?
pcmcia*	at tcic? controller ? socket ?

# ISA PCMCIA controllers
pcic0	at isa? port 0x3e0 iomem 0xd0000 iosiz 0x10000
pcic1	at isa? port 0x3e2 iomem 0xe0000 iosiz 0x4000
pcic2	at isa? port 0x3e4 iomem 0xe0000 iosiz 0x4000
tcic0	at isa? port 0x240 iomem 0xd0000 iosiz 0x10000

# PCI PCMCIA controllers
pcic0	at pci? dev? function ?

# CardBus bridge support
cbb*		at pci? dev ? function ?
cardslot*	at cbb?

# CardBus bus support
cardbus*	at cardslot?
pcmcia* 	at cardslot?

# Coprocessor Support

# Math Coprocessor support
npx0	at isa? port 0xf0 irq 13	# x86 math coprocessor


# Console Devices

# ISA console
#pc0	at isa? port 0x60 irq 1		# pccons generic PC console driver
# Keyboard layout configuration for pccons
#options 	FRENCH_KBD
#options 	FINNISH_KBD
#options 	GERMAN_KBD
#options 	NORWEGIAN_KBD
# pccons-specific options:
#options 	XSERVER_DDB	# PF12 gets you into DDB when X is running
#options 	XSERVER		# X server support


# wscons
pckbc0		at isa?			# pc keyboard controller
pckbd*		at pckbc?		# PC keyboard
pms*		at pckbc?		# PS/2 mouse for wsmouse
vga0		at isa?
vga*		at pci? dev ? function ?
pcdisplay0	at isa?			# CGA, MDA, EGA, HGA
wsdisplay*	at vga? console ?
wsdisplay*	at pcdisplay? console ?
wskbd* 		at pckbd? console ?
wsmouse*	at pms? mux 0

pcppi0		at isa?
sysbeep0	at pcppi?

# Serial Devices

# PCI serial interfaces
com*	at puc? port ?			# 16x50s on "universal" comm boards

# PCMCIA serial interfaces
com*	at pcmcia? function ?		# Modems and serial cards

pcmcom*	at pcmcia? function ?		# PCMCIA multi-port serial cards
com*	at pcmcom? slave ?		# ...and the slave devices

# CardBus serial interfaces
com*	at cardbus? function ?	# Modems and serial cards

# ISA serial interfaces
com0	at isa? port 0x3f8 irq 4	# Standard PC serial ports
com1	at isa? port 0x2f8 irq 3
com2	at isa? port 0x3e8 irq 5


# Parallel Printer Interfaces

# ISA parallel printer interfaces
lpt0	at isa? port 0x378 irq 7	# standard PC parallel ports
lpt1	at isa? port 0x278
lpt2	at isa? port 0x3bc


# SCSI Controllers and Devices

# SCSI bus support
scsibus* at scsi?
scsibus0 at scsi?

# SCSI devices
sd0	at scsibus0 target 0 lun ?	# SCSI disk drives
sd1	at scsibus0 target 1 lun ?	# SCSI disk drives
sd2	at scsibus0 target 2 lun ?	# SCSI disk drives
sd3	at scsibus0 target 3 lun ?	# SCSI disk drives
sd4	at scsibus0 target 4 lun ?	# SCSI disk drives

sd*	at scsibus? target ? lun ?	# SCSI disk drives
cd*	at scsibus? target ? lun ?	# SCSI CD-ROM drives

# IDE drives
# Flags are used only with controllers that support DMA operations
# and mode settings (e.g. some pciide controllers)
# The lowest order four bits (rightmost digit) of the flags define the PIO
# mode to use, the next set of four bits the DMA mode and the third set the
# UltraDMA mode. For each set of four bits, the 3 lower bits define the mode
# to use, and the last bit must be 1 for this setting to be used.
# For DMA and UDMA, 0xf (1111) means 'disable'.
# 0x0fac means 'use PIO mode 4, DMA mode 2, disable UltraDMA'.
# (0xc=1100, 0xa=1010, 0xf=1111)
# 0x0000 means "use whatever the drive claims to support".

# ATA (IDE) bus support
atabus* at ata?

wd*	at atabus? drive ? flags 0x0000

# ATAPI bus support
atapibus* at atapi?

# ATAPI devices
# flags have the same meaning as for IDE drives.
cd*	at atapibus? drive ? flags 0x0000	# ATAPI CD-ROM drives
sd*	at atapibus? drive ? flags 0x0000	# ATAPI disk drives

piixide* 	at pci? dev ? function ?	# Intel IDE controllers

# Miscellaneous mass storage devices

# ISA floppy
fdc0	at isa? port 0x3f0 irq 6 drq 2	# standard PC floppy controllers
fd*	at fdc? drive ?			# the drives themselves

# Network Interfaces

# PCI network interfaces
fxp*	at pci? dev ? function ?	# Intel EtherExpress PRO 10+/100B
ne* 	at pci? dev ? function ?	# NE2000-compatible Ethernet

rtk*	at cardbus? function ?		# Realtek 8129/8139
ral*	at cardbus? function ?		# Ralink Technology, Inc., RT2500, 802.11
an*	at pcmcia? function ?		# Aironet PC4500/PC4800 (802.11)

# PCMCIA network interfaces
ne*	at pcmcia? function ?		# NE2000-compatible Ethernet

# CardBus network cards
fxp*	at cardbus? function ?	# Intel i8255x

# MII/PHY support
inphy*	at mii? phy ?			# Intel 82555 PHYs

# USB Controller and Devices

# PCI USB controllers
ohci*	at pci?	dev ? function ?	# Open Host Controller
uhci*	at pci?	dev ? function ?	# Universal Host Controller (Intel)

# USB bus support
usb*	at ohci?
usb*	at uhci?

# USB Hubs
uhub*	at usb?
uhub*	at uhub? port ? configuration ? interface ?

# USB HID device
uhidev*	at uhub? port ? configuration ? interface ?

# USB Generic HID devices
uhid*	at uhidev? reportid ?

# USB Printer
ulpt*	at uhub? port ? configuration ? interface ?

# USB Modem
umodem*	at uhub? port ? configuration ?
ucom*	at umodem?

# USB Mass Storage
umass*	at uhub? port ? configuration ? interface ?

# USB audio
uaudio*	at uhub? port ? configuration ?

# USB MIDI
umidi* at uhub? port ? configuration ?

# USB IrDA
# USB-IrDA bridge spec
uirda* at uhub? port ? configuration ? interface ?
irframe* at uirda?

# SigmaTel STIr4200 USB/IrDA Bridge
ustir* at uhub? port ?
irframe* at ustir?

# USB Generic driver
ugen*	at uhub? port ?

# IrDA and Consumer Ir devices

# Toshiba Oboe
## oboe* 	at pci0 dev 31 function 0
## irframe* at oboe?



# Audio Devices

# PCI audio devices
auich*	at pci? dev ? function ?	# Intel ICH integrated AC'97 Audio

# Audio support
audio*	at audiobus?

# MIDI support
midi*	at midibus?
midi*	at pcppi?		# MIDI interface to the PC speaker


# Pseudo-Devices

# disk/mass storage pseudo-devices
pseudo-device	cgd		8	# cryptographic disk devices

## pseudo-device	ptm

pseudo-device	md		1	# memory disk device (ramdisk)
pseudo-device	vnd		8	# disk-like interface to files

# network pseudo-devices
pseudo-device	bpfilter	8	# Berkeley packet filter
pseudo-device	ipfilter		# IP filter (firewall) and NAT
pseudo-device	loop		2	# network loopback
pseudo-device	ppp		2	# Point-to-Point Protocol
pseudo-device	pppoe			# PPP over Ethernet (RFC 2516)
pseudo-device	irframetty		# IrDA frame line discipline
pseudo-device	tun		5	# network tunneling over tty
pseudo-device	tap		5	# virtual Ethernet

# miscellaneous pseudo-devices
pseudo-device	pty			# pseudo-terminals
pseudo-device	rnd			# /dev/random and in-kernel generator
pseudo-device	clockctl		# user control of clock subsystem

# a pseudo device needed for SMBFS
pseudo-device	nsmb			# experimental - SMB requester

# wscons pseudo-devices
pseudo-device	wsmux			# mouse & keyboard multiplexor
pseudo-device	wsfont


--BOKacYhQ+x31HxR3--