Subject: Re: RaidFrame problems.
To: None <wojtek@wojtek.from.pl>
From: Andrea Franceschini <andrea@cs.tin.it>
List: port-i386
Date: 02/08/2001 00:47:39
wojtek@wojtek.from.pl wrote:
>
> >
> > All this started after i upgraded my kernel in order to support UDMA/66
> > devices.
> >
> > Using Previous kernel disk/raid performance was not so fast (~10MB/s)
> > but neither so slow!!
> >
> > Of course if i don't use raid devices all works fine and very
> > fast(80MB/s on UDMA disks)!!.
>
> 80MB from one disk?!
This really solve my problem,thank you ;)
My disk is too fast so the NIC hangs:)
Sorry ,was a joke:)
Seriously,the problem to me is not disk speed ,but RaidFrame that takes
an huge amount of kernel resources during write operations(60%/70%
according to 'top').
Take a look to the following benchmark!
On raid5 Partition :
-------Sequential Output-------- ---Sequential Input--
--Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block---
--Seeks---
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
/sec %CPU
100 2504 15.4 2468 6.8 2749 7.8 21473 93.2 86272 99.9
2311.4 27.9
^^^^
On non-raid partition :
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
/sec %CPU
100 12802 96.0 15757 74.2 15847 73.0 23237 99.9 86406 100.0
8523.6 100.1
^^^^^
The real problem was not the loss of speed (i can handle that), but that
during disk operation whole system 'freeze' for a while, so everything
is involved (from NICs to user-processes) and this is not acceptable.
So i got my 100/Mb Ether that perform like a dialup connection or fail(
i have to reboot) during raid access..
This is my raid configuration:
START array
1 4 0
START disks
/dev/wd0e
/dev/wd2e
/dev/wd1e
/dev/wd3e
START layout
32 1 1 5
START queue
fifo 100
and this is 'raidctl -s raid0' output:
Components:
/dev/wd0e: optimal
/dev/wd2e: optimal
/dev/wd1e: optimal
/dev/wd3e: optimal
No spares.
Component label for /dev/wd0e:
Row: 0, Column: 0, Num Rows: 1, Num Columns: 4
Version: 2, Serial Number: 11223344, Mod Counter: 248
Clean: No, Status: 0
sectPerSU: 32, SUsPerPU: 1, SUsPerRU: 1
Queue size: 100, blocksize: 512, numBlocks: 39813600
RAID Level: 5
Autoconfig: Yes
Root partition: Yes
Last configured as: raid0
Component label for /dev/wd2e:
Row: 0, Column: 1, Num Rows: 1, Num Columns: 4
Version: 2, Serial Number: 11223344, Mod Counter: 248
Clean: No, Status: 0
sectPerSU: 32, SUsPerPU: 1, SUsPerRU: 1
Queue size: 100, blocksize: 512, numBlocks: 39813600
RAID Level: 5
Autoconfig: Yes
Root partition: Yes
Last configured as: raid0
Component label for /dev/wd1e:
Row: 0, Column: 2, Num Rows: 1, Num Columns: 4
Version: 2, Serial Number: 11223344, Mod Counter: 248
Clean: No, Status: 0
sectPerSU: 32, SUsPerPU: 1, SUsPerRU: 1
Queue size: 100, blocksize: 512, numBlocks: 39813600
RAID Level: 5
Autoconfig: Yes
Root partition: Yes
Last configured as: raid0
Component label for /dev/wd3e:
Row: 0, Column: 3, Num Rows: 1, Num Columns: 4
Version: 2, Serial Number: 11223344, Mod Counter: 248
Clean: No, Status: 0
sectPerSU: 32, SUsPerPU: 1, SUsPerRU: 1
Queue size: 100, blocksize: 512, numBlocks: 39813600
RAID Level: 5
Autoconfig: Yes
Root partition: Yes
Last configured as: raid0
Parity status: clean
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.
My Kernel Config:
# $NetBSD: GENERIC,v 1.400 2001/02/04 17:36:03 perry Exp $
#
# GENERIC -- everything that's currently supported
#
include "arch/i386/conf/std.i386"
#ident "GENERIC-$Revision: 1.400 $"
maxusers 32 # estimated number of users
# CPU support. At least one is REQUIRED.
options I586_CPU
# CPU-related options.
options MATH_EMULATE # floating point emulation
options VM86 # virtual 8086 emulation
options USER_LDT # user-settable LDT; used by WINE
options UCONSOLE # users can use TIOCCONS (for xconsole)
options INSECURE # disable kernel security levels
options RTC_OFFSET=0 # hardware clock is this many mins. west
of GMT
options NTP # NTP phase/frequency locked loop
options KTRACE # system call tracing via ktrace(1)
options SYSVMSG # System V-like message queues
options SYSVSEM # System V-like semaphores
options SYSVSHM # System V-like memory sharing
#options SHMMAXPGS=1024 # 1024 pages is the default
options LKM # loadable kernel modules
# Diagnostic/debugging support options
options DIAGNOSTIC # cheap kernel consistency checks
# Compatibility options
options COMPAT_15 # NetBSD 1.5,
options COMPAT_43 # and 4.3BSD
options COMPAT_386BSD_MBRPART # recognize old partition ID
options COMPAT_LINUX # binary compatibility with Linux
options COMPAT_FREEBSD # binary compatibility with FreeBSD
# File systems
file-system FFS # UFS
file-system LFS # log-structured file system
file-system MFS # memory file system
file-system NFS # Network File System client
file-system CD9660 # ISO 9660 + Rock Ridge file system
file-system MSDOSFS # MS-DOS file system
file-system FDESC # /dev/fd
file-system KERNFS # /kern
file-system NULLFS # loopback file system
file-system OVERLAY # overlay file system
file-system PROCFS # /proc
# File system options
options QUOTA # UFS quotas
options SOFTDEP # FFS soft updates support.
options NFSSERVER # Network File System server
# Networking options
options INET # IP + ICMP + TCP + UDP
# These options enable verbose messages for several subsystems.
# Warning, these may compile large string tables into the kernel!
options NFS_BOOT_DHCP,NFS_BOOT_BOOTPARAM
#
# wscons options
#
# builtin terminal emulations
#options WSEMUL_SUN # sun terminal emulation
options WSEMUL_VT100 # VT100 / VT220 emulation
# different kernel output - see dev/wscons/wsdisplayvar.h
options WS_KERNEL_FG=WSCOL_GREEN
# compatibility to other console drivers
options WSDISPLAY_COMPAT_PCVT # emulate some ioctls
options WSDISPLAY_COMPAT_SYSCONS # emulate some ioctls
options WSDISPLAY_COMPAT_USL # VT handling
options WSDISPLAY_COMPAT_RAWKBD # can get raw scancodes
# see dev/pckbc/wskbdmap_mfii.c for implemented layouts
#options PCKBD_LAYOUT="(KB_DE | KB_NODEAD)"
# allocate a number of virtual screens at autoconfiguration time
#options WSDISPLAY_DEFAULTSCREENS=4
# use a large software cursor that doesn't blink
options PCDISPLAY_SOFTCURSOR
# modify the screen type of the console; defaults to "80x25"
#options VGA_CONSOLE_SCREENTYPE="\"80x24\""
# Kernel root file system and dump configuration.
config netbsd root on ? type ?
#
# Device configuration
#
mainbus0 at root
# com port
# PCI bus support
pci* at mainbus? bus ?
pci* at pchb? bus ?
pci* at ppb? bus ?
# Configure PCI using BIOS information
#options PCIBIOS # PCI BIOS support
#options PCIBIOSVERBOSE # PCI BIOS verbose info
#options PCIBIOS_ADDR_FIXUP # fixup PCI I/O addresses
#options PCIBIOS_BUS_FIXUP # fixup PCI bus numbering
#options PCIBIOS_INTR_FIXUP # fixup PCI interrupt routing
#options PCIBIOS_IRQS_HINT=0x0a00 # PCI interrupts hint. IRQ 9 or
11
#options PCIBIOS_INTR_GUESS # see pcibios(4)
#options PCIINTR_DEBUG # super-verbose PCI interrupt
fixup
# PCI bridges
pchb* at pci? dev ? function ? # PCI-Host bridges
pceb* at pci? dev ? function ? # PCI-EISA bridges
pcib* at pci? dev ? function ? # PCI-ISA bridges
ppb* at pci? dev ? function ? # PCI-PCI bridges
# XXX 'puc's aren't really bridges, but there's no better place for them
here
puc* at pci? dev ? function ? # PCI "universal" comm. cards
# EISA bus support
eisa* at mainbus?
eisa* at pceb?
# ISA bus support
isa* at mainbus?
isa* at pceb?
isa* at pcib?
# ISA Plug-and-Play bus support
isapnp0 at isa?
# Math Coprocessor support
npx0 at isa? port 0xf0 irq 13 # x86 math coprocessor
# Console Devices
# wscons
pckbc0 at isa? # pc keyboard controller
pckbd* at pckbc? # PC keyboard
# "opms" should not be enabled together with "pms" or "pmsi"
pms* at pckbc? # PS/2 mouse for wsmouse
pmsi* at pckbc? # PS/2 "Intelli"mouse for
wsmouse
#opms* at pckbc? # backwards compatible PS/2
mouse
vga0 at isa?
vga* at pci? dev ? function ?
pcdisplay0 at isa? # CGA, MDA, EGA, HGA
wsdisplay* at vga? console ?
wsdisplay* at pcdisplay? console ?
wskbd* at pckbd? console ?
wsmouse* at pms? mux 0
wsmouse* at pmsi? mux 0
pcppi0 at isa?
sysbeep0 at pcppi?
# Serial Devices
# PCI serial interfaces
com* at puc? port ? # 16x50s on "universal" comm
boards
# ISA serial interfaces
#options COM_HAYESP # adds Hayes ESP serial board
support
com0 at isa? port 0x3f8 irq 4 # Standard PC serial ports
com1 at isa? port 0x2f8 irq 3
# Parallel Printer Interfaces
# PCI parallel printer interfaces
lpt* at puc? port ? # || ports on "universal" comm
boards
# ISA parallel printer interfaces
lpt0 at isa? port 0x378 irq 7 # standard PC parallel ports
# Hardware monitors
# LM7[89] and compatible hardware monitors
#lm0 at isa? port 0x290 # other common ports: 0x280,
0x310
# VIA VT82C686A hardware monitor
#viapm* at pci? dev ? function ?
#viaenv* at viapm?
# IDE and related devices
pciide0 at pci? dev ? function ? flags 0x0000
wd0 at pciide0 channel 0 drive 0 flags 0x0fac
wd1 at pciide0 channel 0 drive 1 flags 0x0fac
wd2 at pciide0 channel 1 drive 0 flags 0x0fac
wd3 at pciide0 channel 1 drive 1 flags 0x0fac
atapibus* at pciide? channel ?
cd* at atapibus? drive ? flags 0x0000 # ATAPI CD-ROM drives
# Miscellaneous mass storage devices
# ISA floppy
fdc0 at isa? port 0x3f0 irq 6 drq 2 # standard PC floppy controllers
fd* at fdc? drive ? # the drives themselves
# PCI network interfaces
ex* at pci? dev ? function ? # 3Com 90x[B]
rtk* at pci? dev ? function ? # Realtek 8129/8139
# MII/PHY support
exphy* at mii? phy ? # 3Com internal PHYs
# Audio Devices
# PCI audio devices
auich* at pci? dev ? function ? # Intel ICH integrated AC'97
Audio
auvia* at pci? dev ? function ? # VIA VT82C686A integrated AC'97
Audio
clcs* at pci? dev ? function ? # Cirrus Logic CS4280
clct* at pci? dev ? function ? # Cirrus Logic CS4281
cmpci* at pci? dev ? function ? # C-Media CMI8338/8738
eap* at pci? dev ? function ? # Ensoniq AudioPCI
esm* at pci? dev ? function ? # ESS Maestro-1/2/2e PCI Audio
Accelerator
eso* at pci? dev ? function ? # ESS Solo-1 PCI AudioDrive
fms* at pci? dev ? function ? # Forte Media FM801
neo* at pci? dev ? function ? # NeoMagic 256 AC'97 Audio
sv* at pci? dev ? function ? # S3 SonicVibes
# ISA Plug-and-Play audio devices
ess* at isapnp? # ESS Tech ES1887, ES1888, ES888
audio
guspnp* at isapnp? # Gravis Ultra Sound PnP audio
sb* at isapnp? # SoundBlaster-compatible audio
wss* at isapnp? # Windows Sound System
ym* at isapnp? # Yamaha OPL3-SA3 audio
# ISA audio devices
# the "aria" probe might give false hits
#aria0 at isa? port 0x290 irq 10 # Aria
#ess0 at isa? port 0x220 irq 5 drq 1 drq2 5 # ESS 18XX
gus0 at isa? port 0x220 irq 7 drq 1 drq2 6 # Gravis Ultra Sound
pas0 at isa? port 0x220 irq 7 drq 1 # ProAudio Spectrum
pss0 at isa? port 0x220 irq 7 drq 6 # Personal Sound System
sp0 at pss0 port 0x530 irq 10 drq 0 # sound port
driver
sb0 at isa? port 0x220 irq 5 drq 1 drq2 5 # SoundBlaster
wss0 at isa? port 0x530 irq 10 drq 0 drq2 1 # Windows Sound System
#cms0 at isa? port 0x220 # Creative Music System
# OPL[23] FM synthesizers
#opl0 at isa? port 0x388 # use only if not attached to sound card
opl* at eso?
opl* at ess?
opl* at fms?
opl* at sb?
opl* at sv?
opl* at wss?
opl* at ym?
# Audio support
#audio* at aria?
audio* at auich?
audio* at auvia?
audio* at clcs?
audio* at clct?
audio* at cmpci?
audio* at eap?
audio* at esm?
audio* at eso?
audio* at ess?
audio* at fms?
audio* at gus?
audio* at guspnp?
audio* at neo?
audio* at pas?
audio* at sb?
audio* at sp?
audio* at sv?
audio* at wss?
audio* at ym?
# MPU 401 UARTs
#mpu* at isa? port 0x330 irq 9 # MPU401 or compatible card
mpu* at eso?
mpu* at fms?
mpu* at sb?
mpu* at ym?
# MIDI support
midi* at clcs? # Cirrus Logic CS4280 MIDI port
#midi* at clct? # Cirrus Logic CS4281 MIDI port
#midi* at cms? # Creative Music System
midi* at eap? # 137[01] MIDI port
midi* at mpu? # MPU 401
midi* at opl? # OPL FM synth
midi* at pcppi? # MIDI interface to the PC speaker
midi* at sb? # SB1 MIDI port
# The spkr driver provides a simple tone interface to the built in
speaker.
#spkr0 at pcppi? # PC speaker
# TV cards
# Mice
# Pull in optional local configuration
include "arch/i386/conf/GENERIC.local"
# disk/mass storage pseudo-devices
pseudo-device ccd 4 # concatenated/striped disk
devices
pseudo-device raid 4 # RAIDframe disk driver
options RAID_AUTOCONFIG # auto-configuration of RAID
components
pseudo-device md 1 # memory disk device (ramdisk)
pseudo-device vnd 4 # disk-like interface to files
# network pseudo-devices
pseudo-device bpfilter 8 # Berkeley packet filter
pseudo-device ipfilter # IP filter (firewall) and NAT
pseudo-device loop # network loopback
pseudo-device ppp 2 # Point-to-Point Protocol
pseudo-device sl 2 # Serial Line IP
pseudo-device strip 2 # Starmode Radio IP (Metricom)
pseudo-device tun 2 # network tunneling over tty
pseudo-device gre 2 # generic L3 over IP tunnel
pseudo-device ipip 2 # IP Encapsulation within IP
(RFC 2003)
# miscellaneous pseudo-devices
pseudo-device pty # pseudo-terminals
pseudo-device tb 1 # tablet line discipline
pseudo-device sequencer 1 # MIDI sequencer
# rnd works; RND_COM does not on port i386 yet.
pseudo-device rnd # /dev/random and in-kernel
generator
# mouse & keyboard multiplexor pseudo-devices
pseudo-device wsmux
and last but not least dmesg :
NetBSD 1.5R (NETAPP) #1: Thu Feb 8 05:29:20 UTC 2001
root@netapp:/usr/src/sys/arch/i386/compile/NETAPP
cpu0: AMD K6-2 (586-class), 501.16 MHz
cpu0: features 8021bf<FPU,VME,DE,PSE,TSC,MSR,MCE,CX8>
cpu0: features 8021bf<PGE,MMX>
total memory = 255 MB
avail memory = 234 MB
using 3296 buffers containing 13184 KB of memory
BIOS32 rev. 0 found at 0xfb270
mainbus0 (root)
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled
pchb0 at pci0 dev 0 function 0
pchb0: vendor 0x1106 product 0x0597 (rev. 0x04)
ppb0 at pci0 dev 1 function 0: vendor 0x1106 product 0x8598 (rev. 0x00)
pci1 at ppb0 bus 1
pci1: i/o space, memory space enabled
vga1 at pci1 dev 0 function 0: vendor 0x1039 product 0x6326 (rev. 0x0b)
wsdisplay0 at vga1: console (80x25, vt100 emulation)
pcib0 at pci0 dev 7 function 0
pcib0: vendor 0x1106 product 0x0596 (rev. 0x12)
pciide0 at pci0 dev 7 function 1: VIA Tech VT82C586A IDE Controller
(rev. 0x06)
pciide0: bus-master DMA support present
pciide0: Ultra/66 capable
pciide0: primary channel configured to compatibility mode
wd0 at pciide0 channel 0 drive 0: <QUANTUM FIREBALLlct20 20>
wd0: drive supports 8-sector PIO transfers, LBA addressing
wd0: 19470 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 39876480
sectors
wd0: 32-bit data port
wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
wd1 at pciide0 channel 0 drive 1: <QUANTUM FIREBALLlct20 20>
wd1: drive supports 8-sector PIO transfers, LBA addressing
wd1: 19470 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 39876480
sectors
wd1: 32-bit data port
wd1: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
pciide0: primary channel interrupting at irq 14
wd0(pciide0:0:0): using PIO mode 4, DMA mode 2 (using DMA data
transfers)
wd1(pciide0:0:1): using PIO mode 4, DMA mode 2 (using DMA data
transfers)
pciide0: secondary channel configured to compatibility mode
wd2 at pciide0 channel 1 drive 0: <IBM-DPTA-372050>
wd2: drive supports 16-sector PIO transfers, LBA addressing
wd2: 19574 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 40088160
sectors
wd2: 32-bit data port
wd2: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 4 (Ultra/66)
wd3 at pciide0 channel 1 drive 1: <QUANTUM FIREBALLP LM20.5>
wd3: drive supports 16-sector PIO transfers, LBA addressing
wd3: 19595 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 40132503
sectors
wd3: 32-bit data port
wd3: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 4 (Ultra/66)
pciide0: secondary channel interrupting at irq 15
wd2(pciide0:1:0): using PIO mode 4, DMA mode 2 (using DMA data
transfers)
wd3(pciide0:1:1): using PIO mode 4, DMA mode 2 (using DMA data
transfers)
vendor 0x1106 product 0x3038 (USB serial bus, revision 0x08) at pci0 dev
7 function 2 not configured
pchb1 at pci0 dev 7 function 3
pchb1: vendor 0x1106 product 0x3050 (rev. 0x20)
ex0 at pci0 dev 11 function 0: 3Com 3c905B-TX 10/100 Ethernet (rev.
0x64)
ex0: interrupting at irq 10
ex0: MAC address 00:50:da:45:3e:d3
OUI 0x001018 model 0x0012 rev 0 at ex0 phy 24 not configured
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
lpt0 at isa0 port 0x378-0x37b irq 7
pcppi0 at isa0 port 0x61
midi0 at pcppi0: PC speaker
sysbeep0 at pcppi0
isapnp0 at isa0 port 0x279: ISA Plug 'n Play device support
npx0 at isa0 port 0xf0-0xff: using exception 16
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
isapnp0: no ISA Plug 'n Play devices found
biomask fb65 netmask ff65 ttymask ffe7
Kernelized RAIDframe activated
RAID autoconfigure
Configuring raid0:
RAIDFRAME: protectedSectors is 64
RAIDFRAME: Configure (RAID Level 5): total number of sectors is
119440800 (58320 MB)
RAIDFRAME(RAID Level 5): Using 20 floating recon bufs with head sep
limit 10
boot device: raid0
root on raid0a dumps on raid0b
root file system type: ffs
wsdisplay0: screen 1 added (80x25, vt100 emulation)
wsdisplay0: screen 2 added (80x25, vt100 emulation)
wsdisplay0: screen 3 added (80x25, vt100 emulation)
wsdisplay0: screen 4 added (80x25, vt100 emulation)
wsmux1: connecting to wsdisplay0
ANy Idea?
Please help ,or i have to use vinum on FreeBSD ;)
Bye!
P.S. Of course my disk speed was not 80MB/s it was an error ,sorry.