So, I've upgraded a Xen server to xenkernel413-4.13.2nb5 and
xentools413-4.13.2nb4 and tried my FreeBSD tests again, but with no
luck, and indeed no visible change whatsoever. (dom0 is still 9.99.81)
So, I'm guessing the problem is in NetBSD dom0, not in FreeBSD and not
in Xen.
The performance issues with FreeBSD domU on different hardware remain
most mysterious too, but perhaps this is due to the difference between
real hardware CPUs on the different servers, with these older ones
requiring more help/hinderance from Xen and/or more likely FreeBSD
itself (e.g. for recent hardware vulnerability mitigations). FreeBSD
shows the following on the newer hardware, with the older one shown in
the full output below:
CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz (2400.09-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2
Features=0x1fc3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT>
Features2=0x829a2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,POPCNT,AESNI,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x1<LAHF>
Data TLB0: 2-MByte or 4 MByte pages, 4-way set associative, 32 entries
Data TLB: 4 KB pages, 4-way set associative, 64 entries
Instruction TLB: 2-MByte or 4-MByte pages, fully associative, 7 entries
Instruction TLB: 4KByte pages, 4-way set associative, 64 entries
64-Byte prefetching
Shared 2nd-Level TLB: 4 KByte pages, 4-way associative, 512 entries
L2 cache: 256 kbytes, 8-way associative, 64 bytes/line
The following is with the FreeBSD-12.2 installer mini-memstick.img file
attached to hda, using a type='pvh' domU (and as noted before that image
boots A-OK on real hardware) booting from a 12.2 release kernel:
# xl create -c /etc/xen/fbsd-test-pvh.conf
Parsing config from /etc/xen/fbsd-test-pvh.conf
---<<BOOT>>---
APIC: Using the MADT enumerator.
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-RELEASE-p4 GENERIC amd64
FreeBSD clang version 10.0.1 (git%github.com@localhost:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
PPIM 0: PA=0xfefff000, VA=0xffffffff82710000, size=0x1000, mode=0x6
pmap: large map 8 PML4 slots (4096 Gb)
VT: init without driver.
XEN: Hypervisor version 4.13 detected.
Calibrating TSC clock ... TSC clock: 3158824082 Hz
CPU: Intel(R) Xeon(R) CPU X5460 @ 3.16GHz (3158.82-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x10676 Family=0x6 Model=0x17 Stepping=6
Features=0x1fc3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT>
Features2=0x80082201<SSE3,SSSE3,CX16,SSE4.1,HV>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Instruction TLB: 2M pages, 4-way, 8 entries or 4M pages, 4-way, 4 entries
Instruction TLB: 4 KB Pages, 4-way set associative, 128 entries
64-Byte prefetching
Data TLB0: 4 KByte pages, 4-way associative, 16 entries
Data TLB0: 4 MByte pages, 4-way set associative, 16 entries
2nd-level cache: 6MByte, 24-way set associative, 64 byte line size
1st-level instruction cache: 32 KB, 8-way set associative, 64 byte line size
Data TLB1: 4 KByte pages, 4-way associative, 256 entries
1st-level data cache: 32 KB, 8-way set associative, 64 byte line size
L2 cache: 6144 kbytes, 16-way associative, 64 bytes/line
Hypervisor: Origin = "XenVMMXenVMM"
real memory = 8455716864 (8064 MB)
Physical memory chunk(s):
0x0000000000004000 - 0x00000000001fffff, 2080768 bytes (508 pages)
0x0000000002700000 - 0x00000000ef25cfff, 3971338240 bytes (969565 pages)
0x0000000100000000 - 0x00000001f7fe7fff, 4160651264 bytes (1015784 pages)
avail memory = 8099274752 (7724 MB)
MADT: Found CPU APIC ID 0 ACPI ID 0: enabled
SMP: Added CPU 0 (AP)
MADT: Found CPU APIC ID 2 ACPI ID 1: enabled
SMP: Added CPU 2 (AP)
MADT: Found CPU APIC ID 4 ACPI ID 2: enabled
SMP: Added CPU 4 (AP)
MADT: Found CPU APIC ID 6 ACPI ID 3: enabled
SMP: Added CPU 6 (AP)
Event timer "LAPIC" quality 100
LAPIC: ipi_wait() us multiplier 1 (r 854304886 tsc 3158824082)
ACPI APIC Table: <Xen HVM>
Package ID shift: 3
L2 cache ID shift: 0
L1 cache ID shift: 0
Core ID shift: 0
INTR: Adding local APIC 2 as a target
INTR: Adding local APIC 4 as a target
INTR: Adding local APIC 6 as a target
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
Package HW ID = 0
Core HW ID = 0
CPU0 (BSP): APIC ID: 0
Core HW ID = 2
CPU1 (AP): APIC ID: 2
Core HW ID = 4
CPU2 (AP): APIC ID: 4
Core HW ID = 6
CPU3 (AP): APIC ID: 6
APIC: CPU 0 has ACPI ID 0
APIC: CPU 1 has ACPI ID 1
APIC: CPU 2 has ACPI ID 2
APIC: CPU 3 has ACPI ID 3
module_register_init: MOD_LOAD (x86bios, 0xffffffff81200e80, 0) error 12
lapic0: CMCI unmasked
Pentium Pro MTRR support enabled
arc4random: no preloaded entropy cache
VIMAGE (virtualized network stack) enabled
hostuuid: using 00000000-0000-0000-0000-000000000000
ULE: setup cpu 0
ULE: setup cpu 1
ULE: setup cpu 2
ULE: setup cpu 3
Xen interrupts: unable to register PIRQ EOI map
Xen interrupt system initialized
ACPI: RSDP 0x00000000FC008000 000024 (v02 Xen )
ACPI: XSDT 0x00000000FC007F80 000034 (v01 Xen HVM 00000000 HVML 00000000)
ACPI: FACP 0x00000000FC007D70 00010C (v05 Xen HVM 00000000 HVML 00000000)
ACPI: DSDT 0x00000000FC001050 006C9B (v05 Xen HVM 00000000 INTL 20160930)
ACPI: FACS 0x00000000FC001010 000040
ACPI: APIC 0x00000000FC007E80 00004C (v02 Xen HVM 00000000 HVML 00000000)
MADT: Could not find APIC for SCI IRQ 9
lapic: Divisor 2, Frequency 50001577 Hz
cpu0 BSP:
ID: 0x00000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
lapic6: CMCI unmasked
SMP: AP CPU #3 Launched!
cpu3 AP:
ID: 0x06000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
lapic2: CMCI unmasked
SMP: AP CPU #1 Launched!
cpu1 AP:
ID: 0x02000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
lapic4: CMCI unmasked
SMP: AP CPU #2 Launched!
cpu2 AP:
ID: 0x04000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff
lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff
timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400
TSC timecounter disables C2 and C3.
TSC timecounter discards lower 1 bit(s)
Timecounter "TSC-low" frequency 1579412041 Hz quality -100
wlan: <802.11 Link Layer>
snd_unit_init() u=0x00ff8000 [512] d=0x00007c00 [32] c=0x000003ff [1024]
feeder_register: snd_unit=-1 snd_maxautovchans=16 latency=2 feeder_rate_min=1 feeder_rate_max=2016000 feeder_rate_round=25
random: entropy device external interface
EFI systbl not available
kbd0 at kbdmux0
mem: <memory>
000.000023 [4336] netmap_init netmap: loaded module
null: <full device, null device, zero device>
[ath_hal] loaded
nfslock: pseudo-device
crypto: <crypto core>
module_register_init: MOD_LOAD (vesa, 0xffffffff81114e40, 0) error 19
io: <I/O>
tcp_log: tcp_log device
hpt27xx: RocketRAID 27xx controller driver v1.2.8
hptnr: R750/DC7280 controller driver v1.1.5
hptrr: RocketRAID 17xx/2xxx SATA controller driver v1.2
nexus0
cryptosoft0: <software crypto> on motherboard
crypto: assign cryptosoft0 driver id 0, flags 0x6000000
crypto: cryptosoft0 registers alg 1 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 2 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 3 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 4 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 5 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 16 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 6 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 7 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 32 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 18 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 19 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 20 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 8 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 15 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 9 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 10 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 13 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 14 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 34 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 35 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 36 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 37 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 11 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 22 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 23 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 25 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 24 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 26 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 27 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 28 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 21 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 17 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 29 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 30 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 31 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 40 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 39 flags 0 maxoplen 0
crypto: cryptosoft0 registers alg 38 flags 0 maxoplen 0
acpi0: <Xen> on motherboard
ACPI: 1 ACPI AML tables successfully acquired and loaded
acpi0: could not allocate interrupt
ACPI Error: AE_ALREADY_EXISTS, Unable to install System Control Interrupt handler (20200430/evevent-260)
acpi0: Could not enable ACPI: AE_ALREADY_EXISTS
device_attach: acpi0 attach returned 6
xenpv0: <Xen PV bus> on motherboard
granttable0: <Xen Grant-table Device> on xenpv0
Grant table initialized
xc0: <Xen Console> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xen_et0: registered as a time-of-day clock, resolution 0.000001s
pvcpu0: <Xen PV CPU> on xenpv0
pvcpu1: <Xen PV CPU> on xenpv0
pvcpu2: <Xen PV CPU> on xenpv0
pvcpu3: <Xen PV CPU> on xenpv0
xenstore0: <XenStore> on xenpv0
xsd_dev0: <Xenstored user-space device> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
gntdev0: <Xen grant-table user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
Device configuration finished.
procfs registered
Timecounters tick every 10.000 msec
lo0: bpf attached
vlan: initialized, using hash tables with chaining
IPsec: Initialized Security Association Processing.
arc4random: no preloaded entropy cache
tcp_init: net.inet.tcp.tcbhashsize auto tuned to 65536
usb_needs_explore_all: no devclass
hpt27xx: no controller detected.
hptnr: no controller detected.
hptrr: no controller detected.
xctrl0: <Xen Control Device> on xenstore0
xs_dev0: <Xenstore user-space device> on xenstore0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xenbusb_add_device: Device device/suspend/event-channel ignored. State 6
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn0: bpf attached
xn0: Ethernet address: 00:16:3e:43:73:31
xenbusb_back0: <Xen Backend Devices> on xenstore0
xenballoon0: <Xen Balloon Device> on xenstore0
xbd0: 386MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: attaching as ada0
xbd0: features: flush
xbd0: synchronize cache commands enabled.
GEOM: new disk ada0
xn0: backend features: feature-sg
Trying to mount root from ufs:/dev/ada0 [ro]...
GEOM_PART: partition 2 has end offset beyond last LBA: 791120 > 790527
GEOM_PART: integrity check failed (ada0, MBR)
arc4random: no preloaded entropy cache
Mounting from ufs:/dev/ada0 failed with error 2; retrying for 3 more seconds
arc4random: no preloaded entropy cache
Mounting from ufs:/dev/ada0 failed with error 2; retrying for 2 more seconds
Mounting from ufs:/dev/ada0 failed with error 2; retrying for 1 more second
Mounting from ufs:/dev/ada0 failed with error 2.
Loader variables:
vfs.root.mountfrom=ufs:/dev/ada0
vfs.root.mountfrom.options=ro
Manual root filesystem specification:
<fstype>:<device> [options]
Mount <device> using filesystem <fstype>
and with the specified (optional) option list.
eg. ufs:/dev/da0s1a
zfs:zroot/ROOT/default
cd9660:/dev/cd0 ro
(which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)
? List valid disk boot devices
. Yield 1 second (for background tasks)
<empty line> Abort manual input
mountroot> ?
List of GEOM managed disk devices:
ada0
mountroot> random: unblocking device.
For reference here's what NetBSD gpt(8) has to say about that image
file:
$ /sbin/gpt -vvv show -a /build/images/FreeBSD-12.2-RELEASE-amd64-mini-memstick.img
/build/images/FreeBSD-12.2-RELEASE-amd64-mini-memstick.img: mediasize=405053952; sectorsize=512; blocks=791121
/build/images/FreeBSD-12.2-RELEASE-amd64-mini-memstick.img: MBR at sector 0
/build/images/FreeBSD-12.2-RELEASE-amd64-mini-memstick.img: MBR part: flag=0 type=239, start=1, size=1600
/build/images/FreeBSD-12.2-RELEASE-amd64-mini-memstick.img: MBR part: flag=0x80 type=165, start=1601, size=789520
GPT not found, displaying data from MBR.
start size index contents
0 1 MBR
1 1600 1 MBR part 239
Type: unknown
TypeID: 00000000-0000-0000-0100-000000000000
GUID: cb1f5080-7cbf-0000-0000-000000000000
Size: 800 K
Label:
Attributes: required
1601 789520 2 MBR part 165 (active)
Type: unknown
TypeID: 00000000-0000-0000-0100-000000000000
GUID: cb1f5080-7cbf-0000-0000-000000000000
Size: 386 M
Label:
Attributes: required
--
Greg A. Woods <gwoods%acm.org@localhost>
Kelowna, BC +1 250 762-7675 RoboHack <woods%robohack.ca@localhost>
Planix, Inc. <woods%planix.com@localhost> Avoncote Farms <woods%avoncote.ca@localhost>
Attachment:
pgpjBYodskD0O.pgp
Description: OpenPGP Digital Signature