Re: Symbol debugging support for kernel modules in crash dumps

[David Young <dyoung%pobox.com@localhost>]

Fantastic! Thanks.

Dave

Spilling kerrectud by iPhone

> On May 1, 2020, at 6:34 PM, Christos Zoulas <christos%zoulas.com@localhost> wrote:
> 
> 
> Hi,
> 
> I just added symbol debugging support for modules in kernel dumps.
> Things are not perfect because of what I call "current thread
> confusion" in the kvm target, but as you see in the following
> session it works just fine if you follow the right steps. First of
> all you need a build from HEAD that has the capability to build
> .debug files for kernel modules.  Once that's done, you are all
> set; see how it works (comments prefixed by ####)
> 
> Enjoy,
> 
> christos
> 
> $ gdb /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb
> GNU gdb (GDB) 8.3
> Copyright (C) 2019 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64--netbsd".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
>    <http://www.gnu.org/software/gdb/documentation/>.
> 
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb...
> (gdb) target kvm netbsd.22.core
> 0xffffffff80224375 in cpu_reboot (howto=howto@entry=260, 
>    bootstr=bootstr@entry=0x0) at ../../../../arch/amd64/amd64/machdep.c:718
> warning: Source file is more recent than executable.
> 718                     if (s != IPL_NONE)
> 
> #### Ok we got a stacktrace here, but we don't have a current thread...
> #### So we set it...
> 
> (gdb) info thread
>  Id   Target Id         Frame 
> * 2.1  <kvm>             0xffffffff80224375 in cpu_reboot (
>    howto=howto@entry=260, bootstr=bootstr@entry=0x0)
>    at ../../../../arch/amd64/amd64/machdep.c:718
> 
> No selected thread.  See `help thread'.
> (gdb) thread 2.1
> 
> [Switching to thread 2.1 (<kvm>)]
> #0  0xffffffff80224375 in ?? ()
> 
> #### Note that here we lost all symbol table access when we switched threads
> #### let's load it again..
> 
> (gdb) add-symbol-file /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb
> add symbol table from file "/usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb"
> (y or n) y
> Reading symbols from /usr/src/sys/arch/amd64/compile/QUASAR/netbsd.gdb...
> 
> #### OK, lets load our modules
> 
> (gdb) source /usr/src/sys/gdbscripts/modload 
> (gdb) modload
> add symbol table from file "/stand/amd64/9.99.59/modules/ping/ping.kmod" at
>        .text_addr = 0xffffffff8266e000
>        .data_addr = 0xffffffff8266b000
>        .rodata_addr = 0xffffffff8266c000
> add symbol table from file "/stand/amd64/9.99.59/modules/nfsserver/nfsserver.kmod" at
>        .text_addr = 0xffffffff82a64000
>        .data_addr = 0xffffffff82669000
>        .rodata_addr = 0xffffffff8298e000
> add symbol table from file "/stand/amd64/9.99.59/modules/npf_ext_log/npf_ext_log.kmod" at
>        .text_addr = 0xffffffff82668000
>        .data_addr = 0xffffffff82667000
>        .rodata_addr = 0xffffffff82969000
> add symbol table from file "/stand/amd64/9.99.59/modules/npf_alg_icmp/npf_alg_icmp.kmod" at
>        .text_addr = 0xffffffff82666000
>        .data_addr = 0xffffffff82665000
>        .rodata_addr = 0xffffffff82952000
> add symbol table from file "/stand/amd64/9.99.59/modules/bpfjit/bpfjit.kmod" at
>        .text_addr = 0xffffffff82661000
>        .data_addr = 0x0
>        .rodata_addr = 0xffffffff828dd000
> add symbol table from file "/stand/amd64/9.99.59/modules/sljit/sljit.kmod" at
>        .text_addr = 0xffffffff82945000
>        .data_addr = 0xffffffff82664000
>        .rodata_addr = 0xffffffff828f9000
> add symbol table from file "/stand/amd64/9.99.59/modules/if_npflog/if_npflog.kmod" at
>        .text_addr = 0xffffffff82660000
>        .data_addr = 0xffffffff8265f000
>        .rodata_addr = 0xffffffff828ca000
> add symbol table from file "/stand/amd64/9.99.59/modules/npf/npf.kmod" at
>        .text_addr = 0xffffffff82648000
>        .data_addr = 0xffffffff82647000
>        .rodata_addr = 0xffffffff826d6000
> add symbol table from file "/stand/amd64/9.99.59/modules/bpf/bpf.kmod" at
>        .text_addr = 0xffffffff82622000
>        .data_addr = 0xffffffff82621000
>        .rodata_addr = 0xffffffff826a3000
> add symbol table from file "/stand/amd64/9.99.59/modules/bpf_filter/bpf_filter.kmod" at
>        .text_addr = 0xffffffff8263c000
>        .data_addr = 0x0
>        .rodata_addr = 0xffffffff82627000
> add symbol table from file "/stand/amd64/9.99.59/modules/scsiverbose/scsiverbose.kmod" at
>        .text_addr = 0xffffffff826a2000
>        .data_addr = 0xffffffff82686000
>        .rodata_addr = 0xffffffff82687000
> add symbol table from file "/stand/amd64/9.99.59/modules/usbverbose/usbverbose.kmod" at
>        .text_addr = 0xffffffff82685000
>        .data_addr = 0xffffffff82670000
>        .rodata_addr = 0xffffffff82671000
> add symbol table from file "/stand/amd64/9.99.59/modules/miiverbose/miiverbose.kmod" at
>        .text_addr = 0xffffffff82646000
>        .data_addr = 0xffffffff8263d000
>        .rodata_addr = 0xffffffff8263e000
> 
> #### Finally!
> 
> (gdb) where
> #0  0xffffffff80224375 in cpu_reboot (howto=howto@entry=260, 
>    bootstr=bootstr@entry=0x0) at ../../../../arch/amd64/amd64/machdep.c:718
> #1  0xffffffff80680a7f in kern_reboot (howto=260, bootstr=bootstr@entry=0x0)
>    at ../../../../kern/kern_reboot.c:73
> #2  0xffffffff804b5003 in db_reboot_cmd (addr=<optimized out>, 
>    have_addr=<optimized out>, count=<optimized out>, modif=<optimized out>)
>    at ../../../../ddb/db_command.c:1436
> #3  0xffffffff804b581b in db_command (
>    last_cmdp=last_cmdp@entry=0xffffffff80e2e6a0 <db_last_command>)
>    at ../../../../ddb/db_command.c:940
> #4  0xffffffff804b5b86 in db_command_loop ()
>    at ../../../../ddb/db_command.c:599
> #5  0xffffffff804b968a in db_trap (type=type@entry=1, code=code@entry=0)
>    at ../../../../ddb/db_trap.c:91
> #6  0xffffffff80220c05 in kdb_trap (type=type@entry=1, code=code@entry=0, 
>    regs=regs@entry=0xffffc784a8f58c00)
>    at ../../../../arch/amd64/amd64/db_interface.c:247
> #7  0xffffffff80225ef2 in trap (frame=0xffffc784a8f58c00)
>    at ../../../../arch/amd64/amd64/trap.c:315
> #8  0xffffffff8021ed43 in alltraps ()
> #9  0xffffffff8021f55d in breakpoint ()
> 
> #### Those two are from the module...
> 
> #10 0xffffffff8266e04c in ping_ioctl (self=<optimized out>, cmd=536899586, 
>    data=<optimized out>, flag=<optimized out>, l=<optimized out>)
>    at /net/quasar/src-5/NetBSD/src.acl/sys/modules/examples/ping/ping.c:104
> #11 ping_ioctl (self=<optimized out>, cmd=<optimized out>, 
>    data=<optimized out>, flag=<optimized out>, l=<optimized out>)
>    at /net/quasar/src-5/NetBSD/src.acl/sys/modules/examples/ping/ping.c:98
> 
> 
> #12 0xffffffff806b1f68 in cdev_ioctl (dev=89856, cmd=536899586, 
>    data=0xffffc784a8f58ee0, flag=3, l=0xffffa6841902a140)
>    at ../../../../kern/subr_devsw.c:935
> #13 0xffffffff8073505c in VOP_IOCTL (vp=vp@entry=0xffffa6840d48c380, 
>    command=command@entry=536899586, data=data@entry=0xffffc784a8f58ee0, 
>    fflag=<optimized out>, cred=<optimized out>)
>    at ../../../../kern/vnode_if.c:646
> #14 0xffffffff8072b87e in vn_ioctl (fp=0xffffa68408795100, com=536899586, 
>    data=0xffffc784a8f58ee0) at ../../../../kern/vfs_vnops.c:780
> #15 0xffffffff806d5bc7 in sys_ioctl (l=<optimized out>, 
>    uap=0xffffc784a8f59000, retval=<optimized out>)
>    at ../../../../kern/sys_generic.c:671
> #16 0xffffffff80254aae in sy_call (rval=0xffffc784a8f58fb0, 
> --Type <RET> for more, q to quit, c to continue without paging--
>    uap=0xffffc784a8f59000, l=0xffffa6841902a140, 
>    sy=0xffffffff80e442f0 <sysent+1296>) at ../../../../sys/syscallvar.h:65
> #17 sy_invoke (code=54, rval=0xffffc784a8f58fb0, uap=0xffffc784a8f59000, 
>    l=0xffffa6841902a140, sy=0xffffffff80e442f0 <sysent+1296>)
>    at ../../../../sys/syscallvar.h:94
> #18 syscall (frame=0xffffc784a8f59000)
>    at ../../../../arch/x86/x86/syscall.c:138
> #19 0xffffffff802096ad in handle_syscall ()
> (gdb) q
> A debugging session is active.
> 
>        Inferior 2 [<kvm>] will be killed.
> 
> Quit anyway? (y or n) y
> You can't do that without a process to debug.