3 Debugging kernel and modules via gdb
4 ====================================
6 The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
7 interfaces allow to debug the Linux kernel and its modules during runtime
8 using gdb. Gdb comes with a powerful scripting interface for python. The
9 kernel provides a collection of helper scripts that can simplify typical
10 kernel debugging steps. This is a short tutorial about how to enable and use
11 them. It focuses on QEMU/KVM virtual machines as target, but the examples can
12 be transferred to the other gdb stubs as well.
18 - gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
25 - Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
26 www.qemu.org for more details). For cross-development,
27 https://landley.net/aboriginal/bin keeps a pool of machine images and
28 toolchains that can be helpful to start from.
30 - Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
31 CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
32 CONFIG_FRAME_POINTER, keep it enabled.
34 - Install that kernel on the guest, turn off KASLR if necessary by adding
35 "nokaslr" to the kernel command line.
36 Alternatively, QEMU allows to boot the kernel directly using -kernel,
37 -append, -initrd command line switches. This is generally only useful if
38 you do not depend on modules. See QEMU documentation for more details on
39 this mode. In this case, you should build the kernel with
40 CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR.
42 - Build the gdb scripts (required on kernels v5.1 and above)::
46 - Enable the gdb stub of QEMU/KVM, either
48 - at VM startup time by appending "-s" to the QEMU command line
52 - during runtime by issuing "gdbserver" from the QEMU monitor
55 - cd /path/to/linux-build
57 - Start gdb: gdb vmlinux
59 Note: Some distros may restrict auto-loading of gdb scripts to known safe
60 directories. In case gdb reports to refuse loading vmlinux-gdb.py, add::
62 add-auto-load-safe-path /path/to/linux-build
64 to ~/.gdbinit. See gdb help for more details.
66 - Attach to the booted guest::
68 (gdb) target remote :1234
71 Examples of using the Linux-provided gdb helpers
72 ------------------------------------------------
74 - Load module (and main kernel) symbols::
78 scanning for modules in /home/user/linux/build
79 loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
80 loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
81 loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
82 loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
83 loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
85 loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
87 - Set a breakpoint on some not yet loaded module function, e.g.::
89 (gdb) b btrfs_init_sysfs
90 Function "btrfs_init_sysfs" not defined.
91 Make breakpoint pending on future shared library load? (y or [n]) y
92 Breakpoint 1 (btrfs_init_sysfs) pending.
94 - Continue the target::
98 - Load the module on the target and watch the symbols being loaded as well as
101 loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
102 loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
103 loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
104 loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
106 Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
107 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
109 - Dump the log buffer of the target kernel::
112 [ 0.000000] Initializing cgroup subsys cpuset
113 [ 0.000000] Initializing cgroup subsys cpu
114 [ 0.000000] Linux version 3.8.0-rc4-dbg+ (...
115 [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
116 [ 0.000000] e820: BIOS-provided physical RAM map:
117 [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
118 [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
121 - Examine fields of the current task struct(supported by x86 and arm64 only)::
123 (gdb) p $lx_current().pid
125 (gdb) p $lx_current().comm
126 $2 = "modprobe\000\000\000\000\000\000\000"
128 - Make use of the per-cpu function for the current or a specified CPU::
130 (gdb) p $lx_per_cpu("runqueues").nr_running
132 (gdb) p $lx_per_cpu("runqueues", 2).nr_running
135 - Dig into hrtimers using the container_of helper::
137 (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
138 (gdb) p *$container_of($next, "struct hrtimer", "node")
142 __rb_parent_color = 18446612133355256072,
143 rb_right = 0x0 <irq_stack_union>,
144 rb_left = 0x0 <irq_stack_union>
153 function = 0xffffffff81078232 <tick_sched_timer>,
154 base = 0xffff88003fd0d6f0,
157 start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
158 start_comm = "swapper/2\000\000\000\000\000\000"
162 List of commands and functions
163 ------------------------------
165 The number of commands and convenience functions may evolve over the time,
166 this is just a snapshot of the initial version::
169 function lx_current -- Return current task
170 function lx_module -- Find module by name and return the module variable
171 function lx_per_cpu -- Return per-cpu variable
172 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
173 function lx_thread_info -- Calculate Linux thread_info from task variable
174 lx-dmesg -- Print Linux kernel log buffer
175 lx-lsmod -- List currently loaded modules
176 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
178 Detailed help can be obtained via "help <command-name>" for commands and "help
179 function <function-name>" for convenience functions.