libbpf-tools/ksnoop: kernel argument/return value tracing/display using BTF

libbpf-tools/ksnoop: kernel argument/return value tracing/display using BTF

BPF Type Format (BTF) provides a description of kernel data structures.
libbpf support was recently added - btf_dump__dump_type_data() -
that uses the BTF id of the associated type to create a string
representation of the data provided.  For example, to create a string
representation of a "struct sk_buff", the pointer to the skb
data is provided along with the type id of "struct sk_buff".

Here that functionality is utilized to support tracing kernel
function entry and return using k[ret]probes.  The "struct pt_regs"
context can be used to derive arguments and return values, and
when the user supplies a function name we

- look it up in /proc/kallsyms to find its address/module
- look it up in the BTF kernel/module data to get types of arguments
  and return value
- store a map representation of the trace information, keyed by
  function address

On function entry/return we look up info about the arguments (is
it a pointer? what size of data do we copy?) and call bpf_probe_read()
to copy the data into our trace buffers.  These are then sent via
perf event to userspace, and since we know the associated BTF id,
we can dump the typed data using btf_dump__dump_type_data().

ksnoop can be used to show function signatures; for example:

$ ksnoop info ip_send_skb
int  ip_send_skb(struct net  * net, struct sk_buff  * skb);

Then we can trace the function, for example:

$ ksnoop trace ip_send_skb
            TIME  CPU      PID FUNCTION/ARGS
  78101668506811    1     2813 ip_send_skb(
                                   net = *(0xffffffffb5959840)
                                    (struct net){
                                     .passive = (refcount_t){
                                      .refs = (atomic_t){
                                       .counter = (int)0x2,
                                      },
                                     },
                                     .dev_base_seq = (unsigned int)0x18,
                                     .ifindex = (int)0xf,
                                     .list = (struct list_head){
                                      .next = (struct list_head *)0xffff9895
                                      .prev = (struct list_head *)0xffffffff
                                     },
[output truncated]

  78178228354796    1     2813 ip_send_skb(
                                   return =
                                    (int)0x0
                               );

We see the raw value of pointers along with the typed representation
of the data they point to.

Up to five arguments are supported.

The arguments are referred to via name (e.g. skb, net), and
the return value is referred to as "return" (using the keyword
ensures we can never clash with an argument name).

ksnoop can select specific arguments/return value rather
than tracing everything; for example:

$ ksnoop "ip_send_skb(skb)"

 ...will only trace the skb argument.  A single level of
reference is supported also, for example:

$ ksnoop "ip_send_skb(skb->sk)"

or

Simple predicates (==, !=, <, <=, >, >=) can also be specified;
for example, to show skbs where the length is > 255:

$ ksnoop "ip_rcv(skb->len > 0xff,skb)"
            TIME  CPU      PID FUNCTION/ARGS
  32461869484376    1     2955 ip_rcv(
                                   skb->len =
                                    (unsigned int)0x127,
                                   skb = *(0xffff89c99623a000)
                                    (struct sk_buff){
                                     (union){
                                      .sk = (struct sock *)0xffff89c880b37000,
                                      .ip_defrag_offset = (int)0x80b37000,
                                     },

We can also specify a combination of entry/return predicates;
when such a combination is specified, data on entry (assuming
it matches the predicate) is "stashed" for retrieval on return.
This allows us to ask questions like "show entry arguments for
function foo when it returned a non-zero value indicating error";

$ ksnoop "sock_sendmsg(skb, return != 0)"

Multiple functions can be specified also.

In addition, using "stack" (-s) mode, it is possible to specify that
a sequence of functions should be traced, but only if function
A calls function B (either directly or indirectly).  For example,
in specifying

$ ksnoop -s tcp_sendmsg __tcp_transmit_skb  ip_output

...we are saying we are only interested in tcp_sendmsg() function
calls that in turn issue calls to __tcp_transmit_skb(), and these
in turn eventually call ip_output(), and that we only want to
see their entry and return.  This mode is useful for investigating
behaviour with a specific stack signature, allowing us to see
function/argument information for specific call chains only.

Finally, module support is included too, provided module BTF is
present in /sys/kernel/btf :

$ ksnoop iwl_trans_send_cmd
            TIME  CPU      PID FUNCTION/ARGS
  80046971419383    3     1038 iwl_trans_send_cmd(
                                   trans = *(0xffff989564d20028)
                                    (struct iwl_trans){
                                     .ops = (struct iwl_trans_ops *)0xffffff
                                     .op_mode = (struct iwl_op_mode *)0xffff
                                     .trans_cfg = (struct iwl_cfg_trans_para

The goal pursued here is not to add another tracer to the world -
there are plenty of those - but rather to demonstrate feature usage
for deep data display in the hope that other tracing technologies
make use of this functionality.  In the meantime, having a simple
tracer like this plugs the gap and can be quite helpful for kernel
debugging.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>