--- /dev/null
+.TH gentrace 8 "2016-02-11" "USER COMMANDS"
+.SH NAME
+gentrace \- Trace a function and display a histogram or summary of its parameter values. Uses Linux eBPF/bcc.
+.SH SYNOPSIS
+.B gentrace [-h] [-p PID] [-z STRING_SIZE] [-i INTERVAL] [-c COUNT] specifier [specifier ...]
+.SH DESCRIPTION
+gentrace attaches to function entry and exit points, collects specified parameter
+values, and stores them in a histogram or a raw counting collection that counts
+the number of times a parameter value occurred. It can also filter parameter
+values and instrument multiple entry points at once.
+
+This currently only works on x86_64. Check for future versions.
+.SH REQUIREMENTS
+CONFIG_BPF and bcc.
+.SH OPTIONS
+.TP
+\-h
+Print usage message.
+.TP
+\-p PID
+Trace only functions in the process PID. This filter will only apply to user-space
+functions. If you only provide kernel-space probes, the filter is ignored.
+.TP
+\-z STRING_SIZE
+When collecting string arguments (of type char*), collect up to STRING_SIZE
+characters. Longer strings will be truncated.
+.TP
+-i INTERVAL
+Print the collected data every INTERVAL seconds. The default is 1 second.
+.TP
+-c COUNT
+Print the collected data COUNT times and then exit.
+.TP
+SPECIFIER
+One or more probe specifications that instruct gentrace which functions to
+probe, which parameters to collect, how to aggregate them, and whether to perform
+any filtering. See SPECIFIER SYNTAX below.
+.SH SPECIFIER SYNTAX
+The general specifier syntax is as follows:
+
+.B <raw|hist>[-ret]:[library]:function(signature)[:type:expr[:filter]]
+.TP
+Probe type \- "raw", "hist", "raw-ret", "hist-ret".
+Indicates where to place the probe and whether the probe should collect raw
+event information, or aggregate the collected values into a histogram. Raw
+probes will collect the number of times every parameter value was observed,
+whereas histogram probes will collect the parameter values into a histogram.
+Only integral types can be used with histogram probes; there is no such limitation
+for raw probes. Suffix with \-ret to indicate that the probe should be placed
+at function return. This probe can only use the pseudo-variable @retval, which
+is an alias for the function's return value.
+.TP
+Library containing the probe.
+Specify the full path to the .so or executable file where the function to probe
+resides. Alternatively, you can specify just the lib name: for example, "c"
+refers to libc. If no library name is specified, the kernel is assumed.
+.TP
+The function to probe, and its signature.
+The function name must match exactly for the probe to be placed. The signature,
+on the other hand, is only required if you plan to collect parameter values
+based on that signature. For example, if you only want to collect the first
+parameter, you don't have to specify the rest of the parameters in the signature.
+.TP
+The type of the expression to capture.
+This is the type of the keys in the histogram or raw event collection that are
+collected by the probes.
+.TP
+The expression to capture.
+These are the values that are assigned to the histogram or raw event collection.
+You may use the parameters directly, or valid C expressions that involve the
+parameters, such as "size % 10".
+.TP
+A filter applied to the captured data.
+Only parameter values that pass the filter will be collected. This is any valid
+C expression that refers to the parameter values, such as "fd == 1 && length > 16".
+.SH EXAMPLES
+.TP
+Print a histogram of allocation sizes passed to kmalloc:
+#
+.B gentrace.py 'hist::__kmalloc(u64 size):u64:size'
+.TP
+Print a raw count of how many times process 1005 called malloc with an allocation size of 16 bytes:
+#
+.B gentrace.py -p 1005 'raw:c:malloc(size_t size):size_t:size:size==16'
+.TP
+Snoop on all strings returned by gets():
+#
+.B gentrace.py 'raw-ret:c:gets():char*:@retval'
+.TP
+Print raw counts of how many times writes were issued to a particular file descriptor number, in process 1005:
+#
+.B gentrace.py -p 1005 'raw:c:write(int fd):int:fd'
+.TP
+Print a histogram of error codes returned by read() in process 1005:
+#
+.B gentrace.py -p 1005 'hist-ret:c:read()'
+.TP
+Print a histogram of buffer sizes passed to write() across all processes, where the file descriptor was 1 (STDOUT):
+#
+.B gentrace.py 'hist:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1'
+.TP
+Count fork() calls in libc across all processes:
+#
+.B gentrace.py 'raw:c:fork'
+.TP
+Print histograms of sleep() and nanosleep() parameter values:
+#
+.B gentrace.py 'hist:c:sleep(u32 seconds):u32:seconds' 'hist:c:nanosleep(struct timespec { time_t tv_sec; long tv_nsec; } *req):long:req->tv_nsec'
+.TP
+Spy on writes to STDOUT performed by process 2780, up to a string size of 120 characters:
+#
+.B gentrace.py -p 2780 -z 120 'raw:c:write(int fd, char* buf, size_t len):char*:buf:fd==1'
+.SH SOURCE
+This is from bcc.
+.IP
+https://github.com/iovisor/bcc
+.PP
+Also look in the bcc distribution for a companion _examples.txt file containing
+example usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Sasha Goldshtein
# gentrace.py Trace a function and display a histogram or summary of its
# parameter values.
#
-# USAGE: gentrace.py [-h] [-p PID] [-z STRING_SIZE]
-# [-s SPECIFIER [SPECIFIER ...]]
-# [interval] [count]
+# USAGE: gentrace.py [-h] [-p PID] [-z STRING_SIZE] [-i INTERVAL]
+# [-c COUNT] specifier [specifier ...]
#
# Copyright (C) 2016 Sasha Goldshtein.
}
"""
- # <raw|hist>:lib:function(signature)[:type:expr[:filter]]
def __init__(self, specifier, pid):
self.raw_spec = specifier
parts = specifier.strip().split(':')
EXAMPLES:
-gentrace.py -s "hist::__kmalloc(u64 size):u64:size"
+gentrace.py "hist::__kmalloc(u64 size):u64:size"
Print a histogram of allocation sizes passed to kmalloc
-gentrace.py -p 1005 -s "raw:c:malloc(size_t size):size_t:size:size==16"
+gentrace.py -p 1005 "raw:c:malloc(size_t size):size_t:size:size==16"
Print a raw count of how many times process 1005 called malloc with
an allocation size of 16 bytes
-gentrace.py -s "raw-ret:c:gets():char*:@retval"
+gentrace.py "raw-ret:c:gets():char*:@retval"
Snoop on all strings returned by gets()
-gentrace.py -p 1005 -s "raw:c:write(int fd):int:fd"
+gentrace.py -p 1005 "raw:c:write(int fd):int:fd"
Print raw counts of how many times writes were issued to a particular
file descriptor number, in process 1005
-gentrace.py -p 1005 -s "hist-ret:c:read()"
+gentrace.py -p 1005 "hist-ret:c:read()"
Print a histogram of error codes returned by read() in process 1005
-gentrace.py -s "hist:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1"
+gentrace.py "hist:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1"
Print a histogram of buffer sizes passed to write() across all
processes, where the file descriptor was 1 (STDOUT)
-gentrace.py -s "raw:c:fork"
+gentrace.py "raw:c:fork"
Count fork() calls in libc across all processes
Can also use funccount.py, which is easier and more flexible
-gentrace.py -s \\
+gentrace.py \\
"hist:c:sleep(u32 seconds):u32:seconds" \\
"hist:c:nanosleep(struct timespec { time_t tv_sec; long tv_nsec; } *req):long:req->tv_nsec"
Print histograms of sleep() and nanosleep() parameter values
-gentrace.py -p 2780 -s -z 120 "raw:c:write(int fd, char* buf, size_t len):char*:buf:fd==1"
+gentrace.py -p 2780 -z 120 "raw:c:write(int fd, char* buf, size_t len):char*:buf:fd==1"
Spy on writes to STDOUT performed by process 2780, up to a string size
of 120 characters
"""
help="id of the process to trace (optional)")
parser.add_argument("-z", "--string-size", default=80, type=int,
help="maximum string size to read from char* arguments")
-parser.add_argument("interval", nargs="?", default=1, type=int,
+parser.add_argument("-i", "--interval", default=1, type=int,
help="output interval, in seconds")
-parser.add_argument("count", nargs="?", type=int,
+parser.add_argument("-c", "--count", type=int,
help="number of outputs")
-parser.add_argument("-s", "--specifier", nargs="+", dest="specifiers",
+parser.add_argument("specifier", nargs="+",
help="the probe specifiers (see examples below)")
args = parser.parse_args()
specifiers = []
-for specifier in args.specifiers:
+for specifier in args.specifier:
specifiers.append(Specifier(specifier, args.pid))
bpf_source = "#include <uapi/linux/ptrace.h>\n"
For example, suppose you want to find what allocation sizes are common in
your application:
-# ./gentrace.py -p 2420 -s "raw:c:malloc(size_t size):size_t:size"
+# ./gentrace.py -p 2420 "raw:c:malloc(size_t size):size_t:size"
[01:42:29]
raw:c:malloc(size_t size):size_t:size
COUNT EVENT
Now, suppose you wanted a histogram of buffer sizes passed to the write()
function across the system:
-# ./gentrace.py -s "hist:c:write(int fd, void *buf, size_t len):size_t:len"
+# ./gentrace.py "hist:c:write(int fd, void *buf, size_t len):size_t:len"
[01:45:22]
hist:c:write(int fd, void *buf, size_t len):size_t:len
len : count distribution
But these are writes across the board -- what if you wanted to focus on writes
to STDOUT?
-# ./gentrace.py -s "hist:c:write(int fd, void *buf, size_t len):size_t:len:fd==1"
+# ./gentrace.py "hist:c:write(int fd, void *buf, size_t len):size_t:len:fd==1"
[01:47:17]
hist:c:write(int fd, void *buf, size_t len):size_t:len:fd==1
len : count distribution
wanted a histogram of kernel allocation (kmalloc) sizes across the system,
printed twice with 3 second intervals:
-# ./gentrace.py 3 2 -s "hist::__kmalloc(size_t size):size_t:size"
+# ./gentrace.py -i 3 -c 2 "hist::__kmalloc(size_t size):size_t:size"
[01:50:00]
hist::__kmalloc(size_t size):size_t:size
size : count distribution
Occasionally, numeric information isn't enough and you want to capture strings.
What are the strings printed by puts() across the system?
-# ./gentrace.py 10 1 -s "raw:c:puts(char *str):char*:str"
+# ./gentrace.py -i 10 -c 1 "raw:c:puts(char *str):char*:str"
[01:53:54]
raw:c:puts(char *str):char*:str
COUNT EVENT
What about reads? You could trace gets() across the system and print the
strings input by the user:
-# ./gentrace.py 10 1 -s "raw-ret:c:gets():char*:@retval"
+# ./gentrace.py -i 10 -c 1 "raw-ret:c:gets():char*:@retval"
[02:12:23]
raw-ret:c:gets():char*:@retval
COUNT EVENT
Similarly, we could get a histogram of the error codes returned by read():
-# ./gentrace.py 10 1 -s "hist-ret:c:read()"
+# ./gentrace.py -i 10 -c 1 "hist-ret:c:read()"
[02:15:36]
hist-ret:c:read()
(int)ctx->ax : count distribution
USAGE message:
# ./gentrace.py -h
-usage: gentrace.py [-h] [-p PID] [-z STRING_SIZE]
- [-s SPECIFIERS [SPECIFIERS ...]]
- [interval] [count]
+usage: gentrace.py [-h] [-p PID] [-z STRING_SIZE] [-i INTERVAL] [-c COUNT]
+ specifier [specifier ...]
Trace a function and display a summary of its parameter values.
positional arguments:
- interval output interval, in seconds
- count number of outputs
+ specifier the probe specifiers (see examples below)
optional arguments:
-h, --help show this help message and exit
-p PID, --pid PID id of the process to trace (optional)
-z STRING_SIZE, --string-size STRING_SIZE
maximum string size to read from char* arguments
- -s SPECIFIERS [SPECIFIERS ...], --specifier SPECIFIERS [SPECIFIERS ...]
- the probe specifiers (see examples below)
+ -i INTERVAL, --interval INTERVAL
+ output interval, in seconds
+ -c COUNT, --count COUNT
+ number of outputs
Probe specifier syntax:
<raw|hist>[-ret]:[library]:function(signature)[:type:expr[:filter]]
EXAMPLES:
-gentrace.py -s "hist::__kmalloc(u64 size):u64:size"
+gentrace.py "hist::__kmalloc(u64 size):u64:size"
Print a histogram of allocation sizes passed to kmalloc
-gentrace.py -p 1005 -s "raw:c:malloc(size_t size):size_t:size:size==16"
+gentrace.py -p 1005 "raw:c:malloc(size_t size):size_t:size:size==16"
Print a raw count of how many times process 1005 called malloc with
an allocation size of 16 bytes
-gentrace.py -s "raw-ret:c:gets():char*:@retval"
+gentrace.py "raw-ret:c:gets():char*:@retval"
Snoop on all strings returned by gets()
-gentrace.py -p 1005 -s "raw:c:write(int fd):int:fd"
+gentrace.py -p 1005 "raw:c:write(int fd):int:fd"
Print raw counts of how many times writes were issued to a particular
file descriptor number, in process 1005
-gentrace.py -p 1005 -s "hist-ret:c:read()"
+gentrace.py -p 1005 "hist-ret:c:read()"
Print a histogram of error codes returned by read() in process 1005
-gentrace.py -s "hist:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1"
+gentrace.py "hist:c:write(int fd, const void *buf, size_t count):size_t:count:fd==1"
Print a histogram of buffer sizes passed to write() across all
processes, where the file descriptor was 1 (STDOUT)
-gentrace.py -s "raw:c:fork"
+gentrace.py "raw:c:fork"
Count fork() calls in libc across all processes
Can also use funccount.py, which is easier and more flexible
-gentrace.py -s \
+gentrace.py \
"hist:c:sleep(u32 seconds):u32:seconds" \
"hist:c:nanosleep(struct timespec { time_t tv_sec; long tv_nsec; } *req):long:req->tv_nsec"
Print histograms of sleep() and nanosleep() parameter values
-gentrace.py -p 2780 -s -z 120 "raw:c:write(int fd, char* buf, size_t len):char*:buf:fd==1"
+gentrace.py -p 2780 -z 120 "raw:c:write(int fd, char* buf, size_t len):char*:buf:fd==1"
Spy on writes to STDOUT performed by process 2780, up to a string size
of 120 characters