- examples/[bitehist.py](examples/bitehist.py) examples/[bitehist.c](examples/bitehist.c): Block I/O size histogram. [Examples](examples/bitehist_example.txt).
- examples/[disksnoop.py](examples/disksnoop.py) examples/[disksnoop.c](examples/disksnoop.c): Trace block device I/O latency. [Examples](examples/disksnoop_example.txt).
- examples/[hello_world.py](examples/hello_world.py): Prints "Hello, World!" for new processes.
+- examples/[tcpv4connect](examples/tcpv4connect): Trace TCP IPv4 active connections. [Examples](examples/tcpv4connect_example.txt).
- examples/[trace_fields.py](examples/trace_fields.py): Simple example of printing fields from traced events.
- examples/[vfsreadlat.py](examples/vfsreadlat.py) examples/[vfsreadlat.c](examples/vfsreadlat.c): VFS read latency distribution. [Examples](examples/vfsreadlat_example.txt).
- tools/[pidpersec](tools/pidpersec): Count new processes (via fork). [Examples](tools/pidpersec_example.txt).
- tools/[syncsnoop](tools/syncsnoop): Trace sync() syscall. [Examples](tools/syncsnoop_example.txt).
- tools/[tcpaccept](tools/tcpaccept): Trace TCP passive connections (accept()). [Examples](tools/tcpaccept_example.txt).
-- tools/[tcpv4connect](tools/tcpv4connect): Trace TCP IPv4 active connections (connect()). [Examples](tools/tcpv4connect_example.txt).
+- tools/[tcpconnect](tools/tcpconnect): Trace TCP active connections (connect()). [Examples](tools/tcpconnect_example.txt).
- tools/[vfscount](tools/vfscount) tools/[vfscount.c](tools/vfscount.c): Count VFS calls. [Examples](tools/vfscount_example.txt).
- tools/[vfsstat](tools/vfsstat) tools/[vfsstat.c](tools/vfsstat.c): Count some VFS calls, with column output. [Examples](tools/vfsstat_example.txt).
#
# USAGE: tcpv4connect [-h] [-t] [-p PID]
#
+# This is provided as a basic example of TCP connection & socket tracing.
+#
+# All IPv4 connection attempts are traced, even if they ultimately fail.
+#
# Copyright (c) 2015 Brendan Gregg.
# Licensed under the Apache License, Version 2.0 (the "License")
#
-# 25-Sep-2015 Brendan Gregg Created this.
+# 15-Oct-2015 Brendan Gregg Created this.
from __future__ import print_function
from bcc import BPF
-import argparse
-
-# arguments
-examples = """examples:
- ./tcpv4connect # trace all TCP IPv4 connect()s
- ./tcpv4connect -t # include timestamps
- ./tcpv4connect -p 181 # only trace PID 181
-"""
-parser = argparse.ArgumentParser(
- description="Trace TCP IPv4 connects",
- formatter_class=argparse.RawDescriptionHelpFormatter,
- epilog=examples)
-parser.add_argument("-t", "--timestamp", action="store_true",
- help="include timestamp on output")
-parser.add_argument("-p", "--pid",
- help="trace this PID only")
-args = parser.parse_args()
-debug = 0
# define BPF program
bpf_text = """
int kprobe__tcp_v4_connect(struct pt_regs *ctx, struct sock *sk)
{
u32 pid = bpf_get_current_pid_tgid();
- FILTER
// stash the sock ptr for lookup on return
currsock.update(&pid, &sk);
}
if (ret != 0) {
- // failed to send SYNC packet, socket __sk_common.{skc_rcv_saddr, ...}
- // may not be populated properly.
+ // failed to send SYNC packet, may not have populated
+ // socket __sk_common.{skc_rcv_saddr, ...}
currsock.delete(&pid);
return 0;
}
}
"""
-# code substitutions
-if args.pid:
- bpf_text = bpf_text.replace('FILTER',
- 'if (pid != %s) { return 0; }' % args.pid)
-else:
- bpf_text = bpf_text.replace('FILTER', '')
-if debug:
- print(bpf_text)
-
# initialize BPF
b = BPF(text=bpf_text)
# header
-if args.timestamp:
- print("%-9s" % ("TIME(s)"), end="")
print("%-6s %-12s %-16s %-16s %-4s" % ("PID", "COMM", "SADDR", "DADDR",
"DPORT"))
-start_ts = 0
-
def inet_ntoa(addr):
dq = ''
for i in range(0, 4):
(task, pid, cpu, flags, ts, msg) = b.trace_fields()
(saddr_hs, daddr_hs, dport_s) = msg.split(" ")
- if args.timestamp:
- if start_ts == 0:
- start_ts = ts
- print("%-9.3f" % (ts - start_ts), end="")
print("%-6d %-12.12s %-16s %-16s %-4s" % (pid, task,
inet_ntoa(int(saddr_hs, 16)),
inet_ntoa(int(daddr_hs, 16)),
--- /dev/null
+Demonstrations of tcpv4connect, the Linux eBPF/bcc version.
+
+
+This example traces the kernel function performing active TCP IPv4 connections
+(eg, via a connect() syscall; accept() are passive connections). Some example
+output (IP addresses changed to protect the innocent):
+
+# ./tcpv4connect
+PID COMM SADDR DADDR DPORT
+1479 telnet 127.0.0.1 127.0.0.1 23
+1469 curl 10.201.219.236 54.245.105.25 80
+1469 curl 10.201.219.236 54.67.101.145 80
+
+This output shows three connections, one from a "telnet" process and two from
+"curl". The output details shows the source address, destination address,
+and destination port. This traces attempted connections: these may have failed.
+
+The overhead of this tool should be negligible, since it is only tracing the
+kernel function performing a connect. It is not tracing every packet and then
+filtering.
+
+This is provided as a basic example of TCP tracing. See tools/tcpconnect for a
+more featured version of this example (a tool).
--- /dev/null
+.TH tcpconnect 8 "2015-08-25" "USER COMMANDS"
+.SH NAME
+tcpconnect \- Trace TCP active connections (connect()). Uses Linux eBPF/bcc.
+.SH SYNOPSIS
+.B tcpconnect [\-h] [\-t] [\-x] [\-p PID]
+.SH DESCRIPTION
+This tool traces active TCP connections (eg, via a connect() syscall;
+accept() are passive connections). This can be useful for general
+troubleshooting to see what connections are initiated by the local server.
+
+All connection attempts are traced, even if they ultimately fail.
+
+This works by tracing the kernel tcp_v4_connect() and tcp_v6_connect() functions
+using dynamic tracing, and will need updating to match any changes to these
+functions.
+
+Since this uses BPF, only the root user can use this tool.
+.SH REQUIREMENTS
+CONFIG_BPF and bcc.
+.SH OPTIONS
+.TP
+\-h
+Print usage message.
+.TP
+\-t
+Include a timestamp column.
+.TP
+\-p PID
+Trace this process ID only (filtered in-kernel).
+.SH EXAMPLES
+.TP
+Trace all active TCP connections:
+#
+.B tcpconnect
+.TP
+Trace all TCP connects, and include timestamps:
+#
+.B tcpconnect \-t
+.TP
+Trace PID 181 only:
+#
+.B tcpconnect \-p 181
+.SH FIELDS
+.TP
+TIME(s)
+Time of the call, in seconds.
+.TP
+PID
+Process ID
+.TP
+COMM
+Process name
+.TP
+IP
+IP address family (4 or 6)
+.TP
+SADDR
+Source IP address. IPv4 as a dotted quad, IPv6 shows "..." then the last 4
+bytes (check for newer versions of this tool for the full address).
+.TP
+DADDR
+Destination IP address. IPv4 as a dotted quad, IPv6 shows "..." then the last 4
+bytes (check for newer versions of this tool for the full address).
+.TP
+DPORT
+Destination port
+.SH OVERHEAD
+This traces the kernel tcp_v[46]_connect functions and prints output for each
+event. As the rate of this is generally expected to be low (< 1000/s), the
+overhead is also expected to be negligible. If you have an application that
+is calling a high rate of connects()s, such as a proxy server, then test and
+understand this overhead before use.
+.SH SOURCE
+This is from bcc.
+.IP
+https://github.com/iovisor/bcc
+.PP
+Also look in the bcc distribution for a companion _examples.txt file containing
+example usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+tcpaccept(8), funccount(8), tcpdump(8)
+++ /dev/null
-.TH tcpv4connect 8 "2015-08-25" "USER COMMANDS"
-.SH NAME
-tcpv4connect \- Trace TCP IPv4 active connections (connect()). Uses Linux eBPF/bcc.
-.SH SYNOPSIS
-.B tcpv4connect [\-h] [\-t] [\-x] [\-p PID]
-.SH DESCRIPTION
-This tool traces active TCP IPv4 connections (eg, via a connect() syscall;
-accept() are passive connections). This can be useful for general
-troubleshooting to see what connections are initiated by the local server.
-
-This works by tracing the kernel tcp_v4_connect() function using dynamic
-tracing, and will need updating to match any changes to this function.
-
-Since this uses BPF, only the root user can use this tool.
-.SH REQUIREMENTS
-CONFIG_BPF and bcc.
-.SH OPTIONS
-.TP
-\-h
-Print usage message.
-.TP
-\-t
-Include a timestamp column.
-.TP
-\-p PID
-Trace this process ID only (filtered in-kernel).
-.SH EXAMPLES
-.TP
-Trace all active TCP IPv4 connections:
-#
-.B tcpv4connect
-.TP
-Trace all TCP IPv4 connects, and include timestamps:
-#
-.B tcpv4connect \-t
-.TP
-Trace PID 181 only:
-#
-.B tcpv4connect \-p 181
-.SH FIELDS
-.TP
-TIME(s)
-Time of the call, in seconds.
-.TP
-PID
-Process ID
-.TP
-COMM
-Process name
-.TP
-SADDR
-IPv4 source address, as a dotted quad
-.TP
-DADDR
-IPv4 destination address, as a dotted quad
-.TP
-DPORT
-Destination port
-.SH OVERHEAD
-This traces the kernel tcp_v4_connect function and prints output for each event.
-As the rate of this is generally expected to be low (< 1000/s), the overhead is
-also expected to be negligible. If you have an application that is calling a high
-rate of connects()s, such as a proxy server, then test and understand this
-overhead before use.
-.SH SOURCE
-This is from bcc.
-.IP
-https://github.com/iovisor/bcc
-.PP
-Also look in the bcc distribution for a companion _examples.txt file containing
-example usage, output, and commentary for this tool.
-.SH OS
-Linux
-.SH STABILITY
-Unstable - in development.
-.SH AUTHOR
-Brendan Gregg
-.SH SEE ALSO
-tcpaccept(8), funccount(8), tcpdump(8)
--- /dev/null
+#!/usr/bin/python
+#
+# tcpconnect Trace TCP connect()s.
+# For Linux, uses BCC, eBPF. Embedded C.
+#
+# USAGE: tcpconnect [-h] [-t] [-p PID]
+#
+# All connection attempts are traced, even if they ultimately fail.
+#
+# Copyright (c) 2015 Brendan Gregg.
+# Licensed under the Apache License, Version 2.0 (the "License")
+#
+# 25-Sep-2015 Brendan Gregg Created this.
+
+from __future__ import print_function
+from bcc import BPF
+import argparse
+
+# arguments
+examples = """examples:
+ ./tcpconnect # trace all TCP connect()s
+ ./tcpconnect -t # include timestamps
+ ./tcpconnect -p 181 # only trace PID 181
+"""
+parser = argparse.ArgumentParser(
+ description="Trace TCP connects",
+ formatter_class=argparse.RawDescriptionHelpFormatter,
+ epilog=examples)
+parser.add_argument("-t", "--timestamp", action="store_true",
+ help="include timestamp on output")
+parser.add_argument("-p", "--pid",
+ help="trace this PID only")
+args = parser.parse_args()
+debug = 0
+
+# define BPF program
+bpf_text = """
+#include <uapi/linux/ptrace.h>
+#include <net/sock.h>
+#include <bcc/proto.h>
+
+BPF_HASH(currsock, u32, struct sock *);
+
+int trace_connect_entry(struct pt_regs *ctx, struct sock *sk)
+{
+ u32 pid = bpf_get_current_pid_tgid();
+ FILTER
+
+ // stash the sock ptr for lookup on return
+ currsock.update(&pid, &sk);
+
+ return 0;
+};
+
+static int trace_connect_return(struct pt_regs *ctx, short ipver)
+{
+ int ret = ctx->ax;
+ u32 pid = bpf_get_current_pid_tgid();
+
+ struct sock **skpp;
+ skpp = currsock.lookup(&pid);
+ if (skpp == 0) {
+ return 0; // missed entry
+ }
+
+ if (ret != 0) {
+ // failed to send SYNC packet, may not have populated
+ // socket __sk_common.{skc_rcv_saddr, ...}
+ currsock.delete(&pid);
+ return 0;
+ }
+
+ // pull in details
+ struct sock *skp = *skpp;
+ u32 saddr = 0, daddr = 0;
+ u16 dport = 0;
+ bpf_probe_read(&dport, sizeof(dport), &skp->__sk_common.skc_dport);
+ if (ipver == 4) {
+ bpf_probe_read(&saddr, sizeof(saddr),
+ &skp->__sk_common.skc_rcv_saddr);
+ bpf_probe_read(&daddr, sizeof(daddr),
+ &skp->__sk_common.skc_daddr);
+
+ // output
+ bpf_trace_printk("4 %x %x %d\\n", saddr, daddr, ntohs(dport));
+ } else /* 6 */ {
+ // just grab the last 4 bytes for now
+ bpf_probe_read(&saddr, sizeof(saddr),
+ &skp->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr32[3]);
+ bpf_probe_read(&daddr, sizeof(daddr),
+ &skp->__sk_common.skc_v6_daddr.in6_u.u6_addr32[3]);
+
+ // output and flip byte order of addresses
+ bpf_trace_printk("6 %x %x %d\\n", bpf_ntohl(saddr),
+ bpf_ntohl(daddr), ntohs(dport));
+ }
+
+ currsock.delete(&pid);
+
+ return 0;
+}
+
+int trace_connect_v4_return(struct pt_regs *ctx)
+{
+ return trace_connect_return(ctx, 4);
+}
+
+int trace_connect_v6_return(struct pt_regs *ctx)
+{
+ return trace_connect_return(ctx, 6);
+}
+"""
+
+# code substitutions
+if args.pid:
+ bpf_text = bpf_text.replace('FILTER',
+ 'if (pid != %s) { return 0; }' % args.pid)
+else:
+ bpf_text = bpf_text.replace('FILTER', '')
+if debug:
+ print(bpf_text)
+
+# initialize BPF
+b = BPF(text=bpf_text)
+b.attach_kprobe(event="tcp_v4_connect", fn_name="trace_connect_entry")
+b.attach_kprobe(event="tcp_v6_connect", fn_name="trace_connect_entry")
+b.attach_kretprobe(event="tcp_v4_connect", fn_name="trace_connect_v4_return")
+b.attach_kretprobe(event="tcp_v6_connect", fn_name="trace_connect_v6_return")
+
+# header
+if args.timestamp:
+ print("%-9s" % ("TIME(s)"), end="")
+print("%-6s %-12s %-2s %-16s %-16s %-4s" % ("PID", "COMM", "IP", "SADDR",
+ "DADDR", "DPORT"))
+
+start_ts = 0
+
+def inet_ntoa(addr):
+ dq = ''
+ for i in range(0, 4):
+ dq = dq + str(addr & 0xff)
+ if (i != 3):
+ dq = dq + '.'
+ addr = addr >> 8
+ return dq
+
+# format output
+while 1:
+ (task, pid, cpu, flags, ts, msg) = b.trace_fields()
+ (ip_s, saddr_hs, daddr_hs, dport_s) = msg.split(" ")
+
+ if args.timestamp:
+ if start_ts == 0:
+ start_ts = ts
+ print("%-9.3f" % (ts - start_ts), end="")
+ print("%-6d %-12.12s %-2s %-16s %-16s %-4s" % (pid, task, ip_s,
+ inet_ntoa(int(saddr_hs, 16)) if ip_s == "4" else "..." + saddr_hs,
+ inet_ntoa(int(daddr_hs, 16)) if ip_s == "4" else "..." + daddr_hs,
+ dport_s))
--- /dev/null
+Demonstrations of tcpconnect, the Linux eBPF/bcc version.
+
+
+This tool traces the kernel function performing active TCP connections
+(eg, via a connect() syscall; accept() are passive connections). Some example
+output (IP addresses changed to protect the innocent):
+
+# ./tcpconnect
+PID COMM IP SADDR DADDR DPORT
+1479 telnet 4 127.0.0.1 127.0.0.1 23
+1469 curl 4 10.201.219.236 54.245.105.25 80
+1469 curl 4 10.201.219.236 54.67.101.145 80
+11072 ssh 6 ...fe8203ac ...fe82abcd 22
+
+This output shows four connections, one from a "telnet" process, two from
+"curl", and one from "ssh". The output details shows the IP version, source
+address, destination address, and destination port. This traces attempted
+connections: these may have failed.
+
+IPv4 addresses are printed as dotted quads. Only the last 4 bytes of IPv6
+addresses are printed for now (check for updated versions of this tool).
+
+The overhead of this tool should be negligible, since it is only tracing the
+kernel functions performing connect. It is not tracing every packet and then
+filtering.
+
+
+The -t option prints a timestamp column:
+
+# ./tcpconnect -t
+TIME(s) PID COMM IP SADDR DADDR DPORT
+31.871 2482 local_agent 4 10.103.219.236 10.251.148.38 7001
+31.874 2482 local_agent 4 10.103.219.236 10.101.3.132 7001
+31.878 2482 local_agent 4 10.103.219.236 10.171.133.98 7101
+90.917 2482 local_agent 4 10.103.219.236 10.251.148.38 7001
+90.928 2482 local_agent 4 10.103.219.236 10.102.64.230 7001
+90.938 2482 local_agent 4 10.103.219.236 10.115.167.169 7101
+
+The output shows some periodic connections (or attempts) from a "local_agent"
+process to various other addresses. A few connections occur every minute.
+
+
+USAGE message:
+
+# ./tcpconnect -h
+usage: tcpconnect [-h] [-t] [-p PID]
+
+Trace TCP connects
+
+optional arguments:
+ -h, --help show this help message and exit
+ -t, --timestamp include timestamp on output
+ -p PID, --pid PID trace this PID only
+
+examples:
+ ./tcpconnect # trace all TCP connect()s
+ ./tcpconnect -t # include timestamps
+ ./tcpconnect -p 181 # only trace PID 181
+++ /dev/null
-Demonstrations of tcpv4connect, the Linux eBPF/bcc version.
-
-
-This tool traces the kernel function performing active TCP IPv4 connections
-(eg, via a connect() syscall; accept() are passive connections). Some example
-output (IP addresses changed to protect the innocent):
-
-# ./tcpv4connect
-PID COMM SADDR DADDR DPORT
-1479 telnet 127.0.0.1 127.0.0.1 23
-1469 curl 10.201.219.236 54.245.105.25 80
-1469 curl 10.201.219.236 54.67.101.145 80
-
-This output shows three connections, one from a "telnet" process and two from
-"curl". The output details shows the source address, destination address,
-and destination port. This traces attempted connections: these may have failed.
-
-The overhead of this tool should be negligible, since it is only tracing the
-kernel function performing a connect. It is not tracing every packet and then
-filtering.
-
-
-The -t option prints a timestamp column:
-
-# ./tcpv4connect -t
-TIME(s) PID COMM SADDR DADDR DPORT
-31.871 2482 local_agent 10.103.219.236 10.251.148.38 7001
-31.874 2482 local_agent 10.103.219.236 10.101.3.132 7001
-31.878 2482 local_agent 10.103.219.236 10.171.133.98 7101
-90.917 2482 local_agent 10.103.219.236 10.251.148.38 7001
-90.928 2482 local_agent 10.103.219.236 10.102.64.230 7001
-90.938 2482 local_agent 10.103.219.236 10.115.167.169 7101
-
-The output shows some periodic connections (or attempts) from a "local_agent"
-process to various other addresses. A few connections occur every minute.
-
-
-USAGE message:
-
-# ./tcpv4connect -h
-usage: tcpv4connect [-h] [-t] [-p PID]
-
-Trace TCP IPv4 connects
-
-optional arguments:
- -h, --help show this help message and exit
- -t, --timestamp include timestamp on output
- -p PID, --pid PID trace this PID only
-
-examples:
- ./tcpv4connect # trace all TCP IPv4 connect()s
- ./tcpv4connect -t # include timestamps
- ./tcpv4connect -p 181 # only trace PID 181