platform/upstream/bcc.git
8 years ago[cachetop] add example and man page.
chantra [Mon, 18 Jul 2016 22:17:45 +0000 (00:17 +0200)]
[cachetop] add example and man page.

make interval a positional parameter.

8 years ago[cachetop] top-like cachestat
Emmanuel Bretelle [Thu, 14 Jul 2016 20:04:57 +0000 (13:04 -0700)]
[cachetop] top-like cachestat

Alike cachestat.py but providing cache stats at the process level.

8 years agoUSDT Python API and example (#624)
Brendan Gregg [Sat, 23 Jul 2016 00:11:51 +0000 (17:11 -0700)]
USDT Python API and example (#624)

* Python USDT API

Code from @vmg

* Basic USDT example

* retire procstat.py

* improve/fix USDT exceptions

8 years agoAdd profile: a CPU profiler (#620)
Brendan Gregg [Fri, 22 Jul 2016 01:13:24 +0000 (18:13 -0700)]
Add profile: a CPU profiler (#620)

* Add profile: a CPU profiler

* move Perf to common class

8 years agoMerge pull request #618 from palmtenor/master
Brendan Gregg [Tue, 19 Jul 2016 03:19:16 +0000 (20:19 -0700)]
Merge pull request #618 from palmtenor/master

Use errno symbols in offcputime.py

8 years agoUse errno symbol instead of hard-coded numbers in offcputime.py
Teng Qin [Mon, 18 Jul 2016 20:21:10 +0000 (13:21 -0700)]
Use errno symbol instead of hard-coded numbers in offcputime.py

8 years agoMerge pull request #614 from markdrayton/fix-auto-kprobe
Brenden Blanco [Mon, 18 Jul 2016 17:20:43 +0000 (10:20 -0700)]
Merge pull request #614 from markdrayton/fix-auto-kprobe

Fix probe detaching and auto-kprobes

8 years agoMove open_{kprobes,uprobes,tracepoints} into BPF object
Mark Drayton [Fri, 15 Jul 2016 22:55:22 +0000 (23:55 +0100)]
Move open_{kprobes,uprobes,tracepoints} into BPF object

* for #605, this diff moves probe storage from the BPF module to the BPF object,
  letting each instantiation clean up its own probes. A module-level counter
  for all open probes is provided for the quota check. It also adds a
  `cleanup()` function to force cleanup before the `atexit` handler runs.

* for #614, it removes the `len(open_kprobes) == 0` check that prevented more
  than one autoload probe working. It fixes the tests that this change breaks by
  calling the `cleanup()` function added.

8 years agoStore kprobes with string keys, fix num_open_kprobes
Mark Drayton [Fri, 15 Jul 2016 20:19:07 +0000 (21:19 +0100)]
Store kprobes with string keys, fix num_open_kprobes

Prior to this diff we used inconsistent types for keys in `open_kprobes`. The
results from the regex match (`attach_kprobe(event_re=..)`) and the automatic
`kprobe__` features were passed through `str.decode()`, yielding unicode keys,
but specific matches (i.e. from `attach_kprobe(event=..)`) were stored with
string keys passed down from the caller. Only probes under string keys were
released in `cleanup_kprobes`, leaving attached probes on program exit.

This diff makes all the keys regular strings. I erred on the side of using
regular strings over `str.decode()`ing them because a) this data isn't passed
outside of Python, b) it's more Python 3 compatible (there is no `.decode()` on
a regular string object in Python 3 so such a change would ultimately need
removing again).

I also cleaned up a few other things:

* removed the call to `awk` for getting probable functions

* removed the `isinstance` checks when cleaning uprobes/tracepoints -- we
  should only have string keys in these dicts

* made `num_open_kprobes` skip the perf_events buffers. People likely use this
  to check that the right number of probes have been placed so counting
  perf_events buffers doesn't make sense here

8 years agoMerge pull request #611 from cdown/failed_opens
Brendan Gregg [Thu, 14 Jul 2016 17:13:12 +0000 (10:13 -0700)]
Merge pull request #611 from cdown/failed_opens

killsnoop: s/failed opens/failed kill syscalls/

8 years agoMerge branch 'master' into failed_opens
Chris Down [Thu, 14 Jul 2016 08:08:48 +0000 (09:08 +0100)]
Merge branch 'master' into failed_opens

8 years agoMerge pull request #612 from markdrayton/fileslower
Brendan Gregg [Wed, 13 Jul 2016 19:05:24 +0000 (12:05 -0700)]
Merge pull request #612 from markdrayton/fileslower

fileslower: try probing vfs_write if __vfs_write is missing

8 years agoMerge branch 'master' into fileslower
Mark Drayton [Wed, 13 Jul 2016 19:00:35 +0000 (12:00 -0700)]
Merge branch 'master' into fileslower

8 years agofileslower: try probing vfs_write if __vfs_write is missing
Mark Drayton [Wed, 13 Jul 2016 17:24:56 +0000 (18:24 +0100)]
fileslower: try probing vfs_write if __vfs_write is missing

8 years agoMerge pull request #610 from bobrik/debian-dockerfile
Brenden Blanco [Wed, 13 Jul 2016 17:56:34 +0000 (10:56 -0700)]
Merge pull request #610 from bobrik/debian-dockerfile

Build debian packages in docker containers

8 years agokillsnoop: s/failed opens/failed kill syscalls/
Chris Down [Wed, 13 Jul 2016 14:18:35 +0000 (15:18 +0100)]
killsnoop: s/failed opens/failed kill syscalls/

8 years agoBuild debian packages in docker containers
Ivan Babrou [Tue, 12 Jul 2016 22:02:35 +0000 (23:02 +0100)]
Build debian packages in docker containers

8 years agouse new tracepoint support (#608)
Brendan Gregg [Tue, 12 Jul 2016 01:27:01 +0000 (18:27 -0700)]
use new tracepoint support (#608)

8 years agoMerge pull request #602 from goldshtn/auto-tp
Brenden Blanco [Mon, 11 Jul 2016 22:02:25 +0000 (15:02 -0700)]
Merge pull request #602 from goldshtn/auto-tp

Full tracepoint support in Clang front-end

8 years agoLink to Ubuntu Xenial binary section (#604)
Trent Schafer [Mon, 11 Jul 2016 22:02:03 +0000 (08:02 +1000)]
Link to Ubuntu Xenial binary section (#604)

8 years agocc: Use manual parsing instead of std::regex
Sasha Goldshtein [Sat, 9 Jul 2016 12:17:44 +0000 (05:17 -0700)]
cc: Use manual parsing instead of std::regex

Older versions of GCC don't support std::regex even though they support
most of C++11. To avoid breaking the build on older systems, such as
Ubuntu 14.04, use manual parsing instead of std::regex.

8 years agotests: Test new tracepoint support
Sasha Goldshtein [Fri, 8 Jul 2016 17:46:53 +0000 (10:46 -0700)]
tests: Test new tracepoint support

8 years agocc: Rewrite probe functions that refer to tracepoint structures
Sasha Goldshtein [Tue, 5 Jul 2016 16:34:56 +0000 (09:34 -0700)]
cc: Rewrite probe functions that refer to tracepoint structures

When a probe function refers to a tracepoint arguments structure,
such as `struct tracepoint__irq__irq_handler_entry`, add that structure
on-the-fly using a Clang frontend action that runs before any other
steps take place.

Typically, the user will create tracepoint probe functions using
the TRACEPOINT_PROBE macro, which avoids the need for specifying
the tracepoint category and event twice in the signature of the
probe function.

8 years agobcc: Auto-tracepoints similar to auto-kprobes
Sasha Goldshtein [Sun, 3 Jul 2016 08:33:19 +0000 (01:33 -0700)]
bcc: Auto-tracepoints similar to auto-kprobes

When a function in the BPF program starts with "tracepoint__", parse
the rest of the name as a tracepoint category and name and attach the
tracepoint automatically. For example:

```
int tracepoint__sched__sched_switch(...)
```

As a result, the sched:sched_switch tracepoint is enabled and the function
is attached to that tracepoint.

8 years agoFix test failure in test_libbcc (#603)
Brenden Blanco [Fri, 8 Jul 2016 23:21:38 +0000 (16:21 -0700)]
Fix test failure in test_libbcc (#603)

On some systems, was seeing a failure at tests/cc/test_c_api.cc:172 due
to failure to open the /tmp/perf-pid.map file. Looking through the code,
narrowed it down to an invalid use of c_str() on a temporary
std::string. Fix it by storing the string in a variable.

Signed-off-by: Brenden Blanco <bblanco@plumgrid.com>
8 years agoProcSyms: deduplicate symbol names (#598)
Mark Drayton [Fri, 8 Jul 2016 02:49:25 +0000 (03:49 +0100)]
ProcSyms: deduplicate symbol names (#598)

8 years agoFix for C++ api change in LLVM 3.9 (#600)
Brenden Blanco [Fri, 8 Jul 2016 00:56:03 +0000 (17:56 -0700)]
Fix for C++ api change in LLVM 3.9 (#600)

Upstream, params() was renamed to parameters(). In order to support both
old and new LLVM, use the unchanged param_begin and param_end API.

Signed-off-by: Brenden Blanco <bblanco@plumgrid.com>
8 years agoA tracepoint example (#596)
Brendan Gregg [Sat, 2 Jul 2016 01:38:30 +0000 (18:38 -0700)]
A tracepoint example (#596)

8 years agoProcSyms: fix off-by-ones, use binary search to resolve addresses (#594)
Mark Drayton [Fri, 1 Jul 2016 23:47:39 +0000 (00:47 +0100)]
ProcSyms: fix off-by-ones, use binary search to resolve addresses (#594)

* libbcc: fix off-by-one errors in resolving adjacent modules/symbols, add test

* libbcc: use binary search in ProcSyms::Module::find_addr()

8 years agobcc: Add ctypes declarations for new functions (#593)
Sasha Goldshtein [Fri, 1 Jul 2016 23:45:57 +0000 (02:45 +0300)]
bcc: Add ctypes declarations for new functions (#593)

Add ctypes declarations for `bpf_attach_tracepoint` and
`bpf_detach_tracepoint` in libbcc.py. It works anyway, but now
it's a bit safer.

8 years agoMerge pull request #586 from goldshtn/offcpudist
Brenden Blanco [Thu, 30 Jun 2016 22:38:11 +0000 (15:38 -0700)]
Merge pull request #586 from goldshtn/offcpudist

cpudist: Support off-cpu time reports

8 years agoMerge pull request #590 from goldshtn/bcc-tp-support
Brenden Blanco [Thu, 30 Jun 2016 18:59:50 +0000 (11:59 -0700)]
Merge pull request #590 from goldshtn/bcc-tp-support

bcc: Tracepoint support in libbpf and BPF

8 years agobcc: Add test for tracepoint support
Sasha Goldshtein [Thu, 30 Jun 2016 18:07:48 +0000 (11:07 -0700)]
bcc: Add test for tracepoint support

The test asserts that we can enable the sched_switch tracepoint and read
some events from it. The test is also marked to require kernel 4.7 or
later, because that's where the BPF support for tracepoints was introduced.

8 years agocpudist: Protect against potentially negative time deltas
Sasha Goldshtein [Thu, 30 Jun 2016 17:46:27 +0000 (10:46 -0700)]
cpudist: Protect against potentially negative time deltas

It seems from experimentation that the calculated timestamps between
on- and off-CPU switch events can produce incorrect results, with a
later event having a smaller timestamp. Discard events when the
resulting delta time would be negative.

8 years agocpudist: Use `finish_task_switch` kprobe instead of `sched_switch` tracepoint
Sasha Goldshtein [Thu, 30 Jun 2016 14:39:27 +0000 (07:39 -0700)]
cpudist: Use `finish_task_switch` kprobe instead of `sched_switch` tracepoint

The `sched_switch` tracepoint approach requires storing the previous
task's tgid in a map and fetching it from there, because it is not
available as a tracepoint argument. Instead, placing a kprobe on the
`finish_task_switch` function allows cleanly fetching the previous
task's pid and tgid from the task_struct.

8 years agocpudist: Fix extraneous filtering of descheduled tasks
Sasha Goldshtein [Thu, 30 Jun 2016 06:35:43 +0000 (23:35 -0700)]
cpudist: Fix extraneous filtering of descheduled tasks

When the `-O` switch was provided, cpudist was unnecessarily filtering
out scheduling events arising from a task waking up when the previous
task was not running. On an idle system, this happens a lot, and causes
events to be missed. This is now fixed.

8 years agocpudist: Attempt to resolve pid to command
Sasha Goldshtein [Wed, 29 Jun 2016 09:18:06 +0000 (02:18 -0700)]
cpudist: Attempt to resolve pid to command

Use `/proc/$PID/comm`, which may fail, for example if the original
process already exited. This may also produce misleading results
if another process got the same pid, but there's no way around this.

8 years agocpudist: Support off-cpu time reports
Sasha Goldshtein [Wed, 29 Jun 2016 08:48:08 +0000 (01:48 -0700)]
cpudist: Support off-cpu time reports

Add -O switch, which directs cpudist to collect off-CPU time
statistics. Also restructure the code slightly and added examples
as appropriate.

8 years agobcc: Tracepoint support in libbpf and BPF
Sasha Goldshtein [Thu, 30 Jun 2016 13:26:28 +0000 (06:26 -0700)]
bcc: Tracepoint support in libbpf and BPF

Introduce tracepoint support in libbpf via new `bpf_attach_tracepoint`
API, which takes the tracepoint category and name (e.g. "sched",
"sched_switch"). Attach the tracing program to the tracepoint's id
and proceed as usual.

Add `attach_tracepoint` API to Python BPF module, which takes the
tracepoint description as a single string (e.g. "sched:sched_switch").
Load the BPF program with bpf_prog_type set to TRACEPOINT and then
call `bpf_attach_tracepoint` to attach it.

8 years agotrace: Specifying a pid with a kernel probe now works (#589)
Sasha Goldshtein [Thu, 30 Jun 2016 09:16:39 +0000 (12:16 +0300)]
trace: Specifying a pid with a kernel probe now works (#589)

Due to an incorrectly referenced global variable, specifying a pid
to filter with a kernel probe produced an error. This is now fixed,
for example:

```
TIME     PID    COMM         FUNC
23:46:00 29967  bash         sched_switch
23:46:01 29967  bash         sched_switch
23:46:01 29967  bash         sched_switch
^C
```

8 years agocpudist: Summarize task on-CPU time as histograms (#585)
Sasha Goldshtein [Wed, 29 Jun 2016 00:57:01 +0000 (03:57 +0300)]
cpudist: Summarize task on-CPU time as histograms (#585)

* cpudist: summarize on-CPU time per task as a histogram

This is the initial implementation of the tool itself, which uses
the sched:sched_switch tracepoint to probe task switches. This is
a slightly more robust approach than using a kernel function, such
as finish_task_switch.

When native BCC support for tracepoints is introduced, this tool
can be reimplemented and reliance on the current Tracepoint module
can be removed.

* cpudist: add man page and examples

8 years agoMerge pull request #584 from ygrek/master
Brendan Gregg [Mon, 27 Jun 2016 20:11:18 +0000 (13:11 -0700)]
Merge pull request #584 from ygrek/master

tools: fix getting {ext4,btrfs}_file_operations address (ref #583)

8 years agotools: fix getting {ext4,btrfs}_file_operations address (ref #583)
ygrek [Mon, 27 Jun 2016 19:54:55 +0000 (12:54 -0700)]
tools: fix getting {ext4,btrfs}_file_operations address (ref #583)

8 years agoMerge pull request #583 from ygrek/master
Brendan Gregg [Mon, 27 Jun 2016 19:33:31 +0000 (12:33 -0700)]
Merge pull request #583 from ygrek/master

ext4slower: fix getting kallsyms address

8 years agoext4slower: fix getting kallsyms address
ygrek [Mon, 27 Jun 2016 18:07:47 +0000 (11:07 -0700)]
ext4slower: fix getting kallsyms address

$ grep ext4_file_operations /proc/kallsyms
ffffffffc0331340 r ext4_file_operations [ext4]

8 years agoIPv6 support for tcp* tools (#582)
Mark Drayton [Sun, 26 Jun 2016 20:14:44 +0000 (21:14 +0100)]
IPv6 support for tcp* tools (#582)

* tcpretrans: support full IPv6 addresses, fix --lossprobe

* tcpaccept: support full IPv6 addresses, fix timestamps

* tcpconnect: support full IPv6 addresses, fix timestamps

* tcpconnlat: support full IPv6 addresses, fix timestamps

8 years agoMerge pull request #581 from goldshtn/docs-fixes
Brenden Blanco [Sun, 26 Jun 2016 16:15:30 +0000 (09:15 -0700)]
Merge pull request #581 from goldshtn/docs-fixes

Update installation instructions for Fedora

8 years agoMerge branch 'master' into docs-fixes
Sasha Goldshtein [Sun, 26 Jun 2016 13:19:56 +0000 (16:19 +0300)]
Merge branch 'master' into docs-fixes

8 years agoUpdate installation instructions for Fedora
Sasha Goldshtein [Sun, 26 Jun 2016 13:15:39 +0000 (06:15 -0700)]
Update installation instructions for Fedora

clang 3.7.0 is known to be buggy, and we recommend that people use
3.7.1. Update install instructions to point to 3.7.1.

Compiling BCC on Fedora requires elfutils-libelf-devel. Add this
package to the install instructions for Fedora.

8 years agoMerge pull request #579 from bobrik/fix-ipv6-print
Brendan Gregg [Thu, 23 Jun 2016 18:05:29 +0000 (11:05 -0700)]
Merge pull request #579 from bobrik/fix-ipv6-print

Do not throw exception on ipv6 packets in tcpretrans

8 years agoDo not throw exception on ipv6 packets in tcpretrans
Ivan Babrou [Thu, 23 Jun 2016 17:11:25 +0000 (18:11 +0100)]
Do not throw exception on ipv6 packets in tcpretrans

8 years agokernel version guide (#577)
Brendan Gregg [Wed, 22 Jun 2016 00:12:38 +0000 (17:12 -0700)]
kernel version guide (#577)

* kernel version guide

* update

8 years agoMerge pull request #576 from markdrayton/perf-pid-map-fixes
4ast [Tue, 21 Jun 2016 15:51:11 +0000 (08:51 -0700)]
Merge pull request #576 from markdrayton/perf-pid-map-fixes

Fix nits for /tmp/perf-pid.map support

8 years agoFix nits for /tmp/perf-pid.map support
Mark Drayton [Tue, 21 Jun 2016 12:37:27 +0000 (13:37 +0100)]
Fix nits for /tmp/perf-pid.map support

8 years agoMerge pull request #573 from markdrayton/pid-map
4ast [Mon, 20 Jun 2016 16:47:30 +0000 (09:47 -0700)]
Merge pull request #573 from markdrayton/pid-map

Add support for reading symbols from /tmp/perf-pid.map

8 years agoMerge pull request #572 from iovisor/ast_dev
Brendan Gregg [Wed, 15 Jun 2016 18:50:24 +0000 (11:50 -0700)]
Merge pull request #572 from iovisor/ast_dev

fix install instruction to mention llvm >= 3.7.1

8 years agoMerge branch 'master' into ast_dev
Brendan Gregg [Wed, 15 Jun 2016 18:42:05 +0000 (11:42 -0700)]
Merge branch 'master' into ast_dev

8 years agofix install instruction to mention llvm >= 3.7.1
Alexei Starovoitov [Wed, 15 Jun 2016 16:39:28 +0000 (12:39 -0400)]
fix install instruction to mention llvm >= 3.7.1

Signed-off-by: Alexei Starovoitov <ast@fb.com>
8 years agoAdd support for reading symbols from /tmp/perf-pid.map
Mark Drayton [Wed, 15 Jun 2016 10:53:24 +0000 (11:53 +0100)]
Add support for reading symbols from /tmp/perf-pid.map

This adds basic support for /tmp/perf-pid.map. To cope with processes in
containers, it supports:

* mapping from BCC's PID namespace to the target process's PID namespace
  using /proc/pid/status
* resolving a target process's root filesystem using /proc/pid/root

8 years agoMerge pull request #571 from iovisor/fix_568
4ast [Wed, 15 Jun 2016 05:14:57 +0000 (22:14 -0700)]
Merge pull request #571 from iovisor/fix_568

Check for NULL result from bpf_attach_kprobe

8 years agoCheck for NULL result from bpf_attach_kprobe
Brenden Blanco [Wed, 15 Jun 2016 03:57:03 +0000 (20:57 -0700)]
Check for NULL result from bpf_attach_kprobe

The check for NULL in the return value from C functions was comparing to
None incorrectly, causing an error check to pass improperly. Simply
check the truthiness of the return instead seems to be more resilient.

Add a test as well.

Fixes: #568
Signed-off-by: Brenden Blanco <bblanco@plumgrid.com>
8 years agoMerge pull request #563 from qmonnet/debug-output
Brenden Blanco [Mon, 13 Jun 2016 11:59:42 +0000 (04:59 -0700)]
Merge pull request #563 from qmonnet/debug-output

libbpf.c: Prepend BPF syscall error to log_bug when debug is enabled.

8 years agopython: With DEBUG_BPF, print BPF error even if log_buf is not empty.
Quentin Monnet [Thu, 9 Jun 2016 12:55:29 +0000 (14:55 +0200)]
python: With DEBUG_BPF, print BPF error even if log_buf is not empty.

When BPF syscall fails and DEBUG_FLAG has been provided to a Python
caller script, the BPF error string would be printed only if log_buf had
not been filled by the kernel. This commit removes this requirement,
printing the error string even if log_buf is not empty.

Signed-off-by: Quentin Monnet <quentin.monnet@6wind.com>
8 years agopython: Print BPF syscall error if DEBUG_BPF is on but log_buf is empty.
Quentin Monnet [Thu, 9 Jun 2016 12:55:29 +0000 (14:55 +0200)]
python: Print BPF syscall error if DEBUG_BPF is on but log_buf is empty.

Commit 759029fea8066b41b54be5447137db95cb1313c4 provided an option to
store the output from BPF syscall into a buffer (and not to print it
systematically to standard output) on program load in libbpf.c.

But doing so, it only stores the content of attr.log_buf, while the
error string--resulting from a failed BPF syscall--is no more displayed
when the DEBUG_BPF flag is used in the Python script responsible for
converting and injecting the code.

This commit proposes a fix for this bug by printing the error message
(associated to the return value from the syscall) from the Python
caller, when all the following conditions are met:

- the syscall fails,
- the DEBUG_BPF flag has been provided, and
- log_buf is empty (has not been filled by kernel).

Note: when DEBUG_BPF is not provided, the error string is printed in the
C wrapper in libbpf.c (bpf_prog_load) anyway.

Fixes: 759029fe ("Add option for custom log string to bpf_prog_load")

Signed-off-by: Quentin Monnet <quentin.monnet@6wind.com>
8 years agoMerge pull request #562 from evverx/change-kernel-user-stacks-order
Brendan Gregg [Wed, 8 Jun 2016 18:44:49 +0000 (11:44 -0700)]
Merge pull request #562 from evverx/change-kernel-user-stacks-order

Change kernel/user stacks order

8 years agooffcputime: add the -d option
Evgeny Vereshchagin [Tue, 7 Jun 2016 20:33:54 +0000 (06:33 +1000)]
offcputime: add the -d option

Closes #559

8 years agooffcputime: change user/kernel stack order
Evgeny Vereshchagin [Tue, 7 Jun 2016 20:06:33 +0000 (06:06 +1000)]
offcputime: change user/kernel stack order

see #559

8 years agoMerge pull request #561 from evverx/folded-dont-miss-backtrace-part
Brendan Gregg [Tue, 7 Jun 2016 15:49:19 +0000 (08:49 -0700)]
Merge pull request #561 from evverx/folded-dont-miss-backtrace-part

offcputime -f: don't miss backtrace bounds

8 years agooffcputime -f: don't miss backtrace bounds
Evgeny Vereshchagin [Tue, 7 Jun 2016 08:00:01 +0000 (18:00 +1000)]
offcputime -f: don't miss backtrace bounds

See #559

8 years agoMerge pull request #560 from evverx/clarify-offcputime
Brendan Gregg [Mon, 6 Jun 2016 19:45:02 +0000 (12:45 -0700)]
Merge pull request #560 from evverx/clarify-offcputime

Update manpage for offcputime

8 years agoMerge pull request #554 from evverx/fix-p-option
Brendan Gregg [Mon, 6 Jun 2016 18:18:54 +0000 (11:18 -0700)]
Merge pull request #554 from evverx/fix-p-option

Fix tools[/old]/offcputime -p PID

8 years agoUpdate manpage for offcputime
Evgeny Vereshchagin [Mon, 6 Jun 2016 09:25:54 +0000 (09:25 +0000)]
Update manpage for offcputime

Closes #558

8 years agoFix tools[/old]/offcputime -p PID
Evgeny Vereshchagin [Fri, 27 May 2016 06:13:52 +0000 (06:13 +0000)]
Fix tools[/old]/offcputime -p PID

We should extract pid before THREAD_FILTER

8 years agoMerge pull request #555 from markdrayton/execsnoop
4ast [Fri, 3 Jun 2016 01:15:43 +0000 (18:15 -0700)]
Merge pull request #555 from markdrayton/execsnoop

execsnoop: use BPF_PERF_OUTPUT, add PPID

8 years agoexecsnoop: show PPID in output
Mark Drayton [Thu, 2 Jun 2016 09:53:34 +0000 (10:53 +0100)]
execsnoop: show PPID in output

8 years agoexecsnoop: use BPF_PERF_OUTPUT instead of trace pipe
Mark Drayton [Thu, 2 Jun 2016 09:53:20 +0000 (10:53 +0100)]
execsnoop: use BPF_PERF_OUTPUT instead of trace pipe

8 years agoMerge pull request #557 from ryanmiao/mypr
Brenden Blanco [Wed, 1 Jun 2016 01:21:10 +0000 (18:21 -0700)]
Merge pull request #557 from ryanmiao/mypr

set use_errno to True for libbcc.so

8 years agoMerge pull request #544 from abirchall/abirchall_release
Brenden Blanco [Wed, 1 Jun 2016 01:16:06 +0000 (18:16 -0700)]
Merge pull request #544 from abirchall/abirchall_release

Add user space stack traces to offcputime

8 years agoset use_errno to True for libbcc.so
Jincheng Miao [Sun, 29 May 2016 05:17:03 +0000 (13:17 +0800)]
set use_errno to True for libbcc.so

Currently, errno could not be accessed by libbcc.py, like:
```
binding socket to 'eth0'
cannot create raw socket: Operation not permitted
Traceback (most recent call last):
  File "/home/ryan/Software/bcc/examples/networking/http_filter/http-parse-simple.py", line 73, in <module>
    BPF.attach_raw_socket(function_http_filter, interface)
  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 330, in attach_raw_socket
    raise Exception("Failed to open raw device %s: %s" % (dev, errstr))
Exception: Failed to open raw device eth0: Success
```
The exception prints the errno as `Success`, instead of `EPERM`.

This patch would fix this problem.

Signed-off-by: Jincheng Miao <jincheng.miao@gmail.com>
8 years agoMerge pull request #552 from evverx/install-tools-old
Brenden Blanco [Fri, 27 May 2016 00:42:13 +0000 (17:42 -0700)]
Merge pull request #552 from evverx/install-tools-old

Install tools/old

8 years agoPoint people to tools/old
Evgeny Vereshchagin [Thu, 26 May 2016 09:11:44 +0000 (09:11 +0000)]
Point people to tools/old

8 years agoInstall tools/old too
Evgeny Vereshchagin [Thu, 26 May 2016 08:50:50 +0000 (08:50 +0000)]
Install tools/old too

8 years agoMerge pull request #551 from vmg/vmg/lua-para
4ast [Tue, 24 May 2016 20:30:39 +0000 (13:30 -0700)]
Merge pull request #551 from vmg/vmg/lua-para

lua: Allow LuaJIT parameterized types in `open_perf_buffer`

8 years agolua: Allow LuaJIT parameterized types in `open_perf_buffer`
Vicent Marti [Tue, 24 May 2016 15:57:34 +0000 (17:57 +0200)]
lua: Allow LuaJIT parameterized types in `open_perf_buffer`

8 years agoMerge pull request #550 from markdrayton/trace-fields
4ast [Tue, 24 May 2016 15:43:51 +0000 (08:43 -0700)]
Merge pull request #550 from markdrayton/trace-fields

Use BPF_PERF_OUTPUT for fileslower and stacksnoop

8 years agoUse BPF_PERF_OUTPUT for fileslower and stacksnoop
Mark Drayton [Tue, 24 May 2016 14:01:01 +0000 (07:01 -0700)]
Use BPF_PERF_OUTPUT for fileslower and stacksnoop

8 years agoUpdate documentation and man page for offcputime
Andrew Birchall [Tue, 24 May 2016 08:44:41 +0000 (01:44 -0700)]
Update documentation and man page for offcputime

8 years agoAdd user space stack traces to offcputime
Andrew Birchall [Thu, 5 May 2016 17:56:40 +0000 (10:56 -0700)]
Add user space stack traces to offcputime

Summary:
* generalize `KernelSymbolCache` into `SymbolCache` which supports user/kernel space symbols
* create `BPF.sym()` for resolving user/kernel space symbols
* `_ksym_cache` => `_sym_caches` and create `BPF._sym_cache()` to leverage the symbol caches
* update `tools/offcputime.py` to print user space stack traces

Test Plan:
```
dev[bcc](abirchall_next): ~/bcc_run_tool.sh offcputime -U -k -f 1
ERROR: Displaying user stacks for kernel threads doesn't make sense.
devbig680[bcc](abirchall_next): ~/bcc_run_tool.sh offcputime -K -f 1 | grep python2 | head -n 1
python2.7;system_call_fastpath;sys_futex;do_futex;futex_wait;futex_wait_queue_me;schedule 19
dev[bcc](abirchall_next): ~/bcc_run_tool.sh offcputime -U -f 1 | grep python2 | head -n 1
python2.7;clone;start_thread;t_bootstrap;PyEval_CallObjectWithKeywords;PyObject_Call;instancemethod_call;PyObject_Call;function_call;PyEval_EvalCodeEx;PyEval_EvalFrameEx;PyObject_Call;function_call;PyEval_EvalCodeEx;PyEval_EvalFrameEx;time_sleep;PyEval_RestoreThread 5
devbig680[bcc](abirchall_next): ~/bcc_run_tool.sh offcputime -f 1 | grep python2 | head -n 1
python2.7;system_call_fastpath;sys_futex;do_futex;futex_wait;futex_wait_queue_me;schedule;[unknown];__libc_start_main;Py_Main;RunModule;PyObject_Call;function_call;PyEval_EvalCodeEx;PyEval_EvalFrameEx;PyEval_EvalFrameEx;PyEval_EvalCodeEx;PyEval_EvalFrameEx;lock_PyThread_acquire_lock;PyEval_RestoreThread 39
```

8 years agoMerge pull request #535 from MQasimSarfraz/patch-1
Brenden Blanco [Mon, 23 May 2016 02:55:06 +0000 (19:55 -0700)]
Merge pull request #535 from MQasimSarfraz/patch-1

Updated monitor.c to take care of host byte order

8 years agoMerge pull request #546 from iamkafai/master
Brenden Blanco [Fri, 20 May 2016 17:31:30 +0000 (10:31 -0700)]
Merge pull request #546 from iamkafai/master

Fix a few fd leaks

8 years agoThis diff fixes a few fd leaks after reading files under
Martin KaFai Lau [Fri, 20 May 2016 06:04:41 +0000 (23:04 -0700)]
This diff fixes a few fd leaks after reading files under
/sys/kernel/debug/tracing.

8 years agoRemoving htons in tunnel.py for consistency.
Qasim Sarfraz [Thu, 19 May 2016 10:02:26 +0000 (15:02 +0500)]
Removing htons in tunnel.py for consistency.

8 years agoMerge pull request #543 from abirchall/abirchall_release
Brendan Gregg [Tue, 17 May 2016 20:41:33 +0000 (13:41 -0700)]
Merge pull request #543 from abirchall/abirchall_release

add --kernel-threads-only to tools/offcputime

8 years agoadd --kernel-threads-only to tools/offcputime
Andrew Birchall [Wed, 4 May 2016 23:03:34 +0000 (16:03 -0700)]
add --kernel-threads-only to tools/offcputime

Summary:
Adds `--kernel-threads-only` arg
The kernel-threads-only arg is exclusive with pid/user-threads-only via `parser.add_mutually_exclusive_group`.
The output message now indicates what we are tracing (pid/user threads/kernel threads/all threads).
Removed the --verbose arg (unused).

Test Plan:
Run with combinations of the args; validate output looks sane:

// test mutually exclusive group
```
devbig680[bcc](abirchall_dev): ~/bcc_run_tool.sh offcputime -k -u 1
[Running] /data/users/abirchall/bcc/tools/offcputime.py -k -u 1
usage: offcputime.py [-h] [-p PID | -k | -u] [-v] [-f] [duration]
offcputime.py: error: argument -u/--user-threads-only: not allowed with argument -k/--kernel-threads-only
```

// kernel threads only
```
devbig680[bcc](abirchall_dev): ~/bcc_run_tool.sh offcputime -f -k 1
[Running] /data/users/abirchall/bcc/tools/offcputime.py -f -k 1
swapper/21;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 11
swapper/16;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 19
swapper/22;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 20
swapper/31;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 20
swapper/23;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 67
swapper/25;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 74
...
```
`~/bcc_run_tool.sh offcputime -f --kernel-threads-only 1`

// user threads only
`~/bcc_run_tool.sh offcputime -f --user-threads-only 1`
`~/bcc_run_tool.sh offcputime -f -u 1`

// specific pid
`~/bcc_run_tool.sh offcputime -f -p $(pidof hphpi) 1`
```
devbig680[bcc](abirchall_dev): ~/bcc_run_tool.sh offcputime --pid $(pidof mcrouter) 10 | head
[Running] /data/users/abirchall/bcc/tools/offcputime.py --pid 95929 10
Tracing off-CPU time (us) of PID 95929 by kernel stack for 10 secs.
```

Note that this last case (specific PID) doesn't appear to be working; I can debug that in a follow up commit.

8 years agoMerge pull request #533 from abirchall/abirchall_dev
4ast [Tue, 17 May 2016 18:10:48 +0000 (11:10 -0700)]
Merge pull request #533 from abirchall/abirchall_dev

handle ENOMEM in tools/offcputime

8 years agomove ENOMEM comment; create --stack-storage-size arg; misc cleanup
Andrew Birchall [Thu, 12 May 2016 01:31:49 +0000 (18:31 -0700)]
move ENOMEM comment; create --stack-storage-size arg; misc cleanup

Summary:
* move ENOMEM comment from tools/offcputime.py to src/cc/export/helpers.h
* create --stack-storage-size arg to allow the user to set the stack storage size
    * requires a positive_nonzero_int type checking function for argparse
* clean up the rest of the args a bit
    * use parser.add_mutually_exclusive_group instead of manually handling the
      mutual exclusion
    * use positive_nonzero_int for duration arg
    * use positive_int for pid arg
* only print a warning about increasing the storage size if at least one of the
  get_stackid errors was a -ENOMEM
* remove the debug param (we can add this manually when testing)
* fix a bug where all processes are traced when specifying pid of 0

Test Plan:
```
devbig680[bcc](no branch, rebasing abirchall_dev): ~/bcc_run_tool.sh offcputime --help
[Running] /data/users/abirchall/bcc/tools/offcputime.py --help
usage: offcputime.py [-h] [-u | -p PID] [-v] [-f]
                     [--stack-storage-size STACK_STORAGE_SIZE]
                     [duration]

Summarize off-CPU time by kernel stack trace

positional arguments:
  duration              duration of trace, in seconds

optional arguments:
  -h, --help            show this help message and exit
  -u, --useronly        user threads only (no kernel threads)
  -p PID, --pid PID     trace this PID only
  -v, --verbose         show raw addresses
  -f, --folded          output folded format
  --stack-storage-size STACK_STORAGE_SIZE
                        the number of unique stack traces that can be stored
                        and displayed

examples:
    ./offcputime             # trace off-CPU stack time until Ctrl-C
    ./offcputime 5           # trace for 5 seconds only
    ./offcputime -f 5        # 5 seconds, and output in folded format
    ./offcputime -u          # don't include kernel threads (user only)
    ./offcputime -p 185      # trace fo PID 185 only
devbig680[bcc](no branch, rebasing abirchall_dev): ~/bcc_run_tool.sh offcputime -f 0
[Running] /data/users/abirchall/bcc/tools/offcputime.py -f 0
usage: offcputime.py [-h] [-u | -p PID] [-v] [-f]
                     [--stack-storage-size STACK_STORAGE_SIZE]
                     [duration]
offcputime.py: error: argument duration: must be positive and nonzero
devbig680[bcc](no branch, rebasing abirchall_dev): ~/bcc_run_tool.sh offcputime -f -1
[Running] /data/users/abirchall/bcc/tools/offcputime.py -f -1
usage: offcputime.py [-h] [-u | -p PID] [-v] [-f]
                     [--stack-storage-size STACK_STORAGE_SIZE]
                     [duration]
offcputime.py: error: argument duration: must be positive and nonzero
devbig680[bcc](no branch, rebasing abirchall_dev): ~/bcc_run_tool.sh offcputime --stack-storage-size 0 -f 1
[Running] /data/users/abirchall/bcc/tools/offcputime.py --stack-storage-size 0 -f 1
usage: offcputime.py [-h] [-u | -p PID] [-v] [-f]
                     [--stack-storage-size STACK_STORAGE_SIZE]
                     [duration]
offcputime.py: error: argument --stack-storage-size: must be positive and nonzero
devbig680[bcc](no branch, rebasing abirchall_dev): ~/bcc_run_tool.sh offcputime --stack-storage-size 1 -f 1
[Running] /data/users/abirchall/bcc/tools/offcputime.py --stack-storage-size 1 -f 1
swapper/27;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 7
swapper/29;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 102
swapper/25;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 113
swapper/26;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 138
swapper/28;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 164
swapper/30;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 177
swapper/24;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 980
swapper/31;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1556
swapper/1;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2038
swapper/3;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2190
swapper/2;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2235
swapper/15;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2260
swapper/13;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2450
swapper/6;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2646
swapper/12;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2675
swapper/11;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2769
swapper/8;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2885
swapper/4;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 3134
swapper/10;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 3416
swapper/14;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 3702
swapper/7;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 4343
swapper/9;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 5397
swapper/5;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 6836
swapper/17;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 13792
swapper/19;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 14488
swapper/22;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 15298
swapper/18;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 15735
swapper/20;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 17333
swapper/16;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 17645
swapper/23;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 17681
swapper/21;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 18607
WARNING: 92 stack traces could not be displayed. Consider increasing --stack-storage-size.

8 years agohandle ENOMEM in tools/offcputime
Andrew Birchall [Tue, 3 May 2016 23:54:00 +0000 (16:54 -0700)]
handle ENOMEM in tools/offcputime

Summary:
BPF_STACK_TRACE(_name, _size) will allocate space for _size stack traces
(see https://github.com/torvalds/linux/blob/master/kernel/bpf/stackmap.c#L30-L50).

If we've already used all of this space, subsequent calls to bpf_get_stackid()
will return -ENOMEM (see https://github.com/torvalds/linux/blob/master/kernel/bpf/stackmap.c#L173-L176).

This causes our BPF bytecode to store this value in key_t.stack_id and
subsequently causes our python application to crash due to a KeyError when
invoking stack_traces.walk(k.stack_id).

Let's avoid calling stack_traces.walk(k.stack_id) with back stackid's

Test Plan:
Run offcputime.py in an extreme case; with space for only a single stack trace
```
devbig680[bcc](tools): sed_in_file 's/BPF_STACK_TRACE(stack_traces, 1024)/BPF_STACK_TRACE(stack_traces, 1)/' tools/offcputime.py && \
> ~/bcc_run_tool.sh offcputime -f 5; \
> git reset --hard HEAD
swapper/30;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 496
swapper/26;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 553
swapper/28;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 604
swapper/31;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 692
swapper/23;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 713
swapper/18;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 919
swapper/16;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1051
swapper/20;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1056
swapper/21;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1585
swapper/24;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1597
swapper/27;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1610
swapper/17;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 1674
swapper/22;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2390
swapper/25;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2574
swapper/19;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 2589
swapper/29;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 8428
swapper/8;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 15272
swapper/15;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 15591
swapper/11;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 17934
swapper/9;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 18100
swapper/14;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 18266
swapper/10;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 20124
swapper/12;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 20887
swapper/13;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 23453
swapper/3;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 27296
swapper/5;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 29094
swapper/6;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 29799
swapper/7;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 31522
swapper/1;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 32269
swapper/4;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 35585
swapper/2;start_secondary;cpu_startup_entry;schedule_preempt_disabled;schedule 37627
WARNING: 249 stack traces could not be displayed. Consider increasing stack trace storage size.
HEAD is now at d3365e9 [RFC] handle ENOMEM in tools/offcputime`
```

8 years agoMerge pull request #542 from mbertrone/master
4ast [Mon, 16 May 2016 14:23:21 +0000 (07:23 -0700)]
Merge pull request #542 from mbertrone/master

http_filter fixes

8 years agoinvalid access fixed. parameter for specify the interface added
Bertrone Matteo [Mon, 16 May 2016 13:46:01 +0000 (06:46 -0700)]
invalid access fixed. parameter for specify the interface added

8 years agoMerge branch 'master' into patch-1
Qasim Sarfraz [Fri, 13 May 2016 13:15:43 +0000 (18:15 +0500)]
Merge branch 'master' into patch-1