zhenwei pi [Wed, 23 Jun 2021 08:24:11 +0000 (16:24 +0800)]
tcprtt: fix compatibility for python3
Suggested by Yonghong, tcprtt report error on python3:
TypeError: can't concat str to bytes
Both python2 and python3, inet_ntop returns a string type, there is
no need to encode any more.
Test for python2 and python3, both work fine.
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
zhenwei pi [Mon, 21 Jun 2021 07:06:58 +0000 (15:06 +0800)]
tcprtt: support extension summary(average RTT)
Support -e/--extension to show extension summary info, currently
only average RTT is supported.
Also some minor changes to make histogram report easy to read.
Orinally tcprtt does't show lable/header without -b/-B option,
currently it shows like this:
All Addresses = *******
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
chenyuezhou [Tue, 22 Jun 2021 20:57:09 +0000 (16:57 -0400)]
bcc-test: fix test error
masi19bw [Tue, 22 Jun 2021 09:07:49 +0000 (09:07 +0000)]
Add docs about BPF_HASH_OF_MAPS
chenyuezhou [Mon, 21 Jun 2021 14:07:30 +0000 (10:07 -0400)]
tools: funclatency use atomic_increment
masibw [Tue, 22 Jun 2021 06:18:23 +0000 (15:18 +0900)]
Allow the use of custom keys in BPF_HASH_OF_MAPS (#3500)
- Allow the use of custom keys in BPF_HASH_OF_MAPS
- Add both python and C++ tests
Dave Marchevsky [Sat, 19 Jun 2021 05:52:46 +0000 (22:52 -0700)]
libbpf-tools: Don't redefine _GNU_SOURCE to avoid redefinition warning
Similar to past commits like
667988ce9e2a051ff608b727f6c89a5baa01fa67,
my toolchain complains that `_GNU_SOURCE` is redefined. Let's only
define it when it passes `ifndef`
chenyuezhou [Fri, 18 Jun 2021 20:31:52 +0000 (16:31 -0400)]
libbcc: add atomic_increment()
Hengqi Chen [Wed, 16 Jun 2021 16:01:21 +0000 (00:01 +0800)]
bcc/python: fix attach kprobe/kretprobe using regex
Attach kprobe/kretprobe using regular expression should fail
explicitly if no functions are traceable. Currently we catch
all exceptions and if no functions are available, program
continue with no BPF programs attached. In this commit, change
this behavior to explicitly report error to user.
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Hengqi Chen [Wed, 16 Jun 2021 15:29:36 +0000 (23:29 +0800)]
bcc/python: remove unused imports, remove redundant semicolon
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Dave Marchevsky [Wed, 16 Jun 2021 00:53:03 +0000 (17:53 -0700)]
pull out enums from main BPF class to avoid namespace collisions
In #3479, the `bpf_attach_type` enum was pulled into the `BPF` class so
that its members could be used in `attach_func` and `detach_func`
functions introduced to the Python API.
Unfortunately this caused a redefinition of BPF.XDP, which was similarly
pulled in from `bpf_prog_type` enum, thus breaking program loading
(#3489).
Let's pull these enum- and flag-type class variables out into their own
wrapper classes. For backwards compatibility, keep them all (except for
`bpf_attach_type`, which was merged 2 days ago) defined in the BPF
class, but add a comment to not continue doing this.
Spencer Nelson [Fri, 11 Jun 2021 19:07:02 +0000 (12:07 -0700)]
Decode bytes when formatting them as strings
USDTProbe objects (and USDTProbeArguments and USDTProbeLocations) are
instantiated with data that's sourced from libccc calls. That means
that their attributes are bytes-typed, not string-typed.
When a bytes-typed value is rendered into a string with Python's '%s'
formatting directive, it gets a wrapped in single quotes and prefixed
with b. For example, b'probe-location'. This is visually noisy, but
also breaks some tool behavior which uses string-formatted values for
stuff like filters.
This is only an issue in Python 3. In Python 2, the bytes type is just
an alias for the string type, and so byte sequences from libcc were
implicitly decoded as ASCII text.
Dave Marchevsky [Tue, 15 Jun 2021 17:47:45 +0000 (13:47 -0400)]
Merge pull request #3466 from chenhengqi/add-bindsnoop
libbpf-tools: add bindsnoop
Khem Raj [Mon, 14 Jun 2021 19:49:43 +0000 (12:49 -0700)]
Remove APInt/APSInt toString() std::string variants
clang 13+ has removed this in favour of a pair of llvm::toString
() helpers inside StringExtras.h to improve compile speed by avoiding
hits on <string> header
Signed-off-by: Khem Raj <raj.khem@gmail.com>
zcy [Mon, 14 Jun 2021 05:59:22 +0000 (13:59 +0800)]
bcc-python: support attach_func() and detach_func() (#3479)
- support attach_func() and detach_func().
- add an sockmap issue to demonstrate using these two functions.
Hengqi Chen [Fri, 11 Jun 2021 13:39:08 +0000 (21:39 +0800)]
libbpf-tools: remove ext4dist
In #3441, we introduce a new libbpf tools named fsdist, which is
built on the idea by @anakryiko and previous work by @ethercflow.
fsdist extends ext4dist to support multiple file systems in a flexable
way. Now we can replace ext4dist and treat it as an alias to fsdist.
This commit removes ext4dist and replaces it with a symlink to fsdist.
References:
#3430, #3436
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Hengqi Chen [Wed, 12 May 2021 00:43:15 +0000 (08:43 +0800)]
libbpf-tools: add bindsnoop
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Hengqi Chen [Wed, 9 Jun 2021 05:42:54 +0000 (13:42 +0800)]
libbpf-tools: optimize fentry_exists helper
The previous implementation checks fentry support either in vmlinux
or module BTF. So we need two calls to fentry_exists to verify that
whether a symbol exists. This commit updates this behavior to use
the module name provided as a hint, and fallback to vmlinux if module
BTF is not available.
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Hengqi Chen [Sat, 5 Jun 2021 02:44:46 +0000 (10:44 +0800)]
libbpf-tools: migrate xfsslower to fsslower
This commit migrates xfsslower to a generic fsslower which supports
tracing multiple file systems. It works the same way as the original
tool except that the users are supposed to specify which file systems
to trace using -t option.
sudo ./fsslower -t ext4 -m 1
Tracing ext4 operations slower than 1 ms... Hit Ctrl-C to end.
TIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME
10:36:07 code 6896 F LL_MAX 0 2.40 state.vscdb-journal
10:36:07 code 6896 F LL_MAX 0 1.74 state.vscdb-journal
10:36:07 code 6896 F LL_MAX 0 1.78 state.vscdb
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
chenyuezhou [Tue, 8 Jun 2021 16:09:31 +0000 (12:09 -0400)]
libbcc: add msg_redirect_hash() and sk_redirect_hash() for sockhash
zcy [Mon, 7 Jun 2021 16:14:14 +0000 (00:14 +0800)]
libbcc: support BPF_SOCKHASH specify the key type (#3473)
support BPF SOCKHASH specify the key type and update documentation
for BPF_SOCKHASH and map.sock_hash_update().
masibw [Sun, 6 Jun 2021 16:12:32 +0000 (01:12 +0900)]
Add attach_xdp to reference_guide.md (#3450)
- Add attach_xdp to reference_guide.md
- Add description about flags
Emilien Gobillot [Sun, 6 Jun 2021 05:44:26 +0000 (07:44 +0200)]
finish to add support of subset in items_*_batch() (#3440)
finish to add support of subset in items_*_batch()
- rewrite items_lookup_batch() and items_lookup_and_delete_batch() to make it more robust.
- add docstring on items_*_batch()
- update the reference_guide.md
edwardwu [Thu, 3 Jun 2021 04:15:27 +0000 (12:15 +0800)]
Add an option to strip leading zeros from linear histograms
Sometimes histogram gives us too much zero info that we don't really care.
For example:
usec : count distribution
0 : 0 | |
1 : 0 | |
2 : 0 | |
3 : 0 | |
4 : 0 | |
5 : 0 | |
6 : 0 | |
7 : 0 | |
8 : 0 | |
9 : 0 | |
10 : 0 | |
11 : 0 | |
12 : 0 | |
13 : 0 | |
14 : 0 | |
15 : 0 | |
16 : 0 | |
17 : 0 | |
18 : 0 | |
19 : 0 | |
20 : 0 | |
21 : 0 | |
22 : 0 | |
23 : 0 | |
24 : 0 | |
25 : 0 | |
26 : 0 | |
27 : 0 | |
28 : 0 | |
29 : 0 | |
30 : 0 | |
31 : 0 | |
32 : 0 | |
33 : 0 | |
34 : 0 | |
35 : 0 | |
36 : 0 | |
37 : 0 | |
38 : 0 | |
39 : 0 | |
40 : 0 | |
41 : 7 |****************************************|
42 : 2 |*********** |
Such much info is hard to analyze by FIRST glance, especially console view
After supporting strip leading zeros
print_linear_hist("usec", "name", name_print, strip_leading_zero=True)
usec : count distribution
41 : 7 |****************************************|
42 : 2 |************* |
This is what we really care, and it's clear.
Signed-off-by: Edward Wu <edwardwu@realtek.com>
Jerome Marchand [Wed, 2 Jun 2021 12:23:20 +0000 (14:23 +0200)]
libbpf-tool: don't ignore LDFLAGS
Packagers need to be able set linker options according to their
distribution guidelines.
chenyuezhou [Wed, 2 Jun 2021 21:50:45 +0000 (17:50 -0400)]
docs: update description of bcc python BPF()
Hengqi Chen [Wed, 19 May 2021 16:00:32 +0000 (00:00 +0800)]
libbpf-tools: add fsdist
fsdist is a multitool which show filesystem latency.
Currently we support btrfs/ext4/nfs/xfs filesystems.
It behaves the same as its counterpart in BCC tools
named btrfsdist.py/ext4dist.py/nfsdist.py/xfsdist.py
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
chenyuezhou [Fri, 28 May 2021 04:27:11 +0000 (00:27 -0400)]
docs: add description of attach_raw_socket
Nick-nizhen [Thu, 27 May 2021 05:21:59 +0000 (13:21 +0800)]
Update cpudist.py
When calculating the ONCPU time, prev has left the CPU already. It is not necessary to judge whether the process state is TASK_RUNNING or not.
zcy [Thu, 27 May 2021 16:50:23 +0000 (00:50 +0800)]
tools/deadlock: support specifies maxnum of threads and edge cases (#3455)
support to specify maxinum of threads and edge cases. The default values make map taking more than 0.5G memory which cause out-of-memory issue on some systems.
also fix an issue with python `open` so the open file is automatically closed upon file reading is done.
Hengqi Chen [Sat, 22 May 2021 08:07:36 +0000 (16:07 +0800)]
hardirqs: Migrate to kernel tracepoint
The hardirqs tool is not working properly in recent kernels.
This commit migrates hardirqs to use kernel tracepoints
instead of kprobes, just as we already made to softirqs.
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Yonghong Song [Wed, 26 May 2021 02:58:00 +0000 (19:58 -0700)]
Fix a llvm compilation error
Current llvm trunk (https://github.com/llvm/llvm-project)
will cause the following compilation errors:
/home/yhs/work/bcc/src/cc/bcc_debug.cc: In member function ‘void ebpf::SourceDebugger::dump()’:
/home/yhs/work/bcc/src/cc/bcc_debug.cc:135:75: error: no matching function for call to
‘llvm::MCContext::MCContext(llvm::Triple&, std::unique_ptr<llvm::MCAsmInfo>::pointer,
std::unique_ptr<llvm::MCRegisterInfo>::pointer, llvm::MCObjectFileInfo*,
std::unique_ptr<llvm::MCSubtargetInfo>::pointer, std::nullptr_t)’
MCContext Ctx(TheTriple, MAI.get(), MRI.get(), &MOFI, STI.get(), nullptr);
^
......
This is because upstream patch https://reviews.llvm.org/D101921
refactored MCObjectFileInfo initialization and changed MCContext
constructor signature.
This patch fixed the issue by following the new code patterns
in https://reviews.llvm.org/D101921.
Jiri Olsa [Fri, 9 Apr 2021 17:24:12 +0000 (19:24 +0200)]
tools/ttysnoop: Add --datasize/--datacount
Adding the possibility to define transmitting data size
(--datasize option) and number of times we ask for this
amount (--datacount option).
This helps to configure ttysnoop behaviour for the expected
data in the terminal session. For example ncurses applications
like mc or huge sized terminals need bigger buffer to snoop
everything from the buffer.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Jiri Olsa [Fri, 9 Apr 2021 15:14:21 +0000 (17:14 +0200)]
tools/ttysnoop: Use array map to store data
So we can use bigger sizes for the data.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Jiri Olsa [Sun, 9 May 2021 15:36:36 +0000 (17:36 +0200)]
tools/ttysnoop: Fix tty_write probe to use new arguments
Kernel commit [1] changed arguments of tty_write function,
changing the probe function to new prototypes.
Also switching to trampolines.
[1]
9bb48c82aced tty: implement write_iter
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
chenyuezhou [Mon, 24 May 2021 21:33:27 +0000 (17:33 -0400)]
adjust: bpf_attach_xdp report nicer error
Hengqi Chen [Fri, 21 May 2021 01:17:14 +0000 (09:17 +0800)]
libbpf-tools: fix misuse of bpf_get_current_pid_tgid
bpf_get_current_pid_tgid() returns process ID in the upper 32bits,
and thread ID in lower 32 bits (both from userspace's perspective).
biosnoop and gethostlatency misuse this function.
biosnoop takes the thread ID as process ID which is not expected.
gethostlatency uses the process ID as a unique key for BPF map,
which may result in event loss or data corruption.
This commit fixes these problems.
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
zcy [Mon, 24 May 2021 03:31:48 +0000 (11:31 +0800)]
feature: add `XDP_FLAGS*` in python lib (#3447)
add XDP_FLAG macros in python lib so macro names instead of numeric numbers can be used by tools.
Hengqi Chen [Thu, 20 May 2021 14:49:25 +0000 (22:49 +0800)]
tools: filter/display using PID instead of TID
As mentioned in #3407, several BCC tools misuse bpf_get_current_pid_tgid(),
bpf_get_current_pid_tgid() returns process ID in the upper 32bits, and
thread ID in lower 32 bits (both from userspace's perspective).
In this commit, we return process ID to userspace for display, and use
thread ID as BPF map key so that we can avoid event loss or data corruption.
The following tools are fixed in the commit:
* bashreadline
* cachetop
* dcsnoop
* killsnoop
* llcstat
* mdflush
* mysqld_qslower
* wakeuptime
See also #3411, #3427, #3433 .
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
marselester [Fri, 21 May 2021 00:36:49 +0000 (20:36 -0400)]
Update tcpconnect to use "__u32 af" instead of "int af"
It helps to decode an address family in Go frontend
generated by bpf2go tool, here is an example
https://github.com/marselester/libbpf-tools/blob/master/cmd/tcpconnect/main.go
Hengqi Chen [Sat, 15 May 2021 15:15:03 +0000 (23:15 +0800)]
libbpf-tools: parse -h option
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Hengqi Chen [Mon, 10 May 2021 14:28:33 +0000 (22:28 +0800)]
libbpf-tools: add statsnoop
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
chenyuezhou [Mon, 17 May 2021 09:07:20 +0000 (05:07 -0400)]
libbpf-tools: add check BPF_F_MMAPABLE is supported
Hengqi Chen [Sun, 16 May 2021 09:18:27 +0000 (17:18 +0800)]
tools: filter/display using PID intead of TID
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Hengqi Chen [Thu, 13 May 2021 13:46:16 +0000 (21:46 +0800)]
tools: filter using PID intead of TID
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Vitaly Chikunov [Thu, 13 May 2021 08:46:42 +0000 (11:46 +0300)]
libbpf-tools: Fix build dependence for parallel builds
Add LIBBPF_OBJ dependence to `%.o'.
When libbpf-tools built in parallel (with `make -j`) sometimes
`map_helpers.o' is built before `libbpf.a' causing build error:
$ make -j8 -C libbpf-tools BPFTOOL=/usr/sbin/bpftool
...
make: Entering directory '/usr/src/RPM/BUILD/bcc-0.19.0/libbpf-tools'
CC map_helpers.o
In file included from map_helpers.c:7:
./map_helpers.h:6:10: fatal error: 'bpf/bpf.h' file not found
^~~~~~~~~~~
1 error generated.
...
make: Leaving directory '/usr/src/RPM/BUILD/bcc-0.19.0/libbpf-tools'
INSTALL bpf.h libbpf.h btf.h xsk.h libbpf_util.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h libbpf_common.h
...
INSTALL libbpf.a
error: Bad exit status from /usr/src/tmp/rpm-tmp.63536 (%build)
Fixes: #3412
Signed-off-by: Vitaly Chikunov <vt@altlinux.org>
Dominique Martinet [Mon, 5 Apr 2021 23:14:06 +0000 (08:14 +0900)]
libbpf-tools/opensnoop: disable open on aarch64
aarch64 has no open syscall, do not attempt to trace it.
Fixes #3344.
Russ Kubik [Wed, 12 May 2021 19:10:48 +0000 (13:10 -0600)]
Update bcc_exception.h
Hengqi Chen [Sat, 8 May 2021 01:15:25 +0000 (09:15 +0800)]
tools: display PID intead of TID in statsnoop.py
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
chenhengqi [Tue, 11 May 2021 05:22:10 +0000 (13:22 +0800)]
tools: fix typo in help message
Signed-off-by: chenhengqi <chenhengqi@outlook.com>
AnyISalIn [Sat, 8 May 2021 05:02:44 +0000 (13:02 +0800)]
Fix rpmbuild error
Signed-off-by: AnyISalIn <anyisalin@gmail.com>
Andreas Ziegler [Thu, 6 May 2021 07:02:28 +0000 (09:02 +0200)]
SPECS/bcc.spec: add dependencies to libdebuginfod
On Fedora builds we can check the version number and add
build and runtime dependencies to debuginfod for all
currently supported releases (>= 32). Note that the buildbot
only has Fedora 25-28 so it will not try to build libbcc
with debuginfod support as the required packages are not
available on these releases.
For .deb packages there is no easy way to add dependencies
dynamically, so we do not add dependencies to libdebuginfod
there for now. For documentation purposes, however, let's
add a comment indicating which changes are required for
libdebuginfod support for downstream maintainers.
Signed-off-by: Andreas Ziegler <andreas.ziegler@fau.de>
Andreas Ziegler [Thu, 29 Apr 2021 10:18:40 +0000 (12:18 +0200)]
bcc_elf: add support for debug information from libdebuginfod
This change adds debuginfod as a new source for debug
information. By using libdebuginfod we can query a server
for a file containing debug information for a given ELF
binary. The environment variable DEBUGINFOD_URLS has to
be defined to an URL for a debuginfod server providing
debug information files for your distribution or the
federating server provided by the elfutils project:
For example, to use the Fedora server, you would need:
$ export DEBUGINFOD_URLS="https://debuginfod.fedoraproject.org/"
Or for the elfutils server which federates to servers for
openSUSE, Void Linux, Debian and Fedora, among others:
$ export DEBUGINFOD_URLS="https://debuginfod.elfutils.org/"
Calls to the debuginfod_find_debuginfo function from
libdebuginfod will fail if the environment variable is not
set, otherwise the library will attempt to download debug
information for a build ID extracted from the binary in
question and store it in a local cache directory.
Fixes iovisor/bpftrace#1774
Signed-off-by: Andreas Ziegler <andreas.ziegler@fau.de>
Hengqi Chen [Sat, 3 Apr 2021 07:35:16 +0000 (15:35 +0800)]
libbpf-tools: add gethostlatency
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Luigi Baldoni [Thu, 6 May 2021 07:37:32 +0000 (09:37 +0200)]
Use arch-specific libdir with pkgconfig
Yonghong Song [Thu, 6 May 2021 03:53:54 +0000 (20:53 -0700)]
update debian changelog for release v0.20.0
* Support for kernel up to 5.12
* Some basic support for MIPS
* added bpf_map_lookup_batch and bpf_map_delete_batch support
* tools/funclatency.py support nested or recursive functions
* tools/biolatency.py can optionally print out average/total value
* fix possible marco HAVE_BUILTIN_BSWAP redefine warning for kernel >= 5.10.
* new tools: virtiostat
* new libbpf-tools: ext4dist
* doc update and bug fixes
Signed-off-by: Yonghong Song <yhs@fb.com>
Yonghong Song [Thu, 6 May 2021 02:11:13 +0000 (19:11 -0700)]
fix llvm compilation errors
MCContext and InitMCObjectFileInfo name/signatures
are changed due to upstream patch
https://reviews.llvm.org/D101462
Adjust related codes in bcc_debug.cc properly to resolve
the compilation error for llvm13.
Signed-off-by: Yonghong Song <yhs@fb.com>
Yonghong Song [Thu, 6 May 2021 00:48:18 +0000 (17:48 -0700)]
sync with latest libbpf repo
sync submodule libbpf repo to the following commit:
c5389a965bc3 sync: latest libbpf changes from kernel
Signed-off-by: Yonghong Song <yhs@fb.com>
Kenta Tada [Wed, 6 Jan 2021 01:26:10 +0000 (10:26 +0900)]
bcc/libbpf-tools: Use fentry for vfsstat
Use fentry like vfsstat.py when it is available.
Signed-off-by: Kenta Tada <Kenta.Tada@sony.com>
Dave Marchevsky [Wed, 5 May 2021 04:31:42 +0000 (21:31 -0700)]
loader: include bpf_workaround.h header
The intent of #3391 was to have `bpf_workaround.h` included after
`bpf.h` when compiling bcc headers. However, that PR only added the file
to the `headers_` map of files that can be included. To actually
include, need to adjust compiler input flags as well.
Fixes:
d089013e ("Move HAVE_BUILTIN_BSWAP includes to separate header")
Wenbo Zhang [Sat, 1 May 2021 04:22:42 +0000 (12:22 +0800)]
libbpf-tools: add offcputime
Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
Dave Marchevsky [Fri, 30 Apr 2021 05:00:05 +0000 (22:00 -0700)]
Move HAVE_BUILTIN_BSWAP includes to separate header
As reported in #3366, on newer kernels bcc complains about macro
redefinition when compiling bpf programs:
```
include/linux/compiler-clang.h:46:9: warning: '__HAVE_BUILTIN_BSWAP64__' macro redefined [-Wmacro-redefined]
\#define __HAVE_BUILTIN_BSWAP64__
^
<command line>:5:9: note: previous definition is here
\#define __HAVE_BUILTIN_BSWAP64__ 1
```
Since these macros are passed in as `-D` cflags, they appear first
before any \#define statements in code. Since an [upstream kernel
patch](https://lore.kernel.org/linux-csky/
20210226161151.2629097-1-arnd@kernel.org/)
added these defines in a kernel header, we see the warning.
This patch moves these definitions to a separate 'virtual' header that's included
after virtual_bpf.h and adds an ifndef guard. As a result, newer kernels
with the patch will not trigger the warning, while older kernels will
not lose the definition.
This should be safe based on my digging - some existing bcc programs use
`__builtin_bswap` methods, but without checking HAVE_BUILTIN_BSWAP.
Macros that may be conditionally defined based on HAVE_BUILTIN_BSWAP,
like those in `bpf_endian.h`, aren't. If a similar macro or struct def
in virtual_bpf.h - or any header it pulls in - changes depending on
HAVE_BUILTIN_BSWAP this could cause problems on older kernels, but I
don't believe that this is the case, or will be based on how
infrequently the defines are checked.
zcy [Fri, 30 Apr 2021 04:36:53 +0000 (12:36 +0800)]
feature: support create new map and pin it to bpffs as file(BPF_TABLE_PINNED) (#3382)
Support create a new map and pin it if the pinned file is not available.
Co-authored-by: chenyue.zhou <chenyue.zhou@upai.com>
Yonghong Song [Thu, 29 Apr 2021 18:54:02 +0000 (11:54 -0700)]
Revert "add macros offsetof and container_of"
This reverts commit
cda7acdbddb19a5a258f881711845e41463dabbf.
This will tigger ugly compilation macro redefined warnings:
$ sudo ./biolatency.py
In file included from /virtual/main.c:3:
In file included from include/linux/blkdev.h:5:
In file included from include/linux/sched.h:12:
In file included from arch/x86/include/asm/current.h:6:
In file included from arch/x86/include/asm/percpu.h:45:
include/linux/kernel.h:992:9: warning: 'container_of' macro redefined [-Wmacro-redefined]
#define container_of(ptr, type, member) ({ \
^
/virtual/include/bcc/helpers.h:48:9: note: previous definition is here
#define container_of(ptr, type, member) \
^
1 warning generated.
Tracing block device I/O... Hit Ctrl-C to end.
Revert now and let us design how to support it better. For example, may
create a different header file to put common kernel macros there to be
used by the program.
Signed-off-by: Yonghong Song <yhs@fb.com>
Simone Magnani [Wed, 28 Apr 2021 10:04:52 +0000 (12:04 +0200)]
enhanced items_delete_batch() in Python to avoid double list creation
This commit enhances the items_delete_batch() function by accepting a ct.Array instead of a Python list.
This way, the array does not need to be re-created, allowing to directly perform the requested operation.
Signed-off-by: Simone Magnani <simonemagnani.96@gmail.com>
Simone Magnani [Wed, 28 Apr 2021 09:59:24 +0000 (11:59 +0200)]
added bpf_update_batch() API support for Python Maps
This commit aims at introducing items_update_batch, batch operation to update multiple key-value pairs at the same time.
Doc has been updated accordingly, and a test is provided.
Signed-off-by: Simone Magnani <simonemagnani.96@gmail.com>
Simone Magnani [Wed, 28 Apr 2021 09:50:01 +0000 (11:50 +0200)]
modified self.max_entries to be available from all the MAP types
This commit introduces the self.max_entries attribute both into Queue/Stack maps and to all those whwqo extend TableBase
Signed-off-by: Simone Magnani <simonemagnani.96@gmail.com>
Simone Magnani [Wed, 28 Apr 2021 09:47:20 +0000 (11:47 +0200)]
introduced Queue/Stack itervalues
This commit introduces the possibility to iterate over all elements of a Queue/Stack.
To avoid infinite loop, a maximum of MAX_ENTRIES pop() are performed
Signed-off-by: Simone Magnani <simonemagnani.96@gmail.com>
edwardwu [Wed, 28 Apr 2021 10:22:48 +0000 (18:22 +0800)]
tools/biolatency: Extend average/total value
Sometimes log2 range is not enough for throughput tuning.
Especially a little difference in performance downgrade.
Also, this extension uses two bpf helper bpf_map_lookup_elem().
It's a cost on embedded system, therefore it's better to be an option.
Signed-off-by: Edward Wu <edwardwu@realtek.com>
Jerome Marchand [Tue, 27 Apr 2021 13:13:12 +0000 (15:13 +0200)]
Fix BPF(src_file="foo")
Since commit
75f20a15 ("Use string type for comparison to PATH
elements"), src_file isn't working anymore. Somehow, two wrongs
(ArgString __str__() returning a bytes object and joining a bytes and
what was supposed to be a string) did make a right.
It fixes the following error in netqtop and deadlock:
Traceback (most recent call last):
File "/usr/share/bcc/tools/netqtop", line 207, in <module>
b = BPF(src_file = EBPF_FILE)
File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 335, in __init__
src_file = BPF._find_file(src_file)
File "/usr/lib/python3.6/site-packages/bcc/__init__.py", line 255, in _find_file
t = b"/".join([os.path.abspath(os.path.dirname(argv0.__str__())), filename])
TypeError: sequence item 0: expected a bytes-like object, str found
netedwardwu [Wed, 28 Apr 2021 04:53:26 +0000 (12:53 +0800)]
tools/funclatency: Should clear() after display that is what we want (#3380)
BUG:
funclatency memcpy -i 2
nsecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 28 |************ |
4096 -> 8191 : 92 |****************************************|
avg = 4265 nsecs, total: 9413985 nsecs, count: 2207
nsecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 0 | |
512 -> 1023 : 0 | |
1024 -> 2047 : 0 | |
2048 -> 4095 : 38 |****** |
4096 -> 8191 : 248 |****************************************|
avg = 4304 nsecs, total:
11066321 nsecs, count: 2571
After long-run, you can see the count above is totally wrong.
Also, display together before together clearing is important for better accuracy.
Signed-off-by: Edward Wu <edwardwu@realtek.com>
yzhao [Wed, 28 Apr 2021 04:48:34 +0000 (21:48 -0700)]
Add BPFStackTable::free_symcache() to free the symbol cache for an PID (#3371)
This is useful when BPFStackTable is used by a long running program to
limit the memory usage, by removing the cached symbols of an exited
process.
Yonghong Song [Mon, 26 Apr 2021 04:10:35 +0000 (21:10 -0700)]
add macros offsetof and container_of
These are two common kernel macros to manipulate data
structures. Let us add them to helpers.h so bpf program
can use them.
Signed-off-by: Yonghong Song <yhs@fb.com>
Emilien Gobillot [Sun, 25 Apr 2021 17:18:50 +0000 (19:18 +0200)]
add bpf_map_lookup_batch and bpf_map_delete_batch in bcc (#3363)
. add bpf_map_lookup_batch and bpf_map_delete_batch in bcc
. add test_map_batch_ops.py to test batch lookup and delete on map
. add items_lookup_batch() and items_delete_batch() in the reference guide
. add keys as an optional argument to items_delete_batch
Kenta Tada [Fri, 2 Apr 2021 14:42:03 +0000 (23:42 +0900)]
libbpf-tools: fix some minor issues of ext4dist
* Fix the check of the existence of module BTFs
* Initialize the global variable
Signed-off-by: Kenta Tada <Kenta.Tada@sony.com>
Wenbo Zhang [Sat, 24 Apr 2021 00:04:03 +0000 (08:04 +0800)]
libbpf-tools: add two helpers
Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
chenyue.zhou [Wed, 14 Apr 2021 17:37:07 +0000 (13:37 -0400)]
update
chenyue.zhou [Tue, 6 Apr 2021 20:55:48 +0000 (16:55 -0400)]
tools/funclatency: support nested or recursive functions
chendotjs [Sat, 10 Apr 2021 15:51:11 +0000 (23:51 +0800)]
fix minor typo
Hengqi Chen [Tue, 6 Apr 2021 02:14:07 +0000 (10:14 +0800)]
libbpf-tools: resolve KERNEL_VERSION macro redefined
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Hengqi Chen [Sat, 20 Mar 2021 04:18:35 +0000 (12:18 +0800)]
libbpf-tools: fix for block io tracepoints changed
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>
Andrii Nakryiko [Mon, 5 Apr 2021 18:36:56 +0000 (11:36 -0700)]
libbpf: update to latest upstream version
This brings in KERNEL_VERSION macro fix, among few other recent features.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Andrii Nakryiko [Fri, 26 Mar 2021 06:58:19 +0000 (23:58 -0700)]
libbpf-tools: add libbpf's Linux uapi headers to build
Do not rely on up-to-date UAPI headers on the system. Instead use the most
recent ones that are used for libbpf's own build.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Andrii Nakryiko [Fri, 26 Mar 2021 06:56:40 +0000 (23:56 -0700)]
libbpf-tools: remove unecessary custom NULL definitions
Now that libbpf defines NULL in bpf_helpers.h, there is no need for tools to
re-define NULL.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Andrii Nakryiko [Fri, 26 Mar 2021 06:48:04 +0000 (23:48 -0700)]
libbpf: update to latest master
Update libbpf to the latest upstream commit.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Wenbo Zhang [Fri, 26 Mar 2021 02:53:18 +0000 (10:53 +0800)]
libbpf-tools: use raw_tp sched_switch instead of kprobe
Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
Roman Sudarikov [Fri, 26 Mar 2021 07:48:47 +0000 (07:48 +0000)]
some code cleanups
Roman Sudarikov [Wed, 24 Mar 2021 16:01:22 +0000 (16:01 +0000)]
addressing review comments
Roman Sudarikov [Wed, 17 Mar 2021 13:54:22 +0000 (13:54 +0000)]
levereging extern Kconfig LINUX_KERNEL_VERSION, adding comments
Roman Sudarikov [Mon, 15 Mar 2021 19:31:27 +0000 (19:31 +0000)]
block tracepoints no longer have struct request_queue arg
Wenbo Zhang [Fri, 26 Mar 2021 06:01:07 +0000 (14:01 +0800)]
libbpf-tools: add ext4dist
Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
FedeParola [Sun, 21 Mar 2021 11:44:36 +0000 (12:44 +0100)]
doc: add perf_submit_skb() documentation
Jerome Marchand [Sat, 20 Mar 2021 13:33:59 +0000 (14:33 +0100)]
tools: handle renamed lookup_fast function in dcache tools
The lookup_fast function can be rename lookup_fast.constprop.x by gcc
constant propagation optimization and that breaks dcstat and
dcsnoop. Let's look for both name using a regular expression.
Tiezhu Yang [Sat, 20 Mar 2021 01:42:35 +0000 (09:42 +0800)]
bcc: Add some basic support for MIPS
In order to fix the following errors when running bpf program
on the MIPS Loongson64 platform, add some basic support, with
this patch, running hello_world.py can get the expected result.
root@linux:/home/loongson/bcc# python examples/hello_world.py
In file included from <built-in>:3:
In file included from /virtual/include/bcc/helpers.h:51:
In file included from include/uapi/linux/if_packet.h:5:
arch/mips/include/uapi/asm/byteorder.h:17:3: error: "MIPS, but neither __MIPSEB__, nor __MIPSEL__???"
# error "MIPS, but neither __MIPSEB__, nor __MIPSEL__???"
^
In file included from <built-in>:3:
In file included from /virtual/include/bcc/helpers.h:53:
In file included from include/linux/log2.h:12:
In file included from include/linux/bitops.h:32:
In file included from arch/mips/include/asm/bitops.h:19:
In file included from arch/mips/include/asm/barrier.h:11:
arch/mips/include/asm/addrspace.h:13:10: fatal error: 'spaces.h' file not found
#include <spaces.h>
^~~~~~~~~~
2 errors generated.
Traceback (most recent call last):
File "examples/hello_world.py", line 12, in <module>
BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()
File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 364, in __init__
raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>
root@linux:/home/loongson/bcc# python examples/hello_world.py
In file included from <built-in>:3:
In file included from /virtual/include/bcc/helpers.h:53:
In file included from include/linux/log2.h:12:
In file included from include/linux/bitops.h:32:
arch/mips/include/asm/bitops.h:101:3: error: invalid output constraint '+ZC' in asm
__bit_op(*m, __INS "%0, %3, %2, 1", "i"(bit), "r"(~0));
^
arch/mips/include/asm/bitops.h:40:19: note: expanded from macro '__bit_op'
: "=&r"(__temp), "+" GCC_OFF_SMALL_ASM()(mem) \
^
[...]
arch/mips/include/asm/atomic.h:154:1: error: invalid output constraint '+ZC' in asm
arch/mips/include/asm/atomic.h:151:2: note: expanded from macro 'ATOMIC_OPS'
ATOMIC_FETCH_OP(pfx, op, type, c_op, asm_op, ll, sc)
^
arch/mips/include/asm/atomic.h:141:4: note: expanded from macro 'ATOMIC_FETCH_OP'
"+" GCC_OFF_SMALL_ASM() (v->counter) \
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
Traceback (most recent call last):
File "examples/hello_world.py", line 12, in <module>
BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()
File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 364, in __init__
raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Yonghong Song [Sat, 20 Mar 2021 02:28:02 +0000 (19:28 -0700)]
update debian changelog for release v0.19.0
* Support for kernel up to 5.11
* allow BCC as a cmake subproject
* add LPORT support in tcpconnlat and tcpconnect
* added bpf_map_lookup_and_delete_batch support
* new tools: virtiostat
* new libbpf-tools: cpufreq, funclatency, cachestat
* add install target to libbpf-tools
* a few lua fixes
* doc update and bug fixes
Signed-off-by: Yonghong Song <yhs@fb.com>
Yonghong Song [Sat, 20 Mar 2021 00:02:58 +0000 (17:02 -0700)]
sync with latest libbpf
sync with latest libbpf with top commit:
092a60685625 Makefile: fix install flags order
Signed-off-by: Yonghong Song <yhs@fb.com>
Gary Lin [Fri, 12 Mar 2021 03:32:19 +0000 (11:32 +0800)]
cmake: make "-no-pie" optional
The recent linux distros already support PIE so it shouldn't be a
problem to remove "-no-pie". To avoid issue#782, we make "-no-pie"
optional and enable it by default. For the distro with PIE luajit,
just add the following build option:
-DENABLE_NO_PIE=OFF
Then, bcc-lua will be built with PIE support.
Signed-off-by: Gary Lin <glin@suse.com>
Mariusz Barczak [Tue, 16 Mar 2021 20:44:06 +0000 (21:44 +0100)]
Allow to use BCC as a cmake sub-project
Signed-off-by: Mariusz Barczak <mariusz.barczak@intel.com>
Wenbo Zhang [Thu, 11 Mar 2021 09:48:50 +0000 (17:48 +0800)]
libbpf-tools: fix bug report address
Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>
chenhengqi [Tue, 16 Mar 2021 05:58:25 +0000 (13:58 +0800)]
libbpf-tools: initialize global variables in cachestat and funclatency
Signed-off-by: Hengqi Chen <chenhengqi@outlook.com>