Frank Ch. Eigler [Sun, 29 Mar 2020 19:10:37 +0000 (15:10 -0400)]
debuginfod-client default_progressfn: formatting fix
The saga of clean $DEBUGINFOD_PROGRESS=1 output continues. Previous
code would sometimes insert a \n (a blank line) into the output
stream, even if the target file was found in a cache and thus the
progressfn was never called. New code sets a client flag to track
this case more precisely.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Mark Wielaard [Sat, 28 Mar 2020 20:42:04 +0000 (16:42 -0400)]
PR25728: disable debuginfod --port=0
When starting debuginfod with --port=0 it would start using a random port
(possibly different for ipv4 and ipv6). This seems to be an accidental
and not very useful functionality. Just produce an error when started
with --port=0.
https://sourceware.org/bugzilla/show_bug.cgi?id=25728
Signed-off-by: Mark Wielaard <mark@klomp.org>
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Sat, 28 Mar 2020 19:57:09 +0000 (15:57 -0400)]
PR25739: debuginfod correct mtime for fdcache'd files
Files extracted from archives then left in the fdcache need to get get
their mtime set consistently, so that a subsequent cache-hit fetch can
relay the correct mtime to clients.
Reported-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Sat, 28 Mar 2020 00:47:45 +0000 (20:47 -0400)]
PR25722: debuginfod-find: accept /path/names in place of buildid hex
Extend the debuginfod-find command line interface to permit
/file/names instead of only buildid hexadecimal strings.
v2: move code into debuginfod-find; client API remains buildid only.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Wed, 25 Mar 2020 03:46:30 +0000 (23:46 -0400)]
debuginfod: User-Agent and X-Forwarded-For header relay
Extend the debuginfod client API with a function to stuff outgoing
headers into libcurl http transfers. Use this from debuginfod so
federated trees of debuginfod/httpd can trace back to to the
originating client for administrative purposes. Docs & test included.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Thu, 26 Mar 2020 20:44:20 +0000 (16:44 -0400)]
PR25448: debuginfod: add transfer performance metrics
We now export metrics related to the time taken and data sent,
from which prometheus type tools can compute nice time series
with averages.
http_responses_duration_milliseconds_count{code="200"} 63
http_responses_duration_milliseconds_count{code="404"} 2
http_responses_duration_milliseconds_count{code="503"} 1
http_responses_duration_milliseconds_sum{code="200"} 66
http_responses_duration_milliseconds_sum{code="404"} 2
http_responses_duration_milliseconds_sum{code="503"} 0
http_responses_transfer_bytes_count{code="200"} 63
http_responses_transfer_bytes_count{code="404"} 2
http_responses_transfer_bytes_count{code="503"} 1
http_responses_transfer_bytes_sum{code="200"} 425177
http_responses_transfer_bytes_sum{code="404"} 18
http_responses_transfer_bytes_sum{code="503"} 37
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Fri, 27 Mar 2020 01:32:40 +0000 (21:32 -0400)]
PR25583: debuginfod: conditionally prefer bsdtar to dpkg for .deb handling
Prefer /usr/bin/dpkg-deb if installed, as normal on a debian system.
Suggested-by: Mark Wielaard <mark@klomp.org>
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Wed, 25 Mar 2020 14:55:53 +0000 (10:55 -0400)]
PR25583: debuginfod: prefer bsdtar to dpkg for .deb handling
It turns out a bsdtar subshell can do the job of dpkg-deb.
bsdtar comes from/with libarchive so it should be available
everywhere.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Thu, 26 Mar 2020 17:48:20 +0000 (13:48 -0400)]
debuginfod: document and workaround fedora31 zstd compression
Old enough (even RHEL8 era) rpm/libarchive tools cannot grok fedora31
zstd-compressed rpms. Disable those tests if necessary. Document -Z
based workaround for these debuginfod users.
https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Mark Wielaard [Thu, 26 Mar 2020 17:01:29 +0000 (18:01 +0100)]
tests: Add missing debuginfod-rpms/fedora31/hello3-*.rpm to EXTRA_DIST.
The new rpms were used in the new test. Make sure they are in the dist
tar ball.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Frank Ch. Eigler [Thu, 26 Mar 2020 15:12:49 +0000 (11:12 -0400)]
PR25548: CURLOPT_PATH_AS_IS backward compatibility
old libcurl lacks the CURL_AT_LEAST_VERSION macro, so use
LIBCURL_VERSION_NUM testing instead.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Thu, 26 Mar 2020 15:12:49 +0000 (11:12 -0400)]
PR25548: CURLOPT_PATH_AS_IS backward compatibility
libcurl < 7.42 lacks the CURLOPT_PATH_AS_IS flag, but extraneous
client-side canonicalization is mostly harmless.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Reported-by: Mark Wielaard <mark@klomp.org>
Frank Ch. Eigler [Mon, 23 Mar 2020 19:33:56 +0000 (15:33 -0400)]
PR25548: support canonicalized source-path names in debuginfod webapi
Programs are sometimes compiled with source path names containing
noise like /./ or // or /foo/../, and these survive into DWARF. This
code allows either raw or canonicalized pathnames in the webapi, by
letting the client pass things verbatim, and letting the server
store/accept both raw and canonicalized path names for source files.
Tests included & docs updated.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Reported-by: Eli Schwartz <eschwartz@archlinux.org>
Tested-by: Eli Schwartz <eschwartz@archlinux.org>
Frank Ch. Eigler [Wed, 25 Mar 2020 01:57:59 +0000 (21:57 -0400)]
PR25367: improve debuginfod webapi logging
Improve debuginfod logging to show webapi query results including
http status, sizes, and processing times.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Mark Wielaard [Wed, 25 Mar 2020 22:22:55 +0000 (23:22 +0100)]
Update mailinglist subscription info.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Frank Ch. Eigler [Wed, 25 Mar 2020 01:30:02 +0000 (21:30 -0400)]
debuginfod-client thinko: non-default progressfn extra output
A previous commit changed the default_progressfn output format
to \rFOOBAR, to be terminated by an \n when the download finished.
The \n terminator was conditional on the wrong thing (env var
setting, rather than actual progressfn setting), so the \n could
be printed even if an app overrode the default.
Frank Ch. Eigler [Tue, 24 Mar 2020 19:01:58 +0000 (15:01 -0400)]
debuginfod-find: Correct error check for -v source combo
Hardcoding argv[2] is wrong in presence of -v option. Code
immediately following did the correct argv[] indexing already.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Mark Wielaard [Mon, 23 Mar 2020 22:57:51 +0000 (23:57 +0100)]
tests: Fix getphdrnum and run-lfs-symbols.sh testcase.
getphdrnum.c didn't include config.h which is why run-lfs-symbols.sh
flagged it for containing bad (non-lfs) symbols.
run-lfs-symbols.sh was still checking the libebl modules, which we
don't create anymore. But it didn't fail the test for non-existing
tests. Add some extra logging and explicitly check files exist.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Sun, 22 Mar 2020 22:14:24 +0000 (23:14 +0100)]
tests: Add testcase for elf_getphdrnum with more than 65535 entries.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Signed-off-by: Omar Sandoval <osandov@fb.com>
Omar Sandoval [Wed, 18 Mar 2020 20:18:51 +0000 (13:18 -0700)]
libelf: handle PN_XNUM in elf_getphdrnum before shdr 0 is cached
__elf_getphdrnum_rdlock() handles PN_XNUM by getting sh_info from
elf->state.elf{32,64}.scns.data[0].shdr.e{32,64}. However, that is only
a cache that may or may not have been populated by elf_begin() or
elf{32,64}_getshdr(); if it hasn't been cached yet, elf_getphdrnum()
returns 65535 (the value of PN_XNUM) instead. We should explicitly get
the shdr if it isn't cached.
Signed-off-by: Omar Sandoval <osandov@fb.com>
Frank Ch. Eigler [Sat, 21 Mar 2020 01:33:52 +0000 (21:33 -0400)]
debuginfod client API: add get_url function
This function lets a client know, during or after a progressfn
callback, what the url of the winning outgoing download is/was.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Fri, 20 Mar 2020 00:27:11 +0000 (20:27 -0400)]
debuginfod client API: add get/set user_data functions
Add a pair of functions to associate a void* parameter with a client
object. Requested by GDB team as a way to pass file names and such
user-interface data through to a progressfn callback.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Matthias Maennich [Fri, 20 Mar 2020 16:38:55 +0000 (17:38 +0100)]
libelf: {de,}compress: ensure zlib resource cleanup
__libelf_decompress would only cleanup zlib resources via inflateEnd()
in case inflating was successful, but would leak memory if not. Fix this
by calling inflateEnd() unconditionally.
__libelf_decompress did this all the time already, but called
deflateEnd() twice. That is not a (known) issue, but can be cleaned up
by ensuring all error paths use 'return deflate_cleanup' and the success
path calls deflateEnd() only once. Note, the deflate() needs to return
Z_STREAM_END to indicate we are done. Hence change the condition.
Fixes:
272018bba1f2 ("libelf: Add elf_compress and elf_compress_gnu.")
Signed-off-by: Matthias Maennich <maennich@google.com>
Aaron Merey [Tue, 3 Mar 2020 19:11:48 +0000 (14:11 -0500)]
debuginfod-client: Update cache_path when the new default path exists
Update cache_path with the path of the new default directory when this
directory already exists. Previously cache_path was updated only when
creating the new default directory and would otherwise be set to the
old default path.
Signed-off-by: Aaron Merey <amerey@redhat.com>
Mark Wielaard [Wed, 4 Mar 2020 10:35:38 +0000 (11:35 +0100)]
config: Remove rpm, add dpkg as Requires for debuginfod package in spec.
We no longer require rpm2cpio to unpack rpms.
We do now require dpkg-deb to unpack debs.
https://sourceware.org/bugzilla/show_bug.cgi?id=25583
Signed-off-by: Mark Wielaard <mark@klomp.org>
Aaron Merey [Thu, 27 Feb 2020 23:10:01 +0000 (18:10 -0500)]
debuginfod-client: default to XDG cache.
PR25502: debuginfod client should default to XDG cache
Location of client cache now defaults to $XDG_CACHE_HOME/debuginfod_client.
If XDG_CACHE_HOME is not set then fallback to $HOME/.cache/debuginfod_client.
Also maintain backwards compatibility with the previous default path-
if $HOME/.debuginfod_client_cache exists, use that instead of the new
defaults.
Signed-off-by: Aaron Merey <amerey@redhat.com>
Konrad Kleine [Wed, 26 Feb 2020 15:00:43 +0000 (10:00 -0500)]
debuginfod: file:// URLs: handle curl resp. code
When file:// is used for DEBUGINFOD_URLS, then the response code for a
successful server query is 0 and not 200.
Using file:// can be helpful when you want to test your
debuginfod-client integration against a mocked file tree that mimics the
HTTP URLs from the debuginfod server. This way you don't have to run the
debuginfod server at all.
Fixes https://sourceware.org/bugzilla/show_bug.cgi?id=25600
Signed-off-by: Konrad Kleine <kkleine@redhat.com>
Frank Ch. Eigler [Tue, 25 Feb 2020 19:27:33 +0000 (14:27 -0500)]
debuginfod PR25583: map -R to -Z.rpm
It was reported that libarchive (bsdtar) at least as far back as rhel7
(3.1.2) can natively process RPM files, so there's no need to mediate
those accesses through rpm2cpio. There's no noteworthy performance or
testing impact.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Signed-off-by: Mark Wielaard <mark@klomp.org>
Frank Ch. Eigler [Sat, 25 Jan 2020 23:43:07 +0000 (18:43 -0500)]
PR25375: fdcache prefetching to reduce repeated archive decompression
Introduce new option --fdcache-prefetch to accelerate repeated
queries from the same debuginfo archive.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Mark Wielaard [Fri, 21 Feb 2020 12:34:09 +0000 (13:34 +0100)]
tests: Explicitly unset DEBUGINFOD_URLS.
If DEBUGINFOD_URLS is set various tests will try to query the debuginfod
server which can stall the tests a bit. If other evironment variables
like DEBUGINFOD_PROGRESS are set it will make various tests fail because
the expected output doesn't match. Tests should PASS without needing a
debuginfod server, unless they test (and set) one themselves.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Aaron Merey [Tue, 11 Feb 2020 20:54:55 +0000 (15:54 -0500)]
PR25365: debuginfod-client: restrict cleanup to client-pattern files
Avoid deleting general files and directories in case a user
mis-sets $DEBUGINFOD_CACHE_PATH or accidentally moves a file
into the cache.
Signed-off-by: Aaron Merey <amerey@redhat.com>
Mark Wielaard [Sat, 8 Feb 2020 20:40:27 +0000 (21:40 +0100)]
readelf, elflint: Handle PT_GNU_PROPERTY.
binutils 2.32 ld emits a new PT_GNU_PROPERTY segment that overlaps
with the PT_NOTE segment covering the .note.gnu.property section data.
https://sourceware.org/bugzilla/show_bug.cgi?id=25511
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Fri, 7 Feb 2020 17:20:46 +0000 (18:20 +0100)]
elflint: Check sh_info for symtab isn't too big.
The sh_info field of the symtab says how many symbols are in the
section. Make sure at least that many symbols fit in the section.
Reported-by: Ulrich Drepper <drepper@redhat.com>
Signed-off-by: Mark Wielaard <mark@klomp.org>
Konrad Kleine [Wed, 12 Feb 2020 13:03:03 +0000 (14:03 +0100)]
debuginfod/doc: Fix typo debuginfo_progressfn_t -> debuginfod_progressfn_t
I've copied the type name for the progress function from the manual and
noticed that it didn't compile...
Signed-off-by: Konrad Kleine <kkleine@redhat.com>
Frank Ch. Eigler [Mon, 10 Feb 2020 19:33:54 +0000 (14:33 -0500)]
debuginfod testing: SIGUSR2 vs "groom" metric synch
Previous code did not account for a groom job that was already
completed at startup, so the SIGUSR2-triggered one may not have
completed in time for the test. The shell won the race condition on
most buildbot VMs, but a debian builder showed the error of our ways.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Thu, 6 Feb 2020 03:22:13 +0000 (22:22 -0500)]
debuginfod: testsuite for general archive (arch-linux) support
Tested analogously to how RPMs are.
Signed-Off-By: Frank Ch. Eigler <fche@redhat.com>
Eli Schwartz [Wed, 5 Feb 2020 22:09:30 +0000 (17:09 -0500)]
debuginfod arch-linux test binaries
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Mark Wielaard [Fri, 7 Feb 2020 23:17:25 +0000 (18:17 -0500)]
debuginfod: archive processing: handle -Z EXT=cat with direct fopen
Signed-off-by: Mark Wielaard <mark@klomp.org>
Frank Ch. Eigler [Wed, 5 Feb 2020 20:04:18 +0000 (15:04 -0500)]
debuginfod: generalized archive support
Add a '-Z EXT[=CMD]' option to debuginfod, which lets it scan any given
extension and run CMD on it to unwrap distro archives. For example,
for arch-linux pacman files, -Z '.tar.zst=zstdcat' lets debuginfod
grok debug and source content in split-debuginfo files.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Konrad Kleine [Mon, 10 Feb 2020 16:03:02 +0000 (17:03 +0100)]
debuginfod.h: fix typo in comment: legnth -> length
Signed-off-by: Konrad Kleine <kkleine@redhat.com>
Mark Wielaard [Fri, 24 Jan 2020 21:55:48 +0000 (22:55 +0100)]
libdwfl: Fix some GCC10 -Wnull-dereference issues.
GCC10 on some arches will warn about possible NULL derefences.
In the libdwfl linux-kernel-modules.c cases it might be caught already
by earlier calls to get_release (). But it is hard to see that will
really always happen. So do an explicit NULL check just in case.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Frank Ch. Eigler [Mon, 3 Feb 2020 19:21:15 +0000 (14:21 -0500)]
ChangeLog catchup for two recent commits
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Mon, 3 Feb 2020 19:11:53 +0000 (14:11 -0500)]
debuginfod test: assert curl & rpm2cpio during test prologue
On debian & ubuntu hosts, rpm2cpio is not available by default,
and heck, curl might not be either. Check for both these programs
during tests/run-debuginfod-find.sh startup, else exit-77 (skip).
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Mon, 3 Feb 2020 11:29:31 +0000 (06:29 -0500)]
PR24393: debuginfod configury: tolerate CXX= configure invocations
Previous code was fragile with respect to this case and needlessly
tolerant with non-c++11 compilers. New configury makes more muscular
assertion about this, to catch bad $CXX settings or poor language
support.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Wed, 22 Jan 2020 20:27:59 +0000 (15:27 -0500)]
debuginfod: diagnostics verbosity tweak
marxin reported "skipping hat= .... empty comp_dir" messages
were too chatty for the journnal.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Wed, 22 Jan 2020 18:22:31 +0000 (13:22 -0500)]
empty commit for irker test
Frank Ch. Eigler [Mon, 20 Jan 2020 20:37:33 +0000 (15:37 -0500)]
PR25394 cont'd: debuginfod testsuite fix for -USR1 timing
If a SIGUSR1 is sent before the initial traversal, it no longer
results in an extra traversal. That's a sensible effect. The
test case just needs to wait before the kill -USR1.
Frank Ch. Eigler [Mon, 20 Jan 2020 20:37:33 +0000 (15:37 -0500)]
PR25394 cont'd: debuginfod timing fix for fts-traversal thread
The new code neglected to set the last_rescan timestamp, leading
to overly frequent rescanning.
Frank Ch. Eigler [Mon, 20 Jan 2020 19:44:15 +0000 (14:44 -0500)]
PR25394 followup#2: debuginfod casting fixes
Buildbot still reports type warnings in time_t arithmetic.
Explicit (long)er cast pushed as obvious ... or is it? :-)
Frank Ch. Eigler [Mon, 20 Jan 2020 19:44:15 +0000 (14:44 -0500)]
PR25394 followup: debuginfod casting fixes
Buildbot reports type warnings in time_t arithmetic.
Explicit (long) cast pushed as obvious.
Frank Ch. Eigler [Mon, 20 Jan 2020 01:33:32 +0000 (20:33 -0500)]
PR25394: debuginfod mutex between grooming and scanning
Extended the work-queue concept with "idlers" - other threads that
block on the work queue until it becomes empty (rather than normal
consumers that block on it until it becomes non-empty). Use this
facility for the groomer thread to avoid working at the same time as
the scanner threads. Use this for the fts traversal thread for
similar reasons. One user-visible effect: response to SIGUSR1 and
SIGUSR2 will wait until the work queue runs empty, but the man page
was unspecific so does not need changing.
It's not obvious how to test this with a tests/ dataset so small that
scanning takes negligible time, so the former races are very tight.
P.S. We also evaluated using sqlite level transactions to isolate the
scanner thread groups-of-operations from the groomer. These
experiments failed to produce a nominally concurrent debuginfod,
having triggered "database locked" type errors. So we remain
single-threaded (fully serialized) at the sqlite API level.
Frank Ch. Eigler [Sat, 18 Jan 2020 20:32:40 +0000 (15:32 -0500)]
PR25415: fix debuginfod SEGV for empty search path list
Prevent triggering undefined behaviour of fts(3), which causes
a SEGV on fedora glibc 2.29.
Mark Wielaard [Thu, 16 Jan 2020 22:33:52 +0000 (23:33 +0100)]
nm: Fix nm --external sysv format output.
Partial revert of commit
66f4c37d497bdde040a33f299b12163f044b1bf2.
If index zero wasn't a real symbol it has already been filtered out
in show_symbols so don't skip it in show_symbols_sysv.
https://sourceware.org/bugzilla/show_bug.cgi?id=25227
Reported-by: Enzo Matsumiya <ematsumiya@suse.com>
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Fri, 10 Jan 2020 14:46:29 +0000 (15:46 +0100)]
doc: Fix DEBUGINFOD_PROGRESS description to just mention output on stderr.
An earlier variant of the default progress function could write to any
file. Which is still in the documentation. But the actual implementation
just uses stderr. Fix the documentation to match.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Wed, 8 Jan 2020 14:04:50 +0000 (15:04 +0100)]
libasm.h: Don't include libebl.h. Define an opaque Ebl handle.
Using libasm isn't really usable without a way to create an Ebl handle.
But we don't support libebl.h (and libebl itself). Just define the
Ebl handle as an opaque struct. Code that uses it needs to figure out
how to instantiate one itself (they cannot in any supportable way...)
Signed-off-by: Mark Wielaard <mark@klomp.org>
Frank Ch. Eigler [Sun, 12 Jan 2020 16:04:57 +0000 (11:04 -0500)]
elfutils.spec: typo fix
Removed dangling %endif.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Sun, 12 Jan 2020 16:00:20 +0000 (11:00 -0500)]
elfutils.spec: dependency typo fix
Commit
da0959cafda4 introduced a dependency on "debuginfod-client",
whereas it's really "elfutils-debuginfod-client".
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Fri, 27 Dec 2019 00:06:46 +0000 (19:06 -0500)]
debuginfod: extracted-from-archive file cache
Add a facility to service webapi and dwz/altdebug requests that
resolve to archives via a $TMPDIR file cache. This permits
instantaneous dwz resolution during -debuginfo rpm scanning, and also
instantanous duplicate webapi requests. The cache is limited both in
number of entries and in storage space. Heuristics provide
serviceable defaults.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Sat, 11 Jan 2020 21:05:46 +0000 (16:05 -0500)]
debuginfod: print U-A: and X-F-F: request headers
For an incoming webapi request, print two headers that should assist
in the administration of a debuginfod service. At fweimer's
suggestion, added a bit of filtering so the text is more reliably
parseable.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Wed, 1 Jan 2020 01:06:30 +0000 (20:06 -0500)]
debuginfod: rework threading model for file/archive scanning
We switch from a thread per supplied PATH, with a semaphore based
concurrency control, to a fixed number of worker threads collecting
the result of a plain directory traversal being put into a work queue.
This allows maximal continuous concurrency, even if the PATH
directories are dramatically differently sized. There is no more need
to use concurrency-motivated subdirectory wildcards for PATH entries:
just a single top level directory will work fast. doc & tests incl.
Mark Wielaard [Thu, 2 Jan 2020 16:02:42 +0000 (17:02 +0100)]
debuginfod: Use DEBUGINFOD_TIMEOUT as seconds to get at least 100K.
Use just one timeout using CURLOPT_LOW_SPEED_TIME (default 90 seconds)
and CURLOPT_LOW_SPEED_LIMIT (100K).
Signed-off-by: Mark Wielaard <mark@klomp.org>
Frank Ch. Eigler [Mon, 6 Jan 2020 09:29:21 +0000 (04:29 -0500)]
debuginfod: pass a distro-summary User-Agent request header
It may be useful for a debuginfod server operator to know what kinds
of clients make webapi requests. This is mainly as a
telemetry/diagnostic (though the data cannot be really trusted). It
may also be useful to automate downloading of distro packages to a
debuginfod server in the case of an unknown hex buildid. doc/testing
not affected as these are diagnostics.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Signed-off-by: Mark Wielaard <mjw@redhat.com>
Dmitry V. Levin [Wed, 1 Jan 2020 23:42:10 +0000 (02:42 +0300)]
Do not install libdebuginfod.pc unless debuginfod is enabled
Fixes:
288f6b199 ("debuginfod 1/2: client side")
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
Frank Ch. Eigler [Sat, 30 Nov 2019 15:46:44 +0000 (10:46 -0500)]
debuginfod server: support .deb/.ddeb archives
Add support for scanning .deb / .ddeb files, enabled with a new
command line option "-U". Using a synthetic .deb/.ddeb from a Ubuntu
18 machine, extend the debuginfod testsuite with some .deb processing,
if the dpkg-deb binary is installed.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Frank Ch. Eigler [Wed, 4 Dec 2019 20:51:12 +0000 (15:51 -0500)]
debuginfod: usability tweaks, incl. $DEBUGINFOD_PROGRESS client support
This facility allows a default progress-printing function to be
installed if the given environment variable is set. Some larger usage
experience (systemtap fetching kernels) indicates the default timeout
is too short, so forked it into a connection timeout (default short)
and a transfer timeout (default unlimited).
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Omar Sandoval [Thu, 12 Dec 2019 01:29:44 +0000 (17:29 -0800)]
libdwfl: remove broken coalescing logic in dwfl_report_segment
dwfl_report_segment has some logic that detects when a segment is
contiguous with the previously reported segment, in which case it's
supposed to coalesce them. However, in this case, it actually returns
without updating the segment array at all. As far as I can tell, this
has always been broken. It appears that no one uses the coalescing logic
anyways, as they pass IDENT as NULL. Let's just get rid of the logic and
add a test case.
Signed-off-by: Omar Sandoval <osandov@fb.com>
Omar Sandoval [Thu, 12 Dec 2019 00:23:28 +0000 (16:23 -0800)]
libcpu: Compile i386_lex.c with -Wno-implicit-fallthrough
elfutils is compiled with -Wimplicit-fallthrough=5, so the fallthrough
comment in i386_lex.c (generated by flex) doesn't prevent the implicit
fallthrough warning. Add -Wno-implicit-fallthrough to i386_lex_CFLAGS.
Signed-off-by: Omar Sandoval <osandov@fb.com>
Omar Sandoval [Thu, 12 Dec 2019 00:23:27 +0000 (16:23 -0800)]
configure: Fix -D_FORTIFY_SOURCE=2 check when CFLAGS contains -Wno-error
If CFLAGS contains -Wno-error, then the check for -D_FORTIFY_SOURCE=2
won't fail when appropriate. E.g., compiling with:
./configure CFLAGS='-Wno-error -O0' &&
Results in a flood of "_FORTIFY_SOURCE requires compiling with
optimization (-O)" warnings.
Make sure we add -Werror after the user-defined CFLAGS.
Signed-off-by: Omar Sandoval <osandov@fb.com>
Mark Wielaard [Mon, 9 Dec 2019 18:38:19 +0000 (19:38 +0100)]
debuginfod: Check the DEBUGINFOD_URLS environment variable early in client.
If the debuginfod-client isn't configured we should do as little
as possible. Simply return early with ENOSYS if no servers are
configured. This means we won't check
This does change the behavior of the debuginfod_find calls slightly.
Previously we would setup and check the cache if the given build-id
was valid. Which might have provided a result if an earlier client
had run with the same cache and valid server URLs which knew about
that particular build-id. Now we don't return any cached results
unless at least one server is configured.
This prevents selinux errors when the library is used in a confined
setup.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Fri, 6 Dec 2019 16:20:00 +0000 (17:20 +0100)]
Make sure to always build with build-ids.
We really need build-ids for various things. If the system compiler
doesn't generate build-ids warn and generate them anyway for both the
binaries and the tests.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Fri, 6 Dec 2019 14:48:00 +0000 (15:48 +0100)]
tests: Run strip under testrun in run-debuginfod-find.sh
Otherwise strip might run against the system libelf which might be too
old or missing.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Fri, 6 Dec 2019 14:39:38 +0000 (15:39 +0100)]
tests: Add break to avoid implicit-fallthrough warning
For some reason gcc might fail to recognize the assert (0) will never
return and emit an implicit-fallthrough warning. Just add a break to
silence it.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Thu, 5 Dec 2019 14:24:49 +0000 (15:24 +0100)]
config: Update elfutils.spec with explicit sub-package requires.
Consistently require version-release subpackages so that everything
is updated together. Technically this isn't always required (it is
for elfutils-libelf and elfutils-libs). But this makes things more
consistent and prevents multilib version mismatches.
Also update a Summary and description to match the current Fedora
text.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Thu, 5 Dec 2019 13:22:56 +0000 (14:22 +0100)]
tests: Run elfcompress under testrun in run-elfclassify.sh
Otherwise elfcompress might run against the system libelf which might
be too old or missing.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Tue, 3 Dec 2019 23:39:26 +0000 (00:39 +0100)]
debuginfod: Fix implicit conversion from 'CURLcode' to 'CURLMcode'
GCC10 warns when converting the value of one enum type into another:
debuginfod-client.c:530:24: error: implicit conversion from ‘CURLcode’
to ‘CURLMcode’ [-Werror=enum-conversion]
530 | curl_res = curl_easy_getinfo(target_handle,
| ^
libcurl has different error code enums. The "easy" interfaces return
a CURLcode error. The "multi" interface functions return a CURLMcode.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Thu, 5 Dec 2019 14:03:54 +0000 (15:03 +0100)]
libdwfl: Find and handle compressed vmlinuz image.
Both the dwfl_linux_kernel_find_elf callback and the
dwfl_linux_kernel_report_offline reporting function only handled
vmlinix images possibly compressed with .gz, .bz2 or .xz extension.
They did not find or handle the much more common vmlinuz compressed
images.
It is not completely clear why we didn't up to now. Support for
compressed ELF files was added in 2009 and the code was updated to
to try to find the .gz, .bz2 or .xz extension variants in 2011.
But not the vmlinuz named variant.
Reported-by: Aaron Merey <amerey@redhat.com>
Tested-by: Frank Ch. Eigler <fche@redhat.com>
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Thu, 28 Nov 2019 15:17:52 +0000 (16:17 +0100)]
config: Add an explicit Requires elfutils-debuginfod-client for debuginfod.
elfutils-debuginfod depends indirectly already on the client library.
Make that explicit so they are always updated in sync. Which isn't
technically required, but makes things clearer and simpler on updates.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Wed, 27 Nov 2019 02:21:49 +0000 (03:21 +0100)]
config: versioned libdebuginfod libraries should not be in -devel package.
The versioned library (links) should be in the runtime package.
The debuginfod-client-devel package should just contain the .so
(symlink). This isn't fatal, just sloppy. ldconfig will correct
the missing symlinks when it was missing.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Tue, 26 Nov 2019 22:34:27 +0000 (23:34 +0100)]
elfutils.spec.in: Sync with fedora spec, remove rhel/fedora specifics.
This makes the sample elfutils.spec file more like the fedora
elfutils.spec but with all fedora and rhel specifics removed.
In particular it introduces a new elfutils-libs subpackage and
updates the requires accordingly.
Other cleanups are removal of Group tags, remove dot at end of
Summary tags, add post/postun ldconfig for elfutils-libs and
elfutils debuginfod-client subpackages, remove default
defattr(-,root,root) for file lists and order binaries by name.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Tue, 26 Nov 2019 08:52:24 +0000 (09:52 +0100)]
Prepare for 0.178
Set version to 0.178.
Update NEWS and elfutils.spec.in.
Regenerate po/*.po files.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Tue, 26 Nov 2019 07:56:44 +0000 (08:56 +0100)]
Fix BUILD_STATIC build and enable gcov for debuginfod.
libdw now always needs -lpthread and -ld even when BUILD_STATIC.
BUILD_STATIC is only used when doing a gcov enabled build.
Enable gcov coverage also for debuginfod.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Mon, 25 Nov 2019 22:20:35 +0000 (23:20 +0100)]
elfutils.spec: Add BuildRequires: curl
The run-debuginfod-find.sh now relies on curl to fetch the metrics.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Frank Ch. Eigler [Mon, 25 Nov 2019 21:47:18 +0000 (16:47 -0500)]
debuginfod: Tweak groom cycle for memory freeing
Use the sqlite3_db_release_memory() call periodically. It should
have no effect except hopefully smaller server memory usage.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Mark Wielaard [Sun, 24 Nov 2019 16:38:18 +0000 (17:38 +0100)]
tests: Don't sleep in run-debuginfod-find.sh, but wait till ready.
Wait for the debuginfod server to finish and use the metrics to see
when a server is ready for the next test instead of sleeping. Also
remove DEBUGINFOD_TEST_WEBAPI_SLEEP.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Sun, 24 Nov 2019 13:22:17 +0000 (14:22 +0100)]
debuginfod: Add found_{executable,debuginfo,sourcerefs}_total metrics.
Keeps metrics of how many executables, debuginfo and sourcerefs were
found in total for file and rpm scanners.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Sat, 23 Nov 2019 13:09:42 +0000 (14:09 +0100)]
tests: Use 127.0.0.1 in run-debuginfod-find.sh instead of localhost.
The way curl does hostname lookups in multi_curl mode takes a very long
time under valgrind. Replace all localhosti hostnames with 127.0.0.1 so
no lookups are necessary.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Mark Wielaard [Sun, 17 Nov 2019 21:17:26 +0000 (22:17 +0100)]
debuginfod: add client context
Add a mandatory debuginfod_begin()/_end() call pair to manage a client
object that represents persistent but non-global state. From libdwfl,
dlopen the debuginfod.so client library early on. This hopefully
makes sure that the code (and the libcurl.so dependency) is loaded
before the program goes into multi-threaded mode.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Frank Ch. Eigler [Thu, 7 Nov 2019 09:00:16 +0000 (04:00 -0500)]
debuginfod 5: add /metrics endpoint
This webapi extensions allows admins to hook up debuginfod to a
prometheus-compatible monitoring system for general situational
statistics. The metrics are simple enough that local curl requests
can give a user a sense of what's going on. The metrics are
documented as unstable with respect to future versions.
Frank Ch. Eigler [Wed, 6 Nov 2019 23:53:31 +0000 (18:53 -0500)]
debuginfod 4: symbolic link traversal mode
In order to support file/rpm archives that are organized via symlink
trees, add an "-L" option to debuginfod, meaning about the same as for
find(1) or ls(1): to traverse rather than ignore symlinks.
Frank Ch. Eigler [Mon, 4 Nov 2019 21:33:09 +0000 (16:33 -0500)]
debuginfod 3/3: client interruptability
For interactive clients such as gdb, interruptibility is important for
usability during longer downloads. This patchset adds a
download-progress callback function to the debuginfod client library,
with which a caller app can interrupt a download as well as be
notified of its quantitative progress.
Frank Ch. Eigler [Mon, 28 Oct 2019 17:29:26 +0000 (13:29 -0400)]
debuginfod 2/2: server side
Add the server to the debuginfod/ subdirectory. This is a highly
multithreaded c++11 program (still buildable on rhel7's gcc 4.8,
which is only partly c++11 compliant). Includes an initial suite
of tests, man pages, and a sample systemd service.
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Signed-off-by: Aaron Merey <amerey@redhat.com>
Aaron Merey [Mon, 28 Oct 2019 17:29:26 +0000 (13:29 -0400)]
debuginfod 1/2: client side
Introduce the debuginfod/ subdirectory, containing the client for a
new debuginfo-over-http service, in shared-library and command-line
forms. Two functions in libdwfl make calls into the client library to
fetch elf/dwarf files by buildid, as a fallback. Instead of normal
dynamic linking (thus pulling in a variety of curl dependencies),
the libdwfl hooks use dlopen/dlsym. Server & tests coming in patch 2.
Signed-off-by: Aaron Merey <amerey@redhat.com>
Signed-off-by: Frank Ch. Eigler <fche@redhat.com>
Andreas Schwab [Thu, 14 Nov 2019 15:50:26 +0000 (16:50 +0100)]
run-large-elf-file.sh: skip if free memory information is not available
Andreas Schwab [Thu, 14 Nov 2019 13:54:58 +0000 (14:54 +0100)]
dwelf_elf_e_machine_string: Clear errno before calling strtol
Avoid spurious failure if errno is modified by any other library call in
the test.
Signed-off-by: Andreas Schwab <schwab@suse.de>
Mark Wielaard [Sun, 10 Nov 2019 23:15:55 +0000 (00:15 +0100)]
libdw: Introduce libdw_unalloc to stop Dwarf_Abbrev leaks.
In the case of reading an invalid abbrev or when reading an abbrev
concurrently the Dwarf_Abbrev just created might leak because it isn't
needed after all. Introduce libdw_unalloc and libdw_typed_unalloc to
unallocate such Dwarf_Abbrevs so they don't leak.
Signed-off-by: Mark Wielaard <mark@klomp.org>
Jonathon Anderson [Sun, 25 Aug 2019 15:07:00 +0000 (10:07 -0500)]
libdw: Rewrite the memory handler to be more robust.
Pthread's thread-local variables are highly limited, which makes
it difficult to use many Dwarfs. This replaces that with a
less efficient (or elegant) but more robust method.
Signed-off-by: Jonathon Anderson <jma14@rice.edu>
Jonathon Anderson [Sun, 25 Aug 2019 14:53:50 +0000 (09:53 -0500)]
Add configure options for Valgrind annotations.
Signed-off-by: Jonathon Anderson <jma14@rice.edu>
Srđan Milaković [Mon, 4 Nov 2019 16:39:35 +0000 (10:39 -0600)]
libdw: Add and use a concurrent version of the dynamic-size hash table.
Signed-off-by: Srđan Milaković <sm108@rice.edu>
Signed-off-by: Mark Wielaard <mark@klomp.org>
Jonathon Anderson [Fri, 1 Nov 2019 13:14:05 +0000 (08:14 -0500)]
libdw: Don't free uninitialized Dwarf_Abbrev_Hash's of "fake" CUs.
fake_{loc,loclists,addr}_cu are Dwarf_CUs that are created separate from
all the others, so their contents are minimal and mostly initialized by
a calloc. On dwarf_end however, they are freed through the same code path
as all the others, so they call DAH_free like all the others. This changes
that so that these three are exempt from DAH and split-DWARF matters, and
swaps the calloc for a malloc so Memcheck will catch any others.
Signed-off-by: Jonathon Anderson <jma14@rice.edu>
Omar Sandoval [Mon, 7 Oct 2019 09:05:36 +0000 (02:05 -0700)]
libdwfl: only use thread->unwound for initial frame
thread->unwound is only used for set_initial_registers (via
dwfl_thread_state_registers, dwfl_thread_state_register_pc, and a
special case in core_set_initial_registers). At that point,
thread->unwound is always the initial frame, so there's no need to
update it as we unwind the stack. Let's set it to NULL after we do the
initial setup. This simplifies the next change.
Signed-off-by: Omar Sandoval <osandov@fb.com>