sdk/tools/heaptrack.git
9 years agoFix warning about missing debug infos in linux-vdso.so.N
Milian Wolff [Mon, 17 Nov 2014 16:16:45 +0000 (17:16 +0100)]
Fix warning about missing debug infos in linux-vdso.so.N

9 years agoCheck for main index while reading the data in.
Milian Wolff [Mon, 8 Sep 2014 08:37:23 +0000 (10:37 +0200)]
Check for main index while reading the data in.

9 years agoSkip operator new etc. when encountering the trace.
Milian Wolff [Mon, 8 Sep 2014 08:23:16 +0000 (10:23 +0200)]
Skip operator new etc. when encountering the trace.

This makes it much simpler and all backtraces are filtered automatically.

9 years agoWrite histogram to file specified by cli argument
Milian Wolff [Sat, 6 Sep 2014 22:42:18 +0000 (00:42 +0200)]
Write histogram to file specified by cli argument

9 years agoImprove the output to better understand what the numbers mean.
Milian Wolff [Sat, 6 Sep 2014 22:24:05 +0000 (00:24 +0200)]
Improve the output to better understand what the numbers mean.

9 years agoFix compilation on OpenSUSE
David Faure [Sat, 6 Sep 2014 21:22:24 +0000 (23:22 +0200)]
Fix compilation on OpenSUSE

9 years agoFix compilation when passing flags on the command line as well
David Faure [Sat, 6 Sep 2014 21:05:34 +0000 (23:05 +0200)]
Fix compilation when passing flags on the command line as well

9 years agoSkip all operator new locations, even when the IP differs.
Milian Wolff [Fri, 5 Sep 2014 08:18:25 +0000 (10:18 +0200)]
Skip all operator new locations, even when the IP differs.

This was occurring in a real-world data file for me, where
operator new(unsigned long) was encountered twice at different IPs.

9 years agoDon't get confused by operator<, <<, > and >>.
Milian Wolff [Thu, 4 Sep 2014 15:57:41 +0000 (17:57 +0200)]
Don't get confused by operator<, <<, > and >>.

9 years agoShare code.
Milian Wolff [Tue, 26 Aug 2014 00:44:08 +0000 (02:44 +0200)]
Share code.

9 years agoMake it possible to switch between merged and non-merged backtraces.
Milian Wolff [Tue, 26 Aug 2014 00:37:58 +0000 (02:37 +0200)]
Make it possible to switch between merged and non-merged backtraces.

9 years agoAdd some comments, esp. about implementing better trace merging.
Milian Wolff [Tue, 26 Aug 2014 00:30:18 +0000 (02:30 +0200)]
Add some comments, esp. about implementing better trace merging.

9 years agoremove obsolete debug output
Milian Wolff [Tue, 26 Aug 2014 00:26:23 +0000 (02:26 +0200)]
remove obsolete debug output

9 years agoAdd implicit values to program options.
Milian Wolff [Tue, 26 Aug 2014 00:21:05 +0000 (02:21 +0200)]
Add implicit values to program options.

9 years agoMerge code to print merged backtraces :)
Milian Wolff [Tue, 26 Aug 2014 00:16:43 +0000 (02:16 +0200)]
Merge code to print merged backtraces :)

C++' pointer-to-member is a really powerful feature - wow!

9 years agoPrettify output of top-allocator places.
Milian Wolff [Tue, 26 Aug 2014 00:01:56 +0000 (02:01 +0200)]
Prettify output of top-allocator places.

TODO: do something similar in the other places as well.

9 years agoMerge allocation data by instruction pointer.
Milian Wolff [Mon, 25 Aug 2014 23:45:03 +0000 (01:45 +0200)]
Merge allocation data by instruction pointer.

This is safe to do except for the peak memory consumption where
we cannot ignore the temporal dependency.

Otherwise this greatly improves the output as multiple calls to the
same function from different places get merged and thus show up
as one fat data point, instead of multiple thinner ones.

9 years agoSkip operator new(unsigned long) (and new[]) from start of backtraces.
Milian Wolff [Mon, 25 Aug 2014 23:04:13 +0000 (01:04 +0200)]
Skip operator new(unsigned long) (and new[]) from start of backtraces.

9 years agoMake indices typesafe.
Milian Wolff [Mon, 25 Aug 2014 22:21:52 +0000 (00:21 +0200)]
Make indices typesafe.

9 years agoFix name of variables.
Milian Wolff [Mon, 25 Aug 2014 22:01:51 +0000 (00:01 +0200)]
Fix name of variables.

9 years agoAdd a bit of recursion to the callgraph test debugee
Milian Wolff [Mon, 25 Aug 2014 21:56:32 +0000 (23:56 +0200)]
Add a bit of recursion to the callgraph test debugee

9 years agoFix regression and properly stop backtraces at main function again.
Milian Wolff [Mon, 25 Aug 2014 21:51:58 +0000 (23:51 +0200)]
Fix regression and properly stop backtraces at main function again.

9 years agoCleanup code and reduce duplication.
Milian Wolff [Mon, 25 Aug 2014 21:39:17 +0000 (23:39 +0200)]
Cleanup code and reduce duplication.

9 years agoSplit trace and instruction pointer data.
Milian Wolff [Mon, 25 Aug 2014 21:33:11 +0000 (23:33 +0200)]
Split trace and instruction pointer data.

The debug info for a ip is only required once, whereas a trace
contains the mapping between an IP and the parent IP, which can
be outputted many times in different combinations.

This way we don't call the libbacktrace functions as often and
decrease the file size, at the cost of memory consumption during
the debugee run (in heaptrack_interpret). The memory consumption
significantly decreases in heaptrack_print though.

9 years agoReduce amount of cout flushing.
Milian Wolff [Mon, 25 Aug 2014 20:19:55 +0000 (22:19 +0200)]
Reduce amount of cout flushing.

std::endl also flushes the output which is pretty bad for performance.
Instead, only use endl when it makes sense, i.e. between output blocks.

9 years agoCleanup code: encapsulate data reading and make more stuff private.
Milian Wolff [Mon, 25 Aug 2014 20:17:21 +0000 (22:17 +0200)]
Cleanup code: encapsulate data reading and make more stuff private.

9 years agoAdd test for call graphs, this should be merged better.
Milian Wolff [Sun, 24 Aug 2014 14:45:06 +0000 (16:45 +0200)]
Add test for call graphs, this should be merged better.

9 years agoImprove headings of different output sections.
Milian Wolff [Mon, 25 Aug 2014 17:18:12 +0000 (19:18 +0200)]
Improve headings of different output sections.

9 years agoTake dlclose into account and update module cache accordingly.
Milian Wolff [Mon, 25 Aug 2014 16:52:16 +0000 (18:52 +0200)]
Take dlclose into account and update module cache accordingly.

9 years agoMake it possible to validate the correctness of allocation overloads.
Milian Wolff [Mon, 25 Aug 2014 16:38:04 +0000 (18:38 +0200)]
Make it possible to validate the correctness of allocation overloads.

This is still flaky, apparently some function related to TLS allocation is missing.

9 years agoImprove output of -h
Milian Wolff [Mon, 25 Aug 2014 16:06:13 +0000 (18:06 +0200)]
Improve output of -h

9 years agoDo not clash --print-histogram with --help.
Milian Wolff [Mon, 25 Aug 2014 16:06:04 +0000 (18:06 +0200)]
Do not clash --print-histogram with --help.

9 years agoDo not link directly to libheaptrack.
Milian Wolff [Mon, 25 Aug 2014 15:57:47 +0000 (17:57 +0200)]
Do not link directly to libheaptrack.

The interpreter is important for useful data imo.

9 years agoShorten template identifiers by default.
Milian Wolff [Mon, 25 Aug 2014 15:57:28 +0000 (17:57 +0200)]
Shorten template identifiers by default.

9 years agoUse boost::program_options and make output configurable.
Milian Wolff [Mon, 25 Aug 2014 15:57:00 +0000 (17:57 +0200)]
Use boost::program_options and make output configurable.

9 years agoSimplify code, we use the std namespace.
Milian Wolff [Mon, 25 Aug 2014 13:46:37 +0000 (15:46 +0200)]
Simplify code, we use the std namespace.

9 years agoSimplify code a bit.
Milian Wolff [Mon, 25 Aug 2014 13:42:12 +0000 (15:42 +0200)]
Simplify code a bit.

9 years agoOutput pointer adresses without 0x prefix.
Milian Wolff [Mon, 25 Aug 2014 13:35:07 +0000 (15:35 +0200)]
Output pointer adresses without 0x prefix.

This just needlessly increases the file size.
We now cast to uintptr_t and output as %lx.

9 years agoDo not output function, file and line index for unknown locations.
Milian Wolff [Mon, 25 Aug 2014 13:24:41 +0000 (15:24 +0200)]
Do not output function, file and line index for unknown locations.

The null value is the default anyways and we do not need to
output that information and can thereby decrease the filesize a bit
for debugees with many occasions of missing debug information.

9 years agoAlso show individual peak allocations.
Milian Wolff [Sat, 23 Aug 2014 15:30:48 +0000 (17:30 +0200)]
Also show individual peak allocations.

9 years agoAlso show top-ten allocation places (ignoring deallocations).
Milian Wolff [Sat, 23 Aug 2014 15:23:55 +0000 (17:23 +0200)]
Also show top-ten allocation places (ignoring deallocations).

9 years agoUse %p to print out ptr addresses.
Milian Wolff [Sat, 23 Aug 2014 15:16:48 +0000 (17:16 +0200)]
Use %p to print out ptr addresses.

9 years agoAdd some debug output to show whats currently going on.
Milian Wolff [Mon, 11 Aug 2014 22:12:05 +0000 (00:12 +0200)]
Add some debug output to show whats currently going on.

9 years agoOptimize: don't sync cin/cout/cerr with stdio for our single-threaded c++ apps.
Milian Wolff [Wed, 13 Aug 2014 14:00:18 +0000 (16:00 +0200)]
Optimize: don't sync cin/cout/cerr with stdio for our single-threaded c++ apps.

9 years agoCleanup code a bit.
Milian Wolff [Wed, 13 Aug 2014 12:11:30 +0000 (14:11 +0200)]
Cleanup code a bit.

9 years agoAdd comment on why we use stdio here.
Milian Wolff [Wed, 13 Aug 2014 09:55:25 +0000 (11:55 +0200)]
Add comment on why we use stdio here.

9 years agoFixup check for executable debuggee.
Milian Wolff [Wed, 13 Aug 2014 08:52:17 +0000 (10:52 +0200)]
Fixup check for executable debuggee.

Call which first to also find the exe in PATH.

9 years agoAdd heaptrack --help.
Milian Wolff [Sun, 10 Aug 2014 15:15:12 +0000 (17:15 +0200)]
Add heaptrack --help.

9 years agoCleanup output messages to make it simpler to understand heaptrack.
Milian Wolff [Sun, 10 Aug 2014 14:57:49 +0000 (16:57 +0200)]
Cleanup output messages to make it simpler to understand heaptrack.

9 years agoInstall heaptrack shell script and use the proper paths therein.
Milian Wolff [Sun, 10 Aug 2014 14:50:57 +0000 (16:50 +0200)]
Install heaptrack shell script and use the proper paths therein.

We get the relative paths from the install directory to our lib
and libexec target folders and write that into the shell script.

9 years agoSome more cleanup and restructuring.
Milian Wolff [Wed, 6 Aug 2014 21:19:14 +0000 (23:19 +0200)]
Some more cleanup and restructuring.

9 years agoRename to heaptrack and start cleaning up the code structure.
Milian Wolff [Wed, 6 Aug 2014 20:49:54 +0000 (22:49 +0200)]
Rename to heaptrack and start cleaning up the code structure.

9 years agoRemove mallocinfo code, it's in the old repo.
Milian Wolff [Wed, 6 Aug 2014 20:31:56 +0000 (22:31 +0200)]
Remove mallocinfo code, it's in the old repo.

9 years agoDirectly evaluate the backtrace data.
Milian Wolff [Sun, 3 Aug 2014 19:22:47 +0000 (21:22 +0200)]
Directly evaluate the backtrace data.

This makes the final data interpretable even when the binary or
any library changes in between. It also makes it possible to ship
the data file to an external computer for further evaluation.

10 years agoHide anything above __libc_start_main as its garbage.
Milian Wolff [Sat, 7 Jun 2014 00:45:28 +0000 (02:45 +0200)]
Hide anything above __libc_start_main as its garbage.

10 years agoUse named pipe to compress output data on the fly.
Milian Wolff [Thu, 5 Jun 2014 13:31:48 +0000 (15:31 +0200)]
Use named pipe to compress output data on the fly.

This is plain awesome imo :) Really easy to get implemented and
the evaluation code is also more or less untouched thanks to Boost's
iostreams library with nice support for gzip data.

10 years agoFixup example code, use proper delete[].
Milian Wolff [Thu, 5 Jun 2014 10:22:23 +0000 (12:22 +0200)]
Fixup example code, use proper delete[].

10 years agoDo not output new IP id.
Milian Wolff [Wed, 4 Jun 2014 17:47:02 +0000 (19:47 +0200)]
Do not output new IP id.

The data is outputted in a sequentail way and thus this information is obsolete.

10 years agoDirectly link malloctrace into test executables.
Milian Wolff [Wed, 4 Jun 2014 17:43:08 +0000 (19:43 +0200)]
Directly link malloctrace into test executables.

10 years agoOutput all numbers in hex form.
Milian Wolff [Wed, 4 Jun 2014 16:37:43 +0000 (18:37 +0200)]
Output all numbers in hex form.

This reduces the output data file size slightly and simplifies the
evaluation code as we do not have to switch between different
formats all the time.

10 years agoDo not output 0x prefix for addresses.
Milian Wolff [Wed, 4 Jun 2014 16:34:04 +0000 (18:34 +0200)]
Do not output 0x prefix for addresses.

This reduces the size of the output data files.

10 years agoDecrease runtime memory overhead by getting rid of AllocationInfo.
Milian Wolff [Wed, 4 Jun 2014 16:25:30 +0000 (18:25 +0200)]
Decrease runtime memory overhead by getting rid of AllocationInfo.

We now output every allocation's ptr and only output it for a free.
This not only reduces the runtime CPU overhead, but also drastically
reduces the memory overhead as the relatively large AllocationInfo
map is not required anymore.

The size of the output data files increases, but the cost seems to
be bearable and worth the reduces memory overhead imo.

10 years agorealloc with nullptr is the same as pure malloc.
Milian Wolff [Wed, 4 Jun 2014 13:03:46 +0000 (15:03 +0200)]
realloc with nullptr is the same as pure malloc.

In such a case, don't handle the realloc like a free followed by a
malloc. It's just the malloc then.

10 years agoStore backtrace information in a memory efficient tree structure.
Milian Wolff [Wed, 4 Jun 2014 12:18:14 +0000 (14:18 +0200)]
Store backtrace information in a memory efficient tree structure.

Instead of storing all individual traces that ever call any of the
allocation functions, we now store a tree of instruction pointers.

The tree is constructed from top-to-bottom and each node representing
an instruction pointer gets associated with a unique index. This
index is enough to reconstruct the backtrace from that point by
iterating over the parent indices, which is now done in the evaluation
program.

A nice side effect is that the data file becomes smaller for bigger
application evaluation runs, as common parts at the beginning of
every allocation trace can be implicitly shared.

10 years agoCleanup code and make it explicitly work with the unw_backtrace api.
Milian Wolff [Mon, 2 Jun 2014 18:33:23 +0000 (20:33 +0200)]
Cleanup code and make it explicitly work with the unw_backtrace api.

10 years agoUse unw_backtrace instead of manual unw_step loop.
Milian Wolff [Mon, 2 Jun 2014 17:20:37 +0000 (19:20 +0200)]
Use unw_backtrace instead of manual unw_step loop.

The former is potentially much more performant but may require some
work on libunwind, which I'm currently investigating.

10 years agoAdd warning about unknown executable format
Milian Wolff [Mon, 2 Jun 2014 14:50:02 +0000 (16:50 +0200)]
Add warning about unknown executable format

10 years agoEnable per-thread unwind caching.
Milian Wolff [Sat, 31 May 2014 23:34:32 +0000 (01:34 +0200)]
Enable per-thread unwind caching.

This is much more performant as libunwind will not also add another
serialization point, it will also trigger calls to sigprocmask for
every mutex lock/unlock.

10 years agoRefactor DUMP_MALLOC_TRACE_OUTPUT handling.
Milian Wolff [Sat, 31 May 2014 23:16:54 +0000 (01:16 +0200)]
Refactor DUMP_MALLOC_TRACE_OUTPUT handling.

Now we can set it to stderr or stdout/- to print directly to the
console. Other files get $$ replaced by the PID of the host app.
This allows us to set DUMP_MALLOC_TRACE_OUTPUT to /dev/null to
disable output.

10 years agoDo not add garbadge null-ip's from end-of-line.
Milian Wolff [Sat, 31 May 2014 23:16:01 +0000 (01:16 +0200)]
Do not add garbadge null-ip's from end-of-line.

10 years agoPrepend output file with 'malloctrace.' even when env var is not set.
Milian Wolff [Sat, 31 May 2014 22:58:21 +0000 (00:58 +0200)]
Prepend output file with 'malloctrace.' even when env var is not set.

Also add some code comments on future things to do.

10 years agoSimplify code and reduce overhead at trace time.
Milian Wolff [Sat, 31 May 2014 22:45:57 +0000 (00:45 +0200)]
Simplify code and reduce overhead at trace time.

We now evaluate the IP to module connection at evaluation time,
rather than during tracing to reduce the overhead. This also gets
rid of the rather large IP cache, thereby reducing the memory
overhead.

10 years agoSave ptr start and end address instead of start + size.
Milian Wolff [Sat, 31 May 2014 22:04:56 +0000 (00:04 +0200)]
Save ptr start and end address instead of start + size.

Also output that size.

10 years agoReturn early for now, without bzipping output files.
Milian Wolff [Mon, 26 May 2014 19:08:57 +0000 (21:08 +0200)]
Return early for now, without bzipping output files.

10 years agoAdd a threaded test application
Milian Wolff [Mon, 26 May 2014 19:08:28 +0000 (21:08 +0200)]
Add a threaded test application

10 years agoExtract some more information on the allocation data, such as a size histogram.
Milian Wolff [Fri, 23 May 2014 13:04:42 +0000 (15:04 +0200)]
Extract some more information on the allocation data, such as a size histogram.

10 years agoUse stack-allocated trace buffer.
Milian Wolff [Wed, 21 May 2014 20:39:46 +0000 (22:39 +0200)]
Use stack-allocated trace buffer.

This seems to be required in multithreaded applications as otherwise
I get random crashes or deadlocks...

10 years agoBreak when encountering zero-IP.
Milian Wolff [Wed, 28 May 2014 12:37:33 +0000 (14:37 +0200)]
Break when encountering zero-IP.

This is senseless imo. Also, continuing would be very slow as
libunwind would try to ask dwarf info for the next IP for that
address which is bad, performance wise. And again, it will never
find anything and we just pay for nothing.

10 years agoTake trace size into account for hash generation.
Milian Wolff [Wed, 28 May 2014 11:19:52 +0000 (13:19 +0200)]
Take trace size into account for hash generation.

10 years agoSkip instructions which we cannot make sense of.
Milian Wolff [Wed, 21 May 2014 19:59:04 +0000 (21:59 +0200)]
Skip instructions which we cannot make sense of.

10 years agoFixup environment passing via gdb to the library.
Milian Wolff [Wed, 21 May 2014 19:41:29 +0000 (21:41 +0200)]
Fixup environment passing via gdb to the library.

10 years agoAbort on failure.
Milian Wolff [Wed, 21 May 2014 19:41:17 +0000 (21:41 +0200)]
Abort on failure.

10 years agoPreallocate trace buffer once.
Milian Wolff [Wed, 21 May 2014 19:41:09 +0000 (21:41 +0200)]
Preallocate trace buffer once.

10 years agoCleanup environment
Milian Wolff [Wed, 21 May 2014 14:22:06 +0000 (16:22 +0200)]
Cleanup environment

10 years agoDisable error messages when dwarf info is missing.
Milian Wolff [Wed, 21 May 2014 13:57:45 +0000 (15:57 +0200)]
Disable error messages when dwarf info is missing.

10 years agoGet file and line from dwarf info if possible.
Milian Wolff [Wed, 21 May 2014 13:56:00 +0000 (15:56 +0200)]
Get file and line from dwarf info if possible.

10 years agoUpdate trace script.
Milian Wolff [Wed, 21 May 2014 13:19:29 +0000 (15:19 +0200)]
Update trace script.

10 years agoUse vectors everywhere and be pedantic about input data.
Milian Wolff [Wed, 21 May 2014 13:13:56 +0000 (15:13 +0200)]
Use vectors everywhere and be pedantic about input data.

10 years agoPrint total leaked bytes and number of allocations.
Milian Wolff [Wed, 21 May 2014 12:58:28 +0000 (14:58 +0200)]
Print total leaked bytes and number of allocations.

10 years agoDon't do manual reserve calls, its slower than the default way.
Milian Wolff [Wed, 21 May 2014 12:58:09 +0000 (14:58 +0200)]
Don't do manual reserve calls, its slower than the default way.

10 years agoIntroduce a memleak in the test app.
Milian Wolff [Wed, 21 May 2014 12:42:33 +0000 (14:42 +0200)]
Introduce a memleak in the test app.

10 years agoPrint top allocators and all leaks in a sorted way.
Milian Wolff [Wed, 21 May 2014 12:42:12 +0000 (14:42 +0200)]
Print top allocators and all leaks in a sorted way.

10 years agoRefactor code: work with just a single file and use vector for traces.
Milian Wolff [Wed, 21 May 2014 12:34:04 +0000 (14:34 +0200)]
Refactor code: work with just a single file and use vector for traces.

The latter will allow us to sort the traces later on for evaluation
purposes easily.

10 years agoDemangle function names.
Milian Wolff [Wed, 21 May 2014 12:33:44 +0000 (14:33 +0200)]
Demangle function names.

10 years agoRefactor malloc tracing, synchronize most of the stuff after all.
Milian Wolff [Wed, 21 May 2014 10:04:59 +0000 (12:04 +0200)]
Refactor malloc tracing, synchronize most of the stuff after all.

I completely forgot before that a free can follow a malloc in a
different thread which is perfectly fine. To support this, we must
centralize our data handler after all.

This is not too bad as we can merge the multi-thread data into one
file that way and have it interleaved properly which will be good
for later evaluation.

From a contention point of view, I tried to minimize it as much as
possible. Further evaluation and profiling will be done later.

10 years agoBring back trace and ip cache to reduce generated filesize.
Milian Wolff [Tue, 20 May 2014 13:08:14 +0000 (15:08 +0200)]
Bring back trace and ip cache to reduce generated filesize.

Also speeds up things as repeated large prints are actually quite
slow. Furthermore we extend the tool to actually interpret the data.

10 years agoCheck for valid malloc/free before handling it.
Milian Wolff [Tue, 20 May 2014 09:48:58 +0000 (11:48 +0200)]
Check for valid malloc/free before handling it.

10 years agoReserve place for 32 modules.
Milian Wolff [Tue, 20 May 2014 08:06:49 +0000 (10:06 +0200)]
Reserve place for 32 modules.

10 years agoReimplement backtrace reading and offload dwarf info interpretation.
Milian Wolff [Tue, 20 May 2014 07:33:29 +0000 (09:33 +0200)]
Reimplement backtrace reading and offload dwarf info interpretation.

This makes the profiling run much faster. The later evaluation is
not so performance critical then thereafter.

10 years agoFix regression I introduced
Milian Wolff [Tue, 20 May 2014 07:32:22 +0000 (09:32 +0200)]
Fix regression I introduced