Babeltrace Debug Info Analysis
-----------------------------

The babeltrace debug info analysis is a set of features which allow
mapping events from a trace to their location in source code or within
a binary file, based on their `ip` (instruction pointer) field.

Prerequisites
-------------

In order to install a version of babeltrace with debug info support,
the following libraries are required:

    * libelf
    * libdw

Both of them are provided by the elfutils project
(https://fedorahosted.org/elfutils/), and can be installed through the
elfutils package on Ubuntu, Debian, RHEL, and others.

Compiling for Debug Info Analysis
---------------------------------

Traced programs on which debug info analysis is to be performed can be
compiled in a few different ways, and still lead to useful results.

Ideally, one should compile the program in debug mode, which is
achieved on gcc by simply using the `-g` flag. This generates debug
information in the operating system's native format, which is then
used by babeltrace to map an event's source location to a file and
line number, and the name of the surrounding function.

Do note that only debug information in DWARF format, version 2 or
later, is currently supported by babeltrace. Use the `-gdwarf` or
`-gdwarf-(VERSION)` to explicitly generate DWARF debug information.

If the executable is not compiled with `-g` or an equivalent option
enabled, and thus no DWARF information is available, babeltrace will
use ELF symbols from the executable. Instead of providing source file,
line number and function name, however, the analysis will provide the
name of the nearest function symbol, plus an offset in bytes to the
location in the executable from which the event originated.

If the executable has neither ELF symbols nor DWARF information,
babeltrace will be unable to map an event to its source location and
will simply display the instruction pointer (address), as in prior
versions of babeltrace.

Getting the Right Tracer
------------------------

Debug info analysis is performed automatically by babeltrace, provided
the trace contains sufficient information. In order to be able to
trace all the necessary information, the following software is
required:

    * lttng-ust version 2.8.0 or later
    * lttng-tools, corresponding version

You can get these from source at:

    * https://github.com/lttng/lttng-ust
    * https://github.com/lttng/lttng-tools

Ubuntu users also have the option of installing via the LTTng daily
PPA:

    * https://launchpad.net/~lttng/+archive/ubuntu/daily

Tracing for Debug Info Analysis
-------------------------------

Babeltrace needs some extra information from contexts, namely ip and
vpid, to perform its analysis. These can be enabled after the creation
of a tracing session as follows:

   $ lttng add-context --userspace --type ip --type vpid

The tracing can then be performed as it normally would. Once the trace
is collected, it can the be read by babeltrace for analysis.

Analysing the Trace
-------------------

To perform the analysis, the trace can simply be read as it normally
would:

   $ babeltrace <path/to/trace>

Debug info analysis is on by default and will automatically print the
extra source location information if it can find it. A sample output
may look like this:

    [...]
    [16:18:15.845829429] (+0.000011697) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D550E, debug_info = { bin = "libhello.so+0x150e", func = "foo+0xa9", src = "libhello.c:7" }, vpid = 28719 }, { my_string_field = "hello, tracer", my_integer_field = 42 }
    [16:18:15.845841484] (+0.000012055) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D55E0, debug_info = { bin = "libhello.so+0x15e0", func = "bar+0xa9", src = "libhello.c:13" }, vpid = 28719 }, { my_string_field = "recoltes et semailles", my_integer_field = 57 }
    [16:18:15.845844852] (+0.000003368) colossus my_provider:my_other_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D56A5, debug_info = { bin = "libhello.so+0x16a5", func = "baz+0x9c", src = "libhello.c:20" }, vpid = 28719 }, { some_field = 1729 }
    [...]

The interesting part is the debug_info section of the context:

    debug_info = { bin = "libhello.so+0x150e", func = "foo+0xa9", src = "libhello.c:7" }

This is the expected output for events generated by an executable for
which DWARF information is available. It shows the name of the binary
and offset to the tracepoint, the name of the function containing the
tracepoint instance which generated the event ("foo") and the offset
within the function, and its source location ("libhello.c", line 7).

The second event in the sample output is of the same type
("my_first_tracepoint"), but it was generated by a different
tracepoint instance, hence the different source location (line 13) and
function ("bar").

The third event, of a different type, also shows debug information.

If DWARF info is absent, but ELF symbols are not stripped, the output
will instead look like this:

    [...]
    [16:18:15.845829429] (+0.000011697) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D550E, debug_info = { bin = "libhello.so+0x150e", func = "foo+0xa9" }, vpid = 28719 }, { my_string_field = "hello, tracer", my_integer_field = 42 }
    [16:18:15.845841484] (+0.000012055) colossus my_provider:my_first_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D55E0, debug_info = { bin = "libhello.so+0x15e0", func = "bar+0xa9" }, vpid = 28719 }, { my_string_field = "recoltes et semailles", my_integer_field = 57 }
    [16:18:15.845844852] (+0.000003368) colossus my_provider:my_other_tracepoint: { cpu_id = 2 }, { ip = 0x7F4D2A5D56A5, debug_info = { bin = "libhello.so+0x16a5", func = "baz+0x9c" }, vpid = 28719 }, { some_field = 1729 }
    [...]

The debug information now provides both binary and function location
information, but no source location information, as this requires
DWARF. The function names are in fact resolved using ELF symbols, so
there may be a discrepancy with those provided by DWARF (e.g. in the
case of mangling).

Paths to the binary and to the source location (if any) can be
expanded by using the command-line option
--debug-info-full-path. Otherwise, only the filename is shown.

Debug Info and Dynamic Loading
------------------------------

Babeltrace can resolve addresses of events originating from
dynamically loaded libraries, provided that some extra information is
collected at tracing time.

This can be achieved by preloading LTTng UST's libdl helper when
launching the program to be traced, like so:

    $ LD_PRELOAD="liblttng-ust-dl.so" <path/to/executable>

The tracing and analysis can now be performed as described in prior
sections, and events from tracepoints in dlopened libraries will be
resolved automatically by babeltrace.

Separate Debug Info
-------------------

It is possible to store DWARF debug information separate from an
executable, whether for concerns of file size, or simply to facilitate
the sharing of the debug information.

This is usually achieved via one of two mechanisms, namely build ID
and debug link. Both methods permit separate executables and debug
information. Their use and operation is described in GDB's
documentation at:

    https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html

Babeltrace will find separate debug files automatically, provided they
follow the requirements described in the documentation above. The
debug information lookup order is the same as GDB's, that is first
debug info is looked for within the executable, then through the build
ID method in the standard /usr/lib/debug/.build-id/ location, and
finally in the various possible debug link locations. The first debug
information file found is used.

The --debug-info-dir command-line option can be used to override the
default /usr/lib/debug/ directory used in build ID and debug link
lookups. Multiple debug info directories are currently not supported.

Target Prefix
-------------

The debug info analysis uses the paths to the executables as collected
during tracing as one mechanism to resolve DWARF or ELF
information. If the trace was taken on a separate machine, for
instance, it is possible to use --debug-info-target-prefix to specify
a prefix directory, representing the root of the target filesystem,
which will then be used for lookups. For example, if an executable was
located at /usr/bin/foo on the target system, it could be placed at
/home/efficios/target/usr/bin/foo on the system on which the analysis
is performed. In this case, the prefix is /home/efficios/target/.
