[TOC]
Minidump is a file format for storing parts of a program's state for later inspection. Microsoft's Documentation defines the format though the Rust Documentation is sometimes easier to navigate. The minidump implementation and tools used by Chrome are Breakpad and Crashpad. However, the tools of interest here are from the Breakpad project.
When Chrome crashes it writes out a minidump file. The minidump file is written
under the application product directory. On Linux this is
<XDG_CONFIG_HOME>/<app-name>/Crash Reports. The default for XDG_CONFIG_HOME
is ~.config. Common <app-name>s are chromium, google-chrome,
google-chrome-beta, and google-chrome-unstable. A typical example is
~.config/google-chrome/Crash Reports. When a minidump is uploaded it will be
moved between the new, pending, and completed subdirectories. The minidump
file is named something like <uuid>.dmp. If the minidump is uploaded to the
crash reporting system, the <uuid>.meta file will contain the crash report id.
Those with access can find the uploaded report at go/crash/<report-id>, where
the minidump file will be available with a name like
upload_file_minidump-<report-id>.dmp.
To create a minidump, you can use a local build of Chromium or a release
version of Chrome. Run the browser with the environment variable
CHROME_HEADLESS=1, which enables crash dumping but prevents crash dumps from
being uploaded and deleted. Something like $ env CHROME_HEADLESS=1 ./out/debug/chrome-wrapper or $ env CHROME_HEADLESS=1 /opt/google/chrome/google-chrome. Navigate to chrome://crash to trigger a
crash in the renderer process or reproduce your current crash bug. A crash dump
file should appear in the Crash Reports directory.
To get an idea about what is in a minidump file, install the Okteta hex editor and add the Minidump Structure Definition. Open the minidump previously created and explore the information it contains.
One quirk to notice is that there is a ThreadListStream which contains
MINIDUMP_THREADs which contain a MINIDUMP_THREAD_CONTEXT and an
ExceptionStream which also contains a MINIDUMP_THREAD_CONTEXT. The thread
list contains the thread contexts as they existed when the crash reporter was
running. The exception's thread context is the state of the crashing thread at
the time that it crashed, which is generally the most interesting thread
context. When using the Breakpad tools for Linux (like minidump_stackwalk and
minidump-2-core) the thread context from the exception record is used in place
of the thread context associated with the corresponding thread.
Each MINIDUMP_THREAD contains a StackMemoryRva which is a reference to to a
copy of the stack on that thread at the time the crash handler was running.
Parsing a stack usefully requires additional debug information.
minidump_stackwalk or a debugger may be used to parse the stack memory to
create a usable trace.
From a Chromium checkout ninja -C out/release minidump-2-core minidump_stackwalk dump_syms. From a Breakpad
checkout make. It can
be useful to use Breakpad directly on machines where one does not already have
a Chromium checkout.
When working at this level, one will also want to have readelf and
objdump available, which are available from most distributions.
In addition to the minidump, you will need the exact executables of Chromium or
Chrome which produced the minidump and those executable's symbols. If the
minidump was created locally, you already have the executables. Symbols for
Google Chrome's official builds are available from
https://edgedl.me.gvt1.com/chrome/linux/symbols/google-chrome-debug-info-linux64-${VERSION}.zip
where ${VERSION} is any version of Google Chrome that has recently been served
to Stable, Beta, or Unstable (Dev) channels on Linux, like 114.0.5696.0. Those
with access can find both executables and symbols for unreleased builds at
go/chrome-symbols.
For symbols outside of Chrome (like when the crash is happening in a shared library) then symbols for the files of interest must be found. If the minidump was created locally then install the symbol packages from your distribution. If not, you will need to track down the exact symbol files, which can be an interesting exercise. For some distributions using the debuginfod system can be quite helpful.
To ensure the correct binaries and debug symbols are used, the minidump contains
the build-id for each loaded module in the ModuleListStream in the
CvRecordRva's Signature. This build-id is matched against a note
section of type NT_GNU_BUILD_ID, usually named .note.gnu.build-id in the
executable and symbol files. This note can be inspected with readelf -n <file> like readelf -n chrome or readelf -n chrome.debug and looking for
the .note.gnu.build-id section. readelf reports the Build ID as the flat
bytes in the note, but Breakpad binaries like stackwalk_minidimp and
dump_syms will report and expect this truncated to a formatted Type 2 GUID
(without dashes). This means readelf will output a <build-id> like
33221100554477668899AABBCCDDEEFFXXXXXXXX but crashpad binaries will expect and
report this as a <build-uuid> of 00112233445566778899AABBCCDDEEFF.
The .gnu_debuglink section states which debug symbol file to use with a
striped binary. For example readelf --string-dump=.gnu_debuglink chrome
produces chrome.debug. This can be helpful to know for libraries with
interesting debug symbol setup, like libc.so.6.
Given a minidump with the name mini.dmp
minidump_stackwalk mini.dmp > mini.stackwalk.nosym
This will produce a mostly unsymbolized summary of the crash. To symbolize, look
toward the bottom of the output for WARNING: No symbols, <file>, <build-uuid>.
For each <file> which is of interest, mkdir -p symbols/<file>/<build-uuid> then
dump_syms <file> <directory-with-file.debug> > symbols/<file>/<build-uuid>/<file>.sym. Ensure this output <file>.sym
contains the expected <build-uuid>. Then re-run minidump_stackwalk but
with the symbols directory, like minidump_stackwalk mini.dmp symbols/ > mini.stackwalk.
The output of minidump_stackwalk is often quite useful and enough to track
down many issues. However, it does not fully use all of the information from
DWARF, so it is possible sometimes to get much better stack traces from a full
debugger like gdb. This is particularly true when functions have been
aggressively inlined.
minidump-2-core mini.dmp > mini.core
This works best if the binaries, symbols, and core files are all in different
directories to prevent gdb from automatically loading them into the wrong
locations. This is also generally necessary when using a system installed
version of Chrome. For full details see the gdb
manual.
The easiest way is to rename and move the .debug files to a directory structure
like <debugdir>/debug/.build-id/nn/nnnnnnnn.debug where nn are the first
two hex characters of the build-id, and nnnnnnnn are the rest of the hex
characters of the build-id. Note that this build-id is exactly what is
reported by readelf -n <binary> | grep "Build ID" and not the build-uuid
used by Breakpad. Then in gdb use show debug-file-directory to get the
<previous-directories> and set debug-file-directory <previous-directories>:<debugdir>/debug.
The offsets used here are the offsets of the corresponding module from the
output of minidump_stackwalk or (equivalently) the value of
ModuleListStream::Modules[]::BaseOfImage from the minidump file (which can be
read with the structure definition).
$ gdb
(gdb) file <executable>
(gdb) show debug-file-directory
<previous directories>
(gdb) set debug-file-directory <previous directories>:<debugdir>/debug
(gdb) symbol-file <executable> -o <executable-offset>
(gdb) core-file <mini.core>
Running the commands in this order avoids needing to load the symbols twice and
maps the <executable> to the expected location.
To add an additional shared library it is possible to
(gdb) add-symbol-file <shared-library> -o <shared-library-offset>
Source paths in Chrome builds are relative to the out/<build> directory. If you
have a Chromium checkout at or around when the Chrome build was created, it can
be added to the debugger search path, like
(gdb) directory <path-to-chromium>/chromium/src/out/<build>/