General Debugging

Logging

Although it isn't the most advanced procedure, sometimes logging state to the screen is sufficient to figure out a problem. The following outlines how to add logging statements to a desired test.

Procedure

1. Make sure the following line of code is at the top of your test to “route” log statements to terminal: solana_logger::setup()

2. Add any of the following statements to display whatever information you'd like: error!(), info!(), warn!(), debug!(), trace!()

   • Note that the codebase contains these statements which will also be displayed on screen. As such, if you want to reduce "noise", it may be preferable to choose a higher level (error or info).

3. Set the RUST_LOG environment variable, making sure to choose a level equal to or lower than the statement in previous step.

   • For example: export RUST_LOG=solana=error
   • The level can also be specified for a single command: RUST_LOG=solana=error cargo test your_test
   • The level can also be specified on a per-crate basis, such as: RUST_LOG=solana_ledger=trace,solana_core=info cargo test your_test

Adding log statements to executables, such as solana-validator, is similar. Same as before, make sure your executable is run with the proper log level, and consider adding any prints at a high level to avoid excessive output.

For more info, check out the official documentation: https://docs.rs/log/latest/log/index.html

Debugger

Running a failing unit test inside a debugger provides much more detail, and is a useful tool to have in your tool belt. The following outlines the process for setting up and running unit tests with rust-lldb.

Procedure

1. Build the code in debug mode (debug mode is the default mode for cargo build)
   • This is obvious, but worthwhile to mention. It's likely that the code has already built and the tests have been run, which prompted the reason to run the test(s) in a debugger.
2. Locate the binary to debug
   • The unit test binaries are in target/debug/deps.
   • Binaries have a hash appended as a string to the end of the binary's filename.
3. Launch rust-lldb
   • $ rust-lldb target/debug/deps/solana_binary_to_debug-SOMEHASH
4. Set a breakpoint
   • (lldb) breakpoint set -r test_name_here
5. Run
   • (lldb) run -- test_name_here

Core Dumps

Inspecting core dumps can be useful for examining the state of a program at the time of a crash.

Generating Core Dumps

System configuration may not allow the generation of core dumps by default. Perform the following steps to allow generation of core dumps:

1. Check that a program is configured to handle core patterns by printing the contents of /proc/sys/kernel/core_pattern. For example,

$ cat /proc/sys/kernel/core_pattern 
|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h

2. Confirm that the service in step above is enabled; it might be disabled by default.
3. Confirm that the limit for created core files is large enough.

$ ulimit -c
unlimited

If the above is 0, no core files will be created.