Add ASIO completion handler tracking #2681
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Boost asio has a way to output diagnostic information about completion handlers by defining
BOOST_ASIO_ENABLE_HANDLER_TRACKING
you get output like so:Information about what these entries mean can be found here:
https://www.boost.org/doc/libs/1_72_0/doc/html/boost_asio/overview/core/handler_tracking.html
But they can also be turned into a graph: (described later)
By itself it's not that useful as we really care about how long these io thread completion handlers are taking. I've modified the
thread_runner
andsystem
polling to now report any completion handlers which take longer thanNANO_ASIO_HANDLER_TRACKING
(new CMake variable). It outputs the timestamp which can be cross-referenced with the images/text to find what caused it. It is recommended to use less io_threads if trying to diagnose issues with these as multiple completion handlers cause be executing in parallel.Proposed mode of operation:
1 -
cmake -DNANO_ASIO_HANDLER_TRACKING=10
(report anything held for 10ms or greater)2 - Lower
io_threads
in config-node.toml3 - Run suspect code and pipe error output (asio uses that stream):
./core_test --gtest_filter=suspect_test 2> asio_output.txt
4 - For any "io_thread held for ??ms" messages, cross-reference with text and remove extraneous text surrounding it (see below)
5 - Generate image
perl handlerviz.pl < asio_output.txt | dot -Tpng > output.png
. If theasio_output.txt
file is large this can crash or take a very long time, as well as make the image unreadable.dot
binary is from GraphViz and can be installed for free on all platforms.This can be a good first step in helping diagnose issues and inform of any completion handlers taking too long. A further step may be to wrap all completion handlers and break before exiting but that's a much larger effort with greater.