Support ELF core dump creation on guest crash #417

dblnz · 2025-04-14T21:01:10Z

Purpose

Add support for core dump file generation when a guest crashes Improve experience when an unexpected VM exit or crash occurs #310

Description

This depends on PR.
This closes #310

Copilot

Pull Request Overview

This PR adds support for ELF core dump generation on guest crash by introducing crashdump metadata and integrating it into the hypervisor initialization and crashdump context API. Key changes include:

Introducing the SandboxMetadata structure and propagating a new crashdump parameter across hypervisor modules.
Implementing a crashdump_context method that gathers register and memory state from various hypervisors.
Integrating the elfcore dependency and functionality in a new crashdump module.

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/hyperlight_host/src/sandbox/uninitialized_evolve.rs	Passes crashdump metadata to hypervisor handler initialization.
src/hyperlight_host/src/sandbox/uninitialized.rs	Introduces SandboxMetadata to support crash dump context.
src/hyperlight_host/src/hypervisor/windows_hypervisor_platform.rs	Adds a get_xsave function and crash dump integration.
src/hyperlight_host/src/hypervisor/mod.rs	Updates the hypervisor trait to use crashdump_context instead of get_memory_regions.
src/hyperlight_host/src/hypervisor/kvm.rs	Implements crashdump_context with register mapping and xsave conversion.
src/hyperlight_host/src/hypervisor/inprocess.rs	Marks crashdump_context as unsupported in in-process mode.
src/hyperlight_host/src/hypervisor/hyperv_windows.rs	Integrates crashdump_context and replaces the get_memory_regions implementation.
src/hyperlight_host/src/hypervisor/hyperv_linux.rs	Adds a crashdump_context implementation with register mapping and xsave handling.
src/hyperlight_host/src/hypervisor/crashdump.rs	Implements core dump creation using elfcore and adds custom memory reading.
src/hyperlight_host/Cargo.toml	Adds the elfcore dependency for ELF core dump support.

Comments suppressed due to low confidence (2)

src/hyperlight_host/src/hypervisor/hyperv_windows.rs:535

Using unwrap() for OsString to String conversion may lead to a runtime panic if the conversion fails; consider handling conversion errors or using to_string_lossy() instead.

Path::new(&path).file_name().map(|name| name.to_os_string().into_string().unwrap()).unwrap()

src/hyperlight_host/src/hypervisor/hyperv_linux.rs:707

Using unwrap() for converting an OsString to a String here might result in a panic if the conversion fails; consider using a more robust conversion strategy such as to_string_lossy().

Path::new(&path).file_name().map(|name| name.to_os_string().into_string().unwrap()).unwrap()

dblnz · 2025-04-28T16:00:15Z

src/hyperlight_host/Cargo.toml

@@ -46,6 +46,7 @@ tempfile = { version = "3.19", optional = true }
 serde_yaml = "0.9"
 anyhow = "1.0"
 metrics = "0.24.2"
+elfcore = { git = "https://github.com/dblnz/elfcore.git", branch = "split-linux-impl-from-elfcore" }


This needs to be change after the elfcore PR is merged

Copilot

Pull Request Overview

This PR adds support for generating ELF core dump files when a guest crashes. The changes integrate a new crashdump feature across the codebase by introducing a new SandboxMetadata structure, updating various hypervisor implementations with a crashdump_context function, and adding a dedicated crashdump module along with necessary dependency updates.

Introduces crashdump support via conditional compilation (#[cfg(crashdump)]) and metadata propagation.
Implements crashdump_context functions in multiple hypervisor modules.
Adds a new crashdump module and updates Cargo.toml to include the elfcore dependency.

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/hyperlight_host/src/sandbox/uninitialized_evolve.rs	Introduces crashdump metadata propagation in sandbox initialization.
src/hyperlight_host/src/sandbox/uninitialized.rs	Defines SandboxMetadata and integrates metadata handling in guest binary parsing.
src/hyperlight_host/src/hypervisor/windows_hypervisor_platform.rs	Adds a new get_xsave function for crashdump support.
src/hyperlight_host/src/hypervisor/kvm.rs	Implements crashdump_context using vCPU register and XSAVE information.
src/hyperlight_host/src/hypervisor/inprocess.rs	Marks crashdump_context as unimplemented for in-process mode.
src/hyperlight_host/src/hypervisor/hypervisor_handler.rs	Updates hypervisor handler functions to accept crashdump metadata.
src/hyperlight_host/src/hypervisor/hyperv_windows.rs	Adds crashdump_context support and propagates metadata in HypervWindowsDriver.
src/hyperlight_host/src/hypervisor/hyperv_linux.rs	Implements crashdump_context for HypervLinuxDriver with register extraction and metadata.
src/hyperlight_host/src/hypervisor/crashdump.rs	Introduces the crashdump module with GuestView, GuestMemReader, and the core dump generation function.
src/hyperlight_host/Cargo.toml	Adds the elfcore dependency needed for ELF core dump creation.

Copilot · 2025-04-29T13:09:41Z

src/hyperlight_host/src/hypervisor/crashdump.rs

-    println!("Memory dumped to file: {:?}", persist_path);
-    log::error!("Memory dumped to file: {:?}", persist_path);
+    println!("Core dump created successfully: {}", path_string);
+    log::error!("Core dump file: {}", path_string);


Logging a successful core dump creation as an error may be misleading. Consider using 'log::info' or 'log::debug' to indicate a normal, successful operation.

Suggested change

log::error!("Core dump file: {}", path_string);

log::info!("Core dump file: {}", path_string);

ludfjig

Looks great. Left some nit comments.

By the way (unrelated to your PR), we should simplify each driver to include only the core VM functionality, (like run, set_reg, get_regs, etc.). Then, we can rely on a dyn Hypervisor to avoid duplicating functionality across different implementations.

src/hyperlight_host/src/hypervisor/crashdump.rs

src/hyperlight_host/src/sandbox/uninitialized.rs

- the core dump file is an ELF file with special segments that describe the guest's memory when it crashed, the CPU register's values and other special notes that tell the debugger how to set up a debugging session starting from the core dump Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

…he sandbox - only store the binary path for now Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

…b and lldb using vscode Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

…e to generics Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

ludfjig · 2025-05-07T18:59:15Z

docs/debugging-hyperlight.md

-To dump the details of the memory configuration, the virtual processors register state and the contents of the VM memory set the feature `crashdump` and run a debug build. This will result in a dump file being created in the temporary directory. The name and location of the dump file will be printed to the console and logged as an error message.
+## Dumping the guest state to an ELF core dump
+
+To dump the state of the vCPU (general purpose registers, registers) to an `ELF` core dump file set the feature `crashdump` and run a debug build. This will result in a dump file being created in the temporary directory.


btw what happens if you run in release mode?

For now, the crashdump feature is gated by the debug build.
There are two scenarios here:

Run release mode with crashdump which will only make the guest crash and the guest return an error that the guest panicked (no core dump)

After I remove the debug only option for crashdump, the core dump file is correctly generated in the release mode, but the symbols are scarce (it just shows the function names). I haven't checked, but for release builds the debug symbols might be stripped

I think that we should add support for release builds, and maybe document the changes needed to the release profile when building the guest to support full symbols (or perhaps even add a profile that does that as an example) The other option (and maybe preferable) would be to set the option to create full debug info but to use split-debug-info on our release builds, we would still need to test this and document it

simongdavies

A couple of minor comments/suggestions, the main thing is that I think we should enable this for release builds and show how to get debug info in a release build as above

simongdavies · 2025-05-08T10:19:11Z

docs/debugging-hyperlight.md

-To dump the details of the memory configuration, the virtual processors register state and the contents of the VM memory set the feature `crashdump` and run a debug build. This will result in a dump file being created in the temporary directory. The name and location of the dump file will be printed to the console and logged as an error message.
+## Dumping the guest state to an ELF core dump
+
+To dump the state of the vCPU (general purpose registers, registers) to an `ELF` core dump file set the feature `crashdump` and run a debug build. This will result in a dump file being created in the temporary directory.


Unless I am mistaken this only happens if there is an unexpected VM exit or unhandled exception in the guest not just by turning on the feature?

Yes, I'll specify it

simongdavies · 2025-05-08T10:31:05Z

docs/debugging-hyperlight.md

+### Inspecting the core dump
+
+After the core dump has been created, to inspect the state of the guest, load the core dump file using `gdb` or `lldb`.
+A `gdb` version later than `15.0` and `lldb` version later than `17` have been used to test this feature.


Suggested change

A `gdb` version later than `15.0` and `lldb` version later than `17` have been used to test this feature.

**NOTE: This feature has been tested with version `15.0` of `gdb` and version `17` of `lldb` , earlier versions may not work, it is recommended to use these versions or later.**

simongdavies · 2025-05-08T11:45:16Z

src/hyperlight_host/src/hypervisor/crashdump.rs

+        let filename = ctx
+            .filename
+            .as_ref()
+            .map_or_else(|| "<unknown>".to_string(), |s| s.to_string());


Suggested change

.map_or_else(|| "<unknown>".to_string(), |s| s.to_string());

.map_or("<unknown>".to_string(), |s| s.to_string());

simongdavies · 2025-05-08T11:45:39Z

src/hyperlight_host/src/hypervisor/crashdump.rs

+        let cmd = ctx
+            .binary
+            .as_ref()
+            .map_or_else(|| "<unknown>".to_string(), |s| s.to_string());


Suggested change

.map_or_else(|| "<unknown>".to_string(), |s| s.to_string());

.map_or("<unknown>".to_string(), |s| s.to_string());

dblnz added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Apr 14, 2025

dblnz force-pushed the gdb-coredump-elfcore branch from d5892ec to 52c3c71 Compare April 15, 2025 06:47

dblnz self-assigned this Apr 15, 2025

dblnz force-pushed the gdb-coredump-elfcore branch 7 times, most recently from cc8a48d to 0126ce3 Compare April 29, 2025 12:40

dblnz requested a review from Copilot April 29, 2025 12:41

Copilot AI reviewed Apr 29, 2025

View reviewed changes

dblnz commented Apr 29, 2025

View reviewed changes

dblnz force-pushed the gdb-coredump-elfcore branch from 0126ce3 to 51ce291 Compare April 29, 2025 13:07

dblnz requested a review from Copilot April 29, 2025 13:09

Copilot AI reviewed Apr 29, 2025

View reviewed changes

dblnz marked this pull request as ready for review April 29, 2025 14:06

ludfjig previously approved these changes Apr 29, 2025

View reviewed changes

ludfjig reviewed Apr 29, 2025

View reviewed changes

src/hyperlight_host/src/sandbox/uninitialized.rs Outdated Show resolved Hide resolved

dblnz dismissed ludfjig’s stale review via e345e85 April 30, 2025 14:18

dblnz force-pushed the gdb-coredump-elfcore branch 3 times, most recently from 4d4b87f to 964a139 Compare May 7, 2025 10:10

dblnz added 6 commits May 7, 2025 14:27

fixup: add constants instead of hardcoded values and add more comments

4957eca

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

crashdump: add SandboxMetadata field that store relevant data about t…

52380c6

…he sandbox - only store the binary path for now Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

fixup: change default filename to "<unknown>" and aother small updates

d3fd68a

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

crashdump: add documentation on how to load a guest core dump in a gd…

8d9e763

…b and lldb using vscode Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

crashdump: update the use of CoreDumpBuilder to reflect elfcore chang…

94a55d1

…e to generics Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

dblnz force-pushed the gdb-coredump-elfcore branch from 964a139 to 94a55d1 Compare May 7, 2025 11:28

ludfjig approved these changes May 7, 2025

View reviewed changes

simongdavies approved these changes May 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support ELF core dump creation on guest crash #417

Support ELF core dump creation on guest crash #417

dblnz commented Apr 14, 2025 •

edited

Loading

Copilot AI left a comment

dblnz Apr 28, 2025

Copilot AI left a comment

Copilot AI Apr 29, 2025

ludfjig left a comment

ludfjig May 7, 2025

dblnz May 8, 2025

simongdavies May 8, 2025 •

edited

Loading

simongdavies left a comment •

edited

Loading

simongdavies May 8, 2025

dblnz May 8, 2025

simongdavies May 8, 2025

simongdavies May 8, 2025

simongdavies May 8, 2025

	log::error!("Core dump file: {}", path_string);
	log::info!("Core dump file: {}", path_string);

	A `gdb` version later than `15.0` and `lldb` version later than `17` have been used to test this feature.
	NOTE: This feature has been tested with version `15.0` of `gdb` and version `17` of `lldb` , earlier versions may not work, it is recommended to use these versions or later.

	.map_or_else(\|\| "<unknown>".to_string(), \|s\| s.to_string());
	.map_or("<unknown>".to_string(), \|s\| s.to_string());

Support ELF core dump creation on guest crash #417

Are you sure you want to change the base?

Support ELF core dump creation on guest crash #417

Conversation

dblnz commented Apr 14, 2025 • edited Loading

Purpose

Description

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Copilot AI Apr 29, 2025

Choose a reason for hiding this comment

ludfjig left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simongdavies May 8, 2025 • edited Loading

Choose a reason for hiding this comment

simongdavies left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dblnz commented Apr 14, 2025 •

edited

Loading

simongdavies May 8, 2025 •

edited

Loading

simongdavies left a comment •

edited

Loading