Failed tart run command crashes gRCP connection watcher

[ruimarinho]

Hi,

On a M4 Mac Mini running macOS 15.2, sometimes I'm experiencing a VZErrorDomain with message The virtual machine failed to start.. I will be monitoring logs via macOS log stream to see if I can catch the crash..

However, what I have noticed is that this failure causes the gRPC connection to be restarted likely due to a 412 Precondition Failed error. Is this expected?

Version information:

ci ~ % tart --version
2.22.3
ci ~ % orchard --version
orchard version 0.26.2-08769e0

Orchard logs:

{"level":"warn","ts":1736266804.4783812,"msg":"'tart run orchard-01JH0S1T11FDEEZ1127XEZJNVD-849036c3-de49-466a-b3c0-d440783c16d3-0' failed with exit code 1: Error Domain=VZErrorDomain Code=1 \"The virtual machine failed to start.\" UserInfo={NSLocalizedFailure=Internal Virtualization error., NSLocalizedFailureReason=The virtual machine failed to start.}","vm_uid":"849036c3-de49-466a-b3c0-d440783c16d3","vm_name":"01JH0S1T11FDEEZ1127XEZJNVD","vm_restart_count":0}
{"level":"error","ts":1736266804.47844,"msg":"VM failed: tart command returned non-zero exit code: \"Error Domain=VZErrorDomain Code=1 \\\"The virtual machine failed to start.\\\" UserInfo={NSLocalizedFailure=Internal Virtualization error., NSLocalizedFailureReason=The virtual machine failed to start.}\"","vm_uid":"849036c3-de49-466a-b3c0-d440783c16d3","vm_name":"01JH0S1T11FDEEZ1127XEZJNVD","vm_restart_count":0}
{"level":"info","ts":1736266809.1068091,"msg":"syncing 5 local VMs against 5 remote VMs..."}
{"level":"warn","ts":1736266809.5063329,"msg":"failed to sync VMs: API client encountered an API error to make a request: 412 Precondition Failed (cannot update status for a VM in a terminal state)"}
{"level":"warn","ts":1736266809.506541,"msg":"failed to watch RPC: rpc error: code = Canceled desc = context canceled"}
{"level":"info","ts":1736266813.8923771,"msg":"registered worker mac-1234"}
{"level":"info","ts":1736266813.892595,"msg":"connecting to orchard-controller:443 over gRPC"}
{"level":"info","ts":1736266813.892888,"msg":"gRPC connection established, starting gRPC stream with the controller"}
{"level":"info","ts":1736266814.030932,"msg":"syncing on-disk VMs..."}