SectorFinalizeFailed - Snap-Up verified deal to sector - Finalizes with missing parts

[scaseye]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

:~$ lotus --version
lotus version 1.15.1+mainnet+git.731da455d
:~$ lotus-miner --version
lotus-miner version 1.15.1+mainnet+git.731da455d

Describe the Bug

Was snapping verified deals. This sector has error, then finalizes but without all the parts.

30.	2022-04-13 21:04:04 -0700 PDT:	[event;sealing.SectorFinalizeFailed]	{"User":{}}
	finalize sector: storage call error 0: %!w(finalizing sector: clear cache: could not read from path="/data/mainnet32GiB/seal/update-cache/s-t01611097-414/t_aux"

Caused by:
    No such file or directory (os error 2)

Stack backtrace:
   0: filecoin_proofs::api::post_util::clear_cache
   1: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
   2: ffi_toolkit::catch_panic_response
   3: fil_clear_cache
   4: _cgo_205e3dfad3fb_Cfunc_fil_clear_cache
             at ./tmp/go-build/cgo-gcc-prolog:97:11
   5: runtime.asmcgocall
             at ./usr/local/go/src/runtime/asm_amd64.s:667 [Hostname: homemate-server01])

missing files in this directory..

:~$ ls -lct /mnt/cephfs/mainnet32GiB/storage/update-cache/s-t01611097-414
total 67108864
-rw-rw-r--+ 1 scaseye scaseye 68719476704 Apr 13 21:51 sc-02-data-tree-d.dat

[sector-status-414.txt](https://github.com/filecoin-project/lotus/files/8490787/sector-status-414.txt)

greplogs414.txt
greplogs2-414.txt

Logging Information

attached

Repo Steps

not sure how to reproduce.

Running Services for all workers, miner, market, daemon processes. I was snapping 16 sectors with 30+ deals at the same time. 8 workers snapping 8 queued.

[scaseye]

lotus-miner sectors list shows all good
414 Proving YES YES 3275006 (in 1 year 24 weeks) 1 31.61GiB 284.4GiB

however it will fault

lotus-miner proving check --slow 0
0         0          414     bad (stat /mnt/cephfs/mainnet32GiB/storage/update-cache/s-t01611097-414/sc-02-data-tree-r-last-3.dat: no such file or directory)

[rjan90]

Hey @scaseye!

Thanks for the issue report! This PR which has been merged into master should mitigate this issue by moving snapdeal sectors to the long term storage before submitting the proof to the chain - as I suspect some of the files where not moved properly because all SnapDeal-files where transferred at the same time right after the proof where submitted!

Closing this issue for now, but will keep an eye out if this still persists on newer versions of Lotus! Thanks again!