Can't remove or terminate sector.

[xzone]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

Daemon:  1.15.0+mainnet+git.0ac1bbc7a+api1.4.0
Local: lotus-miner version 1.15.0+mainnet+git.0ac1bbc7a

Describe the Bug

cpu usage rate 50%-80%, disk busy 50%-80%,
Sector update remove terminator is not allowed.
In the log, continue to output the truncating sector log {spector: 390} warning output.

Logging Information

log file above.

Repo Steps

./lotus-shed sectors terminate --really-do-it 390
2022-03-15T08:15:34.908+0900 WARN lotus-shed lotus-shed/main.go:103 mpool push message:
main.glob..func83
/root/lotus/cmd/lotus-shed/sectors.go:150

• GasEstimateMessageGas error: estimating gas used: message execution failed: exit 16, reason: failed to terminate sectors in deadline 0: failed to terminate sectors in partition 0: can only terminate live sectors (RetCode=16)

---

can't remove it even with the above command.
log.txt

[llifezou]

44.	2021-03-17 01:14:00 +0900 KST:	[event;sealing.SectorRetryFinalize]	{"User":{}}
45.	2021-03-17 01:16:59 +0900 KST:	[event;sealing.SectorFinalized]	{"User":{}}
46.	2021-03-17 17:49:27 +0900 KST:	[event;sealing.SectorStartPacking]	{"User":{}}
47.	2021-03-17 18:07:05 +0900 KST:	[event;sealing.SectorStartPacking]	{"User":{}}
48.	2021-03-17 18:29:17 +0900 KST:	[event;sealing.SectorStartPacking]	{"User":{}}
49.	2021-03-17 18:35:31 +0900 KST:	[event;sealing.SectorForceState]	{"User":{"State":"PreCommit1"}}
50.	2021-03-17 18:35:31 +0900 KST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {124554 390} from remote(1): sector not found
51.	2021-03-17 18:36:31 +0900 KST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
52.	2021-03-17 18:36:31 +0900 KST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: storage call error 0: failed to acquire sector {124554 390} from remote(1): sector not found
53.	2021-03-17 18:37:31 +0900 KST:	[event;sealing.SectorForceState]	{"User":{"State":"RecoverDealIDs"}}
54.	2021-03-17 18:37:31 +0900 KST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}

state machine error: running planner for state RecoverDealIDs failed: return state not set because of multiple forced state changes

[llifezou]

./lotus-shed sectors terminate --really-do-it 390
2022-03-15T08:15:34.908+0900 WARN lotus-shed lotus-shed/main.go:103 mpool push message:
main.glob..func83
/root/lotus/cmd/lotus-shed/sectors.go:150

GasEstimateMessageGas error: estimating gas used: message execution failed: exit 16, reason: failed to terminate sectors in deadline 0: failed to terminate sectors in partition 0: can only terminate live sectors (RetCode=16)

Are you sure you sent the miner containing sector 390?

[xzone]

I did it exactly.

[llifezou]

I did it exactly.

lotus-shed sectors terminate requires sectors must be proving once, otherwise failed to terminate sectors in deadline 0: failed to terminate sectors in partition 0: can only terminate live sectors (RetCode=16).

need to find the sector files for 390, if it is damaged, try to repair it, no need to terminate

[xzone]

how to repair?

[llifezou]

how to repair?

I know a community tool, https://github.com/filecoin-box/lotus-box or https://github.com/froghub-io/filecoin-sealer-recover , you can try

[xzone]

Since the sector was forcibly updated, the status does not change even if the sector file is restored. Other work is possible only when the state can be modified.