LOTUS_FVM_CONCURRENCY=48 causes chain sync panic on migration

[RobQuistNL]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus fvm/fevm - Lotus FVM and FEVM interactions
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt/WinningPoSt)
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

1.23.0-rc1

Repro Steps

1. Run lotus daemon with LOTUS_FVM_CONCURRENCY=48 (the max as per the docs)
2. See chain won't sync after nv19

Describe the Bug

Setting it to LOTUS_FVM_CONCURRENCY=24 fixed the issue.

Potential fix

Maybe cap the value set in the config, if it causes sync issues like these. Or at least show a warning / make the daemon refuse to start.

[RobQuistNL]

It looks like this has since been updated in the release notes, so thats good :)

[rjan90]

Added a PR for the lotus-docs as well: filecoin-project/lotus-docs#560. Although this value (24) seems to be low enough for most systems, the actual right value depends on the architecture.

The longer term fix would be to consider both the system and whether or not we're going over an upgrade

[Stebalien]

This should be fixed now (for all versions of the FVM) as we now over-subscribe instead of allocating all instances up-front.