Hyperspace nv19=>nv20 (Carbonado.3) migration and change management

[raulk]

Migration logic

In terms of state migration, this is a "lightweight" migration where we load a new actors bundle, weave in a new manifest into the system actor, and patch CodeCIDs in the state tree. There is no need to transform the internal state of any actor, or to perform schema migrations on the state tree, or other data structures.

Special change management

Only these changes require special attention because they transpire to the client level and/or require migrations in other components, or version selection code. All other changes are internal to the behaviour of the EVM and/or ref-fvm.

Event schema changes

The event schema has changed. There is a new codec field in entries. We need the node to accept events with and without the codec field during the transitory period (node updated pre-fork). Probably need to solve with a shim in the FVM VM implementation in Lotus.

EVM events

• EVM events now use keys t{1..4} and d instead of topic{1..4} and data.
• Event values are no longer CBOR framed.
• Leading zeros in entry values are no longer stripped.

These changes are visible in the event manager and the Eth JSON-RPC. Ideally we'd like to support historical events, events during the pre-upgrade period, and events post-upgrade. This means that we need Hyperspace code for:

1. On start, updating the existing events index to add the codec column to the database schema. Backfill with "raw".
2. On start, updating existing entries to migrate the event keys.
3. During operation, rewriting new entries coming in with the old keys during the transitory period.

fvm3 minor version multiplexing in filecoin-ffi

filecoin-project/ref-fvm#1627

Explorers

Need to be aware of schema changes and adapt accordingly. Heads-up comms have already gone out.

Change management

Preview Give feedback

1. Migrate event index table on start to add the codec column.
2. Migrate event entries to use the new (post-upgrade) keys.
3. When on nv19, adjust the FVM invoker so that it translates pre-nv20 events to post-nv20 events. That way the rest of the system only needs to deal with events carrying the codec, the new topic names, no CBOR framing, and 32-byte padded topics.
4. Message conversion and Delegated signatures for bare sends (now calling InvokeContract)

Continuity testing

Part of making this upgrade successful involves ensuring that no contract on Hyperspace is broken.

Continuity testing

Preview Give feedback

1. Wrap up work on Continuity and regression testing tools ref-fvm#1631.
2. Run it since the nv19 upgrade until HEAD.

Raw changelogs

ref-fvm

68811f11 (HEAD -> master, origin/master, origin/HEAD) chore: release fvm_shared (#1647)
291f4dee fix: use u64 for the block gas limit (#1646)
e480bd8b chore: release fvm, fvm_sdk, fvm_shared, and fvm_integration_tests (#1645)
190e975a feat: implement tipset gas (#1644)
e50e8e21 feat: tweak tipset lookback limits (#1642)
09fba48e Add size validation and codecs to events (#1638)
c4c808a1 Add gas parameters for event validation + calibration harnesses. (#1635)
7a113993 feat: remove negative gas charges everywhere (#1630)
af0b3245 EVM: apply read-only early and reject events (#1629)
9a56ceeb feat: send benchmarks and updated prices (#1625)
749458cb feat: state access/update gas (#1521)
f5be0a5c (tag: fvm_shared@v3.0.0-alpha.18, tag: fvm_sdk@v3.0.0-alpha.23, tag: fvm@v3.0.0-alpha.22) release `fvm@v3.0.0-alpha.22`, `fvm_shared@v3.0.0-alpha.18`, `fvm_sdk@v3.0.0-alpha.23`. (#1626)
88deef54 Align events implementation with FIP-0049 (#1481)
cd2c7677 feat: explicitly reject placeholder creation (#1568)
6fc226f8 Integrate fvm-bench and the basics of a testkit (#1493)
e7d972ab chore: release integration testing framework (#1531)
6ab8b204 feat: simplify gas tracking stack (#1526)
07f13499 feat: `CarReader::read_into()` (#1524)
7a97e522 feat: normalize transaction signatures (#1525)
432ee12e fix: expose the effective gas premium (#1512)
f22f8d2c (tag: fvm_ipld_encoding@v0.3.3, tag: fvm@v3.0.0-alpha.21) Release ipld_encoding 0.3.3, FVM 3.0.0-alpha.21 (#1516)
c21b7af1 FVM: Use CBOR instead of DAG_CBOR for message params (#1509)
7d8743df chore: update & audit deps (#1508)
f87f1cb4 fix: fix build when testing isn't enabled (#1511)
26f70c2f Restrict create_actor to the InitActor as caller (#1373)
e7b445be cargo update
2f37370b FVM: Machine: Put the Empty Array CID on creation, if missing (#1506)

builtin-actors

267e8332 runtime: drop unused exit() (#1162)
7de37594 EVM: address some TODOs (#1177)
52d2d4a9 EVM: remove a few more deps (#1176)
1280bb26 EVM: remove Bytes (#1175)
878c1704 Merge branch 'master' into next
5b90c2b8 feat: update fvm and switch to the new event structure (#1171)
0a9539bf chore: update FVM (#1169)
c5cbd5bc EVM: more unit tests (#1163)
66c2cdb0 FIP-0055: Introduce the EthAccount actor (#1166)
4db0f1fd chore: rename uses of serde_bytes to strict_bytes (#1157)
06e5d489 chore: deps: bump to fvm_ipld_bitfield 0.5.4 (#1158)
f6325ac6 feat: init actor: Add Exec4 method (#1156)
ac264944 EVM: add precompile conformance tests (#1165)
d0985499 EVM: fix zero curve encoding (#1164)
a6afa868 re-use DAG_CBOR (#1161)
102c7b98 gitignore: Ignore all car files in output directory (#1152)
6540db3f EVM: Test context opcodes (#1155)
f23e26ad chore: paych: fix incorrect error message (#1159)
7db5cff9 EVM: fix mockruntime tipset-cid (#1154)
0f472bdd Merge branch 'master' into next
1630d6c1 fix: miner and verifreg: make getters read-only (#1137)
3cddc9d2 Runtime: Add several new methods (#1150)
6f2c8e5b Refactor: Runtime: tipset_cid should return a Result (#1151)
5eb65d5a EVM: Unit tests for CALL/RETURN data opcodes (#1128)
93c14679 EVM: apply EIP150 to CREATE/CREATE2 (#1149)
d84bbf13 EVM: use shared evm create in the EAM (#1148)
553d2d3c remove duplicated authenticate_message_test (#1130)
0ec7dd86 Runtime: Drop unused user_get_randomness methods (#1147)
b444b4bf EVM: refactor unit test macro params (#1144)
859a4b4c EVM: Factor out shared crate & reorg (#1145)
e6b24ee1 Merge branch 'master' into next
28c76e4a Runtime: Impl Display for SendError (#1143)
01a1a780 EVM: Test balance ops (#1142)
300d83fd EVM: Test keccak opcode (#1132)
a34f5c63 Runtime: test: Refactor to add expect_send_simple (#1139)
c976f741 Market deal metadata methods to return EX_DEAL_EXPIRED if the deal ID is used (#989)
7fe9b77b Introduce a SendError (#1136)
1d473c5f EVM: LOG{0..4}: adjust event definitions as per FIP-0054. (#1057)
992f7b12 Merge branch 'master' into next
0bccc7a8 test_vm: support creating actors over placeholders (#1131)
7c9d1117 fix selfdestruct test files. (#1134)
90807ba7 refactor: Runtime: Move create and state into default impl (#1133)
b47f5f74 feat: runtime: Add read_only getter (#1127)
b4e957a8 EVM: Use word size const (#1129)
4f450fdc feat: runtime: Add send_generalized (#1126)
c8949133 EVM: Fix `Memory::grow` (#1124)
46945f89 feat: Partial implementation of FIP-0048 (f4/delegated addresses) (#1118)
acfa4a1a EVM: Test memory ops (#1123)
c5592791 EVM: unit tests: Control Flow, Call Context, Call (#1116)
cfaa4e36 EVM: simplify call logic and CALL on transfer (#1122)
28377381 EVM: Test `POP` opcode (#1110)
6ea329b4 EVM: Test Fe lang simplecoin (#1120)
b81009e0 placeholder: drop deps on FVM SDK and shared (#1117)
67b759f2 fix: test_vm: Don't set predictable address for accounts (#1114)
68717bc0 EVM: Test boolean ops (#1115)
47a90adf EVM: test remaining math ops (#1111)
110419ff Merge branch 'master' into next
03d2d535 feat: accept all methods in the placeholder (#1112)
ee5dad08 EVM: Test `PUSH*` opcodes a bit more (#1109)
2d7f62aa EVM: abort if we fail to transfer to the beneficiary on selfdestruct (#1106)
60b992ae Update to FVM SDK 3.0.0-alpha.22, shared 3.0.0-alpha.17, ipld_encoding 0.3.3 (#1095)
520e96f7 EVM: test swap 0 (#1105)
992c534f EVM: fix and test DUP opcodes (#1104)
68f3f79f EVM: fix addmod and test (#1100)
076c7f83 EVM: test send error changes (#1099)
764d4a86 fix: bubble up the error from the callactor precompile (#1097)
2c1db5ad fix: paych: correct error message (#1098)
f1d0fb06 Use is_controlling_address in actors (#948)
ea76792b feat: allow ActorErrors to carry optional data (#1093)
252c3690 fix: paych: correct error message (#1094)
09aa10f0 feat: allow all unrestricted method numbers on accounts (#1086)
89d2aec3 EVM: dont panic if calling i256_neg with 0 (#1090)
ed546923 Merge branch 'master' into next
dfe0676c feat: improve dispatch macro (#1088)
e58acdf2 Merge branch 'master' into next
abfb6086 feat: Use CBOR, not DAG_CBOR, for message params & returns (#1076)
94f92ec3 EVM: Remove StatusCode and return meaningful actor errors (#1083)
a0beee1f EVM: Remove get_randomness precompile (#1082)
8a6a447e test: check placeholder/non-existent behavior on extcodesize (#1077)
02df0ad5 EVM: Remove get_actor_type and allow precompile lookup table to have empty slots (#1080)
b7492852 EVM: Test call output region (#1079)
34d70721 EVM: Tests call actor invalid inputs (#1075)
e28bc904 Avoid redundant read/write of allocation/claims HAMT root when unchanged (#1030)
008bac85 EVM: infallible address conversion (#1074)
d865ac3f Ethaccount: Implement FIP 0044 (#1071)
11cd3fc5 EVM: various call actor tests (#1070)
0592571a EVM: update measurements (#1073)
a05defff fix: correctly decode EAM return on resurrect (#1072)
a600986f Market actor batch datacap token transfers (#722)
fe802af8 chore: update hamt & amt (#1052)
d2924bfd Implement log initialisation inline in FVM runtime (#1019)
b84d38c8 FIP-0050: API between user-programmed actors and built-in actors  (#1004)