Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Desub refactor #42

Merged
merged 69 commits into from
Oct 11, 2021
Merged

Desub refactor #42

merged 69 commits into from
Oct 11, 2021

Conversation

insipx
Copy link
Contributor

@insipx insipx commented Sep 6, 2021

Refactor to better control cursor and datastates. This should consolidate data and cursor values to make them easily observable and add the possibility of saving states at different points of decoding in order to inspect them and possibly 'rewind' them.

Introduces DecodeState to the decoder

needed because it will make it much easier to debug extrinsics which don't decode, while making the code easier to read.

Branch off of #41 in case things go south

Versions Passed with 0 Errors on Kusama/Polkadot

Kusama
  • 1020
  • 1021
  • 1022
  • 1023
  • 1024
  • 1025
  • 1027
  • 1028
  • 1029
  • 1030
  • 1031
  • 1032
  • 1033
  • 1038
  • 1039
  • 1040
  • 1042
  • 1045
  • 1050
  • 1051
  • 1052
  • 1053
  • 1054
  • 1055
  • 1058
  • 1062
  • 2005
  • 2007
  • 2008
  • 2011
  • 2012
  • 2013
  • 2015
  • 2019
  • 2022
  • 2023
  • 2024
  • 2025
  • 2026
  • 2027
  • 2028
  • 2029
  • 2030
  • 9010
  • 9030
  • 9040
  • 9050
  • 9070
  • 9080
  • 9090
Polkadot
  • 0
  • 1
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 9050
  • 9080
  • 9090

@insipx insipx changed the base branch from master to insipx/update September 6, 2021 02:09
@insipx insipx marked this pull request as draft September 6, 2021 02:12
Andrew Plaza added 4 commits October 1, 2021 10:04
Andrew Plaza and others added 12 commits October 8, 2021 12:07
@insipx
Copy link
Contributor Author

insipx commented Oct 8, 2021

As far as the metadata versions go, it might be pertinent to actually add those to frame-metadata. AFAICT it wouldn't be very different than how they are in desub (i'm talking about metadata version pre-v12)

Still doing a last run-through of polkadot spec-by-spec ( I think there was one random error i got from the whole chain but i haven't figured out which block it is yet). But once everything is reviewed it should be ready

In the future I think tx-decoder could be a really useful tool for both regression and unit testing. It would require an archive server running somewhere with just blocks synced, and github actions could connect to it via a read-only user. Not sure if there is some precedent for that already (maybe iffy connected to a postgres server from actions?) but It could keep westend synced and then decode through all the extrinsics for the latest/previous runtime to check for any errors before releasing/deploying to master

@insipx insipx requested a review from dvdplm October 8, 2021 10:44
@insipx insipx merged commit 50f0108 into insipx/update Oct 11, 2021
insipx added a commit that referenced this pull request Oct 11, 2021
* include hacky legacy metadata (before v12) in desub

* add local sp-core and legacy frame-metadata versions directly to desub

* using frame-metadata and updating to v14

* Remove StructUnitOrTuple

Removal of this struct simplifies `RustTypeMarker`/`SubstrateType`

* Make translating from Json to Desub more flexible

- Json Objects/arrays/Null fields are seperated based on tuple/struct/enum representation
- add better error handling

* improve error handling in parse_mod_types

* fix decoder test

* add more logging and add metadata v13, v14

* rewrote most of modules to make JSON parsing more flexible

* typo

* implement fallback

* start working on tx verifier for entire chains of extrinsics

* Desub refactor (#42)

* make tx-decoder actual cli app

* huge refactoring to consolidate cursor and data states

* formatting

* clippy

* progress, missing an increment somewhere causing signatures to fail decoding

* some parts of incrementing the cursor is made clearer by DecodeState

* make Extrinsic bytes into an iterator, fix more extrinsics in 1020

* fix bug in Vec decoding

* all of 1020 decodes

* fix all tests

* Decode up to 1050

* fix spec 1062

* 2008 decodes

* fix definition for ConsensusEngineId

* some upgrades to tx decoder

* rework tx-decoder to use previous spec version

when blocks transition from runtimes, the spec will be `next` but the block extrinsics
still use previous spec version

* mutex -> rwlock

* fix sanitization, v12 metadata conversion

- use index in metadata rather than enumerater() iterator
- fix type name sanitization to support <T as Trait<\w>> and <T as
  Config<\w>>

* Add tx_decoder to logs

* fix runtimes, number => version

* do_decode decides on decode_length independently

* remove do_decode in favor of decode

do_decode is no longer needed since we can inspect the decode length
independently.

* fix v12, add v13 metadata

* Add bitvec type to `SubstrateTypes`

* update v11 to scale-codec 2

* update v9 to scale-codec 2

* update v8 to scale-codec 2

* use one DecodeDifferent from frame-metadata

* remove dependency on legacy sp-core

* update codec to 2 for extras

* Add bitvec type

remove not-used version7 metadata

* Update core/src/decoder.rs

* Update core/src/decoder.rs

* some cleanup

* Update core/src/decoder.rs

* log::error -> println

* check for upgrade block 0

* check some

* fix santiziation of types with empty generic, <()>

* remove all extrinsic integration tests

* fix remaining integration tests

* fix tests/clippy lints

* remove solved todos

* add back unused

* add test for ChunkedExtrinsic

* remove println from test

* remove unused RustTypeMarker variant 'String'

* remove 'impossible' types

* remove more impossible types

* fix tests

* fix comment

* add westend, centrifuge and rococo to FromStr impl

* FromStr should allow for custom chains

* add docs about upgrade blocks

* docs

* Update README.md

fix release docs

* Update README.md

* Update bin/tx-decoder/src/queries.rs

Co-authored-by: Tarik Gul <47201679+TarikGul@users.noreply.github.com>

* Update bin/tx-decoder/src/queries.rs

Co-authored-by: Tarik Gul <47201679+TarikGul@users.noreply.github.com>

* Update bin/tx-decoder/src/queries.rs

Co-authored-by: Tarik Gul <47201679+TarikGul@users.noreply.github.com>

* rustfmt

* minus v14 for now

* add StorageKey definition

* add readme comment about StorageKey

* choose simplest representations for substrate_types

* rusfmt

Co-authored-by: Tarik Gul <47201679+TarikGul@users.noreply.github.com>

Co-authored-by: Tarik Gul <47201679+TarikGul@users.noreply.github.com>
@insipx insipx deleted the desub-new branch October 11, 2021 12:24
@insipx insipx restored the desub-new branch October 11, 2021 12:25
@insipx insipx deleted the desub-new branch October 20, 2021 13:07
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants