Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

(v4) Database sharding #55

Closed
3 of 7 tasks
i-norden opened this issue Mar 22, 2022 · 1 comment
Closed
3 of 7 tasks

(v4) Database sharding #55

i-norden opened this issue Mar 22, 2022 · 1 comment
Assignees

Comments

@i-norden
Copy link
Collaborator

i-norden commented Mar 22, 2022

In order to shard our database we need to

  • Denormalize tables by block_number, so that we can partition tables by block range: denormalize tables by block_number #51
  • Script for creating new partition(s) on shard server
  • Script for linking partition(s) with postgres_fdw on the aggregating server
  • Stored procedures for state and storage snapshot generation
  • Need to work out additional details/scripts for how to manage the automatic creation and linking of partitions as our database syncs new blocks at head
  • Update go-ethereum to work with v4 schema: update sql indexer to use new v4 schema go-ethereum#211
  • Update eth-statediff-service to work with v4 schema

Notes:
Basic research and more details around the sharding approach/design can be found here
This is a child issue of https://github.com/vulcanize/ops/issues/137

@i-norden i-norden self-assigned this Mar 22, 2022
@i-norden i-norden changed the title Database sharding (v4) Database sharding Mar 31, 2022
@i-norden
Copy link
Collaborator Author

Closing this as the remaining tasks need to be considered in the context of TimescaleDB

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant