Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

feat: limit bundle by data size #188

Merged
merged 44 commits into from
Feb 24, 2025
Merged

Conversation

segfault-magnet
Copy link
Contributor

@segfault-magnet segfault-magnet commented Jan 28, 2025

Closes: #157

Bundling can now be configured by two new config parameters:

- **`COMMITTER__APP__BUNDLE__MAX_FRAGMENTS_PER_BUNDLE`**

  - **Description:** Maximum number of fragments allowed per bundle. This limits the size of any individual bundle.
  - **Type:** Positive integer (`NonZeroUsize`)
  - **Example:** `10`
  
  - **`COMMITTER__APP__BUNDLE__BYTES_TO_ACCUMULATE`**

  - **Description:** Byte threshold that, if reached before the accumulation timeout expires, will trigger bundling immediately.
  - **Format:** Human-readable byte string (e.g., `10MB`, `500KB`)
  - **Example:** `1MB`

The idea is that we can configure the committer to give lots of data for the bundler to work with (ie achieve better compression, gas usage) while allowing it to only generate bundles that fit in N eth blobs.

Along with that we also bundle if we accumulate the configured number of blocks, so that we are more lively when we have a large backlog of unbundled blocks that are small, for example.

And there is the traditional timeout that will bundle anyways if we haven't bundled anything for a long time.

Also grouped together blocks before importing them. Mainly to avoid an edgecase mentioned in the bundler code but also to reduce the number of "imported N..=N+1 logs" we had.

@segfault-magnet segfault-magnet self-assigned this Jan 28, 2025
@segfault-magnet segfault-magnet force-pushed the feat/limit_bundle_by_data_size branch from a6715bd to 271115b Compare January 28, 2025 13:57
@segfault-magnet segfault-magnet marked this pull request as ready for review February 5, 2025 22:01
Copy link
Contributor

@hal3e hal3e left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely done. Left some small comments. In addition to the original task, thanks for the docs update and adding chunked fuel block imports.

hal3e
hal3e previously approved these changes Feb 13, 2025
hal3e
hal3e previously approved these changes Feb 14, 2025
Copy link

@digorithm digorithm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Great work on this, looking forward to seeing it live! :shipit:

@segfault-magnet segfault-magnet merged commit 9b576b0 into master Feb 24, 2025
13 checks passed
@segfault-magnet segfault-magnet deleted the feat/limit_bundle_by_data_size branch February 24, 2025 11:48
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Limit final bundle size in B rather than num of blocks
4 participants