Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

feat: Implement WAL-based RaftLog storage #16776

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

feat: Implement WAL-based RaftLog storage #16776

wants to merge 6 commits into from
+1,891 −1,312

Commits on Nov 7, 2024

  1. feat: Implement WAL-based RaftLog storage 

    Replace sled-based storage with a dedicated WAL implementation optimized
for Raft Log operations. The new implementation provides:

Performance improvements:
- Non-blocking batched fdatasync
- FILO caching for latest logs (configurable size/count)
- ~0.5ms write latency

Compatibility:
- Backward compatible with existing data format
- Auto-upgrades from V003 to V004 format on startup
- Supports rolling upgrades (no protocol changes)
- databend-metactl supports both V003/V004 import/export

Technical details:
- New storage format: V004
- Optimized specifically for Raft Log operations
- Preserves all existing functionality
    @drmingdrmer
    drmingdrmer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    1d521c3 View commit details
    Browse the repository at this point in the history

  2. chore: reset node id when restore

    @drmingdrmer
    drmingdrmer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    0e842d2 View commit details
    Browse the repository at this point in the history

  3. refactor: rename raft_log_2 to raft_log_v004; move struct to seperate… 

    … files
    @drmingdrmer
    drmingdrmer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    7e43e97 View commit details
    Browse the repository at this point in the history

  4. chore: refactor import paths

    @drmingdrmer
    drmingdrmer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    697589a View commit details
    Browse the repository at this point in the history

  5. chore: refine names

    @drmingdrmer
    drmingdrmer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    449204c View commit details
    Browse the repository at this point in the history

  6. refactor: OnDisk ensures dir

    @drmingdrmer
    drmingdrmer committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    9ef7883 View commit details
    Browse the repository at this point in the history