Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

ETL refactor #1594

Open
godexsoft opened this issue Aug 12, 2024 · 0 comments
Open

ETL refactor #1594

godexsoft opened this issue Aug 12, 2024 · 0 comments
Labels
enhancement New feature or request refactoring Code improvements without logical changes research testability Makes code testable and/or adds unit-tests
Milestone
2.4

Comments

@godexsoft
Copy link
Collaborator

godexsoft commented Aug 12, 2024 •
edited
Loading

Summary

Many parts of the ETL need improvements. Separate issues will be created and linked here for each sufficiently big component.

Subtasks

  1. Node awareness abstraction - ETL node awareness abstraction #1595
  2. Extractor - ETL ng extractor #1596
  3. Loader - ETL ng loader #1597
  4. Backfill - Backfill #66
  5. Plugins - ETL ng plugins support #1598
  6. Move NFT/bookdir - ETL ng NFT and book dir #1599

Motivation

ETL is one of the last parts of Clio that are still not refactored to be testable and covered by unit tests.
The ETL also does not support certain planned functionality that is too complicated to implement on top of the existing architecture.

Solution

Refactor/rewrite ETL
@godexsoft godexsoft added enhancement New feature or request testability Makes code testable and/or adds unit-tests refactoring Code improvements without logical changes research labels Aug 12, 2024
@github-project-automation github-project-automation bot moved this to 📋 Backlog in Clio Aug 12, 2024
This was referenced Aug 12, 2024
Open
Open
Open
Open
Open
@godexsoft godexsoft added this to the 2.3 milestone Aug 12, 2024
@godexsoft godexsoft modified the milestones: 2.3, 2.4 Oct 21, 2024
godexsoft added a commit that referenced this issue Feb 21, 2025
@godexsoft
feat: ETLng monitor (#1898)
491cd58 
For #1594
PeterChen13579 pushed a commit to PeterChen13579/clio that referenced this issue Mar 2, 2025
@godexsoft @PeterChen13579
feat: ETLng monitor (XRPLF#1898)
ccdb693 
For XRPLF#1594
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
enhancement New feature or request refactoring Code improvements without logical changes research testability Makes code testable and/or adds unit-tests
Projects
Clio
Status: 📋 Backlog
Milestone
2.4
Development

No branches or pull requests
1 participant
@godexsoft