Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[pixels-cache] multi-table caching #636

Open
1 of 5 tasks
bianhq opened this issue Feb 1, 2024 · 0 comments
Open
1 of 5 tasks

[pixels-cache] multi-table caching #636

bianhq opened this issue Feb 1, 2024 · 0 comments
Assignees
@bianhq
Labels
enhancement New feature or request
Milestone
Cache Enhancement

Comments

@bianhq
Copy link
Contributor

bianhq commented Feb 1, 2024 •
edited
Loading

Currently, pixels can only cache one table specified in the pixels.properties. We should finish the following tasks for this issue:

  • remove cache.schema and cache.table from pixels.properties;
  • Remove the cacheBorder in the layout of a table, cached column chunks should be directly specified by the cache plan (job);
  • A cache plan will be related to a path and a layout of a table;
  • Redesign the data model for cache plan in etcd. A cache plan records the set of column chunks to be cached in a file and a list of paths on which the cache plan is applied;
  • Redesign the cache coordinator and cache worker, apply the new data model for cache plan.
@bianhq bianhq added the enhancement New feature or request label Feb 1, 2024
@bianhq bianhq added this to the Cache Enhancement milestone Feb 1, 2024
@bianhq bianhq self-assigned this Feb 1, 2024
bianhq added a commit that referenced this issue Sep 22, 2024
@bianhq
[Issue #636] remove cache schema and table from config file and fix m…
09a0166 
…emory leak in cache writer (#728)

1. Now, the cache schema and table names are specified in the layout_version and cache_version values.
2. Close PixelsPhysicalReader in PixelsCacheWriter to release the off-heap buffers allocated for localFs.
3. Revise the pixels cache document.
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@bianhq bianhq

Labels
enhancement New feature or request
Projects
None yet
Milestone
Cache Enhancement
Development

No branches or pull requests
1 participant
@bianhq