Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Slowdown when error dataset grows #340

Open
dcesari opened this issue Nov 13, 2024 · 0 comments
Open

Slowdown when error dataset grows #340

dcesari opened this issue Nov 13, 2024 · 0 comments
Assignees

Comments

@dcesari
Copy link
Member

dcesari commented Nov 13, 2024

I noticed that when error dataset grows (e.g. tenths of GB in a daily file), the arki-scan process tends to slow down by many times, while this is not noticed when importing to a regular dataset having daily files of the same size (I cannot say whether the slowdown affects also data that do not end in error, it could be the case, but I am not sure). Could this be related to how the error dataset is managed?
It is not a serious problem, because of course error dataset should not grow so much, but if it has a simple solution it would be welcome.

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

No branches or pull requests

2 participants