Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Consistency issue in LDA model during worker-side data migration #1222

Open
wynot12 opened this issue Jul 27, 2017 · 1 comment
Open

Consistency issue in LDA model during worker-side data migration #1222

wynot12 opened this issue Jul 27, 2017 · 1 comment

Comments

@wynot12
Copy link
Contributor

wynot12 commented Jul 27, 2017

When migrating worker-side data in LDA app, we have a problem that model is damaged during migration. It's because some of our apps have also partial models in the input table.

However, ET/EM assumes that worker-side table has only immutable input data. As a result, we design them not to care about worker-side table data consistency.

To resolve this problem, we need to provide a consistency mechanism for worker-side tables.
Or we may change LDA app to not to use local cache or cope with damage in local model.

@wynot12
Copy link
Contributor Author

wynot12 commented Nov 15, 2017

Need to fix it, since LDA is a primary workload in multi-job situation.

@wynot12 wynot12 added the bug label Nov 15, 2017
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

No branches or pull requests

1 participant