Replication: avoid overlapping replication jobs

[okin]

In one of our instances one of our users had a replication set up that, in combination with #16656, led to nearly a thousand running replication jobs.
None of which was able to finish.
The logs showed similar entries like #15415.

The replication pulled every few minutes from another registry and when delays occured it lead to an cascading effect where the first replications weren't able to properly finished but new jobs have been started nontheless.

In order to prevent this I would like the possibility to avoid starting new jobs when for a specific replication when there is already a replication running.

I think one way to address this is to switch the way jobs are spawned from a specified schedule to creating jobs X after the previous job has been finished.

[chlins]

Currently the replication jobs are launched and executed by jobservice, harbor has the limit for jobservice workers, default value is 10 which control the max concurrency workers number. If all the workers are occupied and other jobs will be in pending status then wait until have free worker, so your proposal is expecting to run one replication job which has subtasks sequential rather than parallel?

[chlins]

For the design of harbor replication, from the server side all the tasks each other as independent individuals, so it's not suitable to add dependency with each other such as wait for previous task, that's too tightly coupled a mechanism. But to solve your issue, i think you could consider it from client side, for example, you could call harbor API to fetch task status and only trigger replication when match your customize conditions.

[okin]

@chlins I think that sequential processing of replication jobs makes sense at least if they are grouped per replication policy.
Right now I can't think of a reason why I would want to have overlapping replications that stem from the same policy. Different replication policies should still be able to run overlapping.

Thanks for the suggestion with working through the API. I'll have a look into this.

[Santhosh-Radhakrishna]

@chlins its been over a year, do we have any update on this topic - to enable sequential replications to a job?
because we are seeing different problems at index and s3 level when there are too many executions in parallel of the same job. Happens randomly, but it is hard to nail it and reproduce.

[bupd]

do we have any update on this topic - to enable sequential replications to a job?

Yes, I am currently working on it. Since a lot of people facing this problem. Will soon raise a PR with changes needed to fix this.

Thanks