I as a test creator can't separate service tasks mocking with the same Zeebe Job Type

[aivinog1]

Describe the bug
For example, I have such a process:

serviceTasksWithSameTypeAndErrorHandling.bpmn.zip

If I will test a happy path, it is pretty easy:

completeTask(jobType = "updateStatus")
// Note: I can have a just single completeTask(jobType = "updateStatus") because we have already created Zeebe Worker, but end users may not know end implementation of completeTask
completeTask(jobType = "updateStatus")

But if I will decide to test an error path I will get a flaky test because the job in the element with error handling could be activated by the worker from completeTask or throwError:

completeTask(jobType = "updateStatus")
// we create a worker with the same time without closing the previous one which leads to the problem
throwError(jobType = "updateStatus", errorCode = "updateStatusError")
completeTask(jobType = "doSmthngWithErr") // time to time we can't reach it
// other stuff
verifications {
    elementInstanceState(
        selector = ElementSelector.byName("Do something with error"),
        state = ElementInstanceState.COMPLETED // sometimes could lead to the error in the test
    )
}

To Reproduce
Steps to reproduce the behavior:

1. Create or use the process from above
2. Create a test with error handling verifications
3. Obtain the test error

Expected behavior
I expect that a such test will never be flaky.

Additional context
I see the 3 options that we have to fix it:

1. One-time workers. As soon the worker completes/throws an error a job it will close. But it breaks backward compatibility in some cases (this is doubtful because their tests could be just false positive, I need you to argue with me at this point 🙂 ) and maybe not 100% of the solution (for example, we complete the job, start closing the worker and the next job comes (we could also implement counter for such scenario, but it becomes ugly 🙂 ))
2. Add some sort of matcher or selector for completeTask. Again, as I can see this could be not 100% fixed, because in the worst case scenario, all jobs will be executed by one worker, which will never complete the job (and it will be reactivated again, but could be taken again by the "wrong" worker).
3. Call the GRPC API instead of Zeebe Worker. It is close to the one-time worker but just with JobClient instead of using the Zeebe Worker. For example, as soon as completeJob calling we create a single ActivateJobs command with exactly one job. And complete it via the CompleteJob command. In this case, we could get a situation when a job is not ready for activation, but the ActivateJobs request already came.

So, as you can see I can't find a good solution, maybe you can vote/help me, @saig0? 🙂

[saig0]

@aivinog1 interesting. Thank you for raising it. 👍

What do you think about adding an optional number of invocations to the completeTask() command? Similar to your first option.

If the number is set to x, it completes a job of this type x times at max. If no number is set, it completes all available jobs.

It could look like this:

completeTask(jobType = "updateStatus", invocations = 1)
throwError(jobType = "updateStatus", errorCode = "updateStatusError")
completeTask(jobType = "doSmthngWithErr") 

[aivinog1]

What do you think about adding an optional number of invocations to the completeTask() command? Similar to your first option.

Hey, @saig0!
Thank you for your solution, but I think that we should not complete jobs x times at max, but activate them. Otherwise, there could be a case where a worker that already completes jobs x times constantly tries to activate such jobs, do nothing and the broker makes them activable again, and in the worst case scenario this could be infinitely repeated as I already described here:

Add some sort of matcher or selector for completeTask. Again, as I can see this could be not 100% fixed, because in the worst case scenario, all jobs will be executed by one worker, which will never complete the job (and it will be reactivated again, but could be taken again by the "wrong" worker).

[saig0]

Good point.

But I think about replacing the job worker for completing a job with a simple complete job command. By only using the complete command, we could avoid issues with job activation.