Allow for custom shortfall evaluators

[bpiper]

Hey there,

Great to see the new OptimizingShortfallEvaluator. Any chance this class/interface hierarchy could be made public so that we can extend it to implement our own shortfall evaluation strategies?

The use case I have is where we are scheduling only short-lived tasks on a dedicated auto-scaling group (or possibly groups in the future). If there are tasks that have a lifetime in the order of seconds, then the current shortfall evaluation ends up grossly overestimating resource needs. Ideally we want to pseudo-schedule some pseudo tasks that represent what we think our resource requirements will be for the next n minutes (where n is probably derived from the auto-scaling cooldown period) based on currently running tasks and pending tasks (and maybe some task history that we record as well).

We might even just start with something fairly naive that doesn't even use pseudo-scheduling, so it would be cool just to be able to implement ShortfallEvaluator ourselves.

[spodila]

Thank you.
The OptimizingShortfallEvaluator needs to work in conjunction with TaskSchedulingService and as such wouldn't do well implemented outside. However, the shortfall evaluation can provide plugins based extensibility that you could use to achieve your results. For example, if there was a plugin that the evaluator called with the initial list of tasks for which shortfall analysis will be performed. Then, you can filter the list appropriately.

Would that cover your needs?

[bpiper]

I think it would in the sense that we could specify arbitrary pseudo-tasks (e.g. reduce 30 x 10-second tasks to effectively a single long-running pseudo-task) to achieve a desirable (or at least more accurate) scaling result.

Having said that, I still like the simplicity of ShortfallEvaluator's contract... it provides the things that Fenzo knows about (including TaskSchedulingService if one should want to do some pseudo-scheduling) and potentially allows an arbitrarily complex/custom algorithm to be performed to end up with the final Map<String, Integer> result. I think in the longer term we'd want to not just consider the immediate needs (i.e. tasks that could not be scheduled yet), but also factor in historical data so that we potentially adjust the shortfall based on what we expect might happen in the next 15 - 30 minutes (and thus avoid pointless down-scaling, or indeed do pre-emptive up-scaling). Using kludge factors to come up with a list of pseudo-tasks for pseudo-scheduling would likely become unintelligible pretty quickly (even to the author).

Shortfall evaluation is kind of a mini-science in itself, so I think it would be neat to see Fenzo users come up with different approaches for different use cases, some of which could perhaps be fed back into the Fenzo codebase eventually. From that perspective, I still like the thought of being able to implement ShortfallEvaluator myself and plug that in to the AutoScaler, but certainly anything that allows some custom logic to be plugged in to shortfall evaluation would be welcome.

[spodila]

I can see how this can be customized for your example. And likely other examples as well. My initial thought was more on handling the complexity in the "optimizing shortfall analysis" correctly. That requires tight integration with the scheduling service. However, other strategies for shortfall analysis can be achieved without such an integration with the internal scheduler. Your example analyzer can estimate shortfall without requiring a pseudo scheduling run, I think.

If that matches with your thinking, it might be worth the effort to put some detail on how the custom shortfall analyzers can be plugged in.

I agree with you, shortfall evaluation is kind of a mini-science in itself.

[bpiper]

Ideally shortfall evaluation (for us) needs to consider the duration of tasks... so from that perspective one would, for a given set of tasks that could not yet be scheduled, want to pseudo-schedule over a period of time... e.g. if I have some period of time for which scaling up can yield some benefit (which is going to be arbitrarily longer than the time it takes to bring a new slave node into service), then I want to know if I would actually have a shortfall based on the tasks that I expect to run within that period of time. So possibly we'd still want to do pseudo-scheduling, but just as one piece of a more complex equation... i.e. it's just a data point in time, and what we're concerned with is the shape of the graph over a period of time.

My thinking about this is still under-developed, but yes, just being able to specify a custom ShortfallEvaluator is something that would be useful... and it's conceivable that being able to do so with a sub-class of OptimizingShortfallEvaluator might also be useful.