Can an MLGraphBuilder be reused?

[reillyeon]

While it doesn't seem impossible for implementations to support building multiple MLGraph instances out of a single MLGraphBuilder instance (potentially reusing subsets of the graph by choosing different inputs and outputs) it seems like it adds complexity to implementations to support.

Are there any known use cases for MLGraphBuilder reuse?

[zolkis]

Side info: the discussion in #303 on the relationship between context, builder and graph may be relevant, notably comment, comment, comment, and (mainly) comment.

When getting familiar with this spec, I also ran a few circles around use cases vs the relationships above, some of my doubts answered (e.g. in #302). While the current API shape does have its rationale, not all discussions have been closed, and we might want to record the relevant design decisions as well. I am following this up (on reusing builder for multiple graphs).

[inexorabletash]

Relevant tidbit:

• If MLGraphBuilder can't be re-used, then input() can synchronously validate that input names are unique.
• If MLGraphBuilder can be re-used, then at build() time the graph needs to be traversed to identify inputs connected to the outputs, and uniqueness validated then.

[inexorabletash]

Further observations:

• Currently, since MLGraphBuilder can be re-used, then creating an input() operand that is not connected to any output is permitted - two disconnected graphs could be being constructed in parallel. If we change that, then there's a choice: is creating a disconnected input an error at build() time or not?
  • If it's an error, then we help developers realize they've created unused operands.

[fdwr]

Are there any known use cases for MLGraphBuilder reuse?

🤔 Well certainly most of the time you have one model and one unique set of weights related to that model, but there may be interesting preprocessing cases that could benefit from a graphBuilder's constant reuse. One could load several model weights, preprocess part of the model eagerly (like folding LORA weights into SD via WebNN), then want to construct the full graph reusing the unchanged weights and the new weights, rather than resupply them again (which could reupload old ones to the device).

If MLGraphBuilder can't be re-used, then input() can synchronously validate that input names are unique.

That is a nice property to have.

creating an input() operand that is not connected to any output is permitted ... If it's an error, then we help developers realize they've created unused operands.

I can foresee cases where graphs are generated programmatically, and requiring all inputs to always be consumed could be an annoyance. I've definitely seen unused weight inputs in models before (arguably they should have been cleaned out to save space), and if you have apps that try to straight-forwardly map that model's contents to WebNN calls, it will be a bother for the app to add a lot of extra uniqueness book-keeping just to avoid an error that doesn't give the app much actionable information anyway - the app isn't going to modify the model and send email to the model's author that there was unused content in it 😉, and if after building all nodes for WebNN only to realize none of the nodes included that input, the app can't rewind and un-input the input.

Such diagnostic information would be more useful for people calling the API directly, but the audience for that is pretty slim, kinda like how for WebGPU when displaying a 3D model it's not expected that you would call WebGPU API's directly, but rather WebGPU is an implementation to display existing model assets, be they FBX, COLLADA DAE, Wavefront OBJ...

[huningxin]

Are there any known use cases for MLGraphBuilder reuse?

There is another one I mentioned in #614 .

Today when ONNXRuntime WebNN EP inferencing transformer decoders, like Whisper, because WebNN doesn't support If operator, there will be two WebNN sub-graphs being built. One with past Key Value (KV) cache ("with_past") and another one without past KV cache ("no_past"). The inference code will run the "no_past" sub-graph for the first iteration and run the "with_past" sub-graph for the following iterations. The two sub-graphs actually share some common weights. It would be useful if the same weights being built by builder.constant() can be taken by the operators of two sub-graphs.

[reillyeon]

This use case makes sense but it assumes that implementations are able to optimize the case where multiple graphs share operands (or at least constants). The Chromium implementation so far is not capable of this but I can see ways in which it could be however it does somewhat depend on the underlying platform framework as well. This will have to be an area of future design exploration but given the potential it seems reasonable to at least continue to allow builder reuse even if implementations aren't yet taking advantage of the optimization potential.

One thing to consider however is whether an implementation could require all sub-graphs to be built at the same time rather than piecemeal. It seems optimal for implementations which have to convert to another format (e.g. a TFLite or Core ML model) to be able to do this once rather than needing to support building on additional graphs after the fact.

[reillyeon]

An additional thought I had while discussing this with @a-sully is that if an implementation did try to allow constants to be efficiently reused by multiple graphs but didn't require those graphs to be built at the same time then the builder would need to continue to own a copy of the constant data until it is freed instead of being able to pass ownership of that memory to the underlying platform. This could cause this attempt at memory optimization to backfire.

[huningxin]

@reillyeon

if an implementation did try to allow constants to be efficiently reused by multiple graphs but didn't require those graphs to be built at the same time then the builder would need to continue to own a copy of the constant data until it is freed instead of being able to pass ownership of that memory to the underlying platform.

If a builder allows to bind MLBuffer to a constant, constant MLBuffer proposal, would that solve this issue?

[a-sully]

There are still a number of open questions regarding passing MLBuffers as constants, but (depending on how we end up defining it!) yes it could solve the issue of needing to perpetually keep a copy of the data around.

Taking a step back, there are at least four distinct problems I see with constant() which are worth discussion:

1. Releasing memory when it's no longer needed
2. Sharing weights across MLGraphs
3. Avoiding copies
4. Inability to incrementally upload weights

I think these discussions of (2), (3), and (4) are best left for another issue, where we can discuss other ideas such as allowing fetched data to be streamed into constant() (avoiding the copy to an ArrayBuffer in the first place) and what sharing weights across graphs would actually entail (e.g. see #614 (comment)). I'm inclined to believe that we need a new paradigm for tackling these issues - likely something which is tied to an MLContext (e.g. constant MLBuffers) rather than an MLGraphBuilder. The variant of constant() which takes an ArrayBufferView, assuming we still want to keep it, should reflect constants for that graph. In this case, problem (1) is particularly relevant.

I'll note that the Chromium implementation hid problem (1) for a long time by not actually copying constant() data during the constant() method, as the spec says, but by waiting until the build() method. I just put up a change to fix this, which makes it very obvious how wasteful this strategy is. As long as the corresponding MLGraphBuilder exists, build() may be called on it, which means we cannot release this copy of the constant data until both its builder and corresponding MLOperand are GCed.

Other than sharing weights across graphs (which, again, I think the current MLGraphBuilder-based, ArrayBufferView-based method is not sufficient for), are there reasons for calling build() more than once on an MLGraphBuilder? Do those reasons outweigh the costs of keeping around a copy of all the constant data indefinitely? (if so, do we need a MLGraphBuilder.destroy() method?)