[Feature Request] ORT web API to use WebGL texture as model input #15312
Labels
feature request
request for unsupported feature or enhancement
platform:web
issues related to ONNX Runtime web; typically submitted using template
Comments
github-actions
bot
added
the
platform:web
|
@fs-eire - Can tou please take a look ? |
|
feature request: IO binding (webGL) |
|
@fs-eire any I was wondering if this issue is actively being worked on? I am also interested in contributing myself, but I'd need some guidance to dive into the project. If you'd be willing to help out that'd be great! :) |
|
I am thinking about the API design for this feature. To generalize the requirement, I think it can be expand to:
I can start with a branch for the API part and share it out later |
fs-eire
added a commit
that referenced
this issue
Jun 9, 2023
### Description <!-- Describe your changes. --> refactor tensor type in onnxruntime-common. ### Motivation and Context There major motivation is that I am doing a local change to address the API part of #15312. And I am doing a refactoring of onnxruntime-common anyway (#15772). The `tensor.ts` and `tensor-impl.ts` are too large, so I split contents into multiple files to make the type declarations clearer. The original target of this change is for API only ( ie. do not refactor any implementation.). However, there are a few type/implementation inconsistencies so I also made minimal changes to fix them. ### Changes - extract `TensorUtils` for non-template interfaces - extract `TensorFactory` for all overloads of `Tensor.fromImage()` - refactor options type that used for `Tensor.fromImage()` - fix JSDoc comments to make option descriptions consistent with actual type declarations - fix an inconsistency for `options.format` and `options.bitmapFormat`; change all `bitmapFormat` to `format` - extract `ConversionUtils` for `tensor.toDataURL()` and `tensor.toImageData()` - put implementations into multiple files from `tensor-impl.ts` - fix a bug that cause unittest fail. put comments for future fix.
|
@erikvdplas I created a new PR for this issue. the PR is now in design reviewing. please feel free to comments on it |
fs-eire
added a commit
that referenced
this issue
Aug 29, 2023
[//]: # (## Work In Progress. Feedbacks are welcome!) ### Description This PR adds a few properties, methods and factories to Tensor type to support IO-binding feature. This will allow user to create tensor from GPU/CPU bound data without a force transferring of data between CPU and GPU. This change is a way to resolve #15312 ### Change Summary 1. Add properties to `Tensor` type: a. `location`: indicating where the data is sitting. valid values are `cpu`, `cpu-pinned`, `texture`, `gpu-buffer`. b. `texture`: sit side to `data`, a readonly property of `WebGLTexture` type. available only when `location === 'texture'` c. `gpuBuffer`: sit side to `data`, a readonly property of `GPUBuffer` type. available only when `location === 'gpu-buffer'` 2. Add methods to `Tensor` type (usually dealing with inference outputs): - async function `getData()` allows user to download data from GPU to CPU manually. - function `dispose()` allows user to release GPU resources manually. 3. Add factories for creating `Tensor` instances: a. `fromTexture()` to create a WebGL texture bound tensor data b. `fromGpuBuffer()` to create a WebGPUBuffer bound tensor data c. `fromPinnedBuffer()` to create a tensor using a CPU pinned buffer ### Examples: create tensors from texture and pass to inference session as inputs ```js // when create session, specify we prefer 'image_output:0' to be stored on GPU as texture const session = await InferenceSession.create('./my_model.onnx', { executionProviders: [ 'webgl' ], preferredOutputLocation: { 'image_output:0': 'texture' } }); ... const myImageTexture = getTexture(); // user's function to get a texture const myFeeds = { input0: Tensor.fromTexture(myImageTexture, { width: 224, height: 224 }) }; // shape [1, 224, 224, 4], RGBA format. const results = await session.run(myFeeds); const myOutputTexture = results['image_output:0'].texture; ```
kleiti
pushed a commit
to kleiti/onnxruntime
that referenced
this issue
Mar 22, 2024
[//]: # (## Work In Progress. Feedbacks are welcome!) ### Description This PR adds a few properties, methods and factories to Tensor type to support IO-binding feature. This will allow user to create tensor from GPU/CPU bound data without a force transferring of data between CPU and GPU. This change is a way to resolve microsoft#15312 ### Change Summary 1. Add properties to `Tensor` type: a. `location`: indicating where the data is sitting. valid values are `cpu`, `cpu-pinned`, `texture`, `gpu-buffer`. b. `texture`: sit side to `data`, a readonly property of `WebGLTexture` type. available only when `location === 'texture'` c. `gpuBuffer`: sit side to `data`, a readonly property of `GPUBuffer` type. available only when `location === 'gpu-buffer'` 2. Add methods to `Tensor` type (usually dealing with inference outputs): - async function `getData()` allows user to download data from GPU to CPU manually. - function `dispose()` allows user to release GPU resources manually. 3. Add factories for creating `Tensor` instances: a. `fromTexture()` to create a WebGL texture bound tensor data b. `fromGpuBuffer()` to create a WebGPUBuffer bound tensor data c. `fromPinnedBuffer()` to create a tensor using a CPU pinned buffer ### Examples: create tensors from texture and pass to inference session as inputs ```js // when create session, specify we prefer 'image_output:0' to be stored on GPU as texture const session = await InferenceSession.create('./my_model.onnx', { executionProviders: [ 'webgl' ], preferredOutputLocation: { 'image_output:0': 'texture' } }); ... const myImageTexture = getTexture(); // user's function to get a texture const myFeeds = { input0: Tensor.fromTexture(myImageTexture, { width: 224, height: 224 }) }; // shape [1, 224, 224, 4], RGBA format. const results = await session.run(myFeeds); const myOutputTexture = results['image_output:0'].texture; ```
siweic0
pushed a commit
to siweic0/onnxruntime-web
that referenced
this issue
May 9, 2024
### Description <!-- Describe your changes. --> refactor tensor type in onnxruntime-common. ### Motivation and Context There major motivation is that I am doing a local change to address the API part of microsoft#15312. And I am doing a refactoring of onnxruntime-common anyway (microsoft#15772). The `tensor.ts` and `tensor-impl.ts` are too large, so I split contents into multiple files to make the type declarations clearer. The original target of this change is for API only ( ie. do not refactor any implementation.). However, there are a few type/implementation inconsistencies so I also made minimal changes to fix them. ### Changes - extract `TensorUtils` for non-template interfaces - extract `TensorFactory` for all overloads of `Tensor.fromImage()` - refactor options type that used for `Tensor.fromImage()` - fix JSDoc comments to make option descriptions consistent with actual type declarations - fix an inconsistency for `options.format` and `options.bitmapFormat`; change all `bitmapFormat` to `format` - extract `ConversionUtils` for `tensor.toDataURL()` and `tensor.toImageData()` - put implementations into multiple files from `tensor-impl.ts` - fix a bug that cause unittest fail. put comments for future fix.
siweic0
pushed a commit
to siweic0/onnxruntime-web
that referenced
this issue
May 9, 2024
[//]: # (## Work In Progress. Feedbacks are welcome!) ### Description This PR adds a few properties, methods and factories to Tensor type to support IO-binding feature. This will allow user to create tensor from GPU/CPU bound data without a force transferring of data between CPU and GPU. This change is a way to resolve microsoft#15312 ### Change Summary 1. Add properties to `Tensor` type: a. `location`: indicating where the data is sitting. valid values are `cpu`, `cpu-pinned`, `texture`, `gpu-buffer`. b. `texture`: sit side to `data`, a readonly property of `WebGLTexture` type. available only when `location === 'texture'` c. `gpuBuffer`: sit side to `data`, a readonly property of `GPUBuffer` type. available only when `location === 'gpu-buffer'` 2. Add methods to `Tensor` type (usually dealing with inference outputs): - async function `getData()` allows user to download data from GPU to CPU manually. - function `dispose()` allows user to release GPU resources manually. 3. Add factories for creating `Tensor` instances: a. `fromTexture()` to create a WebGL texture bound tensor data b. `fromGpuBuffer()` to create a WebGPUBuffer bound tensor data c. `fromPinnedBuffer()` to create a tensor using a CPU pinned buffer ### Examples: create tensors from texture and pass to inference session as inputs ```js // when create session, specify we prefer 'image_output:0' to be stored on GPU as texture const session = await InferenceSession.create('./my_model.onnx', { executionProviders: [ 'webgl' ], preferredOutputLocation: { 'image_output:0': 'texture' } }); ... const myImageTexture = getTexture(); // user's function to get a texture const myFeeds = { input0: Tensor.fromTexture(myImageTexture, { width: 224, height: 224 }) }; // shape [1, 224, 224, 4], RGBA format. const results = await session.run(myFeeds); const myOutputTexture = results['image_output:0'].texture; ```
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Describe the feature request
I would like to be able to run models directly on WebGL textures as input. Right now, the only possibility is to first render such a WebGL texture to bitmap, then copying it to a Float32Array which can then be used to initialize an ORT Tensor object, which internally is again copied back to a WebGL texture when used with the WebGL execution backend. All this copying adds quite some overhead, which cannot be neglected, especially in real-time applications.
Describe scenario use case
It is not uncommon for image models to work on webcam/streams/video feeds or in general WebGL data. In the case of video feeds, they can efficiently be copied to a WebGL buffer using the
texImage2Dmethod in a WebGL rendering context (by directly the HTMLVideoElement as the source).If someone is willing to guide me through the process of correctly building the onnxruntime-web package from source locally, I could try to implement this feature myself... but maybe someone more skilled and familiar with the project could help out as well.
The text was updated successfully, but these errors were encountered: