Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Support coordinate transformation modes for Resample2d #270

Closed
Honry opened this issue May 31, 2022 · 5 comments
Closed

Support coordinate transformation modes for Resample2d #270

Honry opened this issue May 31, 2022 · 5 comments

Comments

@Honry
Copy link
Contributor

Honry commented May 31, 2022

Issue: DeepLabV3 contains Resize node in ONNX model and ResizeBilinear in TFLite model. It's corresponding op in WebNN is Resample2d. Resize node contains coordinate_transformation_mode=align_corners, ResizeBilinear node contains align_corners=true and half_pixel_centers=false. While WebNN doesn't support such coordinate transformation modes and its default behavior equals to align_corners=false and half_pixel_centers=true.

Compares all the behaviors in different backends and frameworks as follows:

TFLite (half_pixel_centers, align_corners) ONNX (coordinate_transformation_mode enum) OpenVINO (coordinate_transformation_mode enum) DML WebNN
half_pixel_centers=true half_pixel half_pixel Supported Default behavior
N/A pytorch_half_pixel pytorch_half_pixel Supported N/A
half_pixel_centers=false align_corners=false asymmetric asymmetric Supported N/A
N/A N/A tf_half_pixel_for_nn N/A N/A
align_corners=true align_corners align_corners Supported N/A
N/A tf_crop_and_resize N/A Supported N/A

Note: DML backend can support these modes by calculating its InputPixelOffsets and outputPixelOffsets members.
You can refer to its implementation in ONNXRumtime DML backend.

Open: Can we support these coordinate transformation modes in WebNN? At least for asymmetric, half_pixel_centers and align_corners. Or maybe we could refer to DML by defining InputPixelOffsets and outputPixelOffsets options in Resample2d to implement various coordinate transformation modes.

@fdwr
Copy link
Collaborator

fdwr commented Nov 13, 2024

Most of these resampling transformations were mistakes (yes, I added them to DML for completeness using a generic transform approach, but I don't recommend propagating them all into perpetuity). The correct approach from computer vision/imaging experts is to resample the center point of the pixels, not asymmetrically (which causes image shifting) nor as point samples (which causes misalignment). That approach (half pixel) is used by libraries like OpenCV, SciPy, Matlab. Recent versions of PyTorch and TF do the right thing too, and it's ONNX's default.

Also see: https://medium.com/hackernoon/how-tensorflows-tf-image-resize-stole-60-days-of-my-life-aba5eb093f35

@inexorabletash
Copy link
Member

What do folks think about closing this issue? I think @fdwr has given plenty of evidence here and in #358 and #816 that what WebNN currently provides are what graphics experts recommend, and other coordinate transformation modes are a historical mistake.

@fdwr
Copy link
Collaborator

fdwr commented Feb 14, 2025

What do folks think about closing this issue? I think @fdwr has given plenty of evidence here and in #358 and #816 that what WebNN currently provides are what graphics experts recommend, and other coordinate transformation modes are a historical mistake.

Now mind you, this is from an imaging perspective, which is the primary use for resampling anyway, but if someone was using resampling in some other domain like audio, or for some other clever use (such as quickly initializing a linearly spaced 1D series of numbers 0 to 100 by interpolating a 2-element tensor into a larger target tensor, like numpy.linspace) then we may want to support modes treating the endpoints as point samples rather than pixels. Though, we can always extend WebNN later when we hit the use case. ⌛... Content to close.

@fdwr
Copy link
Collaborator

fdwr commented Feb 14, 2025

I'd normally leave it to the issue opener to close an issue, but I see @Honry liked Joshua's comment along with Ningxin too. So closing. 📫➡️📪

@fdwr fdwr closed this as not planned Won't fix, can't repro, duplicate, stale Feb 14, 2025
@Honry
Copy link
Contributor Author

Honry commented Feb 14, 2025

Thanks @fdwr, @inexorabletash, +1 to close. 😊

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

No branches or pull requests

4 participants