Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add ONNX export support for DinoV2, Hiera, Maskformer, PVT, SigLIP, SwinV2, VitMAE, and VitMSN models #2001

Merged
merged 28 commits into from
Dec 19, 2024

Conversation

xenova
Copy link
Contributor

@xenova xenova commented Aug 29, 2024

What does this PR do?

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

Who can review?

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@xenova xenova changed the title Add ONNX export support for Hiera models Add ONNX export support for DinoV2, Hiera, Maskformer, PVT, SigLIP, SwinV2, VitMAE, and VitMSN models Aug 30, 2024
@SangbumChoi
Copy link

@xenova Hi xenova! I think you missed to add official support for maskformer

ONNX Model for semantic-segmentation, with an all-MLP decode head on top e.g. for ADE20k, CityScapes. This class officially supports segformer.

Copy link
Collaborator

@echarlaix echarlaix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the delay @xenova !! Looks so good, thanks for adding so many new models ❤️

Comment on lines 1207 to 1208
"feature-extraction",
"masked-im",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think "image-feature-extraction" (or "feature-extraction") should be enough https://huggingface.co/datasets/huggingface/transformers-metadata/blob/main/pipeline_tags.json#L1128

Suggested change
"feature-extraction",
"masked-im",
"feature-extraction",

if not the correct class needed to load the model can be added to _CUSTOM_CLASSES : for vit-mae I only see AutoModel or AutoModelForPreTraining

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed in 7a2e94a

"vit-msn": supported_tasks_mapping(
"feature-extraction",
"image-classification",
"masked-im",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same I think can be removed :

Suggested change
"masked-im",

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed in 01929b2

@echarlaix
Copy link
Collaborator

echarlaix commented Dec 18, 2024

Also seems there is an issue for Maskformer for the feature-extraction task

FAILED exporters/onnx/test_exporters_onnx_cli.py::OnnxCLIExportTestCase::test_exporters_cli_pytorch_cpu_357_maskformer_feature_extraction_default_hf_internal_testing_tiny_random_MaskFormerForInstanceSegmentation - RuntimeError: number of output names provided (1) exceeded number of outputs (0)

the expected output name seems to be "transformer_decoder_last_hidden_state" https://github.com/huggingface/transformers/blob/69e31eb1bf123d5bd0183b753da83ab7dd9c2eec/src/transformers/models/maskformer/modeling_maskformer.py#L1615 while feature-extraction gives "last_hidden_state", can be fixed by updating MaskFormerOnnxConfig

edit : should be fixed with 3fa346c

@echarlaix echarlaix merged commit 0c42291 into main Dec 19, 2024
39 of 41 checks passed
@echarlaix echarlaix deleted the add-hiera-onnx branch December 19, 2024 08:47
@xenova
Copy link
Contributor Author

xenova commented Dec 19, 2024

Thanks so much @echarlaix! 🚀

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants