Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

🐛 [Bug] Graph breaks in torch 2.7.0.dev but not in torch 2.6.0 #3375

Closed
HolyWu opened this issue Jan 31, 2025 · 2 comments
Closed

🐛 [Bug] Graph breaks in torch 2.7.0.dev but not in torch 2.6.0 #3375

HolyWu opened this issue Jan 31, 2025 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@HolyWu
Copy link
Contributor

HolyWu commented Jan 31, 2025

Bug Description

The node torch.ops.aten._assert_tensor_metadata.default occurred when using torch 2.7.0.dev, which was the culprit causing graph breaks.

To Reproduce

from __future__ import annotations

import os

import torch
import torch_tensorrt

os.environ["CI_BUILD"] = "1"


class MyModule(torch.nn.Module):
    def __init__(self) -> None:
        super().__init__()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = x + x
        x = x.float()
        x = x * x
        x = x.half()
        x = x - x
        return x


with torch.inference_mode():
    model = MyModule().eval().cuda().half()

    inputs = [torch.randn(1, 3, 5, 7, dtype=torch.half, device="cuda")]

    trt_model = torch_tensorrt.compile(model, "dynamo", inputs, debug=True, min_block_size=1, use_explicit_typing=True)

    torch.testing.assert_close(trt_model(*inputs), model(*inputs), rtol=5e-3, atol=5e-3)
    print("assert_close passed")

Log diff: https://www.diffchecker.com/qe8cYJAS/

@HolyWu HolyWu added the bug Something isn't working label Jan 31, 2025
@narendasan narendasan self-assigned this Feb 18, 2025
@narendasan
Copy link
Collaborator

@peri044 has a fix

@narendasan narendasan assigned peri044 and unassigned narendasan Feb 21, 2025
@peri044
Copy link
Collaborator

peri044 commented Feb 21, 2025

This is fixed in #3395

@HolyWu HolyWu closed this as completed Feb 25, 2025
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants