Incorrect results with MPS #435
Comments
|
@constantinpape My Mac has the M3 chip or more specifically it's M3 Max. I also just checked that on my private Macbook with the M1 chip. There I get the same errors. |
Thanks for checking. And this inconsistency with our tests in the CLI is weird... |
|
I don't see a test error when I run I'm using:
Details of pytest output (click to expand)
Details `conda list` (click to expand)
|
|
Re-reading, it seems you want to know what happens for those specific versions of pytorch and torchvision. Here are those results, but I still don't see any errors and it doesn't seem any different than before.
Details of pytest output
Details of conda environment
|
|
We should double check this again. I am confused that this happens with an M1 chipset for your private laptop @Marei33. |
|
We investigated this a bit further, and the only remaining test error on both of @Marei33 is this one: 
This is not so critical, because the corresponding prompt is not really used in the annotator tools. It's also gotten a bit more difficult to debug this due to #558. (So I will unlink this from the milestone.) For reference, here are the two environments where the error occurs for @Marei33 : Environment on M1(click to expand)
packages in environment at /Users/marei/mambaforge/envs/sam:Name Version Build Channelabsl-py 2.1.0 pyhd8ed1ab_0 conda-forge Environment on M3(click to expand)
packages in environment at /Users/marei/mambaforge/envs/sam:Name Version Build Channelabsl-py 2.1.0 pyhd8ed1ab_0 conda-forge Both use pytorch 2.1.2; my current guess is that this is causing this, I will recheck here once the MAC CI is running again. |
|
Just for the record: |
|
Chiming in as a user of micro-sam, and maintainer of the pytorch package at conda-forge.
Looking at @GenevieveBuckley 's environment, the problem starts with "mixing" I would strongly suggest "not debugging" anything that has packages from Now for some knowledge:
Our conda-forge packages are getting better, but it is very unlikely that we will go back and retroactively fix older versions. Unless you are hitting the bug in conda-forge/pytorch-cpu-feedstock#301, I think the conda-forge packages are pretty good for Mac M1-M4. I would try to recreate the issue with conda-forge only (you will want to merge conda-forge/micro_sam-feedstock#25 for your users to be able to do this). Thanks for the cool software! |
|
Hi @hmaarrfk ,
I see, and that makes sense. A 'historical' problem for me used to be that the GPU Pytorch version from conda-forge were not available, or not working properly, especially w.r.t. enabling choice of the correct CUDA version. But I haven't revisited this since maybe 2020, so things have probably evolved a lot since then, and due to the discontinuation of the pytorch conda channel we should revisit this. Is there some easy write up that explains how to get pytorch versions in a controllable way (i.e. selecting CPU vs. GPU and controlling the CUDA version) for the pytorch conda-forge version? |
yes, this is in fact true..... when i created the "pytorch" (2018/2019...) feedstock at conda-forge i called it Many have come in to help with that!!!! CUDA 12 has immensely helped with system compatibility too. Honestly, the command should in general get you to "the most optimized pytorch". However, sometimes there are packages that have strange pins (such as micro_sam on conda-forge for now!) that might kick you into a lesser constrained pytroch, pytorch without cuda.... So I would keep the following in your back pocket to troubleshoot: and if you are on a CPU machine, and need to trick conda into thinking you have cuda, see Give the GPU package on a linux machine. |
|
|
This should be fixed by the new release, which uses |
I'm very curious to know if you have confirmed this, or if you are simply stating this on the conjecture i shared. |
|
I didn't test it, as I don't have a Mac at hand. However, the corresponding test is running without issues on the Mac CI, so I am quite sure it's due to version issues. But if anyone has a Mac at hand it wouldn't hurt to test this again with a clean, conda-forge only, installation. |
|
On my M1: note that CI doesn't have access to MPS: |
|
There is oddly a whole lot of activity on the pytorch feedstock at conda-forge recently (it isn't always like that) and if you can create a minimum reproducing code that can recreate this issue i think it might be resolvable in the long run.... MPS is known to be experimental though... |
@Marei33 observed incorrect results when using MPS results (including failing tests) when using the MPS backend.
This happened with two torch / torchvision settings:
pytorch = 2.0.1; torchvision: 0.15.2pytorch=2.2.1 und torchvision=0.17.1For some reason this is not picked up by our tests that run on
MAC-Latest@Marei33: what chipset does your Mac have? M1, M2 or M3?
@GenevieveBuckley: could you check if this still works on your MAC (you can just run
test/test_prompt_based_segmentation.py).The text was updated successfully, but these errors were encountered: