Silent failure on Windows without python-magic-bin

[tolomea]

When I import magic the Python process immediately exits with status 9 without printing any error message.

This is due to problems in my environment and I don't mind that it didn't work, the problem is how it failed.

I'm trying to bring up an existing Django project on a new machine. I have
python-magic 0.4.18
Python 3.8.6 64bit
Windows 10.0.19041
and I'm running inside the bash/cygwin environment that came with Git for Windows 2.21.0.windows.1

Critically I did not have python-magic-bin installed.

When I import magic the Python process immediately exits with status 9 without printing any error message.

Of course that import was buried down in some random python file and tracing my silent failure back to python-magic took hours.

[ahupp]

magic should fail with an exception if it can't find the DLL. Is there something in the environment that's preventing that error from being printed?

[ahupp]

Specifically, this error: https://github.com/ahupp/python-magic/blob/master/magic.py#L211

[tolomea]

I will do some further digging next time I have access to that machine.
Specifically I'll try and determine if that line is running and also what is the last statement that runs before the interpreter exits.
I'm not sure if it's relevant at all but according to errno.py the status 9 that the interpreter returns means "EBADF: Bad file number"

[tolomea]

It fails here

python-magic/magic.py

Line 189 in 2a755be

libmagic = ctypes.CDLL(dll)

At which point dll = "C:\Program Files\Git\usr\bin\msys-magic-1.dll"

With python-magic-bin installed dll is "C:\redacted\site-packages\magic\libmagic\libmagic.dll"

Wrapping that line in a try except print shows that there's a "OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed"

As far as I can see somewhere in python importlib land that causes an interpreter exit.

  File "manage.py", line 27, in <module>
    execute_from_command_line(sys.argv)
  File "C:\redacted\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "C:\redacted\site-packages\django\core\management\__init__.py", line 377, in execute
    django.setup()
  File "C:\redacted\site-packages\django\__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "C:\redacted\site-packages\django\apps\registry.py", line 122, in populate
    app_config.ready()
  File "C:\redacted\site-packages\django\contrib\admin\apps.py", line 24, in ready
    self.module.autodiscover()
  File "C:\redacted\site-packages\django\contrib\admin\__init__.py", line 24, in autodiscover
    autodiscover_modules('admin', register_to=site)
  File "C:\redacted\site-packages\django\utils\module_loading.py", line 51, in autodiscover_modules
    import_module("%s.%s" % (app_config.name, module_to_search))
  File "C:\redacted\Python38\lib\importlib\__init__.py", line 134, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\redacted\admin\__init__.py", line 41, in <module>
    from redacted.forms import (
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\redacted\forms.py", line 4, in <module>
    import magic
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\redacted\site-packages\magic.py", line 207, in <module>
    traceback.print_stack()

I can see the error go past

  File "C:\redacted\admin\__init__.py", line 41, in <module>
    from redacted.forms import (

but the interpreter never makes it back to

  File "C:\redacted\Python38\lib\importlib\__init__.py", line 134, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

[ahupp]

I can't say why that DLL is failing to load, maybe you have a 32-bit install of msys and are running 64-bit python?

python-magic could fix this by catching the exception, logging and it moving on to the next DLL. I'm wary though that this would be even less obvious; you didn't get any output from this issue (which is a bug in your application I guess?) and this is even more likely to get lost.

Did you figure out why the error wasn't getting reported on startup?

[tolomea]

Only that it seems like import lib converts the exception into a sys.exit

[tolomea]

fwiw it looks like an AMD64 bit dll and it's python definitely reports AMD64

[ahupp]

Does this happen if you directly call import magic, outside of a django environment?

[ahupp]

It would be surprising if importlib exited. Seems more likely that there's an intermediate module doing something like this:

try:
import blah
except:
sys.exit()

And then blah here depends on magic (or is magic, I guess). Could you run this in the debugger, put import pdb; pdb.set_trace() where you catch the exception in magic.py, and then step through until you find the exit?

[tolomea]

pdb was a good suggestion, I should have thought of that
unfortunately it doesn't help much, it just disappears here
https://github.com/python/cpython/blob/db455296be5f792b8c12b7cd7f3962b52e4f44ee/Lib/importlib/__init__.py#L127
the last thing I see in pdb is _gcd_import returning None
I don't see anything in import_module, it just exits

[ahupp]

Did you step on _gcd_import(), or from the spot farther down in magic.py where we throw?

[tolomea]

I started in magic.py and stepped until it exited

[EllieTheYeen]

I just had this issue and it was a pain to have until I stumbled on this post and installed python-magic-bin which made it work again.

I run Python 3.8.5 64 bit on Windows where it crashed with a segfault.

On Pypy 3.7.4 it instead failed gracefully with
ImportError: failed to find libmagic. Check your installation
until I installed python-magic-bin

[tolomea]

also often I need to uninstall and reinstall python-magic-bin, this tends to be necessary around other version updates

[tolomea]

I haven't had time to look into that part of it, but it seems like it's critical that python-magic-bin is installed after python-magic, so if python-magic gets updated it becomes necessary to uninstall and reinstall python-magic-bin

[ahupp]

Merging into #293