Python 3.4 changed pkgutil.get_loader behavior

[mitgr81]

In doing a test upgrade to python 3.4 I encountered the following traceback (snippet)

  File "/my/virtualenv/wonderboom34/lib/python3.4/site-packages/flask/app.py", line 319, in __init__
    template_folder=template_folder)
  File "/my/virtualenv/wonderboom34/lib/python3.4/site-packages/flask/helpers.py", line 742, in __init__
    self.root_path = get_root_path(self.import_name)
  File "/my/virtualenv/wonderboom34/lib/python3.4/site-packages/flask/helpers.py", line 631, in get_root_path
    loader = pkgutil.get_loader(import_name)
  File "/usr/local/Cellar/python3/3.4.0/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pkgutil.py", line 467, in get_loader
    return find_loader(fullname)
  File "/usr/local/Cellar/python3/3.4.0/Frameworks/Python.framework/Versions/3.4/lib/python3.4/pkgutil.py", line 488, in find_loader
    return spec.loader
AttributeError: 'NoneType' object has no attribute 'loader'

I was able to resolve it with the following:

diff --git a/helpers.py b/helpers.py
index 62a64bb..f671c27 100644
--- a/helpers.py
+++ b/helpers.py
@@ -628,7 +628,10 @@ def get_root_path(import_name):
         return os.path.dirname(os.path.abspath(mod.__file__))

     # Next attempt: check the loader.
-    loader = pkgutil.get_loader(import_name)
+    try:
+        loader = pkgutil.get_loader(import_name)
+    except AttributeError:
+        loader = None

     # Loader does not exist or we're referring to an unloaded main module
     # or a main module without path (interactive sessions), go with the
@@ -658,7 +661,10 @@ def find_package(import_name):
     folder structure exists (lib, share etc.).
     """
     root_mod_name = import_name.split('.')[0]
-    loader = pkgutil.get_loader(root_mod_name)
+    try:
+        loader = pkgutil.get_loader(root_mod_name)
+    except AttributeError:
+        loader = None

     if loader is None or import_name == '__main__':
         # import name is not found, or interactive/main module

If you want a formal pull request, I'd be happy to provide one, and do it "right", this feels pretty brute-force, I'm sure there's a better way.

[mitgr81]

Some forgotten information: Python 3.4 changed pkgutil.get_loader to align with PEP-451.

[mitgr81]

That PR was done just using the inline editor. If I get time sometime I'll get another one with tests and such, but there's at least that.

[zooba]

I chatted with Eric Snow about this at PyCon (Eric did the pkgutil work) and he's fixed it for 3.4.1. Until then, this is probably the best available solution.

[mitsuhiko]

Closing this here. The latest code already deals with 3.4 support.

[pfmoore]

The latest code already deals with 3.4 support.

Can you clarify? I have Flask 0.10.1 (the lastest, I believe) and Python 3.4.1, and I'm getting this error (specifically if I install snakemake and do snakemake --gui I get this error). What do I need to upgrade to fix this?

[methane]

@pfmoore It's because case problem.
https://bitbucket.org/johanneskoester/snakemake/src/1869115556819b7a165dc695a5b25330e3e3ea81/snakemake/gui.py?at=master#cl-11
package name is snakemake but it calls Flask('Snakemake', ...

[pfmoore]

@methane - ah, thanks. I hadn't spotted that

[rsyring]

Can Flask revisit fixing this for Python 3.4.0? With Ubuntu Trusty having 3.4.0 installed, there are environments where upgrading to 3.4.1 to deal with this bug is a pain.

[untitaker]

Considering for next bugfix release.

[rsyring]

thank you for reconsidering!

[untitaker]

I am currently reconsidering fixing this issue. I don't think it's a good idea for Flask to work around this behavior, see #1481.

The fix seems easy, but Flask/Werkzeug have already far too much cruft accumulated that results in accepting workarounds like these. I also don't think that Python 3 currently has a userbase big enough for our workaround to matter, which would probably be released when it's already too late and Ubuntu has updated its Python 3 package.

If you really have to use 3.4.0, doing this fix seems easy enough to do in usercode through monkeypatching:

import pkgutil
orig_get_loader = pkgutil.get_loader
def get_loader(name):
    try:
        return orig_get_loader(name)
    except AttributeError:
        pass
pkgutil.get_loader = get_loader

I'll still leave this issue open, because I'm not the only one to decide this, and we already recieved a considerable amount of bug reports because of this issue.

[rawrgulmuffins]

I can relate with not wanting to add cruft to a code base. My understanding is that since Django switched to Python3 by default that the user base for 3 is a decent chunk (numbers I heard at PyCon ranged from 4-15%). I found some stats but I'm not sure how reliable they are.

I'm unsure how large the user base really is at this point. But I am fairly sure it's large enough that if Monkey patch is an acceptable answer than a documentation change should happen (at least) because I spent a hefty chunk of time tracking this issue down.

[jeffwidman]

The fix seems easy, but Flask/Werkzeug have already far too much cruft accumulated that results in accepting workarounds like these. I also don't think that Python 3 currently has a userbase big enough for our workaround to matter, which would probably be released when it's already too late and Ubuntu has updated its Python 3 package.

Agree completely.

a documentation change should happen (at least) because I spent a hefty chunk of time tracking this issue down.

Thanks for tracking this down!

This issue (including the monkey-patch solution) should suffice for docs as it's easily found by googling the error message. For now, I'm going to go ahead and close this. If any of the other maintainers disagree, feel free to re-open.