Python 2.x and 3.x

[samueljohn]

Introduction

To avoid having multiple formulae with endings -py2 and -py3, we will handle support for different pythons (2.x vs. 3.x) in the same formula. Further brewed vs. external python will be transparently supported.

New python Requirements:

depends_on :python => :recommended
depends_on :python3 => :optional  # so --with-python3 option will be generated
# or to declare a specific version:
depends_on PythonInstalled.new("2.6")  

Most formulae should just work with this little change.

The python object that will support formula dev

In the Formula class there will be a python object available (python2 and python3, too), that supports some convenience methods which do the complicated work of figuring out the information no matter which python version or brewed vs. python from OS X:

python.site_packages # the site-packages in the Cellar for the current formulae that will later be linked globally
python.global_site_packages
python.binary # the full path to the python binary. For brewed python: HOMEBREW_PREFIX/opt/python/bin/python2
python.prefix
python.version
python.version.major
python.version.minor
python.xy # => e.g. "python2.7"
python.incdir # includes of python
python.libdir # the python dylib library
python.pkg_config_path # used internally by brew
python.from_osx?
python.framework?
python.universal?
python.pypy?
python.standard_caveats # The usual text to set PYTHONPATH that is only shown for python.from_osx?
python.if3then3 # => "" for 2.x and to "3" for 3.x.

Instead of using python, there is also python2 and python3 to explicitly refer to 2.x and 3.x resp..

Executing code only for 2.x or 3.x

It is also possible to

if python2
  # do something specific for python 2.x
end

if python3
  # do something specific for python 3.x
end

unless python
  # do something if there is no python (or python support was disabled --without-python --without-python3)
end

The python do ... end block

And one of the most important features is the python-block which creates the python.site_packages dir and already sets the PYTHONPATH correctly. This block is executes once for each python requirement.

Formulae that use a setup.py to install python bindings, can do

depends_on :python
depends_on :python3

python do
  # sets up PYTHONPATH and site-packages.
  # The python object within this block points to the python 2.x in the first round
  # and to 3.x in the second round.
  system python, "setup.py", "install", "--prefix=#{prefix}"
end

Are now all formulae automatically available for python 3.x?

No.

Adding Python 3.x support for individual formulae and testing that will take longer and I won't do this right now for all of the formulae. However, I did this for some to test things out.

Tasks:

• depends_on :python Requirement (python is a synonym for python2 here)
• depends_on :python3 Requirement
• Write the correct sitecustomize.py also if the requirement is used.
• Improve sitecustomize.py so that multiple versions (and external) are better supported.
• Write the distutils.cfg if the requirement is used.
• Update Python formulae to reuse sitecustomize.py
• Allow formulae to use setup.py --prefix=#{prefix} instead of the complicated stuff as is.
• pip will install executable scripts to HOMEBREW_PREFIX (solves a few issues)
• Provide python.standard_caveats
• Add Python's pkg-config during build
• Use a temporary build environment to not touch stuff outside of the python block
• Provide python.global_site_packages
• Provide python.binary to indirect the call to python
• Replace which_python by python.xy
• Provide python do block
• Provide python2 do block which is only run if a python 2.x has been found and unless --without-python.
• Provide python.incdir (sometimes needed for cmake based projects)
• Provide python.libdir (sometimes needed for cmake based projects)
• Provide python.version.major and python.version.minor
• Provide python.if3then3 which evaluates to "" for 2.x and to "3" for 3.x.
• Provide python.framework (is nil for non-framework builds)
• Provide python.universal?
• Provide python.pypy?
• Provide python.brewed?
• Provide python.from_osx?
• test :hg download strategy
• Write test_python.rb
• Do we want brew create --python ?
• Check that the user has not python -> python3 symlink, which is not recommended.
• Update audit to tell people to use the blocks and requirements

Formulae in homebrew core to update accordingly (list created via grep for python):

• bazaar
• beecrypt
• boost
• boost149
• bulk_extractor
• bup
• cantera
• cmu-sphinxbase
• collectd
• cracklib
• cvs2svn
• denyhosts
• diffuse
• disco
• discodex
• distcc
• duplicity
• enchant
• fail2ban
• fontforge
• fsh
• gdal
• git-cola
• git
• glade
• gnome-doc-utils
• google-app-engine
• gpsd
• grass
• grc
• gtk-doc
• io
• ldns
• ledger
• libcaca
• libdnet
• liblunar
• libming
• libplist
• libpst
• librets
• libstfl
• libxml2
• lilypond
• little-cms
• llvm
• lorem
• macvim
• mapnik
• mapserver
• mercurial
• node
• omni
• open-babel
• opencolorio
• orfeo
• oscats
• owfs
• pil
• portmidi
• postgres-xc
• postgresql
• py2cairo
• pyexiv2
• pygtk
• pygtkglext
• pygtksourceview
• pyqt
• pyqwt
• pyside-tools
• pyside
• python
• python3
• qscintilla2
• rabbitmq-c
• rabbitmq
• rdiff-backup
• rubber
• scons
• shiboken
• shocco
• sickbeard
• sip
• subversion
• swatchbooker
• thrift
• treeline
• uwsgi
• vim
• vte
• weechat
• wireshark
• wxmac
• xapian
• xchat
• yasm
• yle-dl
• zbar
• zookeeper

[samueljohn]

In that other comment is a short demo: #16439 (comment) how the python block looks. I post this here because the diff in the PR is large.

[jacknagel]

Regarding testing. It is fairly difficult to test things that are integrated with Formula due to the amount of things that are coupled to the filesystem. The best strategy is to push as much behavior as possible into external objects that can be tested in isolation. My suggestion would be to push the python code in formula.rb into a new class, and pass the formula object into it so you can still instance_eval stuff in the correct context, e.g.

def python(*args, &block)
  SomePythonThing.new(self, *args, &block)
end

From there it is much easier to write tests without dragging the formula machinery into it.

[samueljohn]

I am learning from the best.

[jacknagel]

I looked at this a bit more and I see that what I suggested isn't an exact solution, because python needs to respond differently based on block_given? and it keeps state. So it would have to be more along the lines of

def some_python_thing
  @some_python_thing ||= SomePythonThing.new
end

def python(*args, &block)
  some_python_thing.call(*args, &block)
end

so that the object is only constructed once. But the principle is still the same.

BTW, if we're ever online at the same time we should hop on IRC and chat about the requirements API, I have some ideas I want to run by you.

[thiessenstuart]

Should I still try out the steps you have outlined in #16439 or will it be better to wait until you have this ironed out? Just wanted to check before trying it out.

[samueljohn]

@qrider70 you don't have to but I am thankful for feedback. I assume you know to to use git to pull and reset the original state of your homebrew - if not; don't pull this. What should work already is the (re-)installation of python, python3, sip and pyqt. I am working on the other formulae...

[ghost]

You have depends_on :python, :python3, but python, python2 blocks. For me this seems strange. But might be that I don't get the logic ;) Should not python without a number always be the latest? Sorry to take up your time.

[samueljohn]

@MindTooth thanks for asking.

I imagine the following:

• depends_on :python and depends_on :python2 are synonyms. The reasoning is that the python and python2 executables are also synonyms.
• python do is the generic Python block that will be run for each python defined by the depends_on.
• python2 do is only run if a python 2.x is present.
• python3 do is only run if a python 3.x is present.

It is disputable if depends_on should only allow "python2" and "python3".

This allows us to write

depends_on :python2
depends_on :python3

def install
  python do
    <stuff that should be done for 2.x and 3.x>
  end

  python2 do
    <2.x specific handling here>
  end

  python3 do
    <3.x specific handling here>
  end
end

[thiessenstuart]

Ok. I haven't used git to do that yet. I could learn. :) At this point, what you mention as working (python, python3, sip, and pyqt) is all that I need afaik. Is there a way to use what you have to get just that, or is it better that I just wait until you have this fully ready? I am flexible either way for now. Thanks for your work on this. :)

[thiessenstuart]

Any news?

[samueljohn]

soonish :-)

[samueljohn]

Some formulae provide python bindings but don't need further work during the make step. Therefore, I made the python do ... end block return false if no python binding was requested (either by default or by --without-python) and then its possible to write:

if python do 
  system 'cmake', *args # python environment set up
end
else
  system 'cmake', *args # without python
end

This is really the same as

if python { system 'cmake', *args }
else
  system 'cmake', *args
end

where the latter makes it a bit more clear that python takes a block.

[samueljohn]

This takes me so long, because I have to rework all python-using formulae a little bit ...
Today, I made good progress by getting ten more formulae to work with this modification.

[freaky-m0]

Is there any progress in PySide for Python3? I would love to see it! Good work so far! :-)

[samueljohn]

@freaky-m0 my changes in the core (on which I am working since days) raise the need to adapt each formula that uses python. That takes a bit of time. Therefore, I concentrate to first enable python 2.x support (that is the status-quo, but with my new system). Then I will push the changes, and after that, I will care to see which packages work with python3 (sometimes I have to rename executable scripts etc.)

To answer your question, I have not yet added the --with-python3 option to pyside. But from the project website it should work. I don't expect big trouble there.

I got working alraedy python3 bindings for numpy, pillow (PIL), PyQt, PyQwt, sip. But not yet pyside.

[samueljohn]

So with these changes, when you get the error
Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
it means probably that the formula needs a depends_on :python.

[samueljohn]

rebased on current master. For the guys who want to try out.

[samueljohn]

I am almost done with this.
Good news is that I got PySide (and Shiboken) working with 2.x and 3.x in parallel, because it was requested quite often.

I'll rebase and make the commits more beautiful and then push. Even if I succesfully built each and every of the above listed formulae on my Mac (successfully), I except some rough edges with other OS X versions and with system python in few cases. So I'll watch the tracker. Maintainers, please tag issues with "python". All: Please mention my name in Python related topics.

Then, I will update the Wiki and the formulae in homebrew/science (and homebrew/versions etc).
I guess posting to the mailing list would be good.