Writing a Fixer

Abstract

Given a code pattern that needs to be fixed for any reason, including:

• Upgrade source from Python 2 to Python 3
• Backporting source from Python 3 to Python 2
• Refactoring existing Python 3 code to improve it

The ideal approach is to automate the fix so the community at large can benefit from it. Note that external fixer suites (outside of a fork, that is) will be supported once #12 is complete and crosswind is available on pypi.

Determine Fixer Suite

So you know what fixer you want to create and a few samples of the pattern to work off of. Now you need to figure out what suite is most suitable. The suites that exist as of this writing are:

Name	Description
two_to_three	Forked from lib2to3 and enhanced to help port source from Python 2 to 3
three_to_two	Forked from lib3to2 and refined to support backporting from Python 3 to Python 2.7
pessimist	Collection of new fixers that are pessimistic in nature and likely over defensive but helpful for safe porting

Once you decide on a suite it is time to open an issue with these details to communicate to the project and community a desire to create a fixer. This can help collect further code samples to help ensure the fixer is robust for a variety of codebases.

Create Fixer File

Once the issue is created you're welcome to get started on the required code changes while discussion is ongoing. The first step is to create a new git branch from master so you can keep your changes isolated (particularly useful if you're working on many fixers or other contributions at the same time). Then, create a new python file named fix_<name>.py in the desired fixer suite's fixes directory where the name is any word or underscored collection of words to describe the fix accurately and succinctly.

Anatomy of a Fixer

A fixer has a few key sections:

• A module docstring talking about the reason for the fixer, code samples of what it tries to find, and code samples for what it transforms to
• Imports including several from the crosswind library
• A class that inherits from fixer_base.BaseFix; the name of the class must match the name of the file (example: fix_my_issue.py would be FixMyIssue)
• A PATTERN that identifies the grammar pattern for the fixer to match against
• A transform function to mutate existing nodes/leafs or create new nodes/leafs to alter the source

A minimal example/template would be:

"""
<module docstring>
"""

from crosswind import fixer_base
from crosswind.fixer_util import Node


class FixMyIssue(fixer_base.BaseFix):

    PATTERN = "<pattern>"

    def transform(self, node, results):
        pass

Identifying the PATTERN

The script find_pattern.py greatly aids in identifying a PATTERN for your fixer. The idea is to feed it a snippet of code through file or stdin and it will cycle through ever broader snippets for which it can generate a pattern. To continue to next snippet, hit "Enter". To accept the current snippet and have the script return the identified pattern, type "y" and then hit "Enter".

Note that patterns can be combined using pipes (|) and grouped with parentheses. Additionally, you can capture portions of the match by prepending <name>= before any of the tokens. The name can be any valid Python identifier.

See the other fixers in the fixer_suites directory for more concrete examples.

Writing a Transform

The transform method is called whenever the PATTERN is successfully matched. You will be handed the root of the match as the node argument and a dictionary of your named captures in results (the keys of the dict being the names of the captures).

At this point you will want to introspect the node and results data to determine what state the tree is in and what you need to do to transform it into a more valid tree and accomplishing the objective of your fixer.

Note that you can either mutate the nodes/leafs in place using the replace API (see the pytree.py file) or construct a completely custom tree from new nodes/leafs. For both approaches you will want to familiarize yourself with the various premade nodes/leafs in both the fixer_util.py file and in fixer_util_3to2.py file. The latter is simply the fixer_util from lib3to2 and has not yet been merged into the main file from lib2to3.

Now you're on to the cycle of implementing, testing, and so on. Good luck and have fun!

Create Test File

During development you will need a way to both test that your fixer is working as you expect and to aid in making sure future changes don't break it unintentionally. All fixer suites have pytest based tests to ensure this. The test file should have the same name as the fixer file except replacing fix_ with test_.

Anatomy of the Test File

The test file is comprised of only a few distinct sections:

• Imports for pytest and potentially other required modules (likely only needing pytest though since the test fixtures ensure your fixer gets loaded)
• Optional tests to confirm behaviors (mostly a convenient place to help validate your transformed code works as you expect)
• PyTest fixture (uses a suite fixer that itself consumes the base fixture to ensure the fixer suite is loaded and only the declared fixer runs)
• Tests that consume the fixture and do before-and-after style verification (as well as several to verify that other code samples are not changed)

Creating the Fixture

The fixture depends heavily on your selection of what fixer suite this fixer belongs to. As a concrete example, if you were adding a new fixer to two_to_three with a fixer called fix_my_issue it would look like:

@pytest.fixture(name="fixer")
def fixer_fixture(two_to_three_test_case):
    return two_to_three_test_case("my_issue")

You'll notice a few things:

• The string passed is the name of the fixer file except the fix_ is left off. Additionally, the fixture is named "fixer" by convention which helps the tests read fluidly (fixer passed to the test function and fixer.check and other APIs for assertions).
• two_to_three_test_case is the base fixture for the two_to_three fixer suite. You can find these base fixtures in the conftest.py files located at the root of each fixer suite test folder (here is the one for two_to_three)
• The base fixtures use a pattern of returning a factory function that is used to construct the actual test fixture.

Writing Tests

The next section are the actual tests that consume this fixture. If you're not familiar with pytest, simply adding an argument of fixer to the arguments will create and supply the fixture to the function:

def test_my_issue_changes_as_i_expect(fixer):

You then have access to a variety of assertion methods such as check and unchanged. These are used for two broad category of tests:

Method	Description
check	Ensure that, given a specific code sample, the correct transformed code is produced.
unchanged	Ensure that the given code sample is not transformed by the fixer.

See the support.py file in the tests directory for more details on the available methods on the fixture.

The entire test suite across all of crosswind can be run using make tests. It is probably best to save that for later in the process (once you're more confident of the change) and instead call pytest directly so you can provide additional arguments to filter it down to only your tests. Given the example of your test file being called test_my_issue.py you can run the following to run only those tests:

poetry run pytest -k test_my_issue

-k allows specifying a test pattern to run and providing the filename will run only those tests.

Finalizing the Change and Preparing a Pull Request

Once everything is looking good you will want to prepare your changes for contribution. You will want to run the automatic code formatting using make format. This will reformat the code according to our configuration for isort and black.

Then, make sure you've run the tests in whole at least once (make tests).

And finally, run the validate make target - it does a minimal recreation of the CI build process and is an ideal final check.

Note that these targets can be combined to format and validate your changes in a single pass: make format validate.

Now it is time to open your pull request. Thanks in advance!