Skip to content

Maintainer Notes

Jump to bottom
vfdev edited this page Jun 24, 2021 · 37 revisions

How to create new release

  • Cut a branch from master:
    • Branch name should be "X.Y.Z" without prefix "v"
    • Optionally, revert commits that should not go to the release
  • Draft new release note
  • Tag name: "vX.Y.Z"
  • Make sure to point to the branch (not master)

Created release binaries will be uploaded to pypi and conda.

See corresponding GitHub Actions:

Automatic PyPI wheels+tar upload with Travis (out-dated)

PyPI wheels and tars can be built and upload using Travis as deploy phase in the test stage:

# PyPI Deployment: https://docs.travis-ci.com/user/deployment/pypi/
deploy:
  provider: pypi
  user: vfdev-5
  # If password contains non alphanumeric characters
  # https://github.com/travis-ci/dpl/issues/377
  # pass it as secured variable
  password: $PYPI_TOKEN
  # otherwise, follow "How to encrypt the password": https://docs.travis-ci.com/user/encryption-keys/
  # `travis encrypt deploy.password="password"`
  #  secure: "secured_password"

  skip_cleanup: true
  distributions: "sdist bdist_wheel"
  on:
    tags: true
    python: "3.5"

Automatic Conda wheels+tar upload (out-dated)

This is done almost manually, we build 4 python versions (2.7, 3.5, 3.6, 3.7) for single platform (linux-64) and convert the artifacts to osx-64 and win-64:

before_deploy:
  # Conda deploy if on tag
  # ANACONDA_TOKEN should be provided by Travis
  # How to generate ANACONDA_TOKEN: https://docs.anaconda.com/anaconda-cloud/user-guide/tasks/work-with-accounts#creating-access-tokens
  # We need a token with checked "Allow all API operations"

  # https://conda.io/docs/user-guide/tasks/build-packages/install-conda-build.html
  - conda install -y conda-build conda-verify anaconda-client
  - conda config --set anaconda_upload no
  - conda build --quiet --no-test --output-folder conda_build conda.recipe
  # Convert to other platforms: OSX, WIN
  - conda convert --platform win-64 conda_build/linux-64/*.tar.bz2 -o conda_build/
  - conda convert --platform osx-64 conda_build/linux-64/*.tar.bz2 -o conda_build/
  # Upload to Anaconda
  # We could use --all but too much platforms to uploaded
  - ls conda_build/*/*.tar.bz2 | xargs -I {} anaconda -v -t $ANACONDA_TOKEN upload -u pytorch {}

The recipe meta.yaml to build package is provided in the folder conda.recipe.

Documentation automatic generation

The documentation is automatically built for master and all tags and deployed when a PR is merged to master. To build multiple versions we are using a fork of sphinxcontrib-versioning. Documentation is deployed at https://pytorch.org/ignite and contains a selector of versions. History of builds is not conserved, so if you push manually some changes, they will be rewritten by the next doc deployment.

Automatic deployment is done in .travis.yml in the stage docs:

# GitHub Pages Deployment: https://docs.travis-ci.com/user/deployment/pages/
    - stage: Docs
      python: "3.5"
      if: branch != nightly
      # Use previously defined before_install
      before_install: *before_install

      install:
        - pip install -r docs/requirements.txt
        - pip install git+https://github.com/anmolsjoshi/sphinxcontrib-versioning.git

      script:
        - sphinx-versioning --use-master-conf build --whitelist-branches master docs/source docs/build/html
        # Create .nojekyll file to serve correctly _static and friends
        - touch docs/build/html/.nojekyll
      after_success: # Nothing to do

      # Deploy built docs when PR is merged to master
      deploy:
        provider: pages
        skip-cleanup: true
        github-token: $GITHUB_TOKEN  # Set in the settings page of your repository, as a secure variable
        keep-history: false
        local_dir: docs/build/html
        on:
          branch: master

How to manually create a release

How to create and upload pip/conda builds

At first, we build universal wheels and tars:

git checkout vX.Y.Z
python setup.py sdist bdist_wheel

Upload to pypi 

twine upload dist/*

or for testing purposes it is possible to upload to test.pypi:

twine upload --repository-url https://test.pypi.org/legacy/ dist/*

How to manually update documentation

All you have to do to update the site is to modify the gh-pages branch. For example, regenerating docs is:

cd docs
pip install -r requirements.txt
pip install git+https://github.com/anmolsjoshi/sphinxcontrib-versioning.git
sphinx-versioning --use-master-conf build --whitelist-branches master docs/source docs/build/html
touch docs/build/html/.nojekyll
# copy build/html into gh-pages branch, commit, push

README

Side-by-side code compare

Image is created with PyCharm (Dracula Theme) with "Compare files" function and a screenshot. Line spacing ~1.1

Ignite (left side):

model = Net()
train_loader, val_loader = get_data_loaders(train_batch_size, val_batch_size)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.8)
criterion = torch.nn.NLLLoss()

max_epochs = 10
validate_every = 100
checkpoint_every = 100

trainer = create_supervised_trainer(model, optimizer, criterion)
evaluator = create_supervised_evaluator(model, metrics={'accuracy': Accuracy()})

@trainer.on(Events.ITERATION_COMPLETED(every=validate_every))
def validate(trainer):
    evaluator.run(val_loader)
    metrics = evaluator.state.metrics
    print("After {} iterations, binary accuracy = {:.2f}"
          .format(trainer.state.iteration, metrics['accuracy']))


checkpointer = ModelCheckpoint(checkpoint_dir, n_saved=3, create_dir=True)
trainer.add_event_handler(Events.ITERATION_COMPLETED(every=checkpoint_every),
                          checkpointer, {'mymodel': model})

trainer.run(train_loader, max_epochs=max_epochs)

and bare pytorch snippet (right side):

model = Net()
train_loader, val_loader = get_data_loaders(train_batch_size, val_batch_size)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.8)
criterion = torch.nn.NLLLoss()

max_epochs = 10
validate_every = 100
checkpoint_every = 100

def validate(model, val_loader):
    model = model.eval()
    num_correct = 0
    num_examples = 0
    for batch in val_loader:
        input, target = batch
        output = model(input)
        correct = torch.eq(torch.round(output).type(target.type()), target).view(-1)
        num_correct += torch.sum(correct).item()
        num_examples += correct.shape[0]
    return num_correct / num_examples


def checkpoint(model, optimizer, checkpoint_dir):
    # ...
    pass

iteration = 0

for epoch in range(max_epochs):
    for batch in train_loader:
        model = model.train()
        optimizer.zero_grad()
        input, target = batch
        output = model(input)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

        if iteration % validate_every == 0:
            binary_accuracy = validate(model, val_loader)
            print("After {} iterations, binary accuracy = {:.2f}"
                  .format(iteration, binary_accuracy))

        if iteration % checkpoint_every == 0:
            checkpoint(model, optimizer, checkpoint_dir)
        iteration += 1

GIF teaser

Get your contributors for this release

import requests


def get_datetime_of_latest_release():
    r = requests.get('https://api.github.com/repos/pytorch/ignite/releases/latest')
    assert r.status_code == 200
    res = r.json()
    assert "created_at" in res, "{}".format(res)
    return res["created_at"]


def get_contributors_since_latest_release():
    dt = get_datetime_of_latest_release()
    contributors = []
    page = 1
    while True:
        r = requests.get('https://api.github.com/repos/pytorch/ignite/commits?since={}&per_page=100&page={}'.format(dt, page))
        assert r.status_code == 200, r
        commits = r.json()
        if len(commits) < 1:
            break
        unique_contribs = list(set([
            c['author']['login'] for c in commits if ('author' in c) and (c['author'] is not None)
        ]))
        contributors += unique_contribs
        page += 1

    contributors = list(set(contributors))
    contributors = ["@{}".format(c) for c in contributors]
    return sorted(contributors)


print(", ".join(get_contributors_since_latest_release()))

PyTorch-Ignite presented to you with love by PyTorch community

Clone this wiki locally