Sphinx 3.3 is roughly 3.2x slower than sphinx 3.2.1

[sgugger]

Describe the bug

We recently noticed the doc builds for the 🤗 Transformers library were a lot slower with the recent release. We pinned the version to 3.2.1 for now and it solved the issue, but we wanted to let you know about the slowdown.

To Reproduce
Steps to reproduce the behavior:

$ git clone https://github.com/huggingface/transformers/
$ cd transformers
$ pip install --upgrade pip
$ pip install .[tf,torch,sentencepiece,docs]
$ cd docs
$ make html SPHINXOPTS="-W"

With sphinx 3.3.0, the build takes 5min21s on a CircleCI Docker Medium (the size doesn't really impact execution time), whereas sphinx 3.2.1 completes the build in 1min40s (on the same machine). Most of the slowdown is coming in the "[reading sources]" part of the build.

Expected behavior
I'd expect the change of version to not impact the time of the build in such a way.

Your project
Link to your sphinx project, or attach zipped small project sample.

Environment info

• OS: Linux
• Python version: 3.7.9
• Sphinx version: 3.2.1/3.3.0
• Sphinx extensions:
  • 'sphinx.ext.autodoc',
  • 'sphinx.ext.coverage',
  • 'sphinx.ext.napoleon',
  • 'recommonmark',
  • 'sphinx.ext.viewcode',
  • 'sphinx_markdown_tables',
  • 'sphinx_copybutton'

[tk0miya]

Note: reproduce the slowdown with following Dockerfile:

FROM python:3.8-slim

RUN apt update; apt install -y build-essential curl git unzip vim
RUN git clone https://github.com/huggingface/transformers/
WORKDIR /transformers
RUN pip install --upgrade pip
RUN pip install ".[tf,torch,sentencepiece,docs]"
RUN pip install Sphinx==3.3.0
WORKDIR /transformers/docs
RUN make html SPHINXOPTS=-vW