[DEPR] Devstack

[kdmccormick]

Timeline

• Proposed: 2022-03-01

• Updated Acceptance Date: 2024-03-12

• Archived: 2024-03-19 (pre-Redwood)

  • Anyone wishing to continue to use devstack will have to make their own forks in order to make changes or keep it updated/maintained.
  • 2U will likely maintain its own fork, but there will be a number of 2U-specific changes and will not be open to outside development. It is possible that this fork will be made private.

• Removal of devstack support from application repositories:

  • Original deadline: 2024-10-10
  • New deadline: 30 days after lms/envs/development.py and cms/envs/development.py are merged into edx-platform master.
    • Estimate: development.py will merge 2024-10-20, old devstack.py will be removed on 2024-11-20.

Replacement

The replacement for Devstack is Tutor. Tutor is "the Docker-based Open edX distribution designed for peace of mind". It is both a deployment tool and a development environment. Tutor has been the only community-support production deployment method since Maple, although the Open edX docs and website do not officially recommend it as a development environment yet.

Rationale

Devstack has several issues:

• It is not pluggable. This leads to two issues:
  • It is large. Out of the box, it comes with a dozen IDAs and another dozen MFEs; new Open edX developers generally needs two IDAs (LMS and CMS) a few MFEs. In the past couple years, improvements have been made such that only a subset of IDAs are run by default. Still, cloning and provisioning are bound by the full weight of all services.
  • Some IDAs like enterprise-catalog, in order to avoid being part of the large Devstack core, make a plugin-esque docker-compose.yml file that hooks into Devstack. This complicates the developer experience on those IDAs and also makes it harder to modify Devstack itself (because of the implicit plugin interface that such IDAs are using).
• Its images are slow to download. The Docker images used by devstack are built by running Ansible in a Dockerfile instead of using native Dockerfile commands. This means that the images are large and make poor use of the Docker layer cache, and thus take a long time for Devstack users to download.
• Many users find it difficult to diagnose and fix Devstack errors. Often, they find it more time/effort-efficient to destroy their volumes and provision from scratch instead of fixing the immediate issue.
• Provisioning takes a long time. This exacerbates the previous issue.
• There are several "upkeep" steps that must be independently done on a regular basis in order to keep Devstack working reliably. These include git-pulling devstack, git-pulling multiple app repos, pulling Docker images, and running migrations.
• Setting up a service in Devstack is completely tangential to setting it up for production deployment. Once it is set up, it takes attention and effort to keep the setup consistent between Devstack and production.

Tutor shows promise in many of these areas:

• It features a documented plugin API. Many IDAs (forums, discovery, credentials, et al) are implemented as plugins to Tutor. The core of Tutor includes just LMS and CMS, and there is even work being discussed to factor those out into plugins.
• Tutor's Dockerfiles are written using native Dockerfile syntax and are intentionally designed to be as small and cache-friendly as reasonably possible.
• Tutor has actively-maintained documentation including a Troubleshooting section. Beyond that, Tutor has a dedicated forum community where maintainers and users help each other. Tutor also makes a point to have a consistent, documented, and debuggable CLI. For example, the tutor dev segment of the CLI is implemented entirely on top of docker-compose and transparently prints out the underlying docker-compose commands that it runs to aid in user debugging, using terminal colors to call out commands vs tutor logging vs actual output.
• Once Tutor is installed, provisioning involves one command (tutor local quickstart) and runs fairly quickly. Provisioning can be safely re-run without destroying existing data.
• Because Tutor uses the code in the Docker images by default, the user only needs to regularly git-pull the Tutor repo and the repos that they host-mount in, if any.
• Since Tutor is also the community-supported deployment method, one can support development and and production usage of an IDA all in a single plugin. Furthermore, using Tutor for both dev and prod allows operators to keep their dev and prod environments very similar with less effort.

For areas where Tutor could be improved, there is an ongoing Tutor Adoption Initiative which aims to support the transition from Devstack to Tutor through a mix of education, plugin development, and core Tutor improvements. Within the initiative roadmap, the "Dev Quality-of-Life" epic contains stories that should ideally close any existing gaps between Tutor and Devstack.

Migration

Many developers can start using Tutor now, and many already are). Particularly, developers whose projects are encompassed by the LMS, CMS, MFEs, and IDAs covered by existing official plugins can generally transition their workflows over to Tutor. On the other hand, for teams that run IDAs with less community adoption (notably, teams at 2U), several new Tutor plugins will be needed in order for Tutor to fully replace Devstack.

There are no plans to help users migrate data from a Devstack instance to a Tutor instance. We recommend that folks start fresh when switching from Devstack to Tutor.

Deprecation & Removal

Deprecation steps

• Update the development page on open.edx.org to recommend Tutor instead of Devstack.
• Ctrl+F the docs for "Devstack", and where appropriate, update them to point at Tutor.
• Add a deprecation warning to devstack's readme
• Make devstack's dev.up Makefile target print a deprecation warning
• Update the configuration docker role's readme with a deprecation message.
• In the description of https://dockerhub.com/u/edxops, warn that all images in the org are deprecated.

### Removal steps
- [x] Archive https://github.com/openedx/devstack
- [ ] Update OEP-10, which still mentions "devstack" as a supported distro
- [x] https://github.com/openedx/edx-platform/issues/34430
- [ ] Remove env.devstack files from frontend apps
- [ ] Scour our docs for other references to devstack; update them to Tutor.
- [ ] Delete the various `devstack.py` and `development.py` settings files unless otherwise depended upon.
- [ ] Delete the various `docker-compose.yml` files and Makefile targets that some IDAs (eg enterprise-catalog) use in order to integrate with Devstack.
- [ ] Delete the `docker/` role (incl. Dockerfiles) from github.com/openedx/configuration.
- [ ] Follow up: Decide whether or not to keep the Dockerfiles that exist in various repos, which aren't used in Tutor.
- [ ] Update edx-cookiecutters

[kdmccormick]

Relates to openedx/wg-build-test-release#138

[dianakhuang]

Since this is primarily a 2U project right now, we are planning on announcing this and gathering feedback.

2U is still planning on using this for the near to medium term until we're confident we can migrate successfully onto Tutor, but we can move it out of the openedx GitHub org to minimize confusion.

[kdmccormick]

In general, sounds great!

Only thing is... MFE development currently sucks in Tutor >.< So nearly all frontend devs still use Devstack when they're trying to change MFEs.

Brian Smith is actively working on rectifying this, with promising results. Would you be willing to delay the deprecation further until Tutor has MFE dev sorted out?

[dianakhuang]

Yup! I think we're still trying to understand the requirements needed here. I'll push out the acceptance date for another few months.