Validating CKAN2.9 & Python3 framework with catalog

[jbrown-xentity]

User Story

In order to increase confidence that we are ready to run in a Python 3 environment, the team wants to validate our applications with all their extensions under Py3.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN All relevant ckan extension is upgraded to CKAN 2.9 + PY3
  When The testing environment for catalog.data.gov supports CKAN 2.9 + PY3
  THEN the catalog.data.gov tests can run and pass \

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]
Since they're not all consistent, we want to make sure we're using all the right branches/versions of the various extensions in inventory and catalog, and ensure they're all at least minimally working correctly in an integrated deployment.

Security Considerations (required)

None.

Sketch

Any necessary changes go on main (we have no expectation to backport to CKAN 2.8/Py2)

• Use the cloud.gov-provided Python buildpack (which is Py3)
• If we need an integration test to validate each extension is working correctly, we add it
• The local test environment is updated to use the openknowledge foundation ckan-dev image

[jbrown-xentity]

I debated about trying to use the current bats tests vs moving to cypress tests. After adding cypress basic tests, they were definitely the way to go. Already found bugs in downstream extension templates that we will need to resolve, using API would not have found these bugs.

[jbrown-xentity]

Harvests are working in tests. Note that the harvest will complete with the correct number of harvested datasets, but the harvest source will show no datasets associated with the source. If a reindex is run (ckan search-index rebuild), then the datasets are associated appropriately. Not sure why at this time.

[jbrown-xentity]

See final PR here: GSA/catalog.data.gov#328
This represents some major changes in catalog dev environment, including:

• OKFN base image (2.9 with Py3)
• Cypress testing over bats testing
• requirements management (pip instead of poetry)
• CKAN command usage
• Deprecate usage of the following extensions: report, archiver, and qa

[nickumia-reisys]

Summary of Related Work

It was an honor to be a part of this Python 3 migration/upgrade effort because of how cross-platform, cross-disciplinary, open-source and integrated CKAN is in the overall data community. This was a very large effort because of the various communities that rely on the software and the various software abstractions that exist from Python, Front-end Technologies (React, HTML/JS/CSS), web networking, Database Integrations, Search Integration, External services (harvesting methodologies, S3 integration, and more). Even the main data.gov web apps are wrapped in a cloud.gov app that has further layers of abstraction. The following list encompasses the highlights associated with Python 3 compatability,

• Main App Upgrades:
  • Feature/py3 catalog.data.gov#328
  • Feature/py3 full integration inventory-app#359
• GSA-specific Extenstion Upgrades:
  • Py3 Support ckanext-datagovcatalog#25
  • Updates for CKAN 2.9 + py3 ckanext-datagovtheme#88
  • Feature/py3 py2 conditional ckanext-datajson#100
  • Feature/py3 ckanext-geodatagov#190
  • Feature/py3 ckanext-googleanalyticsbasic#6
  • Feature/py3 ckanext-dcat_usmetadata#266
  • Feature/py3 ckanext-usmetadata#53
  • Datagov/local testing ckanext-envvars#1
• Community Upgrades:
  • Main CKAN
  • Python 3 support ckan/ckanext-dcat#174
  • Python 3 support ckan/ckanext-harvest#392
  • Test with python 3.6, 3.7 and 3.8 keitaroinc/ckanext-saml2auth#23
  • Supports Python3 ckan/ckanext-spatial#249
  • Conversion to support both Python 2.7 and Python 3.5 asl2/PyZ3950#5
  • Python3 support ckan/ckanext-xloader#113

Note: There were various bug fixes after these initial passes were complete.