-
Notifications
You must be signed in to change notification settings - Fork 4
Future Testing Requirement Details
This page fleshes out requirement details for testing Momentum model components.
Many tests of simulation models require significant computational resources and run over a longer period of (wall-clock) time, making it unpractical or even impossible to integrate them in GitHub Actions or similar CI services. The public cloud is a possible choice for running such tests in an automated fashion.
As a shared system, the cloud platform should typically run tests that are of general interest and do not require private or sensitive configurations or data.
- A given test can run on the most appropriate set of resources to satisfy its needs around processor type and capability, memory requirements, and storage system requirements.
- A large variety of architectures and compilers can be used for testing to ensure portability and improve overall test coverage.
- Developers from across the Momentum partnership can be given access, which is typically much more difficult to arrange on in-house resources at partner sites.
- Test outcomes can be made visible to developers much more easily, who can take action and gather information required to debug a problem.
- Cloud services come with sophisticated automation tools and services, which is ideal for integrating test systems with the GitHub repository.
- Developers from across the partnership can use the cloud service to access a wide range of tests, while they have to rely on their site support to enable running these tests on in-house systems otherwise.
- Efforts for developing and maintaining tests and test platforms can be shared across the partnership.
- Cloud resources can be expensive, depending on the type and level of utilisation.
- Using cloud services can add a lot of complexity around security, automation, vendor-specific constraints, etc.
- The cloud services should be accessible by developers from across the Momentum partnership.
- The cloud service needs to integrate with GitHub so that tests can be triggered automatically.
Many partner sites are interested in running a subset of tests on their own platform to ensure ongoing support and code portability. They may also want to add highly site-specific tests that are either not of interest to other sites, or that rely on private data or sensitive configurations. In these cases, partners need to be able to run tests on their own on-premise platforms.
- Partner platforms can be tested directly.
- Easier to restrict access to satisfy privacy, security, and regulatory requirements for data and configurations.
- Easier to integrate with some in-house systems (e.g., private GitLab CI instances).
- More difficult to integrate on-premise testing with GitHub - for example, using a self-hosted GitHub runner can potentially collide with site security requirements.
- More difficult to make test results visible to the developer community.
- Setting up and maintaining site-specific testing requires additional effort.
- It is not necessary for on-site testing to integrate with GitHub, but integration should be possible if that is desired.
- Sites need to be able to set up testing according to their site requirements around privacy, security, and regulatory compliance.
Continuous Integration (CI) methods and tools are essential for a robust and transparent code development process, ensuring high code quality by providing immediate feedback to the developer if their changes work as expected for meeting their targets without breaking the code for other users. Low-expense tests can run frequently and without resulting in long delays in the development workflow. They are also typically fully automated, to make it as easy as possible for developers to run the tests. Platforms like GitHub and Gitlab offer highly flexible and powerful CI integration that facilitates this work.
- Quick to run and highly automated to ensure that all developers can (and will) run these tests.
- CI platform constraints require tests with higher resource needs to be run separately, which adds complexity to the test setup and increases maintenance efforts.
- CI tests must be available for all developers.
- The tests must be fully automated to make running them effortless.
- Test results must be visible to all developers, at least in case of pull requests/merge requests.