-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Add ability to track status of a test #300
Comments
Over the past weeks, this issue has been quite heavily referenced in the context of allowing PRs to be merged as soon as possible, reducing the likelihood of merge conflicts down the line. The idea being that tests can be merged after more minimal review, making sure that they don't break anything or have any egregious problems. More nuanced review can then be carried out by members of the community group by executing or reviewing the tests in place, with new PRs being opened to resolve any concerns. A flag will be needed, to allow people to distinguish between test plans in a draft state and those that are ready to run/finalised. During the March 4, 2021 community group meeting though, @mcking65 raised concerns that this issue may not be as simple as a single binary state flag. The concept of whether tests are ready-to-run may be more nuanced, incorporating aspects of test versioning, AT vendor consensus/review, etc. However, we'd still like to find a way forward; maybe a short-term fix to ensure that PR merging is not unnecessarily blocked. |
Notes from March 11, 2021 CG meeting:
With all of the above in mind, the group seemed to agree that a "wide review" status was appropriate after draft. This would be a window of opportunity for third parties to file concerns with the project about the content of a test plan; the proposed time period was 60 days. The ARIA-AT project would have a longer window to resolve any raised issues, the proposed period was 120 days which would always give us a minimum of 60 days to carry out discussion and remediation (if a vendor only engaged on the very last day of the wide review window). There was also discussion about how wide review windows should be scheduled. Should there be several throughout the year? |
Tests can be draft or final/ready to run.
Discussed adding a column to the CSV file where we write assertions.
The text was updated successfully, but these errors were encountered: