Backlog

We try to keep this somewhat organized, but it's pretty dynamic.

Bugs

1. Turn off G node reporting

2. Results think that the endPosition failed when it was actually successful

High Priority

1. Test status code returned in {f} response footer. Call this "status". Change status report machine status to "stat". In this case if the status (return code) fails we will probably want to record that and skip the manual test.

2. UPDATED: Instead this is part of endCondition and endState as rc.

3. Provide a test runner that let's you select one ore more YAML files to include in a test run. Currently grunt will only run one named file. Files are being copied to be that file name so they can be run.

4. UPDATED: Instead we allow a key focus to be true to specify only those tests to be run. if any tests have focus: true then those are the only ones that are run.

5. Find some way to manage before_all and before_each scoping to accommodate multiple test sets in a test suite file.

6. All tests are part of a "suite". If a test has a tests key that contains an array, then that test becomes a "suite", and can then have beforeAll, beforeEach, setValues, and setValuesEach. Also, timeout and endCondition values will be inherited from the parent suite.

On The List

1. Some way to drop into a console using G module. For diagnosing problems in the tests - e.g. show status, list settings, change values, etc.

2. More control over manual tests. Change syntax for manual tests so they can be turned on and off. Something like:

manualTest: ${manual}
prompt: Text to be sent to the user for a manual test

This will allow a test suite to be run without visual inspection or operator intervention being required. standard endPosition, status and stat tests would still apply. The manual portion is omitted or always passes.

1. This now takes the form of manualPrompt which can either be an arbitrary string which will be what is displayed, or the value true in which case the test name will be shown as the prompt.

2. Add a (q)uit to the manual test options Y and N. Quits the current test run (set or suite) and reports results collected up to that point.

3. Timeout should skip to next test. If a test times out it should fail the current test and start the next test. The timeout may occur in either the automatic or manual portion of the test.

4. The tester should indicate if the .js file is missing. I inadvertently deleted it and the test completed successfully and "0 tests completed successfully - Done, with no errors".

5. If a variable cannot be looked up (failure) the tester should fail the entire test run and report the missing variable. Right now it's silent and runs the test with something undefined in place of that variable.

Parking Lot

1. Provide a way to scope variables at the test set or suite level. If this comes "for free" with the before_all, before_each scoping then great, otherwise it's not important enough to address right now.

2. Can you still send bells to the console (control G)? One bell would be a good indication that the auto test passed and the manual test is ready. 3 bells in rapid succession indicates the auto failed and we are skipping the manual test.