Outcomes of testing

In brief

There are several different outcomes in the hs-test library.

1. Wrong answer
2. Presentation Error
3. Exception
4. Error
5. Unexpected error
6. Syntax Error (Python) / Compilation Error (Java)

Wrong answer

This outcome appears when the user's program has successfully completed, but the check method (or dynamic input function) returns False (or throws WrongAnswer). In this case, the user is also shown feedback.

If the user gets the wrong answer with the feedback Your program didn't output a number! in test 1, the output will look like the following:

Wrong answer in test #1  

Your program didn't output a number!

And in the IDE it will look like this:

If the user writes this program below (also incorrect):

Example of wrong answer feedback in test 2:

Wrong answer in test #2  

You should output a sum of numbers

So, you may notice that the library automatically calculates how many tests have been passed and which test has not been passed on the account and displays the information about the number of the failed test together with feedback.

Exception

This outcome appears when the student program is terminated with an exception. In this case, the student is shown the test number and the stack trace of the exception. You can see an example below:

The user will get the exception outcome. It will look like follows:

Exception in test #1

Traceback (most recent call last):
  File "sum/main.py", line 9, in <module>
    print(get_sum())
  File "sum/main.py", line 5, in get_sum
    num2 = int(nums[1])
IndexError: list index out of range

Presentation Error

Presentation error is shown when test expects a certain structure of the output that is not presented.

An example of the feedback can be seen below:

Presentation error in test #1

The following output contains wrong number of integers (expected to be equal to 5, found 0):
Hello World

Error

This outcome occurs when a user has clearly done something wrong, but it has not led to an exception. This outcome is shown in the following cases:

1. Time limit exceeded. It the user's program has gone into an infinite loop tests will wait 15 seconds by default and then stop the test with an error. An example of the feedback:

Error in test #1

In this test, the program is running for a long time, more than 15 seconds. Most likely, the program has gone into an infinite loop.

2. Main method not found (Java) / Module not found (Python). In this case, the user most likely deleted the main method / deleted the file being tested. An example of the feedback:

Error in test #1

No main method found in class Main

3. In case the student forgot to close the file after opening it and reading it in the code. An example of the feedback:

Error in test #1

The file text.txt can't be deleted after the end of the test. Probably you didn't close it.

4. In case the user's program has printed something that is not a number, but tests tried to parse it as a number. An example of the feedback:

Error in test #1

Cannot parse a number from the output part "number is 12"

5. In case the hs-test library detects infinite loop from the output. An example of the feedback:

Error in test #1

Infinite loop detected.
Last 96 lines your program printed have 12 blocks of 8 lines of the same text.

Unexpected error

This outcome appears when the student's program has completed successfully, but an exception occurred during checking. In this case, it is the fault of the person who wrote the tests and the tests should be corrected. The feedback It will look like the following:

Unexpected error in test #1

We have recorded this bug and will fix it soon.

OS Windows 10
Java 11.0.1
Vendor Oracle Corporation
Testing library version 7.2

Stack trace:
...

With a bunch of useful information attached to the report.

Syntax Error (Python) / Compilation Error (Java)

Syntax and compilation errors occur when a student has written a program that cannot be run.

Here is an example of compilation error feedback in Java:

Compilation error

src/sum/Main.java:10: error: cannot find symbol
        System.
              ^
  symbol:   class Scanner
  location: class System
1 error

SyntaxError is a regular Python exception: it happens while executing the user's program.

Exception in test #1

Traceback (most recent call last):
  File "main.py", line 1, in <module>
    print(eval(")"))
  File "<string>", line 1
    )
    ^
SyntaxError: