Outcomes of testing

In brief

There are several different outcomes in the hs-test library.

1. Wrong answer
2. Presentation error
3. Exception
4. Error
5. Unexpected error
6. Syntax Error (Python) / Compilation Error (Java)

Wrong answer

This outcome appears when the user's program has successfully completed, but the check method (or dynamic input function) returns false (or throws WrongAnswer). In this case, the user is also shown feedback.

Let's consider such a test case - the program must add two numbers located on different lines. You may see an example of tests below:

import sum.Main;
import org.hyperskill.hstest.stage.StageTest;
import org.hyperskill.hstest.testcase.CheckResult;
import org.hyperskill.hstest.testcase.TestCase;

import java.util.List;


public class SumTest extends StageTest<Integer> {
    public SumTest() {
        super(Main.class);
    }

    @Override
    public List<TestCase<Integer>> generate() {
        return List.of(
            new TestCase<Integer>()
                .setInput("12\n34")
                .setAttach(46),

            new TestCase<Integer>()
                .setInput("43\n23")
                .setAttach(66)
        );
    }

    @Override
    public CheckResult check(String reply, Integer attach) {

        int replied;

        try {
             replied = Integer.parseInt(reply.trim());
        } catch (NumberFormatException ex) {
            return CheckResult.wrong(
                "Your program didn't output a number!"
            );
        }

        if (attach != replied) {
            return CheckResult.wrong(
                "You should output a sum of numbers"
            );
        }

        return CheckResult.correct();
    }
}

So, if the user writes this program below:

...
public static void main(String[] args) {
    System.out.println("Hello world!");
}
...

The user will get the wrong answer outcome. It will look like follows:

Wrong answer in test #1  

Your program didn't output a number!

And in the IDE it will look like this:

If the user writes this program below (also incorrect):

...
public static void main(String[] args) {
    System.out.println(46);
}
...

The user will get another wrong answer outcome. It will look like follows:

Wrong answer in test #2  

You should output a sum of numbers

So, you may notice that the library automatically calculates how many tests have been passed and which test has not been passed on the account and displays the information about the number of the failed test together with feedback.

Presentation Error

Presentation error is shown when test expects a certain structure of the output that is not presented.

An example of the feedback can be seen below:

Presentation error in test #1

The following output contains wrong number of integers (expected to be equal to 5, found 0):
Hello World

Exception

This outcome appears when the student program is terminated with an exception. In this case, the student is shown the test number and the stack trace of the exception. You can see an example below (tests are the same):

...
static int getSum() {
    Scanner scanner = new Scanner(System.in);
    String[] nums = scanner.nextLine().split(" ");
    int num1 = Integer.parseInt(nums[0]);
    int num2 = Integer.parseInt(nums[1]);
    return num1 + num2;
}

public static void main(String[] args) {
    System.out.println(getSum());
}
...

The user will get the exception outcome. It will look like follows:

Exception in test #1

java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
	at converter.Main.getSum(Main.java:11)
	at converter.Main.main(Main.java:16)

The same outcome in python looks like this:

Exception in test #1

Traceback (most recent call last):
  File "sum/main.py", line 9, in <module>
    print(get_sum())
  File "sum/main.py", line 5, in get_sum
    num2 = int(nums[1])
IndexError: list index out of range

Error

This outcome occurs when a user has clearly done something wrong, but it has not led to an exception. This outcome is shown in the following cases:

1. Time limit exceeded. It the user's program has gone into an infinite loop tests will wait 15 seconds by default and then stop the test with an error. An example of the feedback:

Error in test #1

In this test, the program is running for a long time, more than 15 seconds. Most likely, the program has gone into an infinite loop.

2. Main method not found (Java) / Module not found (Python). In this case, the user most likely deleted the main method / deleted the file being tested. An example of the feedback:

Error in test #1

No main method found in class Main

3. In case the student forgot to close the file after opening it and reading it in the code. An example of the feedback:

Error in test #1

java.nio.file.FileSystemException

The file text.txt can't be deleted after the end of the test. Probably you didn't close File or Scanner.

4. In case the user's program has printed something that is not a number, but tests tried to parse it as a number. An example of the feedback:

Error in test #1

Cannot parse Integer from the output part "number is 12"

Also, prior to version 7 of hs-test and version 2 of hs-test-python, it was prohibited to force an exit from the user's program.

In Java there are two ways to do that:

1. System.exit(0);
2. Runtime.getRuntime().exit(0);

In Python, there are a bit more ways:

1. sys.exit(0)
2. os._exit(0)
3. exit(0)
4. quit(0)
5. CTRL+C (raise KeyboardInterrupt)
6. os.kill(os.getpid(), signal.SIGINT)
7. os.killpg(os.getpid(), signal.SIGINT)
8. signal.pthread_kill(threading.get_ident(), signal.SIGINT)
9. signal.siginterrupt(signal.SIGINT, True)
10. multiprocessing.current_process().kill()

The reason for that was that after calling these methods Java and Python interpreters shut down along with the testing process, so no feedback was given to the user. From hs-test v7 and hs-test-python v2 it is possible to use the abovementioned methods to force exit the user's program and such action doesn't result in tests' exit.

Unexpected error

This outcome appears when the student's program has completed successfully, but an exception occurred during checking. In this case, it is the fault of the person who wrote the tests and the tests should be corrected. The example below shows this problem. The student should write a program that adds two numbers from the standard input and displays the result.

Tests look like this:

import sum.Main;
import org.hyperskill.hstest.stage.StageTest;
import org.hyperskill.hstest.testcase.CheckResult;
import org.hyperskill.hstest.testcase.TestCase;

import java.util.List;


public class SumTest extends StageTest<Integer> {
    public SumTest() {
        super(Main.class);
    }

    @Override
    public List<TestCase<Integer>> generate() {
        return List.of(
            new TestCase<Integer>()
                .setInput("12\n34")
                .setAttach(46),

            new TestCase<Integer>()
                .setInput("43\n23")
                .setAttach(66)
        );
    }

    @Override
    public CheckResult check(String reply, Integer attach) {

        int replied = Integer.parseInt(reply.trim());

        if (attach != replied) {
            return CheckResult.wrong(
                "You should output a sum of numbers"
            );
        }

        return CheckResult.correct();
    }
}

But the student wrote the program like this:

...
public static void main(String[] args) throws Exception {
    Scanner scanner = new Scanner(System.in);
    int num1 = scanner.nextInt();
    int num2 = scanner.nextInt();
    System.out.println("Sum is: " + (num1 + num2));
}
...

When testing such a solution, the check method raises an exception on the Integer.parseInt(reply.trim()) line because the student printed not only the number itself, but also pre-printed it with the "Sum is: " line. Of course, the student made a mistake after reading the task incorrectly, but it is necessary to think through such scenarios and show the user the corresponding feedback. It would be more correct to first check if the student's program outputted a number, and only then compare this number, like in the code block below. Throwing any Throwable objects are not allowed in check method.

int replied;
try {
    replied = Integer.parseInt(reply.trim());
} catch (NumberFormatException ex) {
    return CheckResult.wrong(
        "Your program didn't output a number!"
    );
}

If an exception in the check method occurs, the user will get an unexpected error outcome. It will look like follows:

Unexpected error in test #1

We have recorded this bug and will fix it soon.

OS Windows 10
Java 11.0.1
Vendor Oracle Corporation
Testing library version 7.2

...

With a bunch of useful information attached to the report.

Syntax Error (Python) / Compilation Error (Java)

Syntax and compilation errors occur when a student has written a program that cannot be run.

Here is an example of compilation error feedback in Java:

Compilation error
src/sum/Main.java:10: error: cannot find symbol
        System.
              ^
  symbol:   class Scanner
  location: class System
1 error

Add here is an example of syntax error in Python:

File "sum\main.py", line 1
print().
SyntaxError: invalid syntax