Skip to content

Commit

Permalink
Added functionality to check limits of AssertResults.
Browse files Browse the repository at this point in the history
Linted markdown/documentation files.
See:
#190 - "AddAssertResult not checking limits of AssertResults array"
#193 - "Fix overflow of assert array limits"
  • Loading branch information
sagatowski committed Nov 27, 2023
1 parent 89738d5 commit 4a48bdd
Show file tree
Hide file tree
Showing 16 changed files with 285 additions and 105 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Check out the [TcUnit-Runner project](https://github.com/tcunit/TcUnit-Runner).

**Want to contribute to the project?**
That's fantastic! There are two ways to do this.

1. Contribute with your time and knowledge by fixing issues or adding new features. Please read the [CONTRIBUTING](CONTRIBUTING.md) first.
2. By [becoming a sponsor](https://github.com/sponsors/tcunit).

Expand Down
55 changes: 33 additions & 22 deletions TcUnit-Verifier/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# TcUnit-Verifier

This project is can be used to verify the different functions of [TcUnit](https://github.com/tcunit/TcUnit).
It has various test suites to test different functions in TcUnit.
It consists of two separate programs:

- TcUnit-Verifier_TwinCAT
- TcUnit-Verifier_DotNet

Expand All @@ -10,12 +12,14 @@ When verifying that TcUnit works as expected a developer needs to add tests to b
![TcUnit-Verifier concept](img/TcUnit-Verifier_Concept_1280.png)

## TcUnit-Verifier_TwinCAT

TcUnit-Verifier_TwinCAT (TcVT) is as the name implies a TwinCAT project.
It holds a reference to and instantiates the TcUnit framework.
It is defined by several test suites (function blocks) that make use of TcUnit and its different functionalitites.
An example of a test is to verify that TcUnit prints a message if an assertion of an INT has failed (expected value not equals to asserted value).
This means per definition that the verifier will have failed tests as results, because that is what we want to test.
Everytime a test suite is created in TcVT that tests the functionality of TcUnit it is expected that either:

- Some specific output is created by TcUnit (error-log), for example asserting a value that differs from the expected value
- Some specific output is **not** created by TcUnit (error-log), for example asserting a value that is equal to the expected value

Expand All @@ -24,8 +28,10 @@ This means that running this TwinCAT project is not enough to know whether TcUni
Because we don't manually want to check that the output provided by TcUnit-Verifier_TwinCAT everytime it's running, another program is necessary that verifies that the output of the TcVT is as expected.

## TcUnit-Verifier_DotNet

The TcUnit-Verifier_DotNet (TcVD) is a C# program that opens and runs the TcUnit-Verifier_TwinCAT project by the usage of the TwinCAT automation interface.
It basically does the following:

- Starts Visual Studio (using the same version that was used developing TcVT)
- Opens TcVT
- Does a clean/build of TcVT
Expand All @@ -41,8 +47,9 @@ The C# classes verify and test that the output from the ST function blocks is co
Thus a complete test of a specific function in TcUnit needs to be developed in pair, both a FB of tests needs to be added to TcVT as well a class of tests needs to be added to TcVD.

All test classes are instantiated in the class `Program.cs` starting from the lines:
```
/* Insert the test classes here */

```csharp
// Insert the test classes here
new FB_PrimitiveTypes(errors);
new FB_AssertTrueFalse(errors);
new FB_AssertEveryFailedTestTwice(errors);
Expand All @@ -52,41 +59,45 @@ new FB_AssertEveryFailedTestTwice(errors);
```

To create a new test class and make sure that it will be running all that is necessary is to make sure to instantiate it with the argument `errors` (just as above).
If you have added a test in TcVT that is supposed to fail, and thus adding an additional failed test to the output, you need to increment the variable `expectedNumberOfFailedTests` in TcVD by one for every failed test that you have added.
If you have added a test in TcVT that is supposed to fail, and thus adding an additional failed test to the output, you need to increment the variable `expectedNumberOfFailedTests` in TcVD by one for every failed test that you have added.

For example, if we in the PRG_TEST-program of TcVT have a function block instantiated in this way:
```

```structuredtext
PROGRAM PRG_TEST
VAR
AssertEveryFailedTestTwiceArrayVersion : FB_AssertEveryFailedTestTwiceArrayVersion;
END_VAR
```

The equivalent test class in TcVD needs to be instantiated in the following way:
```

```csharp
new FB_AssertEveryFailedTestTwiceArrayVersion(errors);
```

This is an example of how it can look running the TcUnit-Verifier_DotNet:

```
```cmd
C:\Code\GitHub_TcUnit\TcUnit\TcUnit-Verifier\TcUnit-Verifier_DotNet\TcUnit-Verifier\bin\Debug>TcUnit-Verifier.exe -v "C:\Code\GitHub_TcUnit\TcUnit\TcUnit-Verifier\TcUnit-Verifier_TwinCAT\TcUnit-Verifier_TwinCAT.sln"
2023-04-29 22:15:29 [INFO ] - Starting TcUnit-Verifier...
2023-04-29 22:15:29 [INFO ] - In Visual Studio solution file, found visual studio version 16.0
2023-04-29 22:15:29 [INFO ] - Loading the Visual Studio Development Tools Environment (DTE)...
2023-04-29 22:15:41 [INFO ] - Cleaning and building TcUnit-Verifier_TwinCAT solution...
2023-04-29 22:15:41 [INFO ] - Generating TcUnit-Verifier_TwinCAT boot project...
2023-04-29 22:15:54 [INFO ] - Activating TcUnit-Verifier_TwinCAT configuration...
2023-04-29 22:15:57 [INFO ] - Restarting TwinCAT...
2023-04-29 22:15:57 [INFO ] - Waiting for TcUnit-Verifier_TwinCAT to finish running tests...
2023-04-29 22:16:07 [INFO ] - ... got 347 report lines so far.
2023-04-29 22:16:17 [INFO ] - ... got 872 report lines so far.
2023-04-29 22:16:28 [INFO ] - ... got 1398 report lines so far.
2023-04-29 22:16:39 [INFO ] - ... got 1618 report lines so far.
2023-04-29 22:16:41 [INFO ] - Asserting results...
2023-04-29 22:16:42 [INFO ] - Done.
2023-04-29 22:16:42 [INFO ] - Closing the Visual Studio Development Tools Environment (DTE), please wait...
2023-04-29 22:17:02 [INFO ] - Exiting application...
2023-11-27 20:56:27 [INFO ] - Starting TcUnit-Verifier...
2023-11-27 20:56:27 [INFO ] - In Visual Studio solution file, found visual studio version 16.0
2023-11-27 20:56:27 [INFO ] - Loading the Visual Studio Development Tools Environment (DTE)...
2023-11-27 20:56:34 [INFO ] - Cleaning and building TcUnit-Verifier_TwinCAT solution...
2023-11-27 20:56:34 [INFO ] - Generating TcUnit-Verifier_TwinCAT boot project...
2023-11-27 20:56:48 [INFO ] - Activating TcUnit-Verifier_TwinCAT configuration...
2023-11-27 20:56:51 [INFO ] - Restarting TwinCAT...
2023-11-27 20:56:51 [INFO ] - Waiting for TcUnit-Verifier_TwinCAT to finish running tests...
2023-11-27 20:57:01 [INFO ] - ... got 376 report lines so far.
2023-11-27 20:57:12 [INFO ] - ... got 901 report lines so far.
2023-11-27 20:57:23 [INFO ] - ... got 1392 report lines so far.
2023-11-27 20:57:24 [INFO ] - Asserting results...
2023-11-27 20:57:25 [INFO ] - Done.
2023-11-27 20:57:25 [INFO ] - Closing the Visual Studio Development Tools Environment (DTE), please wait...
2023-11-27 20:57:45 [INFO ] - Exiting application...
```

If there was an error in the TcUnit framework this would be shown between the lines `Asserting results...` and `Done.`.
If nothing is shown between these lines (like in the example) it means that TcUnit behaves according to the tests defined in TcUnit-Verifier.
As TcVD starts the TwinCAT runtime it is required that a (trial) runtime license for TwinCAT is activated on the PC of where TcVD and TcVT will be running.
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
using System;
using System.Collections.Generic;

namespace TcUnit.Verifier
{
class FB_AssertCountExceedsMaxNumber : TestFunctionBlockAssert
{
public FB_AssertCountExceedsMaxNumber(IEnumerable<ErrorList.Error> errors, string testFunctionBlockInstance = null) : base(errors, testFunctionBlockInstance)
{
string testName;
string testMessage;

testName = "Assert_SameEntryInOneCycle";
testMessage = "AssertResults.TotalAsserts invalid";
AssertDoesNotContainMessage(testName, testMessage, EnvDTE80.vsBuildErrorLevel.vsBuildErrorLevelHigh);

testName = "Assert_SameArrayEntryInOneCycle";
testMessage = "AssertArrayResults.TotalArrayAsserts invalid";
AssertDoesNotContainMessage(testName, testMessage, EnvDTE80.vsBuildErrorLevel.vsBuildErrorLevelHigh);
}

}
}
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ static void Main(string[] args)
bool testsFinishedRunningFirstLineFound = false;
bool numberOfTestSuitesLineFound = false;
bool numberOfTestsLineFound = false;
bool numberOfSuccesfulTestsLineFound = false;
bool numberOfSuccessfulTestsLineFound = false;
bool numberOfFailedTestsLineFound = false;
bool durationLineFound = false;
bool testsFinishedRunningLastLineFound = false;
Expand Down Expand Up @@ -138,7 +138,7 @@ static void Main(string[] args)
if (error.Description.Contains("| Tests:"))
numberOfTestsLineFound = true;
if (error.Description.Contains("| Successful tests:"))
numberOfSuccesfulTestsLineFound = true;
numberOfSuccessfulTestsLineFound = true;
if (error.Description.Contains("| Failed tests:"))
{
numberOfFailedTestsLineFound = true;
Expand All @@ -158,7 +158,7 @@ static void Main(string[] args)
testsFinishedRunningFirstLineFound
&& numberOfTestSuitesLineFound
&& numberOfTestsLineFound
&& numberOfSuccesfulTestsLineFound
&& numberOfSuccessfulTestsLineFound
&& numberOfFailedTestsLineFound
&& durationLineFound
&& testsFinishedRunningLastLineFound
Expand All @@ -185,7 +185,7 @@ static void Main(string[] args)
)
);

/* Insert the test classes here */
// Insert the test classes here
new FB_PrimitiveTypes(errors);
new FB_ExtendedTestInformation(errors);
new FB_AssertTrueFalse(errors);
Expand All @@ -209,6 +209,7 @@ static void Main(string[] args)
new FB_TestFinishedNamed(errors);
new FB_TestNumberOfAssertionsCalculation(errors);
new FB_EmptyAssertionMessage(errors);
new FB_AssertCountExceedsMaxNumber(errors);

log.Info("Done.");

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
<Compile Include="AutomationInterface.cs" />
<Compile Include="Constants.cs" />
<Compile Include="ErrorList.cs" />
<Compile Include="FB_AssertCountExceedsMaxNumber.cs" />
<Compile Include="FB_EmptyAssertionMessage.cs" />
<Compile Include="FB_TestXmlControl.cs" />
<Compile Include="FB_TestFileControl.cs" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,16 +32,14 @@ public TestFunctionBlockAssert(IEnumerable<ErrorList.Error> errors, string testF
protected string CreateFailedTestMessage(string method, string expected, string actual, string message)
{
string returnString;
returnString = "FAILED TEST 'PRG_TEST." + _testFunctionBlockInstance + "@" +method +"', "
+"EXP: " +expected + ", " +"ACT: " +actual + ", " + "MSG: " +message;
returnString = CreateFailedTestCommonString(method) + ", " + "EXP: " + expected + ", " + "ACT: " + actual + ", " + "MSG: " + message;
return returnString;
}

protected string CreateFailedTestMessageNoAssertionMessage(string method, string expected, string actual)
{
string returnString;
returnString = "FAILED TEST 'PRG_TEST." + _testFunctionBlockInstance + "@" + method + "', "
+ "EXP: " + expected + ", " + "ACT: " + actual;
returnString = CreateFailedTestCommonString(method) + ", " + "EXP: " + expected + ", " + "ACT: " + actual;
return returnString;
}

Expand All @@ -56,6 +54,11 @@ private bool AreErrorItemsContainingTestMessage(string testMessage, vsBuildError
return _errors.Any(e => Regex.Match(e.Description, testMessage, RegexOptions.IgnoreCase).Success && e.ErrorLevel.Equals(errorLevel));
}

private bool AreErrorItemsContainingTestMessage(string testIdentText, string testMessage, vsBuildErrorLevel errorLevel)
{
return _errors.Any(e => (e.Description.Contains(testIdentText.ToUpper())) && (e.Description.Contains(testMessage.ToUpper())) && e.ErrorLevel.Equals(errorLevel));
}

private int CountErrorItemsContainingTestMessage(string testMessage, vsBuildErrorLevel errorLevel)
{
// no regex needed, do a fast check
Expand Down Expand Up @@ -146,5 +149,22 @@ protected void AssertDoesNotContainMessage(string message, vsBuildErrorLevel err
log.Info("Test suite " + _testFunctionBlockInstance + " reports: " + message);
}
}

protected void AssertDoesNotContainMessage(string testName, string message, vsBuildErrorLevel errorLevel)
{

string testIdentText = CreateFailedTestCommonString(testName);
if (AreErrorItemsContainingTestMessage(testIdentText, message, errorLevel))
{
log.Info("Test " + _testFunctionBlockInstance + "." + testName + " reports: " + message);
}
}

protected string CreateFailedTestCommonString(string method)
{
string returnString;
returnString = "FAILED TEST 'PRG_TEST." + _testFunctionBlockInstance + "@" + method + "'";
return returnString;
}
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<TcSmProject xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.beckhoff.com/schemas/2012/07/TcSmProject" TcSmVersion="1.0" TcVersion="3.1.4022.32">
<?xml version="1.0"?>
<TcSmProject xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.beckhoff.com/schemas/2012/07/TcSmProject" TcSmVersion="1.0" TcVersion="3.1.4024.53">
<Project ProjectGUID="{3B151CEE-1DB6-4543-9B44-7AFACBDFB147}" TargetNetId="127.0.0.1.1.1" Target64Bit="true" ShowHideConfigurations="#x3c7">
<System>
<Tasks>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@
<Compile Include="Test\FB_ArrayPrimitiveTypes.TcPOU">
<SubType>Code</SubType>
</Compile>
<Compile Include="Test\FB_AssertCountExceedsMaxNumber.TcPOU">
<SubType>Code</SubType>
</Compile>
<Compile Include="Test\FB_AssertEveryFailedTestTwice.TcPOU">
<SubType>Code</SubType>
</Compile>
Expand Down Expand Up @@ -142,11 +145,6 @@
<Namespace>TcUnit</Namespace>
</PlaceholderReference>
</ItemGroup>
<ItemGroup>
<PlaceholderResolution Include="TcUnit">
<Resolution>TcUnit, * (www.tcunit.org)</Resolution>
</PlaceholderResolution>
</ItemGroup>
<ItemGroup>
<None Include="TcUnitVerifier.tmc">
<SubType>Content</SubType>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
<?xml version="1.0" encoding="utf-8"?>
<TcPlcObject Version="1.1.0.1">
<POU Name="FB_AssertCountExceedsMaxNumber" Id="{5ccd4520-25c7-4bc8-a1a7-a6751f509f85}" SpecialFunc="None">
<Declaration><![CDATA[(*
This test-suite runs several asserts with the same parameters (same expected, actual, message), calling the asserts
in different cycles and using the same test.
On top of this, we will call new asserts (though with same parameters) in later cycles.
The reason we want the framework to support this is that theoretically the user could want to do an assert with the
same message, using the same test with the same data values for both expected and actual, and splitting the asserts
on different cycles.
*)
FUNCTION_BLOCK FB_AssertCountExceedsMaxNumber EXTENDS TcUnit.FB_TestSuite
VAR
END_VAR]]></Declaration>
<Implementation>
<ST><![CDATA[Assert_SameEntryInOneCycle();
Assert_SameArrayEntryInOneCycle();]]></ST>
</Implementation>
<Method Name="Assert_SameArrayEntryInOneCycle" Id="{6f972df2-c1ab-4b10-9350-030d31b8135f}">
<Declaration><![CDATA[METHOD PRIVATE Assert_SameArrayEntryInOneCycle
VAR
a : ARRAY[1..ARRAY_LENGTH] OF DWORD := [16#12345678, 16#23456789];
b : ARRAY[1..ARRAY_LENGTH] OF DWORD := [16#90ABCDEF, 16#0ABCDEF1];
i : UINT;
END_VAR
VAR CONSTANT
ARRAY_LENGTH : UINT := 2;
MAX_INDEX : UINT := (TcUnit.GVL_Param_TcUnit.MaxNumberOfAssertsForEachTestSuite + 100);
END_VAR]]></Declaration>
<Implementation>
<ST><![CDATA[TEST('Assert_SameArrayEntryInOneCycle');
// Only checks if PLC crashes
FOR i := 1 TO MAX_INDEX DO
AssertArrayEquals_DWORD(Expecteds := a,
Actuals := a,
Message := 'Values differ');
END_FOR;
AssertEquals_UINT(Expected := GVL_Param_TcUnit.MaxNumberOfAssertsForEachTestSuite,
Actual := AssertArrayResults.TotalArrayAsserts,
Message := 'AssertArrayResults.TotalArrayAsserts invalid');
TEST_FINISHED();]]></ST>
</Implementation>
</Method>
<Method Name="Assert_SameEntryInOneCycle" Id="{79d00f1b-a496-49a2-8093-dbe58274f7f4}">
<Declaration><![CDATA[METHOD PRIVATE Assert_SameEntryInOneCycle
VAR
a : DWORD := 16#12345678;
b : DWORD := 16#90ABCDEF;
i : UINT;
END_VAR
VAR CONSTANT
MAX_INDEX : UINT := (TcUnit.GVL_Param_TcUnit.MaxNumberOfAssertsForEachTestSuite + 100);
END_VAR]]></Declaration>
<Implementation>
<ST><![CDATA[TEST('Assert_SameEntryInOneCycle');
// Only checks if PLC crashes
FOR i := 1 TO MAX_INDEX DO
AssertEquals_DWORD(Expected := a,
Actual := a,
Message := 'Values differ');
END_FOR;
AssertEquals_UINT(Expected := GVL_Param_TcUnit.MaxNumberOfAssertsForEachTestSuite,
Actual := AssertResults.TotalAsserts,
Message := 'AssertResults.TotalAsserts invalid');
TEST_FINISHED();]]></ST>
</Implementation>
</Method>
</POU>
</TcPlcObject>
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ VAR
TestStreamBuffer : FB_TestStreamBuffer;
TestFinishedNamed : FB_TestFinishedNamed;
EmptyAssertionMessage : FB_EmptyAssertionMessage;
AssertCountExceedsMaxNumber : FB_AssertCountExceedsMaxNumber;
(* The testsuite below is not active, as it will make TcUnit to abort. Uncomment if you want
to test the function of where a test with a name that doesn't exist is set to finished *)
Expand Down
Loading

0 comments on commit 4a48bdd

Please # to comment.