Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Fix "Complex Example" testset for Julia 1.9+ #99

Merged
merged 3 commits into from
Apr 29, 2024

Conversation

omus
Copy link
Member

@omus omus commented Apr 26, 2024

I noticed this failure on Julia 1.9.4:

┌ Info: Reference Test for "complexexample.txt" failed:
│ - DIFF ------------------------
│ """
│   <?xml version=\"1.0\" encoding=\"UTF-8\"?>
│   <testsuites tests=\"19\" failures=\"6\" errors=\"4\"><testsuite name=\"Math/Multiplication\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"0\"><testcase name=\"1 * 3 == 3\" id=\"1\" classname=\"Math/Multiplication\" time=\"0.0\"/><testcase name=\"1 * 2 == 5\" id=\"2\" classname=\"Math/Multiplication\" time=\"0.0\"><failure message=\"2 == 5\" type=\"test\">Test Failed
│     Expression: 1 * 2 == 5
│ -    Evaluated: 2 == 5</failure></testcase><testcase name=\"1 * 4 == 4\" id=\"3\" classname=\"Math/Multiplication\" time=\"0.0\"/></testsuite><testsuite name=\"Math/addition/negative addition\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"1\"><testcase name=\"1 + -1 == 0\" id=\"1\" classname=\"Math/addition/negative addition\" time=\"0.0\"/><testcase name=\"1 + -2 == 1\" id=\"2\" classname=\"Math/addition/negative addition\" time=\"0.0\"><failure message=\"-1 == 1\" type=\"test\">Test Failed
│ +    Evaluated: 2 == 5
│ + </failure></testcase><testcase name=\"1 * 4 == 4\" id=\"3\" classname=\"Math/Multiplication\" time=\"0.0\"/></testsuite><testsuite name=\"Math/addition/negative addition\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"1\"><testcase name=\"1 + -1 == 0\" id=\"1\" classname=\"Math/addition/negative addition\" time=\"0.0\"/><testcase name=\"1 + -2 == 1\" id=\"2\" classname=\"Math/addition/negative addition\" time=\"0.0\"><failure message=\"-1 == 1\" type=\"test\">Test Failed
│     Expression: 1 + -2 == 1
│ -    Evaluated: -1 == 1</failure></testcase><testcase name=\"10 + -5 == 5\" id=\"3\" classname=\"Math/addition/negative addition\" time=\"0.0\"/></testsuite><testsuite name=\"Math/addition\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"2\"><testcase name=\"1 + 1 == 2\" id=\"1\" classname=\"Math/addition\" time=\"0.0\"/><testcase name=\"1 + 2 == 5\" id=\"2\" classname=\"Math/addition\" time=\"0.0\"><failure message=\"3 == 5\" type=\"test\">Test Failed
│ +    Evaluated: -1 == 1
│ + </failure></testcase><testcase name=\"10 + -5 == 5\" id=\"3\" classname=\"Math/addition/negative addition\" time=\"0.0\"/></testsuite><testsuite name=\"Math/addition\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"2\"><testcase name=\"1 + 1 == 2\" id=\"1\" classname=\"Math/addition\" time=\"0.0\"/><testcase name=\"1 + 2 == 5\" id=\"2\" classname=\"Math/addition\" time=\"0.0\"><failure message=\"3 == 5\" type=\"test\">Test Failed
│     Expression: 1 + 2 == 5
│ -    Evaluated: 3 == 5</failure></testcase><testcase name=\"1 + 4 == 5\" id=\"3\" classname=\"Math/addition\" time=\"0.0\"/></testsuite><testsuite name=\"Math/other\" tests=\"4\" failures=\"0\" errors=\"3\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"3\"><testcase name=\"sqrt(-1)\" id=\"1\" classname=\"Math/other\" time=\"0.0\"><skip/></testcase><error message=\"Expression evaluated to non-Boolean\" type=\"Expression evaluated to non-Boolean\" classname=\"Math/other\" time=\"0.0\" id=\"2\"></error><error message=\"Nooo\" type=\"ErrorException\" classname=\"Math/other\" time=\"0.0\" id=\"3\">Nooo
│ +    Evaluated: 3 == 5
│ + </failure></testcase><testcase name=\"1 + 4 == 5\" id=\"3\" classname=\"Math/addition\" time=\"0.0\"/></testsuite><testsuite name=\"Math/other\" tests=\"4\" failures=\"0\" errors=\"3\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"3\"><testcase name=\"sqrt(-1)\" id=\"1\" classname=\"Math/other\" time=\"0.0\"><skip/></testcase><error message=\"Expression evaluated to non-Boolean\" type=\"Expression evaluated to non-Boolean\" classname=\"Math/other\" time=\"0.0\" id=\"2\"></error><error message=\"Nooo\" type=\"ErrorException\" classname=\"Math/other\" time=\"0.0\" id=\"3\">Nooo
│   </error><error message=\"Got correct result, please change to @test if no longer broken.\" type=\"Unexpected Pass\" classname=\"Math/other\" time=\"0.0\" id=\"4\"></error></testsuite><testsuite name=\"Math/Error outside of tests\" tests=\"0\" failures=\"0\" errors=\"1\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"4\"><error message=\"Got exception outside of a @test\" type=\"ErrorException\" classname=\"Math/Error outside of tests\" time=\"0.0\">Outside of tests
│   </error></testsuite><testsuite name=\"Math/Different failures\" tests=\"2\" failures=\"2\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"5\"><testcase name=\"throw(ArgumentError(&quot;1&quot;))\" id=\"1\" classname=\"Math/Different failures\" time=\"0.0\"><failure message=\"Wrong exception type thrown\" type=\"test_throws_wrong\">Test Failed
│     Expression: throw(ArgumentError(\"1\"))
│       Expected: DimensionMismatch
│ -       Thrown: ArgumentError</failure></testcase><testcase name=\"true\" id=\"2\" classname=\"Math/Different failures\" time=\"0.0\"><failure message=\"No exception thrown\" type=\"test_throws_nothing\">Test Failed
│ +       Thrown: ArgumentError
│ + </failure></testcase><testcase name=\"true\" id=\"2\" classname=\"Math/Different failures\" time=\"0.0\"><failure message=\"No exception thrown\" type=\"test_throws_nothing\">Test Failed
│     Expression: true
│       Expected: DimensionMismatch
│ -   No exception thrown</failure></testcase></testsuite><testsuite name=\"Math/using function from a module\" tests=\"1\" failures=\"0\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"6\"><testcase name=\"nthreads() &gt; 0\" id=\"1\" classname=\"Math/using function from a module\" time=\"0.0\"/></testsuite><testsuite name=\"Math\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"7\"><testcase name=\"4 % 2 == 0\" id=\"1\" classname=\"Math\" time=\"0.0\"/><testcase name=\"sqrt(20) == 5\" id=\"2\" classname=\"Math\" time=\"0.0\"><failure message=\"4.47213595499958 == 5\" type=\"test\">Test Failed
│ +   No exception thrown
│ + </failure></testcase></testsuite><testsuite name=\"Math/using function from a module\" tests=\"1\" failures=\"0\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"6\"><testcase name=\"nthreads() &gt; 0\" id=\"1\" classname=\"Math/using function from a module\" time=\"0.0\"/></testsuite><testsuite name=\"Math\" tests=\"3\" failures=\"1\" errors=\"0\" time=\"0.0\" timestamp=\"0\" hostname=\"localhost\" id=\"7\"><testcase name=\"4 % 2 == 0\" id=\"1\" classname=\"Math\" time=\"0.0\"/><testcase name=\"sqrt(20) == 5\" id=\"2\" classname=\"Math\" time=\"0.0\"><failure message=\"4.47213595499958 == 5\" type=\"test\">Test Failed
│     Expression: sqrt(20) == 5
│ -    Evaluated: 4.47213595499958 == 5</failure></testcase><testcase name=\"16 == 16\" id=\"3\" classname=\"Math\" time=\"0.0\"/></testsuite></testsuites>
│ +    Evaluated: 4.47213595499958 == 5
│ + </failure></testcase><testcase name=\"16 == 16\" id=\"3\" classname=\"Math\" time=\"0.0\"/></testsuite></testsuites>
│
│   """
│ -------------------------------
│   reference = "/Users/cvogt/.julia/dev/TestReports/test/references/complexexample.txt"
└   actual = "/var/folders/j4/g03w3zw11_74xnzmms8hqq4w0000gp/T/jl_vkpCeI/complexexample.txt"
Complex Example: Error During Test at /Users/cvogt/.julia/dev/TestReports/test/reportgeneration.jl:14
  Got exception outside of a @test
  To update the reference images either run the tests interactively with 'include("test/runtests.jl")',
  or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE`
  to "true" and re-run the tests via Pkg.

@omus
Copy link
Member Author

omus commented Apr 26, 2024

Also fixed the Documenter workflow which was running into this issue:

[ Info: CheckDocument: running document checks.
┌ Error: 10 docstrings not included in the manual:
│
│     TestReports.compatible :: Tuple{VersionNumber, VersionNumber}
│     TestReports.getdeps :: Tuple{Any}
│     TestReports.check_manifest :: NTuple{4, Any}
│     TestReports.check_manifest :: Tuple{Any, Any, Any}
│     TestReports.check_testreports_compatability :: Tuple{Any, Any, Any}
│     TestReports.hasdeps :: Tuple{Any}
│     TestReports.check_project :: Tuple{Any, Any, Any, Vector}
│     TestReports.check_project :: Tuple{Nothing, Vararg{Any}}
│     TestReports.check_env :: Tuple{Pkg.Types.EnvCache, Any, Any, Any}
│     TestReports.get_dep_entries
│
│ These are docstrings in the checked modules (configured with the modules keyword)
│ that are not included in canonical @docs or @autodocs blocks.
└ @ Documenter ~/.julia/packages/Documenter/pA5Sa/src/utilities/utilities.jl:44

@omus
Copy link
Member Author

omus commented Apr 26, 2024

I just noticed locally that I forgot to commit the new test/reference/complexexample.txt file. It's unfortunate that the CI checks still passed here as this could have resulted in this PR being merged with the reference tests being in a broken state.

Copy link
Member

@oxinabox oxinabox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@oxinabox oxinabox merged commit f13d00a into master Apr 29, 2024
10 checks passed
@omus omus deleted the cv/fix-complex-example-1.9 branch April 29, 2024 13:54
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants