How to test correctness of spoon.metamodel.Metamodel?

[pvojtechovsky]

I compared results returned by new spoon.metamodel.Metamodel with hardcoded Metamodel which was in PR #1686 and it found some problems.

Many problems were fixed by readding contract "unsettable implies derived" from #1904, which was removed during Metamodel refactorings.

Here are remaining problems

Field CtInterface#modifier<java.util.Set<spoon.reflect.declaration.ModifierKind>>.derived hardcoded value = false but computed value is true
Field CtInterface#interface<java.util.Set<spoon.reflect.reference.CtTypeReference<? extends java.lang.Object>>>.derived hardcoded value = false but computed value is true
Field CtTypeParameter#modifier<java.util.Set<spoon.reflect.declaration.ModifierKind>>.unsettable hardcoded value = true but computed value is false
Field CtTypeParameter#superType<spoon.reflect.reference.CtTypeReference<? extends java.lang.Object>>.derived hardcoded value = false but computed value is true
Field CtConstructor#name<java.lang.String>.derived hardcoded value = true but computed value is false
Field CtConstructor#name<java.lang.String>.unsettable hardcoded value = true but computed value is false
Field CtAnonymousExecutable#name<java.lang.String>.derived hardcoded value = true but computed value is false
Field CtAnonymousExecutable#name<java.lang.String>.unsettable hardcoded value = true but computed value is false
Field CtWildcardReference#name<java.lang.String>.derived hardcoded value = true but computed value is false
Field CtWildcardReference#name<java.lang.String>.unsettable hardcoded value = true but computed value is false
Field CtEnum#label<java.lang.String>.derived hardcoded value = true but computed value is false
Field CtEnum#label<java.lang.String>.unsettable hardcoded value = true but computed value is false
Field CtEnum#modifier<java.util.Set<spoon.reflect.declaration.ModifierKind>>.derived hardcoded value = false but computed value is true
Field CtEnum#interface<java.util.Set<spoon.reflect.reference.CtTypeReference<? extends java.lang.Object>>>.derived hardcoded value = false but computed value is true
Field CtClass#label<java.lang.String>.derived hardcoded value = true but computed value is false
Field CtClass#label<java.lang.String>.unsettable hardcoded value = true but computed value is false
Field CtClass#modifier<java.util.Set<spoon.reflect.declaration.ModifierKind>>.derived hardcoded value = false but computed value is true
Field CtClass#superType<spoon.reflect.reference.CtTypeReference<? extends java.lang.Object>>.derived hardcoded value = false but computed value is true
Field CtClass#interface<java.util.Set<spoon.reflect.reference.CtTypeReference<? extends java.lang.Object>>>.derived hardcoded value = false but computed value is true
Field CtAnnotationType#modifier<java.util.Set<spoon.reflect.declaration.ModifierKind>>.derived hardcoded value = false but computed value is true

[pvojtechovsky]

After some more playing with Unsettable implies derive the list shorten to:

Field CtTypeParameter#modifier<java.util.Set<spoon.reflect.declaration.ModifierKind>>.derived hardcoded value = true but computed value is false
Field CtTypeParameter#modifier<java.util.Set<spoon.reflect.declaration.ModifierKind>>.unsettable hardcoded value = true but computed value is false
Field CtConstructor#name<java.lang.String>.derived hardcoded value = true but computed value is false
Field CtConstructor#name<java.lang.String>.unsettable hardcoded value = true but computed value is false
Field CtAnonymousExecutable#name<java.lang.String>.derived hardcoded value = true but computed value is false
Field CtAnonymousExecutable#name<java.lang.String>.unsettable hardcoded value = true but computed value is false
Field CtWildcardReference#name<java.lang.String>.derived hardcoded value = true but computed value is false
Field CtWildcardReference#name<java.lang.String>.unsettable hardcoded value = true but computed value is false
Field CtEnum#label<java.lang.String>.derived hardcoded value = true but computed value is false
Field CtEnum#label<java.lang.String>.unsettable hardcoded value = true but computed value is false
Field CtClass#label<java.lang.String>.derived hardcoded value = true but computed value is false
Field CtClass#label<java.lang.String>.unsettable hardcoded value = true but computed value is false

note: hardcoded value is correct. Comuputed value is wrong - returned by spoon.metamodelMetamodel

[pvojtechovsky]

@monperrus There is a question how to assure correctness of computed metamodel in future. Current Spoon master tests are not sufficient ... and we planned to replace hardcoded metamodel from #1686 by current spoon.metamodel.Metamodel ... so there will be no test code which can check it's correctness.

Solution
A) to move hardcoded metamodel from #1686 to test package and to change it manually in case of real change of spoon metamodel. This let's us write a test which checks whethere these two metamodels are equal.
B) To persist the metamodel somehow and commit the persisted representation into git. Then tests can check if metamodel is equal like persisted version
C) ?

Note: CtScanner based metamodel test is good, but it checks only roles whose values extends CtElement. The primitive role attributes (name, modifier, label, ...) are not tested here. Therefore the problem was not visible.

[monperrus]

Answering in #2050