Adding new fields to service tables and reflect them in the ledger

[jumaffre]

As discussed with @eddyashton, and something we've discussed before while planning for compatibility (see for example #2181), but that's more concrete now that we have a 1.x release branch:

We will add new fields to core CCF service tables that were not present in previous versions (in fact, it's already been done for 2.x - e.g. #2844). This means that:

1. The new field will have to be make use of the DECLARE_JSON_OPTIONAL_FIELDS macro, so that the old ledger can be deserialised successfully, since the new field won't exist in the old ledger.
2. The new field will either have to be defined as i) std::optional or ii) a non-std::optional type, with a sensible default value.

A consequence of this is that the new field would not be reflected in the ledger unless it is explicitly written to the store, by a new node. This may make auditing of the ledger more awkward - e.g. if a new bool policy is added in 2.x (defaults to true), it will be enforced by 2.x nodes, without being reflected in the ledger. Auditors would have then to scrutinise the corresponding code ID to find out whether the policy was enforced. Note that auditors have to trust the code ID already, but this adds complexity to the already-complex auditing process.

A few more implementation-specific details:

• If a new field is defined as std::optional, it has to remain as such for its entire lifetime as new nodes may be deserialising old ledgers (e.g. historical queries or recovery from scratch). This is awkward for CCF developers who have to deal with that when using the new field (e.g. using value_or(default)).
• If a new field is added as non-std::optional (e.g. bool with a default value of true), then the current JSON serialisation macros won't serialise this field to the ledger if its value stays the same. We would have to break the symmetry of the serialisation macros so that the field is serialised even if its current value is equal to its default value.

In any case, there would need to be some sort of manual process to serialise the new fields to the ledger, once the code update process is complete (member proposal? internal job?).

Opinions and ideas welcome.

[achamayou]

This may make auditing of the ledger more awkward - e.g. if a new bool policy is added in 2.x (defaults to true), it will be enforced by 2.x nodes, without being reflected in the ledger. Auditors would have then to scrutinise the corresponding code ID to find out whether the policy was enforced. Note that auditors have to trust the code ID already, but this adds complexity to the already-complex auditing process.

I'm sceptical about this. If I understand the argument correctly, you're saying that serialising defaults makes audit easier, because it includes slightly more information in the ledger. That's definitely true, but I don't think it absolves the auditor from understanding the semantic of those values in that version of the code, on the contrary.

Looking at a "null" value, and assuming more from it than from its absence does not strike me as sound auditing practice. Even if the project had a written policy about serialising new fields, that's still only 1. a social claim, 2. applicable to new fields. How empty new fields are serialised is far less important than what the variant of the code the primary was running at the time does with them.

Because all governance tables are done in JSON, I also think it's reasonable to set fields only when they're used, rather than populate lots of them to "null" when they aren't needed, but that's only opinion.