[Reducing DB interactions at mongoBackend] Avoid update csub when notifications are sent

[fgalan]

Currently, each time a notification is sent, the csub is updated to increase count and set the new lastNotification. This is inefficent.

A better approach should be implemented. A couple of ideas:

• Not using at all count or lastNotification in DB (although it could be a problem considering the current NGSIv2 JSON for subscriptions, which takes into account both fields)
• Accumulate all the changes (maybe using the cache structures?) and consolidate in DB in a regular basic (maybe at cache refresh frecuency?). The drawback is the risk of data lost and eventual consistency in the case of Active-Active configurations.

[fgalan]

Considering option 2 (accumulate and cosolidate at refresh time) a solution is to use one update() for each field:

To update count, being <n> the amount to sum to DB:

update({ _id: ObjectId(<subId>)}, {count: {$inc: <n>}})

To update lastNotified, being the last notification time:

update({ _id: ObjectId(<subId>), lastNotification: {$lt: <lnTime>}}, {lastNotification: <lnTime>}})

(Not sure if the above is 100% correct syntax)

Actually, the above is expressed in mongo query language. In our code, the collectionUpdate() function has to be used. The first argument of the update() above should be mapped to a BSONObj in the q argument, while the second argument should be mapped to a BSONObj in the doc argument. upsert should be set to false (it is assumed that the document in the csub collection already exists).

bool collectionUpdate
(
  const std::string&  col,
  const BSONObj&      q,
  const BSONObj&      doc,
  bool                upsert,
  std::string*        err
)

Of course 2 update() is worse than 1 update() :). However, I cannot figure out how to do this with just one operation. Maybe an important optimization would be to use a bulk update to update all the cache documents: https://docs.mongodb.org/manual/reference/method/Bulk.find.updateOne/#Bulk.find.updateOne (not essential for #1476 completion, as we can assume that refresh wouldn't be too frecuent in production cases, i.e. in the order of tens of senconds; an issue should be opened about it).

As additional comment -subCacheIval 0 is no longer a good idea, as count/lastNotification should be sync at DB from time to time (to prevent that a CB crash lost too much information). In addition, orionExit() function should execute the refresh part to sync count/lastNotification fields.

(Final thought: maybe "refresh" is not a good name for the operation. At least I tend to think in "refresh" as an exclusive DB -> cache operation. Maybe "sync" is better. Not sure anyway).

[kzangeli]

Yeah, I think mongoSubCacheSync might be a better name when this is included.

Now, the current mongoSubCacheRefresh* function throws away the cache and replaces it with what was found in the db. When lastNotificationTime and count is no longer maintained in the DB, this will have to be changed. mongoSubCacheRefresh must get smarter.

1. Add count to CachedSubscriptions, set it to 0 after each mongoSubCacheSync (to count the increment only)
2. Before mongoSubCacheRefresh, save lastNotificationTime and count for each notification in the cache (as a vector of struct { subId, lastNotificationTime, count } ), after that
3. refresh from DB, and then,
4. compare lastNotificationTime from the "new" cache to the "old" value - the "old" value might be more recent than what was found in the DB ...
5. These lastNotificationTime must now be flushed to DB and the accumulated counts also.

[fgalan]

Implemented in PR #1536. However, some modifications has bene deferred related with -noCache. From comments in that PR:

• https://github.com/telefonicaid/fiware-orion/pull/1536/files#r45840363: "I see that 35 test were modified in order to fit to the new way. However, maybe to have a sample of them (e.g. 5 tests?) with cache disabled (it should be a matter of adjusting a global/library variable) is a good idea, as a "watchdog" to ensure that old functionality still working."
• https://github.com/telefonicaid/fiware-orion/pull/1536/files#r45840038 "I'd suggest to get a copy of the original subscriptions_onchange.test file (e.g. subscriptions_onchange_nocache.test) in order to ensure that the -noCache functionaly still working in the old way (subscriptions_onchange_nocache.test would need an adjustment in the brokerStart line to use the -noCache option, of course)."
• https://github.com/telefonicaid/fiware-orion/pull/1536/files#r45840090: "Same XXX-nocache.test idea here"

[fgalan]

Finished in PR #1544