[ETCM-685] Improve pivot block selection

[robinraju]

Description

If Mantis connects with a peer with the best block number bigger than other nodes (the difference bigger than pivotBlockOffset) it will wait for a minimum number of voters infinitely and never choose pivot block and will never start to sync.

Proposed Solution

If a particular pivot block cannot be confirmed,

• the pivot block number is pushed back by a configured amount of blocks and the confirmation process repeats.
• if a maximum number of retries is reached, the task fails and the whole process is restarted.

Important Changes Introduced

PivotBlockSelector.scala

Testing

Unit tests were added.

[jvdp]

It gets the job done and nothing is standing out but that PivotBlockSelector is pretty smelly and is getting just a bit worse with these additions.