Fix/remove old generations on rebuild

[bkapustik]

Motivation

• Changes retention policy to only store the current Lucene index generation in the file system

Checklist

• Code follows coding conventions held in this repo
• Automated tests have been added
• Tests are passing
• Docs have been updated (if applicable)
• Temporary settings (e.g. variables used during development and testing) have been reverted to defaults

How to test

If manual testing is required, what are the steps?