Use STIX-2 as internal format

[0snap]

📔 Description

• Use STIX-2 (version 2.1) as internal transport format for Indicators and Sightings. This breaks all existing plugins.
• Migrate in-memory backbone plugin
• Migrate RabbitMQ backbone plugin
• Migate ZMQ-App plugin
• The unit tests for all un-migrated plugins and pyvast-threatbus are disabled to make the CI happy. They will be migrated in separate PRs

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/threatbus, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

This PR can already be reviewed. Documentation will follow in the next iteration.

• Go commit-by-commit
• Use tests/utils to test all functionality

Walkthrough:

• Install a fresh virtualenv
• Start a local RabbitMQ (e.g., simply via docker)
• Install TB + only those plugins that changed in this PR (in-mem, rabbitmq, zmq)
• Start Threat Bus with the example config

Then you can use the tests/utils/zmq_sender.py and tests/utils/zmq_receiver.py to mimick small "apps". They connect to the ZMQ plugin and send messages back and forth. This should work independent of the backbone (in-mem or rabbit). Try both.

[lgtm-com]

This pull request introduces 2 alerts when merging 5ef2c14 into 671a0ea - view on LGTM.com

new alerts:

• 1 for Except block handles 'BaseException'
• 1 for Unused import

[rolandpeelen]

I ran this branch locally, and tested it. RabbitMQ backbone / ZMQ sender / receiver and was able to send messages across. I also tried to send some gibberish and that failed as expected.

I went through the code commit-by-commit. I'm not intimitely familiar with Threatbus nor Python, but I did not see anything out of the ordinary. I saw some points where I would possibly break some elements out to functions (like the double big try catch block in rabbitmq_publisher), but at the same time, breaking it out to a different file would not necessarily make it more clear.

If anyone wants to weigh in as well, feel free. But from my pov this is good to go :)