diff --git a/.github/workflows/python-egg.yml b/.github/workflows/python-egg.yml index ab824c86..7d274a3c 100644 --- a/.github/workflows/python-egg.yml +++ b/.github/workflows/python-egg.yml @@ -74,13 +74,13 @@ jobs: username: ${{ secrets.DOCKERHUB_USER }} password: ${{ secrets.DOCKERHUB_PASSWORD }} tags: latest,${{ github.sha }} - - name: Publish pyvast-threatbus Docker Image + - name: Publish vast-threatbus Docker Image uses: elgohr/Publish-Docker-Github-Action@3.04 with: - name: tenzir/pyvast-threatbus + name: tenzir/vast-threatbus username: ${{ secrets.DOCKERHUB_USER }} password: ${{ secrets.DOCKERHUB_PASSWORD }} - dockerfile: docker/pyvast-threatbus/Dockerfile + dockerfile: docker/vast-threatbus/Dockerfile tags: latest,${{ github.sha }} - name: Publish stix-shifter-threatbus Docker Image uses: elgohr/Publish-Docker-Github-Action@3.04 diff --git a/README.md b/README.md index 16a1d7c2..7ed268a0 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ A threat intelligence dissemination layer for open-source security tools. - **Official Plugins**: We maintain many plugins right in the official Threat Bus repository. Check out our integrations for [MISP][misp], [Zeek][zeek], [CIFv3][cif], and generally apps that connect via [ZeroMQ][zmq], like - [pyvast-threatbus][pyvast-threatbus] and our + [vast-threatbus][vast-threatbus] and our [OpenCTI connector][opencti-connector]. - **Snapshotting**: The snapshot feature allows subscribers to directly request @@ -256,7 +256,7 @@ Threat Bus comes with a [3-clause BSD license][license-url]. [misp-zmq-config]: https://github.com/MISP/misp-book/tree/master/misp-zmq#misp-zeromq-configuration [plugin-types]: https://docs.tenzir.com/threatbus/plugins/overview [plugin-development]: https://docs.tenzir.com/threatbus/plugins/plugin-development -[pyvast-threatbus]: https://github.com/tenzir/threatbus/tree/master/apps/vast +[vast-threatbus]: https://github.com/tenzir/threatbus/tree/master/apps/vast [pypi-badge]: https://img.shields.io/pypi/v/threatbus.svg [pypi-url]: https://pypi.org/project/threatbus diff --git a/apps/vast/CHANGELOG.md b/apps/vast/CHANGELOG.md index 6669d018..d06a39c0 100644 --- a/apps/vast/CHANGELOG.md +++ b/apps/vast/CHANGELOG.md @@ -12,6 +12,11 @@ Every entry has a category for which we use the following visual abbreviations: ## Unreleased +- ⚡️ We renamed PyVAST Threat Bus to VAST Threat Bus for clarity. The PyPI + package name and the binary name change from `pyvast-threatbus` to + `vast-threatbus` accordingly. + [#159](https://github.com/tenzir/threatbus/pull/159) + - 🎁 Live matching with VAST works again! [#156](https://github.com/tenzir/threatbus/pull/156) diff --git a/apps/vast/README.md b/apps/vast/README.md index 61b9f28a..7a838805 100644 --- a/apps/vast/README.md +++ b/apps/vast/README.md @@ -1,9 +1,10 @@ -PyVAST Threat Bus App -===================== +VAST Threat Bus App +=================== -Threat Bus is a publish-subscribe broker for threat intelligence. It is expected -that applications register themselves at the bus. Since VAST cannot do so on its -own (yet), `pyvast-threatbus.py` implements that functionality in the meantime. +Threat Bus is a publish-subscribe broker for threat intelligence. It is +expected that applications register themselves at the bus. Since VAST cannot do +so on its own (yet), `vast-threatbus.py` implements that functionality in the +meantime. The application provides a thin layer around [PyVAST](https://docs.tenzir.com/vast/python-bindings/overview), VAST's Python @@ -12,12 +13,12 @@ instance. ## Installation -Install `pyvast-threatbus` via `pip`. Optionally, use a virtual environment. +Install `vast-threatbus` via `pip`. Optionally, use a virtual environment. ``` virtualenv venv # optional source venv/bin/activate # optional -python -m pip install pyvast-threatbus +python -m pip install vast-threatbus ``` ## Development @@ -47,7 +48,7 @@ Bus, or pass a path to configuration file via `-c /path/to/config.yaml`. Start the application: ```sh -pyvast-threatbus +vast-threatbus ``` ### Docker @@ -56,20 +57,20 @@ You can also run this app via Docker. - Build it: ``` - docker build . -t tenzir/pyvast-threatbus:latest + docker build . -t tenzir/vast-threatbus:latest ``` - Run it to print the helptext. ``` - docker run tenzir/pyvast-threatbus:latest + docker run tenzir/vast-threatbus:latest ``` - Run and mount a custom config file into the container: ``` - docker run --net=host -v /path/to/your/conf.yaml:/opt/tenzir/threatbus/pyvast-threatbus/config.yaml tenzir/pyvast-threatbus:latest -c config.yaml + docker run --net=host -v /path/to/your/conf.yaml:/opt/tenzir/threatbus/vast-threatbus/config.yaml tenzir/vast-threatbus:latest -c config.yaml ``` ## Features -This section explains the most important features of `pyvast-threatbus`. +This section explains the most important features of `vast-threatbus`. ### IoC Matching @@ -82,13 +83,13 @@ VAST's [live matching](https://docs.tenzir.com/vast/features/threat-intel-matching) works as [continuous query](https://docs.tenzir.com/vast/cli/vast/export/#documentation). -`pyvast-threatbus` subscribes to those continuous query results and reports all +`vast-threatbus` subscribes to those continuous query results and reports all new IoC matches from VAST to Threat Bus as `Sightings`. You can enable live matching in the config file by setting `live_match: true`. #### Retro Matching -`pyvast-threatbus` supports retro matching. You can enable it in the config file +`vast-threatbus` supports retro matching. You can enable it in the config file by setting `retro_match: true`. This instructs the application to translate IoCs from Threat Bus to normal VAST queries instead of feeding the IoCs to a live matcher. @@ -99,7 +100,7 @@ VAST by setting the config option `retro_match_max_events` to a positive integer ### Sighting Context Transformation -You can configure `pyvast-threatbus` to invoke another program for parsing +You can configure `vast-threatbus` to invoke another program for parsing Sighting `context` data via the config option `transform_context`. If set, the app translates the `x_threatbus_sighting_context` field of a STIX-2 @@ -122,7 +123,7 @@ actually matched IoC. ### Custom Sinks for Sightings -`pyvast-threatbus` offers to send Sighting context to a configurable `sink` +`vast-threatbus` offers to send Sighting context to a configurable `sink` _instead_ of reporting them back to Threat Bus. This can be configured via the `sink` configuration parameter. The special placeholder `STDOUT` can be used to print the Sighting context to `STDOUT`. diff --git a/apps/vast/config.yaml.example b/apps/vast/config.yaml.example index 45e4a5a1..d8c2175f 100644 --- a/apps/vast/config.yaml.example +++ b/apps/vast/config.yaml.example @@ -6,7 +6,7 @@ logging: console_verbosity: INFO file: false file_verbosity: INFO - filename: pyvast-threatbus.log + filename: vast-threatbus.log metrics: interval: 10 # set to 0 to disable metrics diff --git a/apps/vast/setup.py b/apps/vast/setup.py index 8aa3175a..e56b56d6 100644 --- a/apps/vast/setup.py +++ b/apps/vast/setup.py @@ -23,7 +23,7 @@ ], description="Connect the open source telemetry engine VAST with Threat Bus, the open source threat intelligence dissemination layer", entry_points={ - "console_scripts": ["pyvast-threatbus=pyvast_threatbus.pyvast_threatbus:main"] + "console_scripts": ["vast-threatbus=vast_threatbus.vast_threatbus:main"] }, include_package_data=True, install_requires=[ @@ -47,8 +47,8 @@ license="BSD 3-clause", long_description=long_description, long_description_content_type="text/markdown", - name="pyvast-threatbus", - packages=["pyvast_threatbus"], + name="vast-threatbus", + packages=["vast_threatbus"], python_requires=">=3.7", setup_requires=["setuptools", "wheel"], url="https://github.com/tenzir/threatbus", diff --git a/apps/vast/pyvast_threatbus/__init__.py b/apps/vast/vast_threatbus/__init__.py similarity index 100% rename from apps/vast/pyvast_threatbus/__init__.py rename to apps/vast/vast_threatbus/__init__.py diff --git a/apps/vast/pyvast_threatbus/message_mapping.py b/apps/vast/vast_threatbus/message_mapping.py similarity index 99% rename from apps/vast/pyvast_threatbus/message_mapping.py rename to apps/vast/vast_threatbus/message_mapping.py index 28831007..092867af 100644 --- a/apps/vast/pyvast_threatbus/message_mapping.py +++ b/apps/vast/vast_threatbus/message_mapping.py @@ -6,7 +6,7 @@ from typing import Tuple, Union import logging -logger_name = "pyvast-threatbus" +logger_name = "vast-threatbus" logger = logging.getLogger(logger_name) vast_ioc_type_map = { diff --git a/apps/vast/pyvast_threatbus/metrics.py b/apps/vast/vast_threatbus/metrics.py similarity index 100% rename from apps/vast/pyvast_threatbus/metrics.py rename to apps/vast/vast_threatbus/metrics.py diff --git a/apps/vast/pyvast_threatbus/test_message_mapping.py b/apps/vast/vast_threatbus/test_message_mapping.py similarity index 100% rename from apps/vast/pyvast_threatbus/test_message_mapping.py rename to apps/vast/vast_threatbus/test_message_mapping.py diff --git a/apps/vast/pyvast_threatbus/test_metrics.py b/apps/vast/vast_threatbus/test_metrics.py similarity index 100% rename from apps/vast/pyvast_threatbus/test_metrics.py rename to apps/vast/vast_threatbus/test_metrics.py diff --git a/apps/vast/pyvast_threatbus/pyvast_threatbus.py b/apps/vast/vast_threatbus/vast_threatbus.py similarity index 98% rename from apps/vast/pyvast_threatbus/pyvast_threatbus.py rename to apps/vast/vast_threatbus/vast_threatbus.py index e113f44c..b4fc8882 100755 --- a/apps/vast/pyvast_threatbus/pyvast_threatbus.py +++ b/apps/vast/vast_threatbus/vast_threatbus.py @@ -31,7 +31,7 @@ import time import zmq -logger_name = "pyvast-threatbus" +logger_name = "vast-threatbus" logger = logging.getLogger(logger_name) matcher_name = None # List of all running async tasks of the bridge. @@ -82,7 +82,7 @@ def validate_config(config: Settings): when=Validator("logging.file", eq=True), default="INFO", ), - Validator("logging.filename", default="pyvast-threatbus.log"), + Validator("logging.filename", default="vast-threatbus.log"), Validator("vast", default="localhost:42000"), Validator("vast_binary", default="vast"), Validator("threatbus", default="localhost:13370"), @@ -261,7 +261,7 @@ async def write_metrics(every: int, to: str): @param to the filepath to write to """ while True: - line = f"pyvast-threatbus,host={socket.getfqdn()} " + line = f"vast-threatbus,host={socket.getfqdn()} " start_length = len(line) for m in metrics: if not m.is_set: @@ -314,7 +314,7 @@ async def receive(pub_endpoint: str, topic: str, indicator_queue: asyncio.Queue) continue # the topic is suffixed with the message type if not topic.endswith("indicator"): - # pyvast-threatbus is not (yet) interested in Sightings or SnapshotRequests + # vast-threatbus is not (yet) interested in Sightings or SnapshotRequests logger.debug(f"Skipping unsupported message: {msg}") continue await indicator_queue.put(msg) @@ -789,7 +789,7 @@ def main(): config = Dynaconf( settings_files=settings_files, load_dotenv=True, - envvar_prefix="PYVAST_THREATBUS", + envvar_prefix="VAST_THREATBUS", ) try: @@ -828,7 +828,7 @@ def main(): if user_exit: # Tasks were cancelled because the user stopped the app. return - logger.info("Restarting pyvast-threatbus ...") + logger.info("Restarting vast-threatbus ...") if __name__ == "__main__": diff --git a/docker/pyvast-threatbus/Dockerfile b/docker/vast-threatbus/Dockerfile similarity index 86% rename from docker/pyvast-threatbus/Dockerfile rename to docker/vast-threatbus/Dockerfile index d96e9f7e..c9b82e56 100644 --- a/docker/pyvast-threatbus/Dockerfile +++ b/docker/vast-threatbus/Dockerfile @@ -19,14 +19,14 @@ COPY threatbus threatbus RUN python3 -m pip install . # Install the app. -WORKDIR /opt/tenzir/threatbus/pyvast-threatbus +WORKDIR /opt/tenzir/threatbus/vast-threatbus COPY apps/vast/setup.py . COPY apps/vast/README.md . -COPY apps/vast/pyvast_threatbus pyvast_threatbus +COPY apps/vast/vast_threatbus vast_threatbus RUN python3 -m pip install . RUN echo "Adding threatbus user" && useradd -m -d /home/threatbus --user-group threatbus RUN chown -R threatbus . USER threatbus:threatbus -ENTRYPOINT ["pyvast-threatbus"] +ENTRYPOINT ["vast-threatbus"] diff --git a/plugins/apps/threatbus_zmq/README.md b/plugins/apps/threatbus_zmq/README.md index f6b7e991..66e2de72 100644 --- a/plugins/apps/threatbus_zmq/README.md +++ b/plugins/apps/threatbus_zmq/README.md @@ -22,7 +22,7 @@ pip install threatbus-zmq ## Configuration The plugin uses ZeroMQ to communicate with applications, like -[pyvast-threatbus](https://github.com/tenzir/threatbus/tree/master/apps/vast). +[vast-threatbus](https://github.com/tenzir/threatbus/tree/master/apps/vast). The plugin serves three ZeroMQ endpoints to connect with. One endpoint for managing subscriptions (and thus snapshot requests). The other two endpoints exist for pub-sub operations. diff --git a/systemd/README.md b/systemd/README.md index cb538ca1..72500cf6 100644 --- a/systemd/README.md +++ b/systemd/README.md @@ -2,22 +2,22 @@ Threat Bus Systemd Unit ======================= This folder provides `systemd` service units to run Threat Bus and -`pyvast-threatbus` as system services. The services are sandboxed and run with +`vast-threatbus` as system services. The services are sandboxed and run with limited privileges. ## Prepare the Host System The systemd unit declares a private user. Hence, all logs go to `/var/log/private` by default. The following section explains how to configure -file-logging for Threat Bus and `pyvast-threatbus`. Skip the following +file-logging for Threat Bus and `vast-threatbus`. Skip the following instructions if you configure your applications to use console-logging. Find the `logging` config section at the top of your Threat Bus or -`pyvast-threatbus` configuration file and change it to use the private log +`vast-threatbus` configuration file and change it to use the private log directory: - `/var/log/private/threatbus/threatbus.log` (Threat Bus) -- `/var/log/private/pyvast-threatbus/pyvast-threatbus.log` (`pyvast-threatbus`) +- `/var/log/private/vast-threatbus/vast-threatbus.log` (`vast-threatbus`) See the following YAML snippet for a configuration example. @@ -34,9 +34,9 @@ logging: Before you begin, find the line beginning with `ExecStart=` at the very bottom of the `[Service]` section in the unit file. Depending on your installation path -you might need to change the location of the `threatbus` and `pyvast-threatbus` +you might need to change the location of the `threatbus` and `vast-threatbus` executable packages and configuration files. Similarly, you need to change the -environmentvariables `THREATBUSDIR` and `PYVAST_THREATBUSDIR` according to your +environmentvariables `THREATBUSDIR` and `vast_THREATBUSDIR` according to your installation paths. - Threat Bus @@ -45,17 +45,17 @@ installation paths. ExecStart=/installation/path/threatbus --config=/installation/path/threatbus/config.yaml ``` -- `pyvast-threabus` +- VAST Threat Bus ```bash - Environment="PYVAST_THREATBUSDIR=/installation/path" - ExecStart=/installation/path/pyvast-threatbus --config=/installation/path/pyvast-threatbus/config.yaml + Environment="vast_THREATBUSDIR=/installation/path" + ExecStart=/installation/path/vast-threatbus --config=/installation/path/vast-threatbus/config.yaml ``` Then copy (or symlink) the unit file to `/etc/systemd/system`. ```bash systemctl link "$PWD/threatbus.service" -systemctl link "$PWD/pyvast-threatbus.service" +systemctl link "$PWD/vast-threatbus.service" ``` To have the services start up automatically with system boot, you can `enable` @@ -64,6 +64,6 @@ them via `systemd`. Otherwise, just `start` it to run it immediately. ```bash systemctl enable threatbus systemctl start threatbus -systemctl enable pyvast-threatbus -systemctl start pyvast-threatbus +systemctl enable vast-threatbus +systemctl start vast-threatbus ``` diff --git a/systemd/pyvast-threatbus.service b/systemd/vast-threatbus.service similarity index 81% rename from systemd/pyvast-threatbus.service rename to systemd/vast-threatbus.service index dc472f73..7e7d9ae5 100644 --- a/systemd/pyvast-threatbus.service +++ b/systemd/vast-threatbus.service @@ -7,7 +7,7 @@ After=network-online.target Type=simple DynamicUser=yes -LogsDirectory=pyvast-threatbus +LogsDirectory=vast-threatbus # capabilities RestrictNamespaces=yes @@ -28,7 +28,7 @@ ProtectKernelLogs=yes # service specifics TimeoutStopSec=600 -ExecStart=/opt/tenzir/threatbus/pyvast-threatbus -c /opt/tenzir/threatbus/pyvast-threatbus.yaml +ExecStart=/opt/tenzir/threatbus/vast-threatbus -c /opt/tenzir/threatbus/vast-threatbus.yaml [Install] WantedBy=multi-user.target