3.4.0 Feature release

[sebix]

Documentation: https://docs.intelmq.org/latest/

NEWS

Most important changes potentially requiring administration attention

Requirements

Python 3.8 or newer is required.

Bots

CIF 3 API Output deprecation

The CIF 3 API Output bot is not compatible with Python version greater or equal to 3.12 and will be removed in the future due to lack of maintenance.
See https://lists.cert.at/pipermail/intelmq-users/2024-December/000474.html for more information.

Twitter Collector removal

As the bot does not work anymore and uses an unmaintained library, it is removed from IntelMQ.
Please remove if from your setup.

intelmqctl check and intelmqctl upgrade-config command warns if you have the bot in use.

Twitter Parser renaming

The Twitter parser is renamed to IoC Extractor Parser (intelmq.bots.parsers.ioc_extractor).
intelmqctl upgrade-config will automatically adapt the configuration.

The previous module name is left as a stub to load the IoC Extractor parser for backwards-compatibility.

Packaging

Packages are now also available for Ubuntu 24.04.
To upgrade an Ubuntu 22.04 installation to 24.04 please refer to the Ubuntu documentation: https://documentation.ubuntu.com/server/how-to/software/upgrade-your-release/index.html

Full changelog

Configuration

Core

• AMQP: Fix maintaining pipeline connection when during interrupted connections (PR#2533 by Kamil Mankowski).
• Python 3.8 or newer is required (PR#2541 by Sebastian Wagner).
• intelmq.lib.utils.list_all_bots/intelmqctl check: Fix check for bot executable in $PATH by using the bot name instead of the import path (fixes Extension bots: check for executable wrong #2559, PR#2564 by Sebastian Wagner).

Bots

Collectors

• intelmq.bots.collectors.shadowserver.collector_reports_api.py:
  • Fixed behaviour if parameter types value is empty string, behave the same way as not set, not like no type.
• intelmq.bots.collectors.misp: Use PyMISP class instead of deprecated ExpandedPyMISP (PR#2532 by Radek Vyhnal)
• intelmq.bots.collectors.http.collector_http: Log the downloaded size in bytes to ease troubleshooting (PR#2554 by Sebastian Wagner).
• intelmq.bots.collectors.mail.collector_mail_url:
  • Log the downloaded size in bytes to ease troubleshooting (PR#2554 by Sebastian Wagner).
  • Fix import for Timeout exception preventing another exception (fixes Mail URL Collector: Missing requests import for requests.exceptions.Timeout #2555, PR#2556 by Sebastian Wagner).
• Remove intelmq.bots.collectors.twitter as it uses an unmaintained library and does not work any more (fixes Twitter API changes #2346, Twitter collector issues #2441, PR#2568 by Sebastian Wagner).

Parsers

• intelmq.bots.parsers.shadowserver._config:
  • fix error message formatting if schema file is absent (PR#2528 by Sebastian Wagner).
• intelmq.bots.parsers.shadowserver.parser:
  • Fix to avoid schema download if not configured ShadowServer parser attempts to download schema even if not configured #2530.
• intelmq.bots.parsers.misp.parser: Replace deprecated datetime function utcfromtimestamp for Ubuntu 24.04 compatibility (PR#2577 by Sebastian Wagner, fixes CleanMX and MISP parser call deprecated utcfromtimestamp #2576, Ubuntu 24.04 packages #2571).
• intelmq.bots.parsers.cleanmx.parser: Replace deprecated datetime function utcfromtimestamp for Ubuntu 24.04 compatibility (PR#2577 by Sebastian Wagner, fixes CleanMX and MISP parser call deprecated utcfromtimestamp #2576, Ubuntu 24.04 packages #2571).
• Renamed intelmq.bots.parsers.twitter to intelmq.bots.parser.ioc_extractor (PR#2568 by Sebastian Wagner).
  • Added intelmq.bots.parsers.twitter as a stub to load the IoC Extractor parser.

Experts

• intelmq.bots.experts.securitytxt:
  • Added new bot (PR#2538 by Frank Westers and Sebastian Wagner).
• intelmq.bots.experts.misp: Use PyMISP class instead of deprecated ExpandedPyMISP (PR#2532 by Radek Vyhnal).
• intelmq.bots.experts.fake.expert: New expert to fake data (PR#2567 by Sebastian Wagner).

Outputs

• intelmq.bots.outputs.cif3.output:
  • The requirement can only be installed on Python version < 3.12.
  • Add a check on the Python version and exit if incompatible.
  • Add a deprecation warning (PR#2544 by Sebastian Wagner).
• intelmq.bots.outputs.sql.output:
  • Treat an empty string fields parameter as unset parameter, fixing a crash in default configuration (PR#2548 by Sebastian Wagner, fixes SQL Output bot: SQL syntax error if fields is emty string (default) #2548).

Documentation

• docs/admin/installation/linux-packages: Add [signed-by=] options, add wget command as alternative to curl (PR#2547 by Sebastian Wagner).
• Add documentation on the Redis pipeline (databases, configuration), fix generic pipeline documentation and add missing information on parameters, add unlinked intelmqctl docs to the index and TOC (PR#2560 by Sebastian Wagner).
• Remove empty page tutorials/intelmq-manager (PR#2562 by Sebastian Wagner).

Packaging

• Packages for Ubuntu 24.04 (by Sebastian Wagner, fixes Ubuntu 24.04 packages #2571).

Tests

• Install build dependencies for pymssql on Python 3.8 as there are no wheels available for this Python version (PR#2542 by Sebastian Wagner).
• Install psql explicitly for workflow support on other platforms such as act (PR#2542 by Sebastian Wagner).
• Create intelmq user & group if running privileged to allow dropping privileges (PR#2542 by Sebastian Wagner).
• intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge: Also skip on Python 3.11 and 3.12 besides on 3.8 when running on CI (PR#2542 by Sebastian Wagner).
• Full pytest workflow: Version-independent install of postgres client, for Ubuntu 24.04 (default on GitHub now) test environment compatibility (PR#2557 by Sebastian Wagner).
• Debian package build workflow: Use artifact upload v4 instead of v3 (PR#2565 by Sebastian Wagner).

Known issues

This is short list of the most important known issues. The full list can be retrieved from GitHub.

• intelmqctl: interactive run ignores custom log level (ctl: interactive run ignores custom log level #2563).
• intelmq.parsers.html_table may not process invalid URLs in patched Python version due to changes in urllib (Regression on parsing invalid URLs #2382).
• Breaking changes in 'rt' 3.0 library (Breaking changes in 'rt' 3.0 library #2367).
• Type error with SQL output bot's prepare_values returning list instead of tuple (Type error with SQL output bot's prepare_values returning list instead of tuple #2255).
• intelmq_psql_initdb does not work for SQLite (intelmq_psql_initdb does not work for SQLite #2202).
• intelmqsetup: should install a default state file (intelmqsetup: should install a default state file #2175).
• Misp Expert - Crash if misp event already exist (Misp Expert - Crash if misp event already exist #2170).
• Spamhaus CERT parser uses wrong field (Spamhaus CERT parser uses wrong field #2165).
• Custom headers ignored in HTTPCollectorBot (Custom headers ignored in HTTPCollectorBot  #2150).
• intelmqctl log: parsing syslog does not work (intelmqctl log: parsing syslog does not work #2097).
• Bash completion scripts depend on old JSON-based configuration files (Bash completion scripts depend on old JSON-based configuration files #2094).
• Bots started with IntelMQ-API/Manager stop when the webserver is restarted (Bots started with IntelMQ-API/Manager stop when the webserver is restarted #952).
• Corrupt dump files when interrupted during writing (Corrupt dump files when interrupted during writing #870).

---

This discussion was created from the release 3.4.0 Feature release.