searchd leaves binlog.meta and binlog.001 after clean stop

[sanikolaev]

If you stop Manticore, it leaves binlog.meta and binlog.001 which can then result in difficulties when upgrading to a version with a new binlog format and even a crash - #1364

root@1b89ba28833b:/# mysql -P9306 -h0 -e "show status like 'version'"
+---------+-----------------------------------------------------------------------------------------+
| Counter | Value                                                                                   |
+---------+-----------------------------------------------------------------------------------------+
| version | 6.2.0 45680f95d@230804 (columnar 2.2.0 dc33868@230804) (secondary 2.2.0 dc33868@230804) |
+---------+-----------------------------------------------------------------------------------------+

root@1b89ba28833b:/# ls -la /var/lib/manticore/binlog/
total 16
drwx------ 2 manticore manticore 4096 Aug 17 01:14 .
drwxr-xr-x 3 manticore manticore 4096 Aug 17 01:14 ..
-rw------- 1 manticore manticore    8 Aug 17 01:14 binlog.001
-rw------- 1 manticore manticore    0 Aug 17 01:14 binlog.lock
-rw------- 1 manticore manticore   11 Aug 17 01:14 binlog.meta

root@1b89ba28833b:/# systemctl stop manticore

root@1b89ba28833b:/# tail /var/log/manticore/searchd.log
[Thu Aug 17 01:14:36.412 2023] [2189] accepting connections
[Thu Aug 17 01:14:36.441 2023] [2201] [BUDDY] started v1.0.18 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://127.0.0.1:9312  --threads=32' at http://127.0.0.1:36195
[Thu Aug 17 01:14:36.625 2023] [2214] [BUDDY] Loaded plugins:
[Thu Aug 17 01:14:36.625 2023] [2214] [BUDDY]   core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
[Thu Aug 17 01:14:36.625 2023] [2214] [BUDDY]   local:
[Thu Aug 17 01:14:36.625 2023] [2214] [BUDDY]   extra:
[Thu Aug 17 01:46:34.008 2023] [2189] caught SIGTERM, shutting down
[Thu Aug 17 01:46:34.018 2023] [2189] shutdown daemon version '6.2.0 45680f95d@230804 (columnar 2.2.0 dc33868@230804) (secondary 2.2.0 dc33868@230804)' ...
[Thu Aug 17 01:46:34.018 2023] [2189] shutdown complete
[Thu Aug 17 01:46:34.020 2023] [2188] watchdog: main process 2189 exited cleanly (exit code 0), shutting down
root@1b89ba28833b:/#

root@1b89ba28833b:/# ls -la /var/lib/manticore/binlog/
total 16
drwx------ 2 manticore manticore 4096 Aug 17 01:46 .
drwxr-xr-x 3 manticore manticore 4096 Aug 17 01:46 ..
-rw------- 1 manticore manticore   18 Aug 17 01:46 binlog.001
-rw------- 1 manticore manticore   11 Aug 17 01:14 binlog.meta

The tasks are:

• Make sure that we really need to retain the binlog files in this situation.
• If we do, enhance the searchd behavior so that a new version can start smoothly with old-format binlog files. Currently, it does not, as shown in Crash after upgrading from 6.0.4 to 6.2.0 #1364.

[tomatolog]

the FATAL message on reading empty binlog file with the old format fixed at 828d139

And daemon now could start with the binlog of old format if the binlog empty or has only empty ADD_CACHE operation.

[tomatolog]

added removal of the last binlog file if there was no transactions at 0940411 on daemon shutdown

Now start then stop of the daemon without indexes should not leave binlog.001 file