Accumulated archive xdr files are not removed

[b-yap]

💥 DISK USAGE High 💥
- reported constantly in aws, the past few weeks.

Possible reason are these files:

-rw-r--r--   1 root root  7012400 Jan 12 14:59 txs-02f92d3f.xdr
-rw-r--r--   1 root root   113313 Jan 24 12:00 02fbddbf.xdr
-rw-r--r--   1 root root  3377498 Jan 24 12:00 txs-02fbddbf.xdr
-rw-r--r--   1 root root   109698 Feb 21 10:49 0302167f.xdr
-rw-r--r--   1 root root  4221382 Feb 21 10:49 txs-0302167f.xdr
-rw-r--r--   1 root root   106607 Feb 21 11:58 0302197f.xdr
-rw-r--r--   1 root root  4539384 Feb 21 11:58 txs-0302197f.xdr
-rw-r--r--   1 root root   101380 Feb 28 14:42 0303a93f.xdr
-rw-r--r--   1 root root  4702975 Feb 28 14:42 txs-0303a93f.xdr
-rw-r--r--   1 root root    84272 Feb 29 12:18 0303e1ff.xdr
-rw-r--r--   1 root root  8330294 Feb 29 12:19 txs-0303e1ff.xdr
-rw-r--r--   1 root root   115431 Mar  1 00:56 0303feff.xdr
-rw-r--r--   1 root root  1695793 Mar  1 00:56 txs-0303feff.xdr
-rw-r--r--   1 root root   109342 Mar  8 20:33 0305b43f.xdr
-rw-r--r--   1 root root  6481192 Mar  8 20:33 txs-0305b43f.xdr
-rw-r--r--   1 root root    27242 Mar 10 17:19 0305cfff.xdr
-rw-r--r--   1 root root   102574 Mar 13 21:01 0306cc7f.xdr
-rw-r--r--   1 root root  6423701 Mar 13 21:01 txs-0306cc7f.xdr
-rw-r--r--   1 root root   102623 Mar 19 12:01 030802ff.xdr
-rw-r--r--   1 root root  6701406 Mar 19 12:01 txs-030802ff.xdr
-rw-r--r--   1 root root   110281 Mar 19 18:01 030813ff.xdr
-rw-r--r--   1 root root  5563178 Mar 19 18:01 txs-030813ff.xdr

These files are created whenever the vault has to build a proof via the archive.
SCP:

spacewalk/clients/vault/src/oracle/collector/proof_builder.rs

Line 237 in b1dc81f

let scp_archive_result = scp_archive_storage.get_archive(slot).await;

TXSET:

spacewalk/clients/vault/src/oracle/collector/proof_builder.rs

Line 326 in b1dc81f

let transactions_archive = tx_archive_storage.get_archive(slot).await;

where after downloading and extraction, these files are never removed:

spacewalk/clients/vault/src/oracle/storage/traits.rs

Lines 103 to 104 in b1dc81f

	download_file_and_save(&url, &file_name).await?;
	result = Self::try_gz_decode_archive_file(&file_name);

TODO:
remove the file after reading/storing to variable

[b-yap]

@pendulum-chain/product, @ebma this is an assessment from the report in #devops-incidents:

[[FIRING:1] DISK Usage High
**Firing**
Value: B=100, C=1
Labels:
- alertname = DISK Usage High
- grafana_folder = devops_folder
- job = Spacewalk

happened last week, and still occurring until today.

[prayagd]

@b-yap can we move this ticket to development as i see you are already working on it?

[ebma]

Nice find @b-yap 👌