refactor: create_standby: add semaphore during delta generating to limit pending tasks in memory

[Bodong-Yang]

Description

In #308 , the active slot scanning is improved by waiting for all tasks instead of waiting for each folder being scanned, but this brings an issue that if the files processing speed is not fast enough, the pending tasks will pile up.
This PR implements the limitation of total ongoing files scanning entries, and define the number of thread workers.

Tests

• local pytest passed.
• test with VM passed.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/ota_metadata/legacy
__init__.py	11	0	100%
parser.py	326	35	89%	145, 150, 186–187, 197–198, 201, 213, 271, 281–284, 323–326, 406, 409, 417–419, 432, 441–442, 445–446, 720, 723, 735–740
types.py	84	13	84%	37, 40–42, 112–116, 122–125
src/ota_proxy
__init__.py	36	10	72%	59, 61, 63, 72, 81–82, 102, 104–106
__main__.py	7	7	0%	16–18, 20, 22–23, 25
_consts.py	15	0	100%
cache_control.py	68	4	94%	71, 91, 113, 121
config.py	18	0	100%
db.py	146	15	89%	75, 81, 103, 113, 116, 145–147, 166, 199, 208–209, 229, 258, 300
errors.py	5	0	100%
orm.py	112	10	91%	92, 97, 102, 108, 114, 141–142, 155, 232, 236
ota_cache.py	401	84	79%	98–99, 218, 229, 256–258, 278, 294–295, 297, 320–321, 327, 331, 333, 360–362, 378, 482–483, 553, 566–569, 619, 638–639, 671–672, 683, 717–721, 725–727, 729, 731–738, 740–742, 745–746, 750–751, 755, 802, 810–812, 891–894, 898, 901–902, 916–917, 919–921, 925–926, 932–933, 964, 970, 997, 1026–1028
server_app.py	138	39	71%	76, 79, 85, 101, 103, 162, 171, 213–214, 216–218, 221, 226–228, 231–232, 235, 238, 241, 244, 257–258, 261–262, 264, 267, 293–296, 299, 313–315, 321–323
utils.py	23	1	95%	33
src/otaclient
__init__.py	5	2	60%	17, 19
__main__.py	1	1	0%	16
log_setting.py	52	5	90%	53, 55, 64–66
src/otaclient/app
__main__.py	1	1	0%	16
configs.py	76	0	100%
errors.py	112	0	100%
interface.py	5	0	100%
main.py	46	5	89%	52–53, 75–77
ota_client.py	392	131	66%	67, 75, 96, 201–203, 214, 260–263, 275–278, 281–284, 294–297, 302–303, 305, 314, 317, 322–323, 326, 332, 334, 337, 379–382, 387, 391, 394, 410–413, 416–423, 426–433, 439–442, 471, 474–475, 477, 480–483, 485–486, 491–492, 495, 509–516, 523, 526–532, 579–582, 590, 626, 631–634, 639–641, 644–645, 647–648, 650–651, 653, 713–714, 717, 725–726, 729, 740–741, 744, 752–753, 756, 767, 786, 813, 832, 850
ota_client_stub.py	394	109	72%	76–78, 80–81, 89–92, 95–97, 101, 106–107, 109–110, 113, 115–116, 119–121, 124–125, 128–130, 135–140, 144, 147–151, 153–154, 162–164, 167, 204–206, 211, 247, 272, 275, 278, 382, 406, 408, 432, 478, 535, 605–606, 645, 664–666, 672–675, 679–681, 688–690, 693, 697–700, 753, 842–844, 851, 881–882, 885–889, 898–907, 914, 920, 923–924, 928, 931
update_stats.py	106	2	98%	162, 172
src/otaclient/app/boot_control
__init__.py	4	0	100%
_common.py	248	112	54%	74–75, 96–98, 114–115, 135–136, 155–156, 175–176, 195–196, 218–220, 235–236, 260–266, 287, 295, 313, 321, 340–341, 344–345, 368, 370–379, 381–390, 392–394, 413, 416, 424, 432, 448–450, 452–457, 550, 555, 560, 673, 677–678, 681, 689, 691–692, 718–719, 721–724, 729, 735–736, 739–740, 742, 749–750, 761–767, 775–777, 781–782, 785–786, 789, 795
_grub.py	417	128	69%	216, 264–267, 273–277, 314–315, 322–327, 330–336, 339, 342–343, 348, 350–352, 361–367, 369–370, 372–374, 383–385, 387–389, 468–469, 473–474, 526, 532, 558, 580, 584–585, 600–602, 626–629, 641, 645–647, 649–651, 710–713, 738–741, 764–767, 779–780, 783–784, 819, 825, 845–846, 848, 860, 863, 866, 869, 873–875, 893–896, 924–927, 932–940, 945–953
_jetson_cboot.py	270	214	20%	69–70, 77–78, 96–105, 117, 124–125, 137, 143–144, 154–156, 168–169, 180–181, 184–185, 188–189, 192–196, 199–200, 204–205, 210–211, 213–217, 219–225, 227–228, 233, 236, 239–240, 243, 247–248, 252–253, 257, 260, 263, 267–273, 275–277, 282, 285, 288, 292, 299, 301–304, 317, 320, 324, 326–328, 332, 339, 341, 344, 350–351, 356, 364, 372–374, 383–384, 386–388, 394, 397–399, 403–404, 406, 409, 418–420, 423, 426, 429–434, 436–438, 441, 444, 448–453, 457–459, 464–465, 469–470, 473, 476, 479–480, 483, 486, 491, 494, 497–498, 500, 502, 505, 508, 510–511, 514–518, 523–524, 526, 534–538, 540, 543, 546, 557–558, 563, 573, 576–584, 589–597, 602–610, 616–618, 621, 624
_jetson_common.py	141	66	53%	50, 74, 129–134, 136, 141–143, 148–151, 159–160, 167–168, 173–174, 190–191, 193–195, 198–200, 203, 207, 211, 215–217, 223–224, 226, 259, 285–286, 288–290, 294–297, 299–300, 302–306, 308, 315–316, 319, 321, 331, 334–335, 338, 340
_rpi_boot.py	286	134	53%	54, 57, 121–122, 126, 134–137, 151–154, 161–162, 164–165, 170–171, 174–175, 184–185, 223, 229–233, 236, 254–256, 260–262, 267–269, 273–275, 285–286, 289, 292, 294–295, 297–298, 300–302, 308, 311–312, 322–325, 333–337, 339, 341–342, 347–348, 355–361, 392, 394–397, 407–410, 414–415, 417–421, 449–452, 471–474, 479, 482, 500–503, 508–516, 521–529, 546–549, 555–557, 560, 563
configs.py	38	0	100%
protocol.py	4	0	100%
selecter.py	38	26	31%	44–46, 49–50, 54–55, 58–60, 63, 65, 69, 77–79, 81–82, 84–85, 89, 91–93, 95, 97
src/otaclient/app/create_standby
__init__.py	12	5	58%	28–30, 32, 34
common.py	225	44	80%	63, 66–67, 71–73, 75, 79–80, 82, 128, 176–178, 180–182, 184, 187–190, 194, 205, 279–280, 282–287, 300, 337, 365, 368–370, 386–387, 401, 405, 427–428
interface.py	5	0	100%
rebuild_mode.py	99	9	90%	94–96, 108–113
src/otaclient/configs
_common.py	8	0	100%
ecu_info.py	57	1	98%	107
proxy_info.py	52	2	96%	88, 90
src/otaclient_api/v2
__init__.py	14	0	100%
api_caller.py	39	6	84%	45–47, 83–85
api_stub.py	17	0	100%
types.py	256	23	91%	86, 89–92, 131, 209–210, 212, 259, 262–263, 506–508, 512–513, 515, 518–519, 522–523, 586
src/otaclient_common
__init__.py	34	8	76%	42–44, 59, 61, 67, 74–75
common.py	154	19	87%	41, 45, 200, 203–205, 220, 227–229, 295–297, 307, 316–318, 364, 368
downloader.py	269	43	84%	72, 85–86, 301, 306, 328–329, 379–383, 402–404, 407–408, 411–412, 433–436, 440–441, 445–446, 450–451, 460, 535–537, 553, 573–575, 579, 581, 584, 589–591
linux.py	61	15	75%	51–53, 59, 69, 74, 76, 108–109, 133–134, 190, 195–196, 198
logging.py	29	1	96%	55
persist_file_handling.py	113	18	84%	112, 114, 146–148, 150, 176–179, 184, 188–192, 218–219
proto_streamer.py	42	8	80%	33, 48, 66–67, 72, 81–82, 100
proto_wrapper.py	398	45	88%	87, 165, 172, 184–186, 205, 210, 221, 257, 263, 268, 299, 303, 307, 402, 462, 469, 472, 492, 499, 501, 526, 532, 535, 537, 562, 568, 571, 573, 605, 609, 611, 625, 642, 669, 672, 676, 707, 713, 760–763, 765
retry_task_map.py	80	7	91%	164–165, 167, 179–182
typing.py	25	0	100%
TOTAL	6015	1408	76%

Tests	Skipped	Failures	Errors	Time
179	0 💤	0 ❌	0 🔥	4m 58s ⏱️

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
87.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud