Skip to content

[Improve](audit) add load info for active tasks and audit log #50802

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

qzsee
Copy link
Contributor

@qzsee qzsee commented May 12, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Table audit_log or backend_active_tasks lack load info for Insert or Stream load

add load_bytes and load_rows

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented May 12, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

morningman
morningman previously approved these changes May 14, 2025
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 14, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@qzsee
Copy link
Contributor Author

qzsee commented May 14, 2025

run buildall

1 similar comment
@dqz123
Copy link

dqz123 commented May 16, 2025

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.31% (1113/1336)
Line Coverage 66.21% (18662/28187)
Region Coverage 65.75% (9255/14076)
Branch Coverage 55.57% (4981/8964)

@doris-robot
Copy link

TPC-H: Total hot run time: 34721 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fc85a6a1be4cf65f4b1d7ae6f38eae12a3f6f4d0, data reload: false

------ Round 1 ----------------------------------
q1	26807	5243	5231	5231
q2	2083	302	208	208
q3	10583	1376	722	722
q4	10246	1077	525	525
q5	7910	2560	2566	2560
q6	204	171	130	130
q7	1002	750	609	609
q8	9313	1514	1306	1306
q9	6856	5203	5295	5203
q10	6931	2367	1894	1894
q11	526	304	278	278
q12	360	393	213	213
q13	17808	3784	3055	3055
q14	251	231	213	213
q15	566	485	486	485
q16	433	441	373	373
q17	619	935	380	380
q18	7697	7335	7153	7153
q19	1748	1106	608	608
q20	348	351	231	231
q21	4255	3282	2388	2388
q22	1082	999	956	956
Total cold run time: 117628 ms
Total hot run time: 34721 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5387	5219	5392	5219
q2	249	342	228	228
q3	2243	2744	2368	2368
q4	1533	1960	1449	1449
q5	4588	4360	4413	4360
q6	256	181	126	126
q7	2059	1920	1771	1771
q8	2847	2860	2720	2720
q9	7145	7324	7140	7140
q10	3090	3229	2781	2781
q11	617	520	494	494
q12	713	811	649	649
q13	3607	3979	3238	3238
q14	279	293	270	270
q15	531	476	477	476
q16	460	508	439	439
q17	1231	1737	1436	1436
q18	7907	7617	7492	7492
q19	885	846	891	846
q20	1966	2020	1824	1824
q21	4962	4512	4395	4395
q22	1074	1059	980	980
Total cold run time: 53629 ms
Total hot run time: 50701 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187461 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fc85a6a1be4cf65f4b1d7ae6f38eae12a3f6f4d0, data reload: false

query1	994	484	512	484
query2	6575	1934	1914	1914
query3	6756	224	221	221
query4	26773	23814	23582	23582
query5	4311	632	467	467
query6	299	211	205	205
query7	4631	506	292	292
query8	291	265	262	262
query9	8624	2662	2631	2631
query10	481	328	287	287
query11	15637	15120	15378	15120
query12	172	111	110	110
query13	1664	544	423	423
query14	9711	6170	6073	6073
query15	212	198	172	172
query16	7563	653	486	486
query17	1208	728	587	587
query18	2058	407	315	315
query19	202	194	170	170
query20	124	118	118	118
query21	218	130	109	109
query22	4012	4095	3934	3934
query23	34164	33551	33255	33255
query24	8477	2426	2409	2409
query25	552	464	399	399
query26	1248	267	157	157
query27	2753	512	348	348
query28	4274	2138	2118	2118
query29	709	643	450	450
query30	277	214	184	184
query31	965	862	783	783
query32	75	64	64	64
query33	553	391	316	316
query34	801	844	530	530
query35	772	839	733	733
query36	953	997	883	883
query37	107	117	72	72
query38	4199	4150	4205	4150
query39	1486	1422	1407	1407
query40	214	120	107	107
query41	61	57	57	57
query42	131	108	104	104
query43	519	535	486	486
query44	1366	839	833	833
query45	179	180	170	170
query46	845	1033	649	649
query47	1758	1783	1700	1700
query48	419	421	317	317
query49	805	528	428	428
query50	640	691	413	413
query51	4180	4237	4082	4082
query52	107	111	99	99
query53	227	264	187	187
query54	584	569	499	499
query55	87	83	84	83
query56	310	302	325	302
query57	1194	1152	1080	1080
query58	266	255	256	255
query59	2681	2859	2673	2673
query60	329	320	325	320
query61	133	122	125	122
query62	806	735	667	667
query63	232	190	194	190
query64	4291	1024	685	685
query65	4368	4243	4280	4243
query66	1075	420	328	328
query67	15646	15704	15237	15237
query68	8677	890	528	528
query69	474	316	267	267
query70	1201	1133	1137	1133
query71	469	330	320	320
query72	5267	4760	4854	4760
query73	739	659	363	363
query74	9015	9086	8933	8933
query75	4035	3226	2862	2862
query76	3627	1206	788	788
query77	779	378	294	294
query78	10128	10153	9363	9363
query79	5287	810	560	560
query80	702	508	474	474
query81	471	254	220	220
query82	406	128	98	98
query83	295	259	241	241
query84	295	106	86	86
query85	752	349	381	349
query86	332	297	275	275
query87	4480	4467	4404	4404
query88	3004	2348	2337	2337
query89	444	323	288	288
query90	2082	219	221	219
query91	144	147	123	123
query92	81	65	59	59
query93	3078	938	571	571
query94	661	413	311	311
query95	382	288	288	288
query96	516	575	298	298
query97	2755	2736	2626	2626
query98	248	215	201	201
query99	1416	1380	1286	1286
Total cold run time: 280576 ms
Total hot run time: 187461 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fc85a6a1be4cf65f4b1d7ae6f38eae12a3f6f4d0, data reload: false

query1	0.03	0.03	0.03
query2	0.12	0.10	0.10
query3	0.24	0.20	0.20
query4	1.59	0.20	0.21
query5	0.43	0.42	0.40
query6	1.16	0.66	0.66
query7	0.03	0.01	0.01
query8	0.04	0.03	0.03
query9	0.59	0.51	0.52
query10	0.57	0.58	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.59
query14	0.78	0.80	0.82
query15	0.87	0.86	0.86
query16	0.38	0.39	0.39
query17	1.08	1.05	1.01
query18	0.23	0.22	0.21
query19	1.89	1.88	1.75
query20	0.01	0.02	0.01
query21	15.40	0.87	0.54
query22	0.76	1.12	0.62
query23	15.05	1.41	0.63
query24	6.69	2.07	0.98
query25	0.49	0.17	0.16
query26	0.69	0.17	0.15
query27	0.04	0.05	0.04
query28	9.44	0.86	0.46
query29	12.57	3.95	3.33
query30	0.25	0.09	0.06
query31	2.84	0.61	0.40
query32	3.22	0.55	0.47
query33	3.04	3.03	3.09
query34	15.76	5.16	4.50
query35	4.55	4.51	4.56
query36	0.65	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.14
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 102.88 s
Total hot run time: 29.23 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 9.52% (2/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.86% (14883/26645)
Line Coverage 44.65% (131884/295376)
Region Coverage 43.76% (66404/151737)
Branch Coverage 38.36% (34021/88690)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (21/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.37% (20820/26232)
Line Coverage 72.57% (214366/295387)
Region Coverage 70.74% (126127/178286)
Branch Coverage 64.53% (65381/101318)

@dqz123
Copy link

dqz123 commented May 29, 2025

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 83.26% (1114/1338)
Line Coverage 66.47% (18802/28287)
Region Coverage 66.14% (9325/14098)
Branch Coverage 56.03% (5029/8976)

@doris-robot
Copy link

TPC-H: Total hot run time: 33840 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 65675f6876dff97b8474a486c2ee532fb6a8ecb3, data reload: false

------ Round 1 ----------------------------------
q1	26160	5081	5054	5054
q2	1954	272	175	175
q3	10396	1233	664	664
q4	10246	991	541	541
q5	7650	2427	2374	2374
q6	191	168	135	135
q7	897	757	605	605
q8	9295	1290	1089	1089
q9	6777	5098	5118	5098
q10	6843	2322	1887	1887
q11	506	294	269	269
q12	346	355	217	217
q13	17766	3721	3139	3139
q14	229	230	219	219
q15	561	484	484	484
q16	439	440	377	377
q17	572	869	358	358
q18	7654	7286	7057	7057
q19	1793	969	554	554
q20	322	334	224	224
q21	3719	3173	2337	2337
q22	1028	1018	983	983
Total cold run time: 115344 ms
Total hot run time: 33840 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5187	5115	5131	5115
q2	243	325	220	220
q3	2174	2663	2292	2292
q4	1344	1764	1370	1370
q5	4562	4461	4411	4411
q6	222	168	124	124
q7	1934	1911	1721	1721
q8	2639	2498	2504	2498
q9	7263	7167	7145	7145
q10	2985	3216	2771	2771
q11	600	509	500	500
q12	658	746	633	633
q13	3492	3884	3269	3269
q14	277	288	293	288
q15	539	489	484	484
q16	439	476	449	449
q17	1154	1555	1412	1412
q18	7752	7397	7368	7368
q19	801	772	884	772
q20	2100	2071	1876	1876
q21	4951	4381	4301	4301
q22	1108	1088	994	994
Total cold run time: 52424 ms
Total hot run time: 50013 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185554 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 65675f6876dff97b8474a486c2ee532fb6a8ecb3, data reload: false

query1	1038	498	494	494
query2	6575	1846	1842	1842
query3	6744	227	222	222
query4	26377	23633	23358	23358
query5	4325	610	465	465
query6	312	216	223	216
query7	4624	480	286	286
query8	262	211	201	201
query9	8600	2662	2662	2662
query10	448	334	267	267
query11	15565	15221	14839	14839
query12	155	104	101	101
query13	1659	504	396	396
query14	8759	6125	6136	6125
query15	217	191	165	165
query16	7128	638	480	480
query17	1184	704	546	546
query18	1951	392	286	286
query19	188	183	156	156
query20	120	120	118	118
query21	215	119	106	106
query22	3986	4052	3908	3908
query23	34039	33071	33243	33071
query24	8369	2360	2358	2358
query25	540	451	379	379
query26	1238	262	150	150
query27	2778	505	341	341
query28	4337	2140	2128	2128
query29	766	542	434	434
query30	280	220	188	188
query31	927	840	768	768
query32	72	64	64	64
query33	572	369	319	319
query34	796	887	514	514
query35	793	798	738	738
query36	961	995	887	887
query37	116	101	80	80
query38	4195	4115	3981	3981
query39	1500	1423	1485	1423
query40	220	124	110	110
query41	71	65	66	65
query42	128	113	110	110
query43	499	505	479	479
query44	1289	815	820	815
query45	191	182	177	177
query46	816	1008	619	619
query47	1719	1793	1720	1720
query48	381	414	312	312
query49	755	505	390	390
query50	629	679	419	419
query51	4080	4168	4104	4104
query52	116	106	98	98
query53	222	245	187	187
query54	573	571	516	516
query55	85	82	83	82
query56	327	299	294	294
query57	1125	1158	1059	1059
query58	271	252	248	248
query59	2593	2647	2654	2647
query60	320	321	296	296
query61	124	124	124	124
query62	799	733	675	675
query63	220	188	189	188
query64	4385	1005	660	660
query65	4239	4171	4127	4127
query66	1177	409	339	339
query67	15895	15481	15145	15145
query68	7807	867	509	509
query69	475	320	276	276
query70	1229	1145	1087	1087
query71	465	318	290	290
query72	5596	4783	4861	4783
query73	701	606	349	349
query74	9100	9106	8860	8860
query75	3758	3186	2665	2665
query76	3623	1183	747	747
query77	785	378	310	310
query78	9866	10223	9419	9419
query79	2011	800	595	595
query80	599	531	449	449
query81	481	258	222	222
query82	422	128	104	104
query83	258	349	233	233
query84	251	118	90	90
query85	810	350	315	315
query86	341	311	281	281
query87	4352	4512	4249	4249
query88	3678	2268	2266	2266
query89	382	319	288	288
query90	1928	220	216	216
query91	140	148	127	127
query92	78	65	68	65
query93	1289	957	571	571
query94	675	422	311	311
query95	378	298	291	291
query96	491	572	285	285
query97	2736	2737	2676	2676
query98	244	214	212	212
query99	1652	1408	1346	1346
Total cold run time: 272561 ms
Total hot run time: 185554 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 65675f6876dff97b8474a486c2ee532fb6a8ecb3, data reload: false

query1	0.04	0.04	0.03
query2	0.14	0.10	0.12
query3	0.25	0.19	0.20
query4	1.60	0.19	0.12
query5	0.44	0.41	0.42
query6	1.17	0.67	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.59	0.52	0.54
query10	0.56	0.58	0.58
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.63	0.61	0.60
query14	0.80	0.84	0.79
query15	0.90	0.85	0.87
query16	0.37	0.39	0.39
query17	1.02	1.03	1.03
query18	0.23	0.21	0.22
query19	1.96	1.77	1.89
query20	0.02	0.01	0.01
query21	15.42	0.92	0.56
query22	0.77	1.27	0.69
query23	14.82	1.39	0.62
query24	7.10	1.53	0.93
query25	0.51	0.24	0.14
query26	0.61	0.17	0.15
query27	0.05	0.05	0.05
query28	9.61	0.89	0.46
query29	12.55	4.05	3.31
query30	0.26	0.11	0.07
query31	2.82	0.62	0.41
query32	3.24	0.57	0.47
query33	3.05	3.10	3.09
query34	15.76	5.11	4.56
query35	4.52	4.59	4.54
query36	0.68	0.50	0.49
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.35 s
Total hot run time: 29.39 s

@morningman morningman self-assigned this May 30, 2025
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants