perf tests: reset BB indices after every iteration

[weissi]

Motivation:

The performance tests are assumed to be run with the same number of iterations, so we didn't reset indices. That however means if one were to increase the iterations by a few orders of magnitude, the perf tests may crash.

Modifications:

Reset the indices every iteration

Result:

Robustness against more iterations.

[weissi]

@swift-nio-bot test perf please

[weissi]

@swift-nio-bot perf test please

[swift-server-bot]

performance report

build id: 150

timestamp: Tue Oct 10 14:12:24 UTC 2023

results

name	min	max	mean	std
write_http_headers	0.043014505	0.043306598	0.0431162306	0.00010731312498188622
http_headers_canonical_form	0.106946909	0.107892021	0.10728716170000001	0.0002833579147995799
http_headers_canonical_form_trimming_whitespace	0.020683159	0.021407913	0.020918072599999997	0.00022976412999093035
http_headers_canonical_form_trimming_whitespace_from_short_string	0.018709077	0.019375857	0.018959682999999998	0.00018499297449362746
http_headers_canonical_form_trimming_whitespace_from_long_string	0.030267107	0.030997091	0.0305039882	0.00025507197759882914
bytebuffer_write_12MB_short_string_literals	0.142563608	0.148590862	0.1436036586	0.0017861804612743517
bytebuffer_write_12MB_short_calculated_strings	0.070222919	0.0708125	0.0704589394	0.00022633425182493926
bytebuffer_write_12MB_medium_string_literals	0.953401421	0.974287472	0.9586733166	0.007822362441176942
bytebuffer_write_12MB_medium_calculated_strings	0.086686734	0.087563136	0.0869350517	0.0002955572711671793
bytebuffer_write_12MB_large_calculated_strings	0.165309369	0.167909327	0.1659703964	0.0007141817839321354
bytebuffer_lots_of_rw	0.043019494	0.051462025	0.0453577275	0.003153924892972573
bytebuffer_write_http_response_ascii_only_as_string	0.028281608	0.029285053	0.0284791386	0.00032904505071366025
bytebuffer_write_http_response_ascii_only_as_staticstring	0.028091182	0.02870517	0.028252014399999997	0.0002329989748228279
bytebuffer_write_http_response_some_nonascii_as_string	0.028558588	0.029064384	0.0286836496	0.0001572473054389871
bytebuffer_write_http_response_some_nonascii_as_staticstring	0.028063191	0.028592289	0.0281558235	0.00015624360285660052
no-net_http1_1k_reqs_1_conn	0.011476718	0.012029496	0.011621876	0.00015324460154421266
http1_1k_reqs_1_conn	0.059931217	0.0614986	0.0609082556	0.0005438390786627156
http1_1k_reqs_100_conns	0.089933409	0.090918834	0.09040405230000001	0.0004361625096449593
future_whenallsucceed_100k_immediately_succeeded_off_loop	0.078928743	0.080169795	0.0793072573	0.00040395791696524563
future_whenallsucceed_100k_immediately_succeeded_on_loop	0.078998407	0.085820761	0.0800438171	0.0020597018117677677
future_whenallsucceed_10k_deferred_off_loop	0.022875388	0.024793183	0.023162864499999998	0.0005914088917095909
future_whenallsucceed_10k_deferred_on_loop	0.014156877	0.014809926	0.014297279800000002	0.00023285527794881966
future_whenallcomplete_100k_immediately_succeeded_off_loop	0.039364529	0.039972442	0.0395825928	0.00025576178173396626
future_whenallcomplete_100k_immediately_succeeded_on_loop	0.039381215	0.040062964	0.039590436	0.00023442427562197745
future_whenallcomplete_10k_deferred_off_loop	0.015573206	0.016101516	0.0157025479	0.00020277311180093237
future_whenallcomplete_100k_deferred_on_loop	0.079627057	0.081879032	0.080198994	0.0006685182591727274
future_reduce_10k_futures	0.01686448	0.017072749	0.016938683399999998	6.779905982665107e-05
future_reduce_into_10k_futures	0.014373277	0.014546166	0.0144299328	6.0007709948713014e-05
channel_pipeline_1m_events	0.097162988	0.097371327	0.0972513708	7.700369765984128e-05
websocket_encode_50b_space_at_front_100k_frames_cow	0.050007398	0.050467892	0.050112158	0.00018718196526553183
websocket_encode_50b_space_at_front_1m_frames_cow_masking	0.659760665	0.663345773	0.6605193692	0.0011637800760828008
websocket_encode_1kb_space_at_front_1m_frames_cow	0.536436371	0.537337117	0.5369617467	0.00023800972323231393
websocket_encode_50b_no_space_at_front_100k_frames_cow	0.050282579	0.05073057	0.050418719300000005	0.00020042058912754794
websocket_encode_1kb_no_space_at_front_100k_frames_cow	0.052822284	0.053289319	0.0529830637	0.0001817920159798318
websocket_encode_50b_space_at_front_100k_frames	0.072336168	0.073171258	0.0727095796	0.0003022669362655764
websocket_encode_50b_space_at_front_10k_frames_masking	0.008694311	0.00881499	0.0087594933	4.4253874580023506e-05
websocket_encode_1kb_space_at_front_10k_frames	0.013595813	0.014064357	0.013674241	0.00014010449835906192
websocket_encode_50b_no_space_at_front_100k_frames	0.071745384	0.072547389	0.0721716852	0.00025465748126227524
websocket_encode_1kb_no_space_at_front_10k_frames	0.012977159	0.013390419	0.013076428300000002	0.00011998371862423688
websocket_decode_125b_10k_frames	0.012317883	0.012792525	0.012389185899999999	0.00014238367965110666
websocket_decode_125b_with_a_masking_key_10k_frames	0.012454907	0.012643234	0.012577122199999999	7.545323480408242e-05
websocket_decode_64kb_10k_frames	0.012485698	0.012966883	0.012560469800000001	0.00014407405133688098
websocket_decode_64kb_with_a_masking_key_10k_frames	0.012812225	0.013290839	0.012919691	0.000142901661896098
websocket_decode_64kb_+1_10k_frames	0.012482082	0.012550232	0.012517223499999999	2.1251053224879508e-05
websocket_decode_64kb_+1_with_a_masking_key_10k_frames	0.012813086	0.013127375	0.0128884979	0.0001034615646373624
circular_buffer_into_byte_buffer_1kb	0.033054848	0.033503522	0.0331557043	0.00017944955839582191
circular_buffer_into_byte_buffer_1mb	0.064676286	0.065181594	0.0648810856	0.00022449795557802186
byte_buffer_view_iterator_1mb	0.017559613	0.017627097	0.017581155100000002	2.0864181079064865e-05
byte_buffer_view_contains_12mb	0.052891645	0.053408531	0.053011115799999994	0.00019578022043483168
byte_to_message_decoder_decode_many_small	0.042069248	0.042601428	0.0422079188	0.0002091237393362003
generate_10k_random_request_keys	0.090362137	0.0907124	0.0904782	0.00011872865139655484
bytebuffer_rw_10_uint32s	0.041825292	0.042425175	0.0419722626	0.00022393559948302583
bytebuffer_multi_rw_10_uint32s	0.075377505	0.075975179	0.0756001154	0.00023707551808475565
lock_1_thread_10M_ops	0.154122763	0.154697059	0.1544698979	0.00017408331866312413
lock_2_threads_10M_ops	0.85417334	0.916802749	0.8836796321	0.016833714655382866
lock_4_threads_10M_ops	0.94391286	0.97990026	0.9618429609	0.0133136438629432
lock_8_threads_10M_ops	0.985340206	1.003434984	0.9957256407000001	0.00690479784006877
schedule_100k_tasks	0.061270598	0.10444348	0.07146500330000001	0.01313605798826484
schedule_and_run_100k_tasks	0.242553134	0.258454392	0.2462510563	0.005227635300873635
execute_100k_tasks	0.102127981	0.103984412	0.102658455	0.0005437932329671704
bytebufferview_copy_to_array_100k_times_1kb	0.011197019	0.011748476	0.011280436100000001	0.00016555787587859842
circularbuffer_copy_to_array_10k_times_1kb	0.019775152	0.020226344	0.0198293829	0.00013966140814321046
deadline_now_1M_times	0.024638332	0.024825877	0.0246844799	5.304881949770404e-05
asyncwriter_single_writes_1M_times	0.398845948	0.399136675	0.3989355288	8.07444075817815e-05
asyncsequenceproducer_consume_1M_times	0.842201723	0.842855327	0.8424539363999999	0.00024983011347633855
udp_10k_writes	0.378236088	0.379017871	0.37864265389999996	0.0002700605930035568
udp_10k_vector_writes	0.204004851	0.207070443	0.20469129170000003	0.0008697534437882759
udp_10k_vector_reads	0.387346315	0.388603096	0.3880963511	0.0003651079837693219
udp_10k_vector_reads_and_writes	0.108641037	0.109062026	0.1088557054	0.0001487834912453527
tcp_100k_messages_throughput	0.657714673	0.681243698	0.6643943569	0.007281486629238503

comparison

name	current	previous	winner	diff
write_http_headers	0.043014505	0.0430229	current	0%
http_headers_canonical_form	0.106946909	0.105630804	previous	1%
http_headers_canonical_form_trimming_whitespace	0.020683159	0.020651751	previous	0%
http_headers_canonical_form_trimming_whitespace_from_short_string	0.018709077	0.018705421	previous	0%
http_headers_canonical_form_trimming_whitespace_from_long_string	0.030267107	0.030118695	previous	0%
bytebuffer_write_12MB_short_string_literals	0.142563608	0.142338887	previous	0%
bytebuffer_write_12MB_short_calculated_strings	0.070222919	0.070262673	current	0%
bytebuffer_write_12MB_medium_string_literals	0.953401421	0.958581625	current	0%
bytebuffer_write_12MB_medium_calculated_strings	0.086686734	0.08320151	previous	4%
bytebuffer_write_12MB_large_calculated_strings	0.165309369	0.165032179	previous	0%
bytebuffer_lots_of_rw	0.043019494	0.043252504	current	0%
bytebuffer_write_http_response_ascii_only_as_string	0.028281608	0.02892041	current	-2%
bytebuffer_write_http_response_ascii_only_as_staticstring	0.028091182	0.028383653	current	-1%
bytebuffer_write_http_response_some_nonascii_as_string	0.028558588	0.02904993	current	-1%
bytebuffer_write_http_response_some_nonascii_as_staticstring	0.028063191	0.028065086	current	0%
no-net_http1_1k_reqs_1_conn	0.011476718	0.011712992	current	-2%
http1_1k_reqs_1_conn	0.059931217	0.059793763	previous	0%
http1_1k_reqs_100_conns	0.089933409	0.090073182	current	0%
future_whenallsucceed_100k_immediately_succeeded_off_loop	0.078928743	0.078844476	previous	0%
future_whenallsucceed_100k_immediately_succeeded_on_loop	0.078998407	0.079507845	current	0%
future_whenallsucceed_10k_deferred_off_loop	0.022875388	0.023129703	current	-1%
future_whenallsucceed_10k_deferred_on_loop	0.014156877	0.014335301	current	-1%
future_whenallcomplete_100k_immediately_succeeded_off_loop	0.039364529	0.039785627	current	-1%
future_whenallcomplete_100k_immediately_succeeded_on_loop	0.039381215	0.039852453	current	-1%
future_whenallcomplete_10k_deferred_off_loop	0.015573206	0.015583899	current	0%
future_whenallcomplete_100k_deferred_on_loop	0.079627057	0.079878193	current	0%
future_reduce_10k_futures	0.01686448	0.01674664	previous	0%
future_reduce_into_10k_futures	0.014373277	0.014551334	current	-1%
channel_pipeline_1m_events	0.097162988	0.097154256	previous	0%
websocket_encode_50b_space_at_front_100k_frames_cow	0.050007398	0.049820596	previous	0%
websocket_encode_50b_space_at_front_1m_frames_cow_masking	0.659760665	0.658743996	previous	0%
websocket_encode_1kb_space_at_front_1m_frames_cow	0.536436371	0.526375362	previous	1%
websocket_encode_50b_no_space_at_front_100k_frames_cow	0.050282579	0.049825271	previous	0%
websocket_encode_1kb_no_space_at_front_100k_frames_cow	0.052822284	0.052533091	previous	0%
websocket_encode_50b_space_at_front_100k_frames	0.072336168	0.073907377	current	-2%
websocket_encode_50b_space_at_front_10k_frames_masking	0.008694311	0.008906724	current	-2%
websocket_encode_1kb_space_at_front_10k_frames	0.013595813	0.013767831	current	-1%
websocket_encode_50b_no_space_at_front_100k_frames	0.071745384	0.074102811	current	-3%
websocket_encode_1kb_no_space_at_front_10k_frames	0.012977159	0.013247764	current	-2%
websocket_decode_125b_10k_frames	0.012317883	0.012193333	previous	1%
websocket_decode_125b_with_a_masking_key_10k_frames	0.012454907	0.012581377	current	-1%
websocket_decode_64kb_10k_frames	0.012485698	0.012626693	current	-1%
websocket_decode_64kb_with_a_masking_key_10k_frames	0.012812225	0.012901974	current	0%
websocket_decode_64kb_+1_10k_frames	0.012482082	0.01260745	current	0%
websocket_decode_64kb_+1_with_a_masking_key_10k_frames	0.012813086	0.012950214	current	-1%
circular_buffer_into_byte_buffer_1kb	0.033054848	0.0330283	previous	0%
circular_buffer_into_byte_buffer_1mb	0.064676286	0.064673017	previous	0%
byte_buffer_view_iterator_1mb	0.017559613	0.017560121	current	0%
byte_buffer_view_contains_12mb	0.052891645	0.052874245	previous	0%
byte_to_message_decoder_decode_many_small	0.042069248	0.041733564	previous	0%
generate_10k_random_request_keys	0.090362137	0.090236081	previous	0%
bytebuffer_rw_10_uint32s	0.041825292	0.038879033	previous	7%
bytebuffer_multi_rw_10_uint32s	0.075377505	0.058782413	previous	28%
lock_1_thread_10M_ops	0.154122763	0.151366825	previous	1%
lock_2_threads_10M_ops	0.85417334	0.653117854	previous	30%
lock_4_threads_10M_ops	0.94391286	0.827757617	previous	14%
lock_8_threads_10M_ops	0.985340206	0.883223176	previous	11%
schedule_100k_tasks	0.061270598	0.061135269	previous	0%
schedule_and_run_100k_tasks	0.242553134	0.232383799	previous	4%
execute_100k_tasks	0.102127981	0.100105199	previous	2%
bytebufferview_copy_to_array_100k_times_1kb	0.011197019	0.011069273	previous	1%
circularbuffer_copy_to_array_10k_times_1kb	0.019775152	0.019764947	previous	0%
deadline_now_1M_times	0.024638332	0.024566638	previous	0%
asyncwriter_single_writes_1M_times	0.398845948	0.420522042	current	-5%
asyncsequenceproducer_consume_1M_times	0.842201723	0.81599794	previous	3%
udp_10k_writes	0.378236088	0.373549021	previous	1%
udp_10k_vector_writes	0.204004851	0.204568654	current	0%
udp_10k_vector_reads	0.387346315	0.382309959	previous	1%
udp_10k_vector_reads_and_writes	0.108641037	0.107802509	previous	0%
tcp_100k_messages_throughput	0.657714673	0.641782205	previous	2%

significant differences found

[weissi]

what's going in here in the lock benchmarks?

[Lukasa]

Unclear: there's no obvious different in the logs.