Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Implement fallback to smaller vector size for swizzle_dyn #433

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Implement fallback to smaller vector size for swizzle_dyn #433

wants to merge 1 commit into from

Conversation

cvijdea-bd
Copy link
Contributor

@cvijdea-bd cvijdea-bd commented Aug 25, 2024 •
edited
Loading

This PR adds a fallback implementation so that e.g. u8x64::swizzle_dyn can be reasonably efficient even when only compiled with 128-bit SSSE.

A "downgraded" swizzle_dyn op on N lanes emits 4 swizzle_dyn ops on N/2 lanes. If the optimizer can deduce that index values are bounded to N/2 or less, then it will generally be more efficient.
For example, u8x64::swizzle_dyn will only emit 4 pshufb instructions on SSSE, instead of 16 in the general case, if the optimizer can prove index values are always <16 (this is generally achieved by preceding the swizzle with a 0xf mask).

Additionaly, for non-power-of-two N values, this PR adds a fallback implementation which zero-extends to the next power of two size.

Benchmarks

Below are benchmark results for the following code, on 5 target-cpu levels:

    b.bytes = (lookups.len() * std::mem::size_of::<Simd<u8, N>>()) as u64;
    b.iter(|| {
        for (lookup, index) in black_box(&lookups).iter().zip(black_box(&indexes).iter()) {
            let lookup = *lookup;
            let index = *index & Simd::splat(MASK);
            black_box(Simd::<u8, N>::swizzle_dyn(lookup, index));
        }
    });
  • x86-64 - baseline, no vectorized shuffles
  • x86-64-v2 - ssse, adds pshufb on u8x16
  • x86-64-v3 - avx2, adds vpshufb on u8x32 (vpshufb is not a true extension of pshufb to 256-bit, instead it's more like 2 pshufb ops ran in parallel, with 4-bit indices in the corresponding lane)
  • x86-64-v4 - avx512, adds vpshufb on u8x64 (again really just 4x pshufb in parallel)
  • icelake-server - avx512vbmi, adds vpermb (this is a true 256/512-bit shuffle with 5/6-bit indices)

N.B. the code used to benchmark includes #431, and removes the src/masks/bitmask.rs avx512f mask implementation to work around the problem discussed here.

The main thing to look for is the performance of swizzle_dyn_64* on non-vbmi targets (v2, v3, v4), and the performance of swizzle_dyn_32* on x86-64-v2. With the previous implementation, sizes without native vector instructions fall back to the scalar implementation, while in the PR version they fall back to a lower vector size for ~4-10x performance over the scalar version.

Benchmark data for old version

-Ctarget-cpu=x86-64

test bench_swizzle_dyn_16        ... bench:   1,758,136.93 ns/iter (+/- 620,537.28) = 1192 MB/s
test bench_swizzle_dyn_16_mask16 ... bench:     884,830.30 ns/iter (+/- 26,690.78) = 2370 MB/s
test bench_swizzle_dyn_32        ... bench:   2,852,244.70 ns/iter (+/- 981,864.91) = 735 MB/s
test bench_swizzle_dyn_32_mask16 ... bench:   1,218,294.30 ns/iter (+/- 15,292.08) = 1721 MB/s
test bench_swizzle_dyn_32_mask32 ... bench:   1,218,707.50 ns/iter (+/- 13,197.90) = 1720 MB/s
test bench_swizzle_dyn_64        ... bench:   5,415,598.00 ns/iter (+/- 173,006.27) = 387 MB/s
test bench_swizzle_dyn_64_mask16 ... bench:   1,274,801.70 ns/iter (+/- 21,257.84) = 1645 MB/s
test bench_swizzle_dyn_64_mask32 ... bench:     818,246.50 ns/iter (+/- 31,324.95) = 2562 MB/s
test bench_swizzle_dyn_64_mask64 ... bench:     825,826.30 ns/iter (+/- 42,892.83) = 2539 MB/s

test bench_swizzle_dyn_24        ... bench:   2,546,501.90 ns/iter (+/- 78,041.70) = 823 MB/s
test bench_swizzle_dyn_24_mask16 ... bench:   1,132,914.90 ns/iter (+/- 33,921.66) = 1851 MB/s
test bench_swizzle_dyn_43        ... bench:   5,561,422.50 ns/iter (+/- 1,310,875.16) = 377 MB/s
test bench_swizzle_dyn_43_mask16 ... bench:     888,501.18 ns/iter (+/- 36,721.91) = 2360 MB/s
test bench_swizzle_dyn_5_mask_8  ... bench:   7,800,009.70 ns/iter (+/- 101,916.37) = 268 MB/s

-Ctarget-cpu=x86-64-v2

test bench_swizzle_dyn_16        ... bench:     129,001.85 ns/iter (+/- 2,045.71) = 16256 MB/s
test bench_swizzle_dyn_16_mask16 ... bench:     126,409.63 ns/iter (+/- 1,106.15) = 16590 MB/s
test bench_swizzle_dyn_32        ... bench:   2,648,847.38 ns/iter (+/- 72,835.19) = 791 MB/s
test bench_swizzle_dyn_32_mask16 ... bench:     638,609.40 ns/iter (+/- 1,205.20) = 3283 MB/s
test bench_swizzle_dyn_32_mask32 ... bench:     638,810.50 ns/iter (+/- 1,368.94) = 3282 MB/s
test bench_swizzle_dyn_64        ... bench:   5,125,435.65 ns/iter (+/- 332,486.48) = 409 MB/s
test bench_swizzle_dyn_64_mask16 ... bench:     817,608.81 ns/iter (+/- 41,425.93) = 2564 MB/s
test bench_swizzle_dyn_64_mask32 ... bench:     814,272.70 ns/iter (+/- 39,830.58) = 2575 MB/s
test bench_swizzle_dyn_64_mask64 ... bench:     817,215.23 ns/iter (+/- 30,396.90) = 2566 MB/s

test bench_swizzle_dyn_24        ... bench:   2,402,568.90 ns/iter (+/- 61,893.97) = 872 MB/s
test bench_swizzle_dyn_24_mask16 ... bench:     610,669.90 ns/iter (+/- 18,419.01) = 3434 MB/s
test bench_swizzle_dyn_43        ... bench:   4,149,922.60 ns/iter (+/- 162,629.27) = 505 MB/s
test bench_swizzle_dyn_43_mask16 ... bench:     868,863.50 ns/iter (+/- 27,605.20) = 2413 MB/s
test bench_swizzle_dyn_5_mask_8  ... bench:   7,755,508.70 ns/iter (+/- 164,755.79) = 270 MB/s

-Ctarget-cpu=x86-64-v3

test bench_swizzle_dyn_16        ... bench:     141,146.06 ns/iter (+/- 207.07) = 14858 MB/s
test bench_swizzle_dyn_16_mask16 ... bench:     138,518.46 ns/iter (+/- 191.52) = 15139 MB/s
test bench_swizzle_dyn_32        ... bench:     121,160.02 ns/iter (+/- 162.73) = 17308 MB/s
test bench_swizzle_dyn_32_mask16 ... bench:     118,747.46 ns/iter (+/- 420.25) = 17660 MB/s
test bench_swizzle_dyn_32_mask32 ... bench:     120,251.52 ns/iter (+/- 186.12) = 17439 MB/s
test bench_swizzle_dyn_64_mask16 ... bench:     734,519.20 ns/iter (+/- 26,480.60) = 2855 MB/s
test bench_swizzle_dyn_64_mask32 ... bench:     755,231.40 ns/iter (+/- 25,220.19) = 2776 MB/s
test bench_swizzle_dyn_64_mask64 ... bench:     756,587.60 ns/iter (+/- 9,162.56) = 2771 MB/s

test bench_swizzle_dyn_24        ... bench:   2,525,430.65 ns/iter (+/- 80,467.50) = 830 MB/s
test bench_swizzle_dyn_24_mask16 ... bench:     618,569.60 ns/iter (+/- 15,202.54) = 3390 MB/s
test bench_swizzle_dyn_43        ... bench:   3,806,743.75 ns/iter (+/- 22,935.83) = 550 MB/s
test bench_swizzle_dyn_43_mask16 ... bench:     885,843.70 ns/iter (+/- 656.71) = 2367 MB/s
test bench_swizzle_dyn_5_mask_8  ... bench:   7,792,672.30 ns/iter (+/- 702,420.66) = 269 MB/s
test bench_swizzle_dyn_64        ... bench:   4,981,829.60 ns/iter (+/- 169,819.79) = 420 MB/s

-Ctarget-cpu=x86-64-v4

test bench_swizzle_dyn_16        ... bench:     144,257.45 ns/iter (+/- 39,790.03) = 14537 MB/s
test bench_swizzle_dyn_16_mask16 ... bench:     143,098.64 ns/iter (+/- 305.36) = 14655 MB/s
test bench_swizzle_dyn_32        ... bench:     140,321.57 ns/iter (+/- 14,479.64) = 14945 MB/s
test bench_swizzle_dyn_32_mask16 ... bench:     122,695.31 ns/iter (+/- 169.55) = 17092 MB/s
test bench_swizzle_dyn_32_mask32 ... bench:     125,273.56 ns/iter (+/- 212.20) = 16740 MB/s
test bench_swizzle_dyn_64        ... bench:   5,133,064.60 ns/iter (+/- 71,121.29) = 408 MB/s
test bench_swizzle_dyn_64_mask16 ... bench:     778,564.40 ns/iter (+/- 3,198.50) = 2693 MB/s
test bench_swizzle_dyn_64_mask32 ... bench:     778,808.50 ns/iter (+/- 3,380.96) = 2692 MB/s
test bench_swizzle_dyn_64_mask64 ... bench:     842,318.40 ns/iter (+/- 34,180.06) = 2489 MB/s

test bench_swizzle_dyn_24        ... bench:   3,409,837.20 ns/iter (+/- 5,918.96) = 615 MB/s
test bench_swizzle_dyn_24_mask16 ... bench:     831,782.30 ns/iter (+/- 1,396.30) = 2521 MB/s
test bench_swizzle_dyn_43        ... bench:   3,845,018.10 ns/iter (+/- 62,559.08) = 545 MB/s
test bench_swizzle_dyn_43_mask16 ... bench:     925,849.20 ns/iter (+/- 1,840.68) = 2265 MB/s
test bench_swizzle_dyn_5_mask_8  ... bench:   7,792,497.60 ns/iter (+/- 169,610.01) = 269 MB/s

-Ctarget-cpu=icelake-server (avx512vbmi)

test bench_swizzle_dyn_16        ... bench:     141,509.67 ns/iter (+/- 431.88) = 14819 MB/s
test bench_swizzle_dyn_16_mask16 ... bench:     140,317.70 ns/iter (+/- 291.18) = 14945 MB/s
test bench_swizzle_dyn_32        ... bench:     120,201.04 ns/iter (+/- 127.38) = 17447 MB/s
test bench_swizzle_dyn_32_mask16 ... bench:     120,402.70 ns/iter (+/- 1,684.98) = 17417 MB/s
test bench_swizzle_dyn_32_mask32 ... bench:     120,711.00 ns/iter (+/- 1,595.57) = 17373 MB/s
test bench_swizzle_dyn_64        ... bench:     127,307.53 ns/iter (+/- 1,356.22) = 16473 MB/s
test bench_swizzle_dyn_64_mask16 ... bench:     127,292.10 ns/iter (+/- 811.17) = 16475 MB/s
test bench_swizzle_dyn_64_mask32 ... bench:     127,435.99 ns/iter (+/- 1,624.41) = 16456 MB/s
test bench_swizzle_dyn_64_mask64 ... bench:     127,359.49 ns/iter (+/- 1,162.59) = 16466 MB/s

test bench_swizzle_dyn_24        ... bench:   3,403,487.80 ns/iter (+/- 20,527.12) = 616 MB/s
test bench_swizzle_dyn_24_mask16 ... bench:     833,268.80 ns/iter (+/- 8,846.27) = 2516 MB/s
test bench_swizzle_dyn_43        ... bench:   3,989,622.70 ns/iter (+/- 136,142.64) = 525 MB/s
test bench_swizzle_dyn_43_mask16 ... bench:     925,779.40 ns/iter (+/- 6,602.74) = 2265 MB/s
test bench_swizzle_dyn_5_mask_8  ... bench:   9,831,859.25 ns/iter (+/- 2,805,111.90) = 213 MB/s
Benchmark data for new version

-Ctarget-cpu=x86-64

test bench_swizzle_dyn_16        ... bench:   1,757,825.75 ns/iter (+/- 55,621.98) = 1193 MB/s
test bench_swizzle_dyn_16_mask16 ... bench:     885,512.05 ns/iter (+/- 25,654.57) = 2368 MB/s
test bench_swizzle_dyn_32        ... bench:   2,805,398.55 ns/iter (+/- 66,245.41) = 747 MB/s
test bench_swizzle_dyn_32_mask16 ... bench:     904,389.00 ns/iter (+/- 13,162.55) = 2318 MB/s
test bench_swizzle_dyn_32_mask32 ... bench:     904,787.30 ns/iter (+/- 25,749.03) = 2317 MB/s
test bench_swizzle_dyn_64        ... bench:   5,401,521.75 ns/iter (+/- 165,573.85) = 388 MB/s
test bench_swizzle_dyn_64_mask16 ... bench:   1,274,823.30 ns/iter (+/- 38,571.44) = 1645 MB/s
test bench_swizzle_dyn_64_mask32 ... bench:     812,329.91 ns/iter (+/- 34,655.58) = 2581 MB/s
test bench_swizzle_dyn_64_mask64 ... bench:     820,164.60 ns/iter (+/- 38,105.78) = 2556 MB/s

test bench_swizzle_dyn_24        ... bench:   2,520,968.30 ns/iter (+/- 101,413.91) = 831 MB/s
test bench_swizzle_dyn_24_mask16 ... bench:   1,133,691.24 ns/iter (+/- 43,145.54) = 1849 MB/s
test bench_swizzle_dyn_43        ... bench:   4,126,429.50 ns/iter (+/- 141,499.13) = 508 MB/s
test bench_swizzle_dyn_43_mask16 ... bench:     886,000.79 ns/iter (+/- 26,400.43) = 2366 MB/s
test bench_swizzle_dyn_5_mask_8  ... bench:   7,799,985.50 ns/iter (+/- 327,156.35) = 268 MB/s

-Ctarget-cpu=x86-64-v2

test bench_swizzle_dyn_16        ... bench:     142,425.58 ns/iter (+/- 2,086.47) = 14724 MB/s
test bench_swizzle_dyn_16_mask16 ... bench:     137,679.62 ns/iter (+/- 2,961.24) = 15232 MB/s
test bench_swizzle_dyn_32        ... bench:     180,386.56 ns/iter (+/- 40,374.21) = 11625 MB/s
test bench_swizzle_dyn_32_mask16 ... bench:     117,059.84 ns/iter (+/- 263.47) = 17915 MB/s
test bench_swizzle_dyn_32_mask32 ... bench:     122,496.84 ns/iter (+/- 5,551.35) = 17120 MB/s
test bench_swizzle_dyn_64        ... bench:     255,301.92 ns/iter (+/- 6,439.50) = 8214 MB/s
test bench_swizzle_dyn_64_mask16 ... bench:     118,240.99 ns/iter (+/- 3,802.15) = 17736 MB/s
test bench_swizzle_dyn_64_mask32 ... bench:     122,202.14 ns/iter (+/- 6,491.88) = 17161 MB/s
test bench_swizzle_dyn_64_mask64 ... bench:     208,805.05 ns/iter (+/- 7,945.68) = 10043 MB/s

test bench_swizzle_dyn_24        ... bench:     237,441.03 ns/iter (+/- 26,824.54) = 8832 MB/s
test bench_swizzle_dyn_24_mask16 ... bench:     134,570.62 ns/iter (+/- 3,092.15) = 15584 MB/s
test bench_swizzle_dyn_43        ... bench:     503,438.20 ns/iter (+/- 919.14) = 4165 MB/s
test bench_swizzle_dyn_43_mask16 ... bench:     120,647.25 ns/iter (+/- 329.87) = 17382 MB/s
test bench_swizzle_dyn_5_mask_8  ... bench:     352,197.78 ns/iter (+/- 34,416.03) = 5954 MB/s

-Ctarget-cpu=x86-64-v3

test bench_swizzle_dyn_16        ... bench:     154,768.45 ns/iter (+/- 2,525.78) = 13550 MB/s
test bench_swizzle_dyn_16_mask16 ... bench:     152,077.10 ns/iter (+/- 280.91) = 13790 MB/s
test bench_swizzle_dyn_32        ... bench:     140,406.54 ns/iter (+/- 5,891.90) = 14936 MB/s
test bench_swizzle_dyn_32_mask16 ... bench:     135,856.60 ns/iter (+/- 1,557.41) = 15436 MB/s
test bench_swizzle_dyn_32_mask32 ... bench:     140,208.33 ns/iter (+/- 1,010.13) = 14957 MB/s
test bench_swizzle_dyn_64        ... bench:     182,426.04 ns/iter (+/- 492.10) = 11495 MB/s
test bench_swizzle_dyn_64_mask16 ... bench:     130,687.91 ns/iter (+/- 300.31) = 16047 MB/s
test bench_swizzle_dyn_64_mask32 ... bench:     133,016.57 ns/iter (+/- 611.21) = 15766 MB/s
test bench_swizzle_dyn_64_mask64 ... bench:     154,378.03 ns/iter (+/- 929.31) = 13584 MB/s

test bench_swizzle_dyn_24        ... bench:     167,513.82 ns/iter (+/- 3,303.83) = 12519 MB/s
test bench_swizzle_dyn_24_mask16 ... bench:     138,114.07 ns/iter (+/- 1,129.57) = 15184 MB/s
test bench_swizzle_dyn_43        ... bench:     248,154.55 ns/iter (+/- 1,220.99) = 8450 MB/s
test bench_swizzle_dyn_43_mask16 ... bench:     138,646.46 ns/iter (+/- 2,367.86) = 15125 MB/s
test bench_swizzle_dyn_5_mask_8  ... bench:     484,605.70 ns/iter (+/- 1,178.23) = 4327 MB/s

-Ctarget-cpu=x86-64-v4

test bench_swizzle_dyn_16        ... bench:     147,056.86 ns/iter (+/- 7,901.26) = 14260 MB/s
test bench_swizzle_dyn_16_mask16 ... bench:     145,653.23 ns/iter (+/- 765.60) = 14398 MB/s
test bench_swizzle_dyn_32        ... bench:     128,586.98 ns/iter (+/- 1,309.26) = 16309 MB/s
test bench_swizzle_dyn_32_mask16 ... bench:     125,415.66 ns/iter (+/- 439.47) = 16721 MB/s
test bench_swizzle_dyn_32_mask32 ... bench:     128,567.36 ns/iter (+/- 236.51) = 16311 MB/s
test bench_swizzle_dyn_64        ... bench:     154,575.97 ns/iter (+/- 179.12) = 13567 MB/s
test bench_swizzle_dyn_64_mask16 ... bench:     124,714.50 ns/iter (+/- 242.85) = 16815 MB/s
test bench_swizzle_dyn_64_mask32 ... bench:     127,455.67 ns/iter (+/- 1,334.81) = 16454 MB/s
test bench_swizzle_dyn_64_mask64 ... bench:     130,072.39 ns/iter (+/- 957.03) = 16123 MB/s

test bench_swizzle_dyn_24        ... bench:     236,086.73 ns/iter (+/- 11,341.07) = 8882 MB/s
test bench_swizzle_dyn_24_mask16 ... bench:     128,803.36 ns/iter (+/- 2,010.54) = 16281 MB/s
test bench_swizzle_dyn_43        ... bench:     373,447.80 ns/iter (+/- 531.31) = 5615 MB/s
test bench_swizzle_dyn_43_mask16 ... bench:     130,216.60 ns/iter (+/- 932.21) = 16104 MB/s
test bench_swizzle_dyn_5_mask_8  ... bench:     313,044.87 ns/iter (+/- 14,027.41) = 6699 MB/s

-Ctarget-cpu=icelake-server (avx512vbmi)

test bench_swizzle_dyn_16        ... bench:     139,096.00 ns/iter (+/- 534.63) = 15077 MB/s
test bench_swizzle_dyn_16_mask16 ... bench:     137,995.96 ns/iter (+/- 2,989.46) = 15197 MB/s
test bench_swizzle_dyn_32        ... bench:     118,391.74 ns/iter (+/- 253.91) = 17713 MB/s
test bench_swizzle_dyn_32_mask16 ... bench:     118,585.59 ns/iter (+/- 10,503.54) = 17684 MB/s
test bench_swizzle_dyn_32_mask32 ... bench:     118,472.72 ns/iter (+/- 1,708.84) = 17701 MB/s
test bench_swizzle_dyn_64        ... bench:     146,002.83 ns/iter (+/- 181.27) = 14363 MB/s
test bench_swizzle_dyn_64_mask16 ... bench:     135,991.02 ns/iter (+/- 120.40) = 15421 MB/s
test bench_swizzle_dyn_64_mask32 ... bench:     135,965.09 ns/iter (+/- 128.03) = 15424 MB/s
test bench_swizzle_dyn_64_mask64 ... bench:     135,971.99 ns/iter (+/- 172.91) = 15423 MB/s

test bench_swizzle_dyn_24        ... bench:     130,304.84 ns/iter (+/- 8,965.54) = 16094 MB/s
test bench_swizzle_dyn_24_mask16 ... bench:     121,858.66 ns/iter (+/- 63,820.69) = 17209 MB/s
test bench_swizzle_dyn_43        ... bench:     186,879.93 ns/iter (+/- 436.80) = 11221 MB/s
test bench_swizzle_dyn_43_mask16 ... bench:     158,348.00 ns/iter (+/- 410.78) = 13243 MB/s
test bench_swizzle_dyn_5_mask_8  ... bench:     420,093.25 ns/iter (+/- 3,317.77) = 4992 MB/s

@cvijdea-bd cvijdea-bd force-pushed the swizzle-dyn-fallback branch from 169c541 to 041e43b Compare August 27, 2024 11:21
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@cvijdea-bd