4:2:2/4:4:4 desync on 4x4 blocks with intra edge filter

[rzumer]

Found through a master branch run. See the log.

This run, though incomplete, shows that this was not introduced by #2224.

Also reproduced on 4:4:4 locally.

Probably due to above/left block contexts not properly taking sampling format into account.

[ycho]

commit ace686d
./target/release/rav1e ~/sequences/flower_garden_422_ntsc.y4m -o test.ivf -r test_rec.y4m --quantizer 20 --speed=2 --limit=3 --rdo-lookahead-frames=1 --low_latency # mismatch, when sub 8x8 used!

• speed levels 0,1,2 all give mismatches between encoder and decoder at 1st frame.

• test input is available from: https://media.xiph.org/video/derf/y4m/flower_garden_422_ntsc.y4m

[rzumer]

#2083 seems to have caused it.

Both native and AVX2 (dav1d) implementations desync, so most likely this is related to the edge buffer initialization, which is native only.

[ycho]

Some more info:
The other smaller frame size video, https://media.xiph.org/video/derf/y4m/football_422_qcif.y4m, also gives the mismatch.
./target/release/rav1e ~/sequences/football_422_qcif.y4m -o test.ivf -r test_rec.y4m --quantizer 20 --speed=2 --limit=3 --rdo-lookahead-frames=1 --low_latency

And, I have diff image for you, which has mismatch in the 1st frame.

The black grid is 64x64, so there are rowxcol = 3x3 SBs, and the 1st mismatch occur at SB (row, col) = (0,2).
Meanwhile, please note that AOM Analyzer does not decode and display 422 bitstream correctly, adding lots of artifacts, which seems wrong. The aomdec or dav1d decode correctly.

[ycho]

To obtain diff image, I have used https://github.com/IENT/YUView.

[barrbrain]

I have confirmed that the reproduction in #2299 (comment) is resolved by #2313.