External mask overviews aren't copied correctly when creating internally-masked GeoTIFFs

[mojodna]

(Follow-on from discussion in rasterio/rasterio#1373.)

Expected behavior and actual behavior.

As part of COG generation, I'm splitting up image data from its corresponding mask. The image data is converted to YCbCr and bicubic/lanczos-resampled overviews are generated. The mask also has nearest neighbor-resampled overviews produced (and has DEFLATE compression applied). To create the final output of the pipeline, the processed image + mask data (replete with overviews) are merged into a COG containing an internal mask.

Steps to reproduce the problem.

# COG produced using this pipeline; we're going to ignore any overviews present and
# regenerate them
curl -O https://mojodna-temp.s3.amazonaws.com/internal-mask.tif

# copy bands 1,2,3 to intermediate.tif, mask to intermediate.tif.msk
gdal_translate -b 1 -b 2 -b 3 -mask mask internal-mask.tif intermediate.tif

# add overviews (possibly resampling in a way that would be disruptive to a mask (i.e.
# introduce values that aren't 0 or 255), e.g. bicubic)
gdaladdo intermediate.tif 2 4 8 16 32

# add overviews to the mask
gdaladdo intermediate.tif.msk 2 4 8 16 32

# create an internally-masked result (this is where COG options would be applied)
gdal_translate \
  -co COPY_SRC_OVERVIEWS=YES \
  --config GDAL_TIFF_INTERNAL_MASK YES \
  intermediate.tif out.tif

out.tif contains all expected overviews, although each of its mask overviews is incorrect (all 255, extracted using tiffcp out.tif,7 mask-ovr1.tif for the first level).

The same overview level from intermediate.tif.msk (tiffcp intermediate.tif.msk,1 intermediate-mask-ovr1.tif for the first level) contains correct values.

Operating system

Ubuntu 16.04 64 bit (Docker)

GDAL version and provenance

GDAL 2.3.1, built according to https://github.com/mojodna/docker-gdal/blob/master/Dockerfile.

[mojodna]

Hurrah, thank you!!