Change #236858

Comments

avfilter/vf_overlay: Use correct alpha when > 8 bits
When chroma subsampling is in use, the filter averages
the corresponding (non subsampled) alpha values to get
the actual alpha value. When vertical subsampling is
in use, the next line is accessed via a[src->linesize[3]],
yet a is an uint16_t* for >8 bit formats and linesize
is always in bytes, so that this actually uses the second
line below the current one. This is fixed in this commit.

No FATE test needed updates, because the filter-overlay-yuv420p10
and filter-overlay-yuv444p10 tests use a yuv420p test file
that has constant opacity after conversion to yuva.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>

Changed files

• libavfilter/vf_overlay.c