Change #236855

Comments

avfilter/vf_overlay: Don't perform UB pointer arithmetic
This happens when the pixel format of the output does not
have an alpha channel. It leads to FATE failures with
the ffmpeg-filter_colorkey, filter-overlay-dvdsub-2397
filter-overlay,
filter-overlay_{gbrp_gbrap,nv12,nv21,yuv420,yuv420_yuva420,
yuv420p10,yuv422_yuva422,yuv422p10,yuv444_yuva444,yuv444p10}
and sub2video tests when using Clang UBSan.

Fix this by only performing the pointer arithmetic when
it is going to be used. This can be checked via variables
that compile-time constants due to inlining, so that the
checks are free. Given that the pointer is potentially
used as a function argument, the compiler could elide
the calculation, but not it can. The size of .text decreased
by 1632B with GCC 14 and by 1392B with Clang 19 (both -O3).

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>

Changed files

• libavfilter/vf_overlay.c