Change #286416

Comments

avfilter/dnn_io_proc: fix stack overflow in NCHW<->frame sws_scale calls
ff_proc_from_dnn_to_frame() and ff_proc_from_frame_to_dnn() each declare a
single `void *middle_data` local and, on the DL_NCHW layout branch, take its
address (`dst_data = &middle_data;` / `src_data = &middle_data;`) and pass
that as sws_scale()'s dst/src plane-array argument after casting to
`uint8_t *const *`. sws_scale()'s dst/src parameters are a 4-element array
(uint8_t *const dst[4]/src[4]), and scale_internal() copies all four
pointers (memcpy of srcSlice/dstSlice into src2/dst2) regardless of how many
planes the actual conversion uses. Since `&middle_data` only ever points at
one pointer-sized stack slot, that copy reads past it into whatever is next
on the stack: an out-of-bounds stack read on
every dnn_processing call whose model uses NCHW layout output or input into
an RGB24/BGR24 frame.

Fix both by giving dst_data/src_data real 4-element backing storage
(dst_data_buf/src_data_buf, zero-initialized) with the allocated buffer in
slot 0, instead of aliasing the single pointer.

Tested on arm64 macOS 27.0 (M1 Ultra), clang-asan toolchain, with
--enable-libopenvino (2026.4.0), --enable-libonnxruntime (1.30.0) and
--enable-libtensorflow, against unmodified upstream master (ff2059afd9) and
against this commit applied on top of the same base:

- Reproduced the stack-buffer-overflow on unmodified master two ways:
  - ff_proc_from_dnn_to_frame(): `ffmpeg -threads 1 -i puppies.ogv -an -vf
    "fps=2,format=rgb24,dnn_processing=dnn_backend=openvino:
    model=proc_valid_720.onnx:input=input:output=output" -f framemd5 out.md5`
    aborts under ASan on the first frame (offset-72 read into the
    `middle_data`/`planar_data` stack slots, from the OpenVINO async
    inference callback thread).
  - ff_proc_from_frame_to_dnn(): the same dnn_processing filter with two real
    ONNX Model Zoo fast-neural-style models (candy-9.onnx,
    sha256 9d11a3529d1e547da6ae07201d93484dbab2ec0a3614535752c8f40f0fe2968a;
    mosaic-9.onnx,
    sha256 fa646dedade881243f8d5a2ceb7de2b93675b21fc24f7482894ac4851a9a0a47;
    both NCHW 1x3x224x224 in/out) via dnn_backend=onnx aborts under ASan the
    same way, this time with the overflow located directly in
    ff_proc_from_frame_to_dnn's `middle_data` frame (offset 72), confirming
    the sibling instance flagged at review is independently live, not just
    structurally identical.
  With this commit applied, all three of the above runs are clean of the
  stack-buffer-overflow report (an unrelated, pre-existing OpenVINO/TBB
  at-exit SEGV under ASan remains on this machine/toolchain/OpenVINO version;
  it reproduces identically before and after this commit, and even on plain
  `ffmpeg -version` with no DNN filter at all, so it is unrelated to this fix
  and not addressed here).
- Whole-clip (puppies.ogv, 589 frames) framemd5 output from a release
  (non-ASan) build is byte-for-byte identical before/after this commit for:
  the NCHW OpenVINO identity model (proc_valid_720.onnx) into rgb24, and for
  both candy-9.onnx and mosaic-9.onnx into rgb24 via dnn_backend=onnx. This
  is expected: the aliasing read pulls garbage bytes that sws_scale never
  writes to a packed destination/never reads for these plane counts, so the
  visible output is unaffected; only ASan's redzone instrumentation makes the
  read observable.
- Unchanged paths are also byte-for-byte identical before/after: the NHWC
  TensorFlow path (tf_rgb_valid.pb into rgb24, which never takes the
  DL_NCHW/middle_data branch) and the planar/gray output path (tf_y_valid.pb
  into gray8, which uses frame->data directly and never allocates
  middle_data).
- `make fate-source` passes with this commit. This tree has no fate-dnn*
  tests to run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jake Fineman <jake@wave.online>

Changed files

• libavfilter/dnn/dnn_io_proc.c