h265parse: Ensure parsing ends on start-code + full header

The parser is used all over the place assuming that after calling
gst_h265_parser_identify_nalu(), the start-code found is can also be
identified. In H264 this works, because scan_for_start_code rely on
gst_byte_reader_masked_scan_uint32() that ensures that 1 byte passed the 3
bytes start code is found. But for HEVC, we need two bytes to identify the
following NAL.

This patch will return NO_NAL_END, even if a start code is found in the case
there was not enough bytes. This solution was chosen to maintain backward
compatibility, and reduce complexicity.

Fixes #1287

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1251>

1 files changed, 9 insertions, 0 deletions

diff --git a/gst-libs/gst/codecparsers/gsth265parser.c b/gst-libs/gst/codecparsers/gsth265parser.c
index dc8db4c2e..e63215dcd 100644
--- a/gst-libs/gst/codecparsers/gsth265parser.c
+++ b/gst-libs/gst/codecparsers/gsth265parser.c
@@ -1484,6 +1484,15 @@ gst_h265_parser_identify_nalu (GstH265Parser * parser,
     return GST_H265_PARSER_NO_NAL_END;
   }
 
+  /* Callers assumes that enough data will available to identify the next NAL,
+   * but scan_for_start_codes() only ensure 1 extra byte is available. Ensure
+   * we have the required two header bytes (3 bytes start code and 2 byte
+   * header). */
+  if (size - (nalu->offset + off2) < 5) {
+    GST_DEBUG ("Not enough bytes identify the next NAL.");
+    return GST_H265_PARSER_NO_NAL_END;
+  }
+
   /* Mini performance improvement:
    * We could have a way to store how many 0s were skipped to avoid
    * parsing them again on the next NAL */