h264_mp4toannexb: Improve extradata overread checks

Currently during parsing the extradata, h264_mp4toannexb checks for
overreads by adding the size of the current unit to the current position
pointer and comparing this to the end position of the extradata. But
pointer comparisons and pointer arithmetic are only defined if it does not
exceed the object it is used on (one past the last element of an array
is allowed, too). In practice, this might lead to overflows. Therefore
the check has been changed to use bytestream2_get_bytes_left() which
means that the pointers get subtracted and the result gets compared to
the available size.

Furthermore, the error code has been fixed.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

1 files changed, 2 insertions, 2 deletions

diff --git a/libavcodec/h264_mp4toannexb_bsf.c b/libavcodec/h264_mp4toannexb_bsf.c
index f31902b506..b9c6b165af 100644
--- a/libavcodec/h264_mp4toannexb_bsf.c
+++ b/libavcodec/h264_mp4toannexb_bsf.c
@@ -101,11 +101,11 @@ static int h264_extradata_to_annexb(AVBSFContext *ctx, const int padding)
         unit_size   = bytestream2_get_be16u(gb);
         total_size += unit_size + 4;
         av_assert1(total_size <= INT_MAX - padding);
-        if (gb->buffer + unit_size > gb->buffer_end) {
+        if (bytestream2_get_bytes_left(gb) < unit_size) {
             av_log(ctx, AV_LOG_ERROR, "Packet header is not contained in global extradata, "
                    "corrupted stream or invalid MP4/AVCC bitstream\n");
             av_free(out);
-            return AVERROR(EINVAL);
+            return AVERROR_INVALIDDATA;
         }
         if ((err = av_reallocp(&out, total_size + padding)) < 0)
             return err;