codecs: h264decoder: update max_dpb_frames only if VUI is present

There are some streams, with HRD, where the the calculated
max_dpb_frames is zero (max_dpb_mbs is less than size mb). In order to
get the dbp size it is required to rely on the VUI parameters if they
are present.

According to the spec Annex E.2.1

**max_dec_frame_buffering** specifies the required size of the HRD
decoded picture buffer (DPB) in units of frame buffers.  It is a
requirement of bitstream conformance that the coded video sequence
shall not require a decoded picture buffer with size of more than
Max(1, max_dec_frame_buffering) frame buffers to enable the output of
decoded pictures at the output times specified by dpb_output_delay of
the picture timing SEI messages. The value of max_dec_frame_buffering
shall be greater than or equal to max_num_ref_frames. An upper bound
for the value of max_dec_frame_buffering is specified by the level
limits in clauses A.3.1, A.3.2, G.10.2.1, and H.10.2.

When the max_dec_frame_buffering syntax element is not present, the
value of max_dec_frame_buffering shall be inferred as follows:

– If profile_idc is equal to 44, 86, 100, 110, 122, or 244 and
constraint_set3_flag is equal to 1, the value of
max_dec_frame_buffering shall be inferred to be equal to 0.

– Otherwise (profile_idc is not equal to 44, 86, 100, 110, 122, or 244
or constraint_set3_flag is equal to 0), the value of
max_dec_frame_buffering shall be inferred to be equal to MaxDpbFrames.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1381>

1 files changed, 6 insertions, 3 deletions

diff --git a/gst-libs/gst/codecs/gsth264decoder.c b/gst-libs/gst/codecs/gsth264decoder.c
index e3d8dd01a..8d4638511 100644
--- a/gst-libs/gst/codecs/gsth264decoder.c
+++ b/gst-libs/gst/codecs/gsth264decoder.c
@@ -1847,7 +1847,7 @@ gst_h264_decoder_process_sps (GstH264Decoder * self, GstH264SPS * sps)
   level = sps->level_idc;
   if (level == 11 && (sps->profile_idc == 66 || sps->profile_idc == 77) &&
       sps->constraint_set3_flag) {
-    /* Leel 1b */
+    /* Level 1b */
     level = 9;
   }
 
@@ -1861,8 +1861,11 @@ gst_h264_decoder_process_sps (GstH264Decoder * self, GstH264SPS * sps)
   max_dpb_frames = MIN (max_dpb_mbs / (width_mb * height_mb),
       GST_H264_DPB_MAX_SIZE);
 
-  max_dpb_size = MIN (max_dpb_frames,
-      MAX (sps->num_ref_frames, sps->vui_parameters.max_dec_frame_buffering));
+  if (sps->vui_parameters_present_flag
+      && sps->vui_parameters.bitstream_restriction_flag)
+    max_dpb_frames = MAX (1, sps->vui_parameters.max_dec_frame_buffering);
+
+  max_dpb_size = MIN (max_dpb_frames, sps->num_ref_frames);
 
   /* Safety, so that subclass don't need bound checking */
   g_return_val_if_fail (max_dpb_size <= GST_H264_DPB_MAX_SIZE, FALSE);