va: h264dec: Try to use ConstrainedBaseline or Main to decode BaseLine.

In the h264, the Baseline profile is widely misused. A lot of streams declare
that they are the Baseline, but in fact they just conform to ConstrainedBaseline.
The features such as FMO and ASO are not used at all.
If the decoder does not strictly conforms to the SPEC, we can just use Baseline
or Main profile to decode it to avoid lots of streams failure.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/2428>

1 files changed, 10 insertions, 2 deletions

diff --git a/sys/va/gstvah264dec.c b/sys/va/gstvah264dec.c
index 4d342a880..aba8bab70 100644
--- a/sys/va/gstvah264dec.c
+++ b/sys/va/gstvah264dec.c
@@ -602,13 +602,21 @@ _get_profile (GstVaH264Dec * self, const GstH264SPS * sps, gint max_dpb_size)
 
   switch (sps->profile_idc) {
     case GST_H264_PROFILE_BASELINE:
-      /* A.2 compliant */
+    {
+      GstH264DecoderCompliance compliance = GST_H264_DECODER_COMPLIANCE_STRICT;
+
+      g_object_get (G_OBJECT (self), "compliance", &compliance, NULL);
+
+      /* A.2 compliant or not strict */
       if (sps->constraint_set0_flag || sps->constraint_set1_flag
-          || sps->constraint_set2_flag) {
+          || sps->constraint_set2_flag
+          || compliance != GST_H264_DECODER_COMPLIANCE_STRICT) {
         profiles[i++] = VAProfileH264ConstrainedBaseline;
         profiles[i++] = VAProfileH264Main;
       }
+
       break;
+    }
     case GST_H264_PROFILE_EXTENDED:
       if (sps->constraint_set1_flag) {  /* A.2.2 (main profile) */
         profiles[i++] = VAProfileH264Main;