raseonsi/vcn: fix a h264 decoding issue

reason:
   some h264 streams have some strange pictures, from
   vaapi input these pictures don't have a reference frame,
   however, they are not intra only pictures, in MB layer
   these pictures are looking for some references, if they
   cannot find it. It could cause PF.

   when reference pictures exist, it will need to set used_for
   reference_flags, therefore if that is set, however the
   number of reference frames is zero, which is odd, it
   should be avoided.

solution:
   In the above case,  to scan the ref list so that it will
   make at least one reference available to avoid crash, since
   this is not accurate enough, it could cause some artifacts.
   And in that case, it will need to be checked individually
   for another solution.

closes: https://gitlab.freedesktop.org/drm/amd/-/issues/1462
closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8401
Cc: mesa-stable

Tested-by: llyyr <llyyr.public@gmail.com>
Reviewed-by: Leo Liu <leo.liu@amd.com>
Signed-off-by: Ruijing Dong <ruijing.dong@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21732>
(cherry picked from commit 0f3370eede23284c09abdaa46714d7c4c795ecd4)

2 files changed, 15 insertions, 1 deletions

diff --git a/.pick_status.json b/.pick_status.json
index 8ab4f3997dd..90e5c5e25df 100644
--- a/.pick_status.json
+++ b/.pick_status.json
@@ -1489,7 +1489,7 @@
         "description": "raseonsi/vcn: fix a h264 decoding issue",
         "nominated": true,
         "nomination_type": 0,
-        "resolution": 0,
+        "resolution": 1,
         "main_sha": null,
         "because_sha": null
     },
diff --git a/src/gallium/drivers/radeonsi/radeon_vcn_dec.c b/src/gallium/drivers/radeonsi/radeon_vcn_dec.c
index 3bdcc7d71d5..1c803832567 100644
--- a/src/gallium/drivers/radeonsi/radeon_vcn_dec.c
+++ b/src/gallium/drivers/radeonsi/radeon_vcn_dec.c
@@ -278,6 +278,20 @@ static rvcn_dec_message_avc_t get_h264_msg(struct radeon_decoder *dec,
       }
    }
 
+   /* if reference picture exists, however no reference picture found at the end
+      curr_pic_ref_frame_num == 0, which is not reasonable, should be corrected. */
+   if (result.used_for_reference_flags && (result.curr_pic_ref_frame_num == 0)) {
+      for (i = 0; i < ARRAY_SIZE(result.ref_frame_list); i++) {
+         result.ref_frame_list[i] = pic->ref[i] ?
+                (uintptr_t)vl_video_buffer_get_associated_data(pic->ref[i], &dec->base) : 0xff;
+         if (result.ref_frame_list[i] != 0xff) {
+            result.curr_pic_ref_frame_num++;
+            result.non_existing_frame_flags &= ~(1 << i);
+            break;
+         }
+      }
+   }
+
    for (i = 0; i < ARRAY_SIZE(result.ref_frame_list); i++) {
       if (result.ref_frame_list[i] != 0xff) {
          dec->h264_valid_ref_num[i]         = result.frame_num_list[i];