avfilter/palettegen: use AV_QSORT()

This makes the sorting of the colors along an axis (r, g or b)
predictible, and thus testable under FATE. The performance is not really
an issue here since the function is called only once at the end and will
need to sort very small number of entries, so an alternative would be to
make the sorting functions (see DECLARE_CMP_FUNC()) fallback on another
axis in case of equality. This approach was actually simpler.

I don't know if there is any advantage in using a multidimensional sort,
but it will affect the final palette one way or another.

1 files changed, 3 insertions, 1 deletions

diff --git a/libavfilter/vf_palettegen.c b/libavfilter/vf_palettegen.c
index f1f472225b..fa8cc12a3c 100644
--- a/libavfilter/vf_palettegen.c
+++ b/libavfilter/vf_palettegen.c
@@ -25,6 +25,7 @@
 
 #include "libavutil/avassert.h"
 #include "libavutil/opt.h"
+#include "libavutil/qsort.h"
 #include "avfilter.h"
 #include "internal.h"
 
@@ -352,7 +353,8 @@ static AVFrame *get_palette_frame(AVFilterContext *ctx)
 
         /* sort the range by its longest axis if it's not already sorted */
         if (box->sorted_by != longest) {
-            qsort(&s->refs[box->start], box->len, sizeof(*s->refs), cmp_funcs[longest]);
+            cmp_func cmpf = cmp_funcs[longest];
+            AV_QSORT(&s->refs[box->start], box->len, const struct color_ref *, cmpf);
             box->sorted_by = longest;
         }