libavutil: Add wchartoutf8(), wchartoansi(), utf8toansi(), getenv_utf8(), freeenv_utf8() and getenv_dup()

wchartoutf8() converts strings returned by WinAPI into UTF-8,
which is FFmpeg's preffered encoding.

Some external dependencies, such as AviSynth, are still
not Unicode-enabled. utf8toansi() converts UTF-8 strings
into ANSI in two steps: UTF-8 -> wchar_t -> ANSI.
wchartoansi() is responsible for the second step of the conversion.
Conversion in just one step is not supported by WinAPI.

Since these character converting functions allocate the buffer
of necessary size, they also facilitate the removal of MAX_PATH limit
in places where fixed-size ANSI/WCHAR strings were used
as filename buffers.

On Windows, getenv_utf8() wraps _wgetenv() converting its input from
and its output to UTF-8. Strings returned by getenv_utf8()
must be freed by freeenv_utf8().

On all other platforms getenv_utf8() is a wrapper around getenv(),
and freeenv_utf8() is a no-op.

The value returned by plain getenv() cannot be modified;
av_strdup() is usually used when modifications are required.
However, on Windows, av_strdup() after getenv_utf8() leads to
unnecessary allocation. getenv_dup() is introduced to avoid
such an allocation. Value returned by getenv_dup() must be freed
by av_free().

Because of cleanup complexities, in places that only test the existence
of an environment variable or compare its value with a string
consisting entirely of ASCII characters, the use of plain getenv()
is still preferred. (libavutil/log.c check_color_terminal()
is an example of such a place.)

Plain getenv() is also preffered in UNIX-only code,
such as bktr.c, fbdev_common.c, oss.c in libavdevice
or af_ladspa.c in libavfilter.

Signed-off-by: Martin Storsjö <martin@martin.st>

1 files changed, 53 insertions, 0 deletions

diff --git a/libavutil/wchar_filename.h b/libavutil/wchar_filename.h
index f36d9dfea3..08de073ed7 100644
--- a/libavutil/wchar_filename.h
+++ b/libavutil/wchar_filename.h
@@ -20,6 +20,8 @@
 #define AVUTIL_WCHAR_FILENAME_H
 
 #ifdef _WIN32
+
+#define WIN32_LEAN_AND_MEAN
 #include <windows.h>
 #include "mem.h"
 
@@ -41,6 +43,57 @@ static inline int utf8towchar(const char *filename_utf8, wchar_t **filename_w)
     return 0;
 }
 
+av_warn_unused_result
+static inline int wchartocp(unsigned int code_page, const wchar_t *filename_w,
+                            char **filename)
+{
+    DWORD flags = code_page == CP_UTF8 ? WC_ERR_INVALID_CHARS : 0;
+    int num_chars = WideCharToMultiByte(code_page, flags, filename_w, -1,
+                                        NULL, 0, NULL, NULL);
+    if (num_chars <= 0) {
+        *filename = NULL;
+        return 0;
+    }
+    *filename = av_malloc_array(num_chars, sizeof *filename);
+    if (!*filename) {
+        errno = ENOMEM;
+        return -1;
+    }
+    WideCharToMultiByte(code_page, flags, filename_w, -1,
+                        *filename, num_chars, NULL, NULL);
+    return 0;
+}
+
+av_warn_unused_result
+static inline int wchartoutf8(const wchar_t *filename_w, char **filename)
+{
+    return wchartocp(CP_UTF8, filename_w, filename);
+}
+
+av_warn_unused_result
+static inline int wchartoansi(const wchar_t *filename_w, char **filename)
+{
+    return wchartocp(CP_ACP, filename_w, filename);
+}
+
+av_warn_unused_result
+static inline int utf8toansi(const char *filename_utf8, char **filename)
+{
+    wchar_t *filename_w = NULL;
+    int ret = -1;
+    if (utf8towchar(filename_utf8, &filename_w))
+        return -1;
+
+    if (!filename_w) {
+        *filename = NULL;
+        return 0;
+    }
+
+    ret = wchartoansi(filename_w, filename);
+    av_free(filename_w);
+    return ret;
+}
+
 /**
  * Checks for extended path prefixes for which normalization needs to be skipped.
  * see .NET6: PathInternal.IsExtended()