Redis Function Libraries (#10004)

# Redis Function Libraries

This PR implements Redis Functions Libraries as describe on: https://github.com/redis/redis/issues/9906.

Libraries purpose is to provide a better code sharing between functions by allowing to create multiple
functions in a single command. Functions that were created together can safely share code between
each other without worrying about compatibility issues and versioning.

Creating a new library is done using 'FUNCTION LOAD' command (full API is described below)

This PR introduces a new struct called libraryInfo, libraryInfo holds information about a library:
* name - name of the library
* engine - engine used to create the library
* code - library code
* description - library description
* functions - the functions exposed by the library

When Redis gets the `FUNCTION LOAD` command it creates a new empty libraryInfo.
Redis passes the `CODE` to the relevant engine alongside the empty libraryInfo.
As a result, the engine will create one or more functions by calling 'libraryCreateFunction'.
The new funcion will be added to the newly created libraryInfo. So far Everything is happening
locally on the libraryInfo so it is easy to abort the operation (in case of an error) by simply
freeing the libraryInfo. After the library info is fully constructed we start the joining phase by
which we will join the new library to the other libraries currently exist on Redis.
The joining phase make sure there is no function collision and add the library to the
librariesCtx (renamed from functionCtx). LibrariesCtx is used all around the code in the exact
same way as functionCtx was used (with respect to RDB loading, replicatio, ...).
The only difference is that apart from function dictionary (maps function name to functionInfo
object), the librariesCtx contains also a libraries dictionary that maps library name to libraryInfo object.

## New API
### FUNCTION LOAD
`FUNCTION LOAD <ENGINE> <LIBRARY NAME> [REPLACE] [DESCRIPTION <DESCRIPTION>] <CODE>`
Create a new library with the given parameters:
* ENGINE - REPLACE Engine name to use to create the library.
* LIBRARY NAME - The new library name.
* REPLACE - If the library already exists, replace it.
* DESCRIPTION - Library description.
* CODE - Library code.

Return "OK" on success, or error on the following cases:
* Library name already taken and REPLACE was not used
* Name collision with another existing library (even if replace was uses)
* Library registration failed by the engine (usually compilation error)

## Changed API
### FUNCTION LIST
`FUNCTION LIST [LIBRARYNAME <LIBRARY NAME PATTERN>] [WITHCODE]`
Command was modified to also allow getting libraries code (so `FUNCTION INFO` command is no longer
needed and removed). In addition the command gets an option argument, `LIBRARYNAME` allows you to
only get libraries that match the given `LIBRARYNAME` pattern. By default, it returns all libraries.

### INFO MEMORY
Added number of libraries to `INFO MEMORY`

### Commands flags
`DENYOOM` flag was set on `FUNCTION LOAD` and `FUNCTION RESTORE`. We consider those commands
as commands that add new data to the dateset (functions are data) and so we want to disallows
to run those commands on OOM.

## Removed API
* FUNCTION CREATE - Decided on https://github.com/redis/redis/issues/9906
* FUNCTION INFO - Decided on https://github.com/redis/redis/issues/9899

## Lua engine changes
When the Lua engine gets the code given on `FUNCTION LOAD` command, it immediately runs it, we call
this run the loading run. Loading run is not a usual script run, it is not possible to invoke any
Redis command from within the load run.
Instead there is a new API provided by `library` object. The new API's: 
* `redis.log` - behave the same as `redis.log`
* `redis.register_function` - register a new function to the library

The loading run purpose is to register functions using the new `redis.register_function` API.
Any attempt to use any other API will result in an error. In addition, the load run is has a time
limit of 500ms, error is raise on timeout and the entire operation is aborted.

### `redis.register_function`
`redis.register_function(<function_name>, <callback>, [<description>])`
This new API allows users to register a new function that will be linked to the newly created library.
This API can only be called during the load run (see definition above). Any attempt to use it outside
of the load run will result in an error.
The parameters pass to the API are:
* function_name - Function name (must be a Lua string)
* callback - Lua function object that will be called when the function is invokes using fcall/fcall_ro
* description - Function description, optional (must be a Lua string).

### Example
The following example creates a library called `lib` with 2 functions, `f1` and `f1`, returns 1 and 2 respectively:
```
local function f1(keys, args)
    return 1
end

local function f2(keys, args)
    return 2
end

redis.register_function('f1', f1)
redis.register_function('f2', f2)
```

Notice: Unlike `eval`, functions inside a library get the KEYS and ARGV as arguments to the
functions and not as global.

### Technical Details

On the load run we only want the user to be able to call a white list on API's. This way, in
the future, if new API's will be added, the new API's will not be available to the load run
unless specifically added to this white list. We put the while list on the `library` object and
make sure the `library` object is only available to the load run by using [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv) API. This API allows us to set
the `globals` of a function (and all the function it creates). Before starting the load run we
create a new fresh Lua table (call it `g`) that only contains the `library` API (we make sure
to set global protection on this table just like the general global protection already exists
today), then we use [lua_setfenv](https://www.lua.org/manual/5.1/manual.html#lua_setfenv)
to set `g` as the global table of the load run. After the load run finished we update `g`
metatable and set `__index` and `__newindex` functions to be `_G` (Lua default globals),
we also pop out the `library` object as we do not need it anymore.
This way, any function that was created on the load run (and will be invoke using `fcall`) will
see the default globals as it expected to see them and will not have the `library` API anymore.

An important outcome of this new approach is that now we can achieve a distinct global table
for each library (it is not yet like that but it is very easy to achieve it now). In the future we can
decide to remove global protection because global on different libraries will not collide or we
can chose to give different API to different libraries base on some configuration or input.

Notice that this technique was meant to prevent errors and was not meant to prevent malicious
user from exploit it. For example, the load run can still save the `library` object on some local
variable and then using in `fcall` context. To prevent such a malicious use, the C code also make
sure it is running in the right context and if not raise an error.

1 files changed, 176 insertions, 14 deletions

diff --git a/src/function_lua.c b/src/function_lua.c
index 864ced809..e6b8a2727 100644
--- a/src/function_lua.c
+++ b/src/function_lua.c
@@ -48,6 +48,9 @@
 #define LUA_ENGINE_NAME "LUA"
 #define REGISTRY_ENGINE_CTX_NAME "__ENGINE_CTX__"
 #define REGISTRY_ERROR_HANDLER_NAME "__ERROR_HANDLER__"
+#define REGISTRY_LOAD_CTX_NAME "__LIBRARY_CTX__"
+#define LIBRARY_API_NAME "__LIBRARY_API__"
+#define LOAD_TIMEOUT_MS 500
 
 /* Lua engine ctx */
 typedef struct luaEngineCtx {
@@ -60,6 +63,27 @@ typedef struct luaFunctionCtx {
     int lua_function_ref;
 } luaFunctionCtx;
 
+typedef struct loadCtx {
+    functionLibInfo *li;
+    monotime start_time;
+} loadCtx;
+
+/* Hook for FUNCTION LOAD execution.
+ * Used to cancel the execution in case of a timeout (500ms).
+ * This execution should be fast and should only register
+ * functions so 500ms should be more than enough. */
+static void luaEngineLoadHook(lua_State *lua, lua_Debug *ar) {
+    UNUSED(ar);
+    loadCtx *load_ctx = luaGetFromRegistry(lua, REGISTRY_LOAD_CTX_NAME);
+    uint64_t duration = elapsedMs(load_ctx->start_time);
+    if (duration > LOAD_TIMEOUT_MS) {
+        lua_sethook(lua, luaEngineLoadHook, LUA_MASKLINE, 0);
+
+        lua_pushstring(lua,"FUNCTION LOAD timeout");
+        lua_error(lua);
+    }
+}
+
 /*
  * Compile a given blob and save it on the registry.
  * Return a function ctx with Lua ref that allows to later retrieve the
@@ -67,25 +91,88 @@ typedef struct luaFunctionCtx {
  *
  * Return NULL on compilation error and set the error to the err variable
  */
-static void* luaEngineCreate(void *engine_ctx, sds blob, sds *err) {
+static int luaEngineCreate(void *engine_ctx, functionLibInfo *li, sds blob, sds *err) {
     luaEngineCtx *lua_engine_ctx = engine_ctx;
     lua_State *lua = lua_engine_ctx->lua;
+
+    /* Each library will have its own global distinct table.
+     * We will create a new fresh Lua table and use
+     * lua_setfenv to set the table as the library globals
+     * (https://www.lua.org/manual/5.1/manual.html#lua_setfenv)
+     *
+     * At first, populate this new table with only the 'library' API
+     * to make sure only 'library' API is available at start. After the
+     * initial run is finished and all functions are registered, add
+     * all the default globals to the library global table and delete
+     * the library API.
+     *
+     * There are 2 ways to achieve the last part (add default
+     * globals to the new table):
+     *
+     * 1. Initialize the new table with all the default globals
+     * 2. Inheritance using metatable (https://www.lua.org/pil/14.3.html)
+     *
+     * For now we are choosing the second, we can change it in the future to
+     * achieve a better isolation between functions. */
+    lua_newtable(lua); /* Global table for the library */
+    lua_pushstring(lua, REDIS_API_NAME);
+    lua_pushstring(lua, LIBRARY_API_NAME);
+    lua_gettable(lua, LUA_REGISTRYINDEX); /* get library function from registry */
+    lua_settable(lua, -3); /* push the library table to the new global table */
+
+    /* Set global protection on the new global table */
+    luaSetGlobalProtection(lua_engine_ctx->lua);
+
+    /* compile the code */
     if (luaL_loadbuffer(lua, blob, sdslen(blob), "@user_function")) {
-        *err = sdsempty();
-        *err = sdscatprintf(*err, "Error compiling function: %s",
-                lua_tostring(lua, -1));
-        lua_pop(lua, 1);
-        return NULL;
+        *err = sdscatprintf(sdsempty(), "Error compiling function: %s", lua_tostring(lua, -1));
+        lua_pop(lua, 2); /* pops the error and globals table */
+        return C_ERR;
     }
-
     serverAssert(lua_isfunction(lua, -1));
 
-    int lua_function_ref = luaL_ref(lua, LUA_REGISTRYINDEX);
+    loadCtx load_ctx = {
+        .li = li,
+        .start_time = getMonotonicUs(),
+    };
+    luaSaveOnRegistry(lua, REGISTRY_LOAD_CTX_NAME, &load_ctx);
 
-    luaFunctionCtx *f_ctx = zmalloc(sizeof(*f_ctx));
-    *f_ctx = (luaFunctionCtx ) { .lua_function_ref = lua_function_ref, };
+    /* set the function environment so only 'library' API can be accessed. */
+    lua_pushvalue(lua, -2); /* push global table to the front */
+    lua_setfenv(lua, -2);
 
-    return f_ctx;
+    lua_sethook(lua,luaEngineLoadHook,LUA_MASKCOUNT,100000);
+    /* Run the compiled code to allow it to register functions */
+    if (lua_pcall(lua,0,0,0)) {
+        *err = sdscatprintf(sdsempty(), "Error registering functions: %s", lua_tostring(lua, -1));
+        lua_pop(lua, 2); /* pops the error and globals table */
+        lua_sethook(lua,NULL,0,0); /* Disable hook */
+        luaSaveOnRegistry(lua, REGISTRY_LOAD_CTX_NAME, NULL);
+        return C_ERR;
+    }
+    lua_sethook(lua,NULL,0,0); /* Disable hook */
+    luaSaveOnRegistry(lua, REGISTRY_LOAD_CTX_NAME, NULL);
+
+    /* stack contains the global table, lets rearrange it to contains the entire API. */
+    /* delete 'redis' API */
+    lua_pushstring(lua, REDIS_API_NAME);
+    lua_pushnil(lua);
+    lua_settable(lua, -3);
+
+    /* create metatable */
+    lua_newtable(lua);
+    lua_pushstring(lua, "__index");
+    lua_pushvalue(lua, LUA_GLOBALSINDEX); /* push original globals */
+    lua_settable(lua, -3);
+    lua_pushstring(lua, "__newindex");
+    lua_pushvalue(lua, LUA_GLOBALSINDEX); /* push original globals */
+    lua_settable(lua, -3);
+
+    lua_setmetatable(lua, -2);
+
+    lua_pop(lua, 1); /* pops the global table */
+
+    return C_OK;
 }
 
 /*
@@ -137,6 +224,64 @@ static void luaEngineFreeFunction(void *engine_ctx, void *compiled_function) {
     zfree(f_ctx);
 }
 
+static int luaRegisterFunction(lua_State *lua) {
+    int argc = lua_gettop(lua);
+    if (argc < 2 || argc > 3) {
+        luaPushError(lua, "wrong number of arguments to redis.register_function");
+        return luaRaiseError(lua);
+    }
+    loadCtx *load_ctx = luaGetFromRegistry(lua, REGISTRY_LOAD_CTX_NAME);
+    if (!load_ctx) {
+        luaPushError(lua, "redis.register_function can only be called on FUNCTION LOAD command");
+        return luaRaiseError(lua);
+    }
+
+    if (!lua_isstring(lua, 1)) {
+        luaPushError(lua, "first argument to redis.register_function must be a string");
+        return luaRaiseError(lua);
+    }
+
+    if (!lua_isfunction(lua, 2)) {
+        luaPushError(lua, "second argument to redis.register_function must be a function");
+        return luaRaiseError(lua);
+    }
+
+    if (argc == 3 && !lua_isstring(lua, 3)) {
+        luaPushError(lua, "third argument to redis.register_function must be a string");
+        return luaRaiseError(lua);
+    }
+
+    size_t function_name_len;
+    const char *function_name = lua_tolstring(lua, 1, &function_name_len);
+    sds function_name_sds = sdsnewlen(function_name, function_name_len);
+
+    sds desc_sds = NULL;
+    if (argc == 3){
+        size_t desc_len;
+        const char *desc = lua_tolstring(lua, 3, &desc_len);
+        desc_sds = sdsnewlen(desc, desc_len);
+        lua_pop(lua, 1); /* pop out the description */
+    }
+
+    int lua_function_ref = luaL_ref(lua, LUA_REGISTRYINDEX);
+
+    luaFunctionCtx *lua_f_ctx = zmalloc(sizeof(*lua_f_ctx));
+    *lua_f_ctx = (luaFunctionCtx ) { .lua_function_ref = lua_function_ref, };
+
+    sds err = NULL;
+    if (functionLibCreateFunction(function_name_sds, lua_f_ctx, load_ctx->li, desc_sds, &err) != C_OK) {
+        sdsfree(function_name_sds);
+        if (desc_sds) sdsfree(desc_sds);
+        lua_unref(lua, lua_f_ctx->lua_function_ref);
+        zfree(lua_f_ctx);
+        luaPushError(lua, err);
+        sdsfree(err);
+        return luaRaiseError(lua);
+    }
+
+    return 0;
+}
+
 /* Initialize Lua engine, should be called once on start. */
 int luaEngineInitEngine() {
     luaEngineCtx *lua_engine_ctx = zmalloc(sizeof(*lua_engine_ctx));
@@ -144,6 +289,18 @@ int luaEngineInitEngine() {
 
     luaRegisterRedisAPI(lua_engine_ctx->lua);
 
+    /* Register the library commands table and fields and store it to registry */
+    lua_pushstring(lua_engine_ctx->lua, LIBRARY_API_NAME);
+    lua_newtable(lua_engine_ctx->lua);
+
+    lua_pushstring(lua_engine_ctx->lua, "register_function");
+    lua_pushcfunction(lua_engine_ctx->lua, luaRegisterFunction);
+    lua_settable(lua_engine_ctx->lua, -3);
+
+    luaRegisterLogFunction(lua_engine_ctx->lua);
+
+    lua_settable(lua_engine_ctx->lua, LUA_REGISTRYINDEX);
+
     /* Save error handler to registry */
     lua_pushstring(lua_engine_ctx->lua, REGISTRY_ERROR_HANDLER_NAME);
     char *errh_func =       "local dbg = debug\n"
@@ -163,11 +320,16 @@ int luaEngineInitEngine() {
     lua_pcall(lua_engine_ctx->lua,0,1,0);
     lua_settable(lua_engine_ctx->lua, LUA_REGISTRYINDEX);
 
-    /* save the engine_ctx on the registry so we can get it from the Lua interpreter */
-    luaSaveOnRegistry(lua_engine_ctx->lua, REGISTRY_ENGINE_CTX_NAME, lua_engine_ctx);
+    /* Save global protection to registry */
+    luaRegisterGlobalProtectionFunction(lua_engine_ctx->lua);
 
-    luaEnableGlobalsProtection(lua_engine_ctx->lua, 0);
+    /* Set global protection on globals */
+    lua_pushvalue(lua_engine_ctx->lua, LUA_GLOBALSINDEX);
+    luaSetGlobalProtection(lua_engine_ctx->lua);
+    lua_pop(lua_engine_ctx->lua, 1);
 
+    /* save the engine_ctx on the registry so we can get it from the Lua interpreter */
+    luaSaveOnRegistry(lua_engine_ctx->lua, REGISTRY_ENGINE_CTX_NAME, lua_engine_ctx);
 
     engine *lua_engine = zmalloc(sizeof(*lua_engine));
     *lua_engine = (engine) {