summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #10532 from oranagra/7.0-rc37.0-rc3Oran Agra2022-04-05171-1704/+5495
|\ | | | | Release 7.0 rc3
| * 7.0-RC3Oran Agra2022-04-052-3/+88
| |
| * Merge remote-tracking branch 'origin/unstable' into 7.0Oran Agra2022-04-05169-1701/+5407
| |\ |/ /
| * update static help.h from redis.io commands.json (#10529)Oran Agra2022-04-051-2/+12
| |
| * Update json command files so they only include syntax related information ↵Madelyn Olson2022-04-0510-55/+16
| | | | | | | | | | | | (#10398) The command json documents should just include information about the "arguments" and the "outputs". I removed all of the 'functional wording' so it's clear.
| * Functions: Move library meta data to be part of the library payload. (#10500)Meir Shpilraien (Spielrein)2022-04-0517-296/+425
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ## Move library meta data to be part of the library payload. Following the discussion on https://github.com/redis/redis/issues/10429 and the intention to add (in the future) library versioning support, we believe that the entire library metadata (like name and engine) should be part of the library payload and not provided by the `FUNCTION LOAD` command. The reasoning behind this is that the programmer who developed the library should be the one who set those values (name, engine, and in the future also version). **It is not the responsibility of the admin who load the library into the database.** The PR moves all the library metadata (engine and function name) to be part of the library payload. The metadata needs to be provided on the first line of the payload using the shebang format (`#!<engine> name=<name>`), example: ```lua #!lua name=test redis.register_function('foo', function() return 1 end) ``` The above script will run on the Lua engine and will create a library called `test`. ## API Changes (compare to 7.0 rc2) * `FUNCTION LOAD` command was change and now it simply gets the library payload and extract the engine and name from the payload. In addition, the command will now return the function name which can later be used on `FUNCTION DELETE` and `FUNCTION LIST`. * The description field was completely removed from`FUNCTION LOAD`, and `FUNCTION LIST` ## Breaking Changes (compare to 7.0 rc2) * Library description was removed (we can re-add it in the future either as part of the shebang line or an additional line). * Loading an AOF file that was generated by either 7.0 rc1 or 7.0 rc2 will fail because the old command syntax is invalid. ## Notes * Loading an RDB file that was generated by rc1 / rc2 **is** supported, Redis will automatically add the shebang to the libraries payloads (we can probably delete that code after 7.0.3 or so since there's no need to keep supporting upgrades from an RC build).
| * Cluster node name sanity check (#10391)bugwz2022-04-043-31/+57
| | | | | | | | | | | | * Limit cluster node id length for CLUSTER commands loading * Cluster node name sanity check for length and values Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
| * delete obsolete REDISMODULE_EXPERIMENTAL_API define in module demos (#10527)judeng2022-04-057-7/+0
| | | | | | This macro was recently removed from redismodule.h, so no longer needed.
| * Remove outdated utils/hashtable/rehashing.c (#10526)Binbin2022-04-042-156/+0
| | | | | | | | | | It is really outdated and i guessing no one will use it now. Remove the corresponding utils/hashtable directory. closes #10235
| * Fix null pointer subtraction warning (#10498)chenyang80942022-04-041-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The warning: ``` pqsort.c:106:7: warning: performing pointer subtraction with a null pointer has undefined behavior [-Wnull-pointer-subtraction] loop: SWAPINIT(a, es); ^~~~~~~~~~~~~~~ pqsort.c:65:47: note: expanded from macro 'SWAPINIT' #define SWAPINIT(a, es) swaptype = ((char *)a - (char *)NULL) % sizeof(long) || \ ``` Clang version: ``` Apple clang version 13.1.6 (clang-1316.0.21.2) Target: x86_64-apple-darwin21.3.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin ```
| * Fix mistake / outdated doc comment (#10521)Bofang Liu2022-04-041-1/+1
| | | | | | Unlike original reference code, Redis uses 64bit variable, so half the bits make it 32bit, not 16
| * Fix #10508, on error, pop function and error handler from Lua stack. (#10519)Meir Shpilraien (Spielrein)2022-04-042-0/+14
| | | | | | | | | | | | | | If, for some reason, Redis decides not to execute the script, we need to pop the function and error handler from Lua stack. Otherwise, eventually the Lua stack will explode. Relevant only for 7.0-rc1 and 7.0-rc2.
| * fix typos in aof.c (#10513)judeng2022-04-041-10/+10
| | | | | | | | | | function aofRewriteLimited in aof.c, deley->delay and NAX->MAX Co-authored-by: judeng <judeng@didiglobal.com>
| * Fix sentinel ACL test. Timing issue. (#10510)Moti Cohen2022-04-031-4/+11
| | | | | | | | | | Fix by replacing in test blind sleep with wait_for_condition(). Co-authored-by: moticless <moticless@github.com>
| * Turn into replica on SETSLOT (#10489)Viktor Söderqvist2022-04-023-3/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix race condition where node loses its last slot and turns into replica When a node has lost its last slot and finds out from the SETSLOT command before the cluster bus PONG from the new owner arrives. In this case, the node didn't turn itself into a replica of the new slot owner. This commit adds the same logic to the SETSLOT command as already exists for the cluster bus PONG processing. * Revert "Fix new / failing cluster slot migration test (#10482)" This reverts commit 0b21ef8d49c47a5dd47a0bcc0cf1b2c235f24fd0. In this test, the old slot owner finds out that it has lost its last slot in a nondeterministic way. Either the cluster bus PONG from the new slot owner and sometimes in a SETSLOT command from redis-cli. In both cases, the result should be the same and the old owner should turn itself into a replica of the new slot owner.
| * Fix failing moduleconfigs tests and memory leak (#10501)sundb2022-03-312-2/+8
| | | | | | | | | | | | | | Fix global `strval` not reset to NULL after being freed, causing a crash on alpine (most likely because the dynamic library loader doesn't init globals on reload) By the way, fix the memory leak of using `RedisModule_Free` to free `RedisModuleString`, and add a corresponding test.
| * Prevent replica failover during manual takeover test (#10499)Madelyn Olson2022-03-311-6/+18
| | | | | | | | | | | | | | | | | | During 11-manual-takeover.tcl, if the killing of the instances happens too slowly, one of the replicas might be able to promote itself. I'm not sure why it was slow, but it was observed taking 6 seconds which is enough time to do an election. I was able to verify the error locally by adding a small delay (1 second) during ASAN CI. A fix is just to disable automated failover until all the nodes are confirmed dead.
| * Fix cluster slot migration test (#10495)Binbin2022-03-301-0/+26
| | | | | | Fix three timing issues in the test
| * Use exit code 1 on error in redis-cli (#10468)Ozan Tezcan2022-03-301-8/+4
| | | | | | On error, redis-cli was returning `REDIS_ERR` on some cases by mistake. `REDIS_ERR` is `-1` which becomes `255` as exit code. This commit changes it and returns `1` on errors to be consistent.
| * Move restart_killed_instances and verify_sentinel_auto_discovery to utils ↵Binbin2022-03-303-23/+25
| | | | | | | | | | | | (#10497) Create a utils.tcl in sentinel/tests/includes, and move two procs to it. Allow sentinel test 08-hostname-conf run on its own.
| * Bump actions/checkout from 2 to 3 (#10390)dependabot[bot]2022-03-305-36/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
| * Bump actions/cache from 2 to 3 (#10463)dependabot[bot]2022-03-301-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Bumps [actions/cache](https://github.com/actions/cache) from 2 to 3. - [Release notes](https://github.com/actions/cache/releases) - [Commits](https://github.com/actions/cache/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
| * Module Configurations (#10285)Nick Chun2022-03-3013-233/+1530
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This feature adds the ability to add four different types (Bool, Numeric, String, Enum) of configurations to a module to be accessed via the redis config file, and the CONFIG command. **Configuration Names**: We impose a restriction that a module configuration always starts with the module name and contains a '.' followed by the config name. If a module passes "config1" as the name to a register function, it will be registered as MODULENAME.config1. **Configuration Persistence**: Module Configurations exist only as long as a module is loaded. If a module is unloaded, the configurations are removed. There is now also a minimal core API for removal of standardConfig objects from configs by name. **Get and Set Callbacks**: Storage of config values is owned by the module that registers them, and provides callbacks for Redis to access and manipulate the values. This is exposed through a GET and SET callback. The get callback returns a typed value of the config to redis. The callback takes the name of the configuration, and also a privdata pointer. Note that these only take the CONFIGNAME portion of the config, not the entire MODULENAME.CONFIGNAME. ``` typedef RedisModuleString * (*RedisModuleConfigGetStringFunc)(const char *name, void *privdata); typedef long long (*RedisModuleConfigGetNumericFunc)(const char *name, void *privdata); typedef int (*RedisModuleConfigGetBoolFunc)(const char *name, void *privdata); typedef int (*RedisModuleConfigGetEnumFunc)(const char *name, void *privdata); ``` Configs must also must specify a set callback, i.e. what to do on a CONFIG SET XYZ 123 or when loading configurations from cli/.conf file matching these typedefs. *name* is again just the CONFIGNAME portion, *val* is the parsed value from the core, *privdata* is the registration time privdata pointer, and *err* is for providing errors to a client. ``` typedef int (*RedisModuleConfigSetStringFunc)(const char *name, RedisModuleString *val, void *privdata, RedisModuleString **err); typedef int (*RedisModuleConfigSetNumericFunc)(const char *name, long long val, void *privdata, RedisModuleString **err); typedef int (*RedisModuleConfigSetBoolFunc)(const char *name, int val, void *privdata, RedisModuleString **err); typedef int (*RedisModuleConfigSetEnumFunc)(const char *name, int val, void *privdata, RedisModuleString **err); ``` Modules can also specify an optional apply callback that will be called after value(s) have been set via CONFIG SET: ``` typedef int (*RedisModuleConfigApplyFunc)(RedisModuleCtx *ctx, void *privdata, RedisModuleString **err); ``` **Flags:** We expose 7 new flags to the module, which are used as part of the config registration. ``` #define REDISMODULE_CONFIG_MODIFIABLE 0 /* This is the default for a module config. */ #define REDISMODULE_CONFIG_IMMUTABLE (1ULL<<0) /* Can this value only be set at startup? */ #define REDISMODULE_CONFIG_SENSITIVE (1ULL<<1) /* Does this value contain sensitive information */ #define REDISMODULE_CONFIG_HIDDEN (1ULL<<4) /* This config is hidden in `config get <pattern>` (used for tests/debugging) */ #define REDISMODULE_CONFIG_PROTECTED (1ULL<<5) /* Becomes immutable if enable-protected-configs is enabled. */ #define REDISMODULE_CONFIG_DENY_LOADING (1ULL<<6) /* This config is forbidden during loading. */ /* Numeric Specific Configs */ #define REDISMODULE_CONFIG_MEMORY (1ULL<<7) /* Indicates if this value can be set as a memory value */ ``` **Module Registration APIs**: ``` int (*RedisModule_RegisterBoolConfig)(RedisModuleCtx *ctx, char *name, int default_val, unsigned int flags, RedisModuleConfigGetBoolFunc getfn, RedisModuleConfigSetBoolFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata); int (*RedisModule_RegisterNumericConfig)(RedisModuleCtx *ctx, const char *name, long long default_val, unsigned int flags, long long min, long long max, RedisModuleConfigGetNumericFunc getfn, RedisModuleConfigSetNumericFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata); int (*RedisModule_RegisterStringConfig)(RedisModuleCtx *ctx, const char *name, const char *default_val, unsigned int flags, RedisModuleConfigGetStringFunc getfn, RedisModuleConfigSetStringFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata); int (*RedisModule_RegisterEnumConfig)(RedisModuleCtx *ctx, const char *name, int default_val, unsigned int flags, const char **enum_values, const int *int_values, int num_enum_vals, RedisModuleConfigGetEnumFunc getfn, RedisModuleConfigSetEnumFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata); int (*RedisModule_LoadConfigs)(RedisModuleCtx *ctx); ``` The module name will be auto appended along with a "." to the front of the name of the config. **What RM_Register[...]Config does**: A RedisModule struct now keeps a list of ModuleConfig objects which look like: ``` typedef struct ModuleConfig { sds name; /* Name of config without the module name appended to the front */ void *privdata; /* Optional data passed into the module config callbacks */ union get_fn { /* The get callback specificed by the module */ RedisModuleConfigGetStringFunc get_string; RedisModuleConfigGetNumericFunc get_numeric; RedisModuleConfigGetBoolFunc get_bool; RedisModuleConfigGetEnumFunc get_enum; } get_fn; union set_fn { /* The set callback specified by the module */ RedisModuleConfigSetStringFunc set_string; RedisModuleConfigSetNumericFunc set_numeric; RedisModuleConfigSetBoolFunc set_bool; RedisModuleConfigSetEnumFunc set_enum; } set_fn; RedisModuleConfigApplyFunc apply_fn; RedisModule *module; } ModuleConfig; ``` It also registers a standardConfig in the configs array, with a pointer to the ModuleConfig object associated with it. **What happens on a CONFIG GET/SET MODULENAME.MODULECONFIG:** For CONFIG SET, we do the same parsing as is done in config.c and pass that as the argument to the module set callback. For CONFIG GET, we call the module get callback and return that value to config.c to return to a client. **CONFIG REWRITE**: Starting up a server with module configurations in a .conf file but no module load directive will fail. The flip side is also true, specifying a module load and a bunch of module configurations will load those configurations in using the module defined set callbacks on a RM_LoadConfigs call. Configs being rewritten works the same way as it does for standard configs, as the module has the ability to specify a default value. If a module is unloaded with configurations specified in the .conf file those configurations will be commented out from the .conf file on the next config rewrite. **RM_LoadConfigs:** `RedisModule_LoadConfigs(RedisModuleCtx *ctx);` This last API is used to make configs available within the onLoad() after they have been registered. The expected usage is that a module will register all of its configs, then call LoadConfigs to trigger all of the set callbacks, and then can error out if any of them were malformed. LoadConfigs will attempt to set all configs registered to either a .conf file argument/loadex argument or their default value if an argument is not specified. **LoadConfigs is a required function if configs are registered. ** Also note that LoadConfigs **does not** call the apply callbacks, but a module can do that directly after the LoadConfigs call. **New Command: MODULE LOADEX [CONFIG NAME VALUE] [ARGS ...]:** This command provides the ability to provide startup context information to a module. LOADEX stands for "load extended" similar to GETEX. Note that provided config names need the full MODULENAME.MODULECONFIG name. Any additional arguments a module might want are intended to be specified after ARGS. Everything after ARGS is passed to onLoad as RedisModuleString **argv. Co-authored-by: Madelyn Olson <madelyneolson@gmail.com> Co-authored-by: Madelyn Olson <matolson@amazon.com> Co-authored-by: sundb <sundbcn@gmail.com> Co-authored-by: Madelyn Olson <34459052+madolson@users.noreply.github.com> Co-authored-by: Oran Agra <oran@redislabs.com> Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>
| * command json files cleanups (#10473)Binbin2022-03-3032-218/+301
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This PR do some command json files cleanups: 1. Add COMMAND TIPS to some commands - command-docs: add `NONDETERMINISTIC_OUTPUT_ORDER` - command-info: add `NONDETERMINISTIC_OUTPUT_ORDER` - command-list: add `NONDETERMINISTIC_OUTPUT_ORDER` - command: change `NONDETERMINISTIC_OUTPUT` to `NONDETERMINISTIC_OUTPUT_ORDER` - function-list: add `NONDETERMINISTIC_OUTPUT_ORDER` - latency-doctor: add `NONDETERMINISTIC_OUTPUT`, `REQUEST_POLICY:ALL_NODES` and `RESPONSE_POLICY:SPECIAL` - latency-graph: add `NONDETERMINISTIC_OUTPUT`, `REQUEST_POLICY:ALL_NODES` and `RESPONSE_POLICY:SPECIAL` - memory-doctor: add `REQUEST_POLICY:ALL_SHARDS` and `RESPONSE_POLICY:SPECIAL` - memory-malloc-stats: add `REQUEST_POLICY:ALL_SHARDS` and `RESPONSE_POLICY:SPECIAL` - memory-purge: add `REQUEST_POLICY:ALL_SHARDS` and `RESPONSE_POLICY:ALL_SUCCEEDED` - module-list: add `NONDETERMINISTIC_OUTPUT_ORDER` - msetnx: add `REQUEST_POLICY:MULTI_SHARD` and `RESPONSE_POLICY:AGG_MIN` - object-refcount: add `NONDETERMINISTIC_OUTPUT` 3. Only (mostly) indentation and formatting changes: - cluster-shards - latency-history - pubsub-shardchannels - pubsub-shardnumsub - spublish - ssubscribe - sunsubscribe 4. add doc_flags (DEPRECATED) to cluster-slots, replaced_by `CLUSTER SHARDS` in 7.0 5. command-getkeysandflags: a better summary (the old one is copy from command-getkeys) 6. adjustment of command parameter types - `port` is integer, not string (`MIGRATE`, `REPLICAOF`, `SLAVEOF`) - `replicationid` is string, not integer (`PSYNC`) - `pattern` is pattern, not string (`PUBSUB CHANNELS`, `SENTINEL RESET`, `SORT`, `SORT_RO`)
| * fix daily.yaml skip filters (#10490)Oran Agra2022-03-291-49/+49
| | | | | | | | | | | | | | | | | | * missing parenthesis meant that the ubuntu and centos jobs were not skipped * the recently divided freebsd, macos, and valgrind jobs, which are now split into distict jobs for redis, modules, sentinel, cluster. were all executed, producing a build, but not running anything. now they're filtered at the job level * iothreads was missing from the skip list defaults, so was not skipped
| * redis-cli: Do DNS lookup before sending CLUSTER MEET (#10436)Viktor Söderqvist2022-03-291-2/+22
| | | | | | Affects `--cluster create` and `--cluster add-node`.
| * show cluster.links in MEMORY STATS (#10302)zhaozhao.zz2022-03-291-1/+4
| |
| * improve malloc efficiency for cluster slots_info_pairs (#10488)Oran Agra2022-03-292-41/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit improve malloc efficiency of the slots_info_pairs mechanism in cluster.c by changing adlist into an array being realloced with greedy growth mechanism Recently the cluster tests are consistently failing when executed with ASAN in the CI. I tried to track down the commit that started it, and it appears to be #10293. Looking at the commit, i realize it didn't affect this test / flow, other than the replacement of the slots_info_pairs from sds to list. I concluded that what could be happening is that the slot range is very fragmented, and that results in many allocations. with sds, it results in one allocation and also, we have a greedy growth mechanism, but with adlist, we just have many many small allocations. this probably causes stress on ASAN, and causes it to be slow at termination.
| * introduce MAX_D2STRING_CHARS instead of 128 const (#10487)Oran Agra2022-03-285-13/+52
| | | | | | | | | | | | | | | | | | There are a few places that use a hard coded const of 128 to allocate a buffer for d2string. Replace these with a clear macro. Note that In theory, converting double into string could take as much as nearly 400 chars, but since d2string uses `%g` and not `%f`, it won't pass some 40 chars. unrelated: restore some changes to auto generated commands.c that got accidentally reverted in #10293
| * Make redis-cli --cluster help output to stdout (#10485)Binbin2022-03-281-14/+14
| | | | | | | | redis-cli --cluster help currently outputs the help on stderr. This is similar to #9124
| * Fix sentinel test SDOWN is triggered by non-responding instance (#10484)Moti Cohen2022-03-281-3/+6
| | | | | | | | | | A timing issue of debug sleep master isn't long enough to ensure that master is down and let the test identify it. Replaced the code with suspend PID until verified master-is-down.
| * fix crash in debug protocol push (#10483)DarrenJiang132022-03-282-1/+8
| | | | | | | | | | | | | | | | a missing of resp3 judgement which may lead to the crash using `debug protocol push` introduced in #9235 Similar improvement in RM_ReplySetAttributeLength in case the module ignored the error that returned from RM_ReplyWithAttribute. Co-authored-by: Oran Agra <oran@redislabs.com>
| * Fix new / failing cluster slot migration test (#10482)Oran Agra2022-03-271-8/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | #10381 fixed an issue in `redis-cli --cluster reshard` that used to fail it (redis-cli) because of a race condition. the race condition is / was that when moving the last slot from a node, sometimes the PONG messages delivering the configuration change arrive to that node before the SETSLOT arrives to it, and it becomes a replica. other times the the SETSLOT arrive first, and then PONG **doesn't** demote it. **however**, the PR also added a new test that suffers from exactly the same race condition, and the tests started failing a lot. The fact is (if i understand it correctly), that this test (the one being deleted here), isn't related to the fix that PR fixed (which was to fix redis-cli). The race condition in the cluster code still happens, and as long as we don't solve it, there's no reason to test it. For now, even if my understandings are wrong, i'm gonna delete that failing test, since as far as i understand, #10381 didn't introduce any new risks for that matter (which are gonna be compromised by removing this check), this race existed since forever, and still exists, and the fact that redis-cli is now immune to it is still being tested. Additional work should be carried to fix it, and i live it for other PRs to handle.
| * Fix Sentinel reconnect test following ACL change (#10480)Moti Cohen2022-03-271-9/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Replace condition with wait_for_condition On "Verify sentinel that restarted failed to reconnect master after ACL change" The reason we reach it, is because the test is fast enough to modify ACL and test sentinel connection status with the server - before its scheduled operation got the chance to update connection status with the server: ``` /* Perform scheduled operations for the specified Redis instance. */ void sentinelHandleRedisInstance(sentinelRedisInstance *ri) { /* ========== MONITORING HALF ============ */ /* Every kind of instance */ sentinelReconnectInstance(ri); ```
| * Cleanups in redis.conf (#10452)Binbin2022-03-271-7/+7
| | | | | | | | | | | | | | | | | | | | Did some cleanups: 1. local local typo 2. replace the only slave word in the file 3. add FUNCTION FLUSH to `lazyfree-lazy-user-flush` description 4. thought it would be better to use these, there are actually "four" options 5. the the typo 6. remove a extra space 7. change comment next to `activedefrag no` to match the default value
| * Add appendonlydir-* in utils/create-cluster/.gitignore (#10471)小令童鞋2022-03-261-0/+1
| |
| * optimize(remove) usage of client's pending_querybuf (#10413)zhaozhao.zz2022-03-256-61/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To remove `pending_querybuf`, the key point is reusing `querybuf`, it means master client's `querybuf` is not only used to parse command, but also proxy to sub-replicas. 1. add a new variable `repl_applied` for master client to record how many data applied (propagated via `replicationFeedStreamFromMasterStream()`) but not trimmed in `querybuf`. 2. don't sdsrange `querybuf` in `commandProcessed()`, we trim it to `repl_applied` after the whole replication pipeline processed to avoid fragmented `sdsrange`. And here are some scenarios we cannot trim to `qb_pos`: * we don't receive complete command from master * master client blocked because of client pause * IO threads operate read, master client flagged with CLIENT_PENDING_COMMAND In these scenarios, `qb_pos` points to the part of the current command or the beginning of next command, and the current command is not applied yet, so the `repl_applied` is not equal to `qb_pos`. Some other notes: * Do not do big arg optimization on master client, since we can only sdsrange `querybuf` after data sent to replicas. * Set `qb_pos` and `repl_applied` to 0 when `freeClient` in `replicationCacheMaster`. * Rewrite `processPendingCommandsAndResetClient` to `processPendingCommandAndInputBuffer`, let `processInputBuffer` to be called successively after `processCommandAndResetClient`.
| * Split daily CI into smaller chunks (#10469)Oran Agra2022-03-221-11/+227
| | | | | | | | this should aid find the CI issues with freebsd and macos runs, and also get faster results from valgrind and tls
| * crash log, print killer pid only when si_code is SI_USER (#10454)Oran Agra2022-03-221-1/+1
| | | | | | | | Avoid printing "Killed by PID" when si_code != SI_USER. Apparently SI_USER isn't always set to 0. e.g. on Mac it's 0x10001 and the check that did <= was wrong.
| * Add new RM_Call flags for script mode, no writes, and error replies. (#10372)Meir Shpilraien (Spielrein)2022-03-2214-57/+256
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The PR extends RM_Call with 3 new capabilities using new flags that are given to RM_Call as part of the `fmt` argument. It aims to assist modules that are getting a list of commands to be executed from the user (not hard coded as part of the module logic), think of a module that implements a new scripting language... * `S` - Run the command in a script mode, this means that it will raise an error if a command which are not allowed inside a script (flaged with the `deny-script` flag) is invoked (like SHUTDOWN). In addition, on script mode, write commands are not allowed if there is not enough good replicas (as configured with `min-replicas-to-write`) and/or a disk error happened. * `W` - no writes mode, Redis will reject any command that is marked with `write` flag. Again can be useful to modules that implement a new scripting language and wants to prevent any write commands. * `E` - Return errors as RedisModuleCallReply. Today the errors that happened before the command was invoked (like unknown commands or acl error) return a NULL reply and set errno. This might be missing important information about the failure and it is also impossible to just pass the error to the user using RM_ReplyWithCallReply. This new flag allows you to get a RedisModuleCallReply object with the relevant error message and treat it as if it was an error that was raised by the command invocation. Tests were added to verify the new code paths. In addition small refactoring was done to share some code between modules, scripts, and `processCommand` function: 1. `getAclErrorMessage` was added to `acl.c` to unified to log message extraction from the acl result 2. `checkGoodReplicasStatus` was added to `replication.c` to check the status of good replicas. It is used on `scriptVerifyWriteCommandAllow`, `RM_Call`, and `processCommand`. 3. `writeCommandsGetDiskErrorMessage` was added to `server.c` to get the error message on persistence failure. Again it is used on `scriptVerifyWriteCommandAllow`, `RM_Call`, and `processCommand`.
| * Fix an off by one error in zzlStrtod (#10465)yiyuaner2022-03-221-2/+2
| | | | | | | | When vlen = sizeof(buf), the statement buf[vlen] = '\0' accessing the buffer buf is an off by one error.
| * config rewrite enhancement, in case of line longer than 1024 (#8009)zhaozhao.zz2022-03-222-10/+27
| | | | | | | | | | | | | | | | When rewrite the config file, we need read the old config file first, but the CONFIG_MAX_LEN is 1024, so if some lines are longer than it, it will generate a wrong config file, and redis cannot reboot from the new config file. Rename CONFIG_MAX_LINE to CONFIG_READ_LEN
| * config rewrite failed save errno in case of being tainted (#10461)郭伟光2022-03-211-2/+4
| | | | | | | | | | errno would be potentially tainted during the serverLog() by the file io functions, such as fopen and fflush.
| * Use exit code 1 if redis-cli fails to connect (#10438)Ozan Tezcan2022-03-211-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | Use exit code 1 if redis-cli fails to connect. Before https://github.com/redis/redis/pull/10382/, on a connection failure, exit code would be 1. After this PR, whether connection is established or not, `noninteractive()` return value is used as the exit code. On a failure, this function returns `REDIS_ERR` which is `-1`. It becomes `255` as exit codes are between `0-255`. There is nothing wrong by returning 1 or 255 on failure as far as I know but it'll break things that expect to see 1 as exit code on a connection failure. This is also how we realized the issue. With this PR, changing behavior back to using 1 as exit code to preserve backward compatibility.
| * BITSET and BITFIELD SET should propagate even if just length changed (#10459)guybe72022-03-212-11/+19
| | | | | | | | | | Bug introduced in #9403, caused inconsistency between master and replica in case just the length (i.e. set a high-index bit to 0) changed.
| * Increase function tests timeout (#10458)Meir Shpilraien (Spielrein)2022-03-211-9/+9
| | | | | | | | Increase function tests timeout to avoid false failures on slow systems.
| * Fix timing issue in shards test and fix displayed TLS port (#10450)Madelyn Olson2022-03-202-7/+9
| |
| * unblockClient: avoid to reset client when the client was shutdown-blocked ↵郭伟光2022-03-202-1/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#10440) fix #10439. see https://github.com/redis/redis/pull/9872 When executing SHUTDOWN we pause the client so we can un-pause it if the shutdown fails. this could happen during the timeout, if the shutdown is aborted, but could also happen from withing the initial `call()` to shutdown, if the rdb save fails. in that case when we return to `call()`, we'll crash if `c->cmd` has been set to NULL. The call stack is: ``` unblockClient(c) replyToClientsBlockedOnShutdown() cancelShutdown() finishShutdown() prepareForShutdown() shutdownCommand() ``` what's special about SHUTDOWN in that respect is that it can be paused, and then un-paused before the original `call()` returns. tests where added for both failed shutdown, and a followup successful one.
| * Restore ::singledb after cluster test (#10441)sundb2022-03-182-1/+7
| | | | | | | | | | When ::singledb is 0, we will use db 9 for the test db. Since ::singledb is set to 1 in the cluster-related tests, but not restored, some subsequent tests associated with db 9 will fail.
| * fix wrong comment about cluster_announce_hostname (#10274)carlosfu2022-03-161-1/+1
| |