summaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
* Sync argv_len var back after command call in execCommand (#10900)Binbin2022-06-261-0/+1
| | | | | This is harmless, we only restore mstate to make sure we free the right pointer in freeClientMultiState, but it'll be nicer to also sync that argv_len var back.
* Replace regular array to array of structs to get rid of false sharing (#10892)Elvina Yakubova2022-06-261-3/+8
| | | | | | There are a lot of false sharing cache misses in line 4013 inside getIOPendingCount function. The reason is that elements of io_threads_pending array access the same cache line from different threads, so it is better to represent it as an array of structures with fields aligned to cache line size. This change should improve performance (in particular, it affects the latency metric, we saw up to 3% improvement).
* changing min,max in ZRANGE -> start stop (#10097)Steve Lorello2022-06-252-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | In 6.2.0 with the introduction of the REV subcommand in ZRANGE, there was a semantic shift in the arguments of ZRANGE when the REV sub-command is executed. Without the sub-command `min` and `max` (the old names of the arguments) are appropriate because if you put the min value and the max value in everything works fine. ```bash 127.0.0.1:6379> ZADD myset 0 foo (integer) 1 127.0.0.1:6379> ZADD myset 1 bar (integer) 1 127.0.0.1:6379> ZRANGE myset 0 inf BYSCORE 1) "foo" 2) "bar" ``` However - if you add the `REV` subcommand, ordering the arguments `min` `max` breaks the command: ```bash 127.0.0.1:6379> ZRANGE myset 0 inf BYSCORE REV (empty array) ``` why? because `ZRANGE` with the `REV` sub-command is expecting the `max` first and the `min` second (because it's a reverse range like `ZREVRANGEBYSCORE`): ```bash 127.0.0.1:6379> ZRANGE myset 0 inf BYSCORE REV (empty array) ```
* migrateGetSocket() cleanup.. (#5546)WuYunlong2022-06-231-1/+1
| | | | I think parameter c is only useful to get client reply. Besides, other commands' host and port parameters may not be the at index 1 and 2.
* When dirCreateIfMissing or openNewIncrAofForAppend fail, set ↵Binbin2022-06-231-1/+5
| | | | | | aof_lastbgrewrite_status to err (#10775) It will be displayed in the `aof_last_bgrewrite_status` field of the INFO command.
* command prompt help: add stdin config usage (#6313)(╯°□°）╯︵ uᴉǝssnH ɐɟɐʇsoW2022-06-231-0/+1
| | | Signed-off-by: Mostafa Hussein <mostafa.hussein91@gmail.com>
* Add bus_port argument in cluster-meet.json (#10304)Binbin2022-06-232-1/+17
| | | | | | | | In `CLUSTER MEET`, bus-port argument was added in 11436b1. For cluster announce ip / port implementation, part of the 4.0-RC1. And in #9389, we add a new cluster-port config and make cluster bus port configurable, part of the 7.0-RC1.
* fix redis-benchmark's bug: check if clients are created successfully in idle ↵judeng2022-06-221-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | mode (#10891) my maxclients config: ``` redis-cli config get maxclients 1) "maxclients" 2) "4064" ``` Before this bug was fixed, creating 4065 clients appeared to be successful, but only 4064 were actually created``` ``` ./redis-benchmark -c 4065 -I Creating 4065 idle connections and waiting forever (Ctrl+C when done) cients: 4065 ``` now : ``` ./redis-benchmark -c 4065 -I Creating 4065 idle connections and waiting forever (Ctrl+C when done) Error from server: ERR max number of clients reached ./redis-benchmark -c 4064 -I Creating 4064 idle connections and waiting forever (Ctrl+C when done) clients: 4064 ```
* Module API docs corrections (#10890)Viktor Söderqvist2022-06-211-4/+4
| | | | * Fix typo `RedisModule_CreatString` -> `RedisModule_CreateString` (multiple occurrences) * Make the markdown gen script change all `RM_` to `RedisModule_` even in code examples, etc.
* Fix crash on RM_Call with script mode. (#10886)Meir Shpilraien (Spielrein)2022-06-211-2/+11
| | | | | | | | | | | | | | | | | | | | | | The PR fixes 2 issues: ### RM_Call crash on script mode `RM_Call` can potentially be called from a background thread where `server.current_client` are not set. In such case we get a crash on `NULL` dereference. The fix is to check first if `server.current_client` is `NULL`, if it does we should verify disc errors and readonly replica as we do to any normal clients (no masters nor AOF). ### RM_Call block OOM commands when not needed Again `RM_Call` can be executed on a background thread using a `ThreadSafeCtx`. In such case `server.pre_command_oom_state` can be irrelevant and should not be considered when check OOM state. This cause OOM commands to be blocked when not necessarily needed. In such case, check the actual used memory (and not the cached value). Notice that in order to know if the cached value can be used, we check that the ctx that was used on the `RM_Call` is a ThreadSafeCtx. Module writer can potentially abuse the API and use ThreadSafeCtx on the main thread. We consider this as a API miss used.
* Outdated comments, replace COUNTER_INIT_VAL with LFU_INIT_VAL, fix typo (#10888)Vlad2022-06-211-5/+5
| | | `COUNTER_INIT_VAL` doesn't exist in the code anymore so we can replace it with `LFU_INIT_VAL` entries
* Set replicas' configEpoch to 0 when loaded from cluster configuration file ↵Bar Shaul2022-06-201-2/+4
| | | | | (#10798) * Changed clusterLoadConfig to set the config epoch of replica nodes to 0 when loaded.
* Optimize the performance of clusterSendPing for large clusters (#10624)judeng2022-06-202-14/+6
| | | Optimize the performance of clusterSendPing by improving speed of checking for duplicate items in gossip.
* Fsync directory while persisting AOF manifest, RDB file, and config file ↵Tian2022-06-206-4/+82
| | | | | | | | | | | | | | | | | | | | (#10737) The current process to persist files is `write` the data, `fsync` and `rename` the file, but a underlying problem is that the rename may be lost when a sudden crash like power outage and the directory hasn't been persisted. The article [Ensuring data reaches disk](https://lwn.net/Articles/457667/) mentions a safe way to update file should be: 1. create a new temp file (on the same file system!) 2. write data to the temp file 3. fsync() the temp file 4. rename the temp file to the appropriate name 5. fsync() the containing directory This commit handles CONFIG REWRITE, AOF manifest, and RDB file (both for persistence, and the one the replica gets from the master). It doesn't handle (yet), ACL SAVE and Cluster configs, since these don't yet follow this pattern.
* Fix a prototype inconsitency of `_serverAssert` between redisassert.h and ↵Masahiro Ide2022-06-191-1/+1
| | | | | redis.h (#10872) both redisassert.c and server.h + debug.c use const, but redisassert.h doesn't.
* optimize zset conversion on large ZRANGESTORE (#10789)Oran Agra2022-06-141-2/+4
| | | | | | when we know the size of the zset we're gonna store in advance, we can check if it's greater than the listpack encoding threshold, in which case we can create a skiplist from the get go, and avoid converting the listpack to skiplist later after it was already populated.
* Script that made modification will not break with unexpected NOREPLICAS ↵Oran Agra2022-06-141-0/+5
| | | | | | | | | | | error (#10855) If a script made a modification and then was interrupted for taking too long. there's a chance redis will detect that a replica dropped and would like to reject write commands with NOREPLICAS due to insufficient good replicas. returning an error on a command in this case breaks the script atomicity. The same could in theory happen with READONLY, MISCONF, but i don't think these state changes can happen during script execution.
* Allow ECHO in loading and stale modes (#10853)Oran Agra2022-06-142-1/+3
| | | | | | I noticed that scripting.tcl uses INFO from within a script and thought it's an overkill and concluded it's nicer to use another CMD_STALE command, decided to use ECHO, and then noticed it's not at all allowed in stale mode. probably overlooked at #6843
* Fix 2 comments in dict.c & redis-cli.c (#10860)hdyztmdqd2022-06-142-2/+2
| | | | | | | | | * fix comment in dict.c fix comment in dict.c * Fix comment in redis-cli.c Fix comment in redis-cli.c
* Throw -TRYAGAIN instead of -ASK on migrating nodes for multi-key commands ↵Huang Zhw2022-06-131-7/+14
| | | | | | | | | | | | | | when the node only has some of the keys (#9526) * In cluster getNodeByQuery when target slot is in migrating state and the slot lack some keys but have at least one key, should return TRYAGAIN. Before this commit, when a node is in migrating state and recevies multiple keys command, if some keys don't exist, the command emits an `ASK` redirection. After this commit, if some keys exist and some keys don't exist, the command emits a TRYAGAIN error. If all keys don't exist, the command emits an `ASK` redirection.
* redis-check-rdb add when_opcode check for module aux (#10859)Binbin2022-06-131-1/+8
| | | | | | | | | | | | | | | | | In #9199, we add a `goto eoferr` in when_opcode check, this means that if the when_opcode check fails, we will abort the rdb loading, but this not reflected in the rdb-check tool. So someone can modify when_opcode to make rdb load fail, but rdb-check will report OK. Just a cleanup or a code consistency issue. ``` serverLog: # Internal error in RDB reading offset 0, function at rdb.c:3055 -> bad when_opcode [offset 0] Checking RDB file dump.rdb [offset 109] \o/ RDB looks OK! \o/ ``` Plus a minor memory leak fix like #9860, note that it will exit immediately after the eoferr, so it is not strictly a leak, so it is also a small cleanup.
* Fix typo in replication.c (#10854)XiongDa2022-06-131-1/+1
|
* Correctly check for vm.overcommit_memory == 0 (#10841)Johannes Truschnigg2022-06-121-1/+1
| | | | | | | A regression caused by #10636 (released in 7.0.1) causes Redis startup warning about overcommit to be missing when needed and printed when not. Also, using atoi() to convert the string's value to an integer cannot discern between an actual 0 (zero) having been read, or a conversion error.
* Fix 3 comments in server.c (#10844)XiongDa2022-06-121-3/+3
|
* Fixed SET and BITFIELD commands being wrongly marked movablekeys (#10837)Binbin2022-06-123-47/+44
| | | | | | | | | | | | | | | | | | | | | | | | | The SET and BITFIELD command were added `get_keys_function` in #10148, causing them to be wrongly marked movablekeys in `populateCommandMovableKeys`. This was an unintended side effect introduced in #10148 (7.0 RC1) which could cause some clients an extra round trip for these commands in cluster mode. Since we define movablekeys as a way to determine if the legacy range [first, last, step] doesn't find all keys, then we need a completely different approach. The right approach should be to check if the legacy range covers all key-specs, and if none of the key-specs have the INCOMPLETE flag. This way, we don't need to look at getkeys_proc of VARIABLE_FLAG at all. Probably with the exception of modules, who may still not be using key-specs. In this PR, we removed `populateCommandMovableKeys` and put its logic in `populateCommandLegacyRangeSpec`. In order to properly serve both old and new modules, we must probably keep relying CMD_MODULE_GETKEYS, but do that only for modules that don't declare key-specs. For ones that do, we need to take the same approach we take with native redis commands. This approach was proposed by Oran. Fixes #10833 Co-authored-by: Oran Agra <oran@redislabs.com>
* Fix crash when overcommit_memory is inaccessible (#10848)Binbin2022-06-112-5/+5
| | | | | | | | | | | | When `/proc/sys/vm/overcommit_memory` is inaccessible, fp is NULL. `checkOvercommit` will return -1 without setting err_msg, and then the err_msg is used to print the log, crash the server. Set the err_msg variables to Null when declaring it, seems safer. And the overcommit_memory error log will print two "WARNING", like `WARNING WARNING overcommit_memory is set to 0!`, this PR also removes the second WARNING in `checkOvercommit`. Reported in #10846. Fixes #10846. Introduced in #10636 (7.0.1)
* Update some comments in stream command docs (#10821)Wen Hui2022-06-091-8/+7
| | | some small documentation fixes.
* Increment the stat_rdb_saves counter in SAVE command (#10827)Binbin2022-06-071-1/+4
| | | | | | | | | | | | | Currently, we only increment stat_rdb_saves in rdbSaveBackground, we should also increment it in the SAVE command. We concluded there's no need to increment when: 1. saving a base file for an AOF 2. when saving an empty rdb file to delete an old one 3. when saving to sockets (not creating a persistence / snapshot file) The stat counter was introduced in #10178 * fix a wrong comment in startSaving
* fix unused argument warning in ae_select.c (#10824)ls-20182022-06-071-0/+1
|
* Documentation fixes of `BITFIELD_RO` and `XINFO STREAM` (#10823)Bjorn Svensson2022-06-073-2/+7
| | | | | | | | | Correcting the introduction version of the command `BITFIELD_RO` Command added by commit: b3e4abf06e Add history info of the `FULL` modifier in `XINFO STREAM` Modifier was added by commit: 1e2aee3919 (Includes output from `./utils/generate-command-code.py`)
* Handle multiple_token flag in generate-command-help.rb (#10822)Binbin2022-06-074-10/+10
| | | | | | | | | | | | Currently generate-command.help.rb dose not handle the multiple_token flag, handle this flag in this PR. The format is the same as redis-cli rendering. ```diff - bitfield_ro key GET encoding offset [encoding offset ...] + bitfield_ro key GET encoding offset [GET encoding offset ...] ``` Re run generate-command-code.py which was forget in #10820. Also change the flag value from string to bool, like "true" to true
* Fix bitfield_ro documentation (#10820)ranshid2022-06-061-0/+1
| | | Current documentation only states a single GET token is needed which is not true. Marking the command with multiple_token.
* update help.h (#10818)Oran Agra2022-06-061-7/+7
| | | before releasing 7.0.1
* crash report instructions (#10816)Oran Agra2022-06-061-0/+2
| | | Trying to avoid people opening crash report issues about module crashes and ARM QEMU bugs.
* Split instantaneous_repl_total_kbps to instantaneous_input_repl_kbps and ↵DarrenJiang132022-06-062-6/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | instantaneous_output_repl_kbps. (#10810) A supplement to https://github.com/redis/redis/pull/10062 Split `instantaneous_repl_total_kbps` to `instantaneous_input_repl_kbps` and `instantaneous_output_repl_kbps`. ## Work: This PR: - delete 1 info field: - `instantaneous_repl_total_kbps` - add 2 info fields: - `instantaneous_input_repl_kbps / instantaneous_output_repl_kbps` ## Result: - master ``` total_net_input_bytes:26633673 total_net_output_bytes:21716596 total_net_repl_input_bytes:0 total_net_repl_output_bytes:18433052 instantaneous_input_kbps:0.02 instantaneous_output_kbps:0.00 instantaneous_input_repl_kbps:0.00 instantaneous_output_repl_kbps:0.00 ``` - slave ``` total_net_input_bytes:18433212 total_net_output_bytes:94790 total_net_repl_input_bytes:18433052 total_net_repl_output_bytes:0 instantaneous_input_kbps:0.00 instantaneous_output_kbps:0.05 instantaneous_input_repl_kbps:0.00 instantaneous_output_repl_kbps:0.00 ```
* Update cluster.c (#10773)Mixficsol2022-06-061-1/+1
| | | | On line 4068, redis has a logical nodeIsSlave(myself) on the outer if layer, which you can delete without having to repeat the decision
* some minor spelling/grammatical fixes to module.c (#10812)Steve Lorello2022-06-031-20/+20
|
* Update time independent string compare to use hash length (#9759)Madelyn Olson2022-06-032-37/+7
| | | * Update time independent string compare to use hash length
* rewrite alias config to original name (#10811)zhaozhao.zz2022-06-021-10/+5
| | | | | | | | | | | | | | | | | | | | | Redis 7 adds some new alias config like `hash-max-listpack-entries` alias `hash-max-ziplist-entries`. If a config file contains both real name and alias like this: ``` hash-max-listpack-entries 20 hash-max-ziplist-entries 20 ``` after set `hash-max-listpack-entries` to 100 and `config rewrite`, the config file becomes to: ``` hash-max-listpack-entries 100 hash-max-ziplist-entries 20 ``` we can see that the alias config is not modified, and users will get wrong config after restart. 6.0 and 6.2 doesn't have this bug, since they only have the `slave` word alias. Co-authored-by: Oran Agra <oran@redislabs.com>
* Only print final log when aof is loaded successfully (#10808)chenyang80942022-06-021-1/+2
| | | Skip the print on AOF_NOT_EXIST status.
* Fix bugs in CONFIG REWRITE, omitting rename-command and include lines, and ↵zhugezy2022-06-021-4/+15
| | | | | | | | | | | | | | | | inserting comments around module and acl configs (#10761) A regression from #10285 (redis 7.0). CONFIG REWRITE would put lines with: `include`, `rename-command`, `user`, `loadmodule`, and any module specific config in a comment. For ACL `user`, `loadmodule` and module specific configs would be re-inserted at the end (instead of updating existing lines), so the only implication is a messy config file full of comments. But for `rename-command` and `include`, the implication would be that they're now missing, so a server restart would lose them. Co-authored-by: Oran Agra <oran@redislabs.com>
* Expose script flags to processCommand for better handling (#10744)Oran Agra2022-06-018-96/+192
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The important part is that read-only scripts (not just EVAL_RO and FCALL_RO, but also ones with `no-writes` executed by normal EVAL or FCALL), will now be permitted to run during CLIENT PAUSE WRITE (unlike before where only the _RO commands would be processed). Other than that, some errors like OOM, READONLY, MASTERDOWN are now handled by processCommand, rather than the command itself affects the error string (and even error code in some cases), and command stats. Besides that, now the `may-replicate` commands, PFCOUNT and PUBLISH, will be considered `write` commands in scripts and will be blocked in all read-only scripts just like other write commands. They'll also be blocked in EVAL_RO (i.e. even for scripts without the `no-writes` shebang flag. This commit also hides the `may_replicate` flag from the COMMAND command output. this is a **breaking change**. background about may_replicate: We don't want to expose a no-may-replicate flag or alike to scripts, since we consider the may-replicate thing an internal concern of redis, that we may some day get rid of. In fact, the may-replicate flag was initially introduced to flag EVAL: since we didn't know what it's gonna do ahead of execution, before function-flags existed). PUBLISH and PFCOUNT, both of which because they have side effects which may some day be fixed differently. code changes: The changes in eval.c are mostly code re-ordering: - evalCalcFunctionName is extracted out of evalGenericCommand - evalExtractShebangFlags is extracted luaCreateFunction - evalGetCommandFlags is new code
* Fix broken protocol in MISCONF error, RM_Yield bugs, RM_Call(EVAL) OOM check ↵Oran Agra2022-06-018-29/+72
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | bug, and new RM_Call checks. (#10786) * Fix broken protocol when redis can't persist to RDB (general commands, not modules), excessive newline. regression of #10372 (7.0 RC3) * Fix broken protocol when Redis can't persist to AOF (modules and scripts), missing newline. * Fix bug in OOM check of EVAL scripts called from RM_Call. set the cached OOM state for scripts before executing module commands too, so that it can serve scripts that are executed by modules. i.e. in the past EVAL executed by RM_Call could have either falsely fail or falsely succeeded because of a wrong cached OOM state flag. * Fix bugs with RM_Yield: 1. SHUTDOWN should only accept the NOSAVE mode 2. Avoid eviction during yield command processing. 3. Avoid processing master client commands while yielding from another client * Add new two more checks to RM_Call script mode. 1. READONLY You can't write against a read only replica 2. MASTERDOWN Link with MASTER is down and `replica-serve-stale-data` is set to `no` * Add new RM_Call flag to let redis automatically refuse `deny-oom` commands while over the memory limit. * Add tests to cover various errors from Scripts, Modules, Modules calling scripts, and Modules calling commands in script mode. Add tests: * Looks like the MISCONF error was completely uncovered by the tests, add tests for it, including from scripts, and modules * Add tests for NOREPLICAS from scripts * Add tests for the various errors in module RM_Call, including RM_Call that calls EVAL, and RM_call in "eval mode". that includes: NOREPLICAS, READONLY, MASTERDOWN, MISCONF
* Moving client flags to a more cache friendly position within client struct ↵filipe oliveira2022-06-011-1/+1
| | | | | | | (#10697) Move the client flags to a more cache friendly position within the client struct we regain the lost 2% of CPU cycles since v6.2 ( from 630532.57 to 647449.80 ops/sec ). These are due to higher rate of calls to getClientType due to changes in #9166 and #10020
* Add module API RM_MallocUsableSize (#10795)menwen2022-05-312-0/+9
| | | | This allows the module to know the usable size of an allocation it made, rather than the consumed size.
* rename channel to shardchannel in sharded pubsub commands (#10738)zhaozhao.zz2022-05-317-20/+20
| | | since the sharded pubsub is different with pubsub, it's better to give users a hint to make it more clear.
* Adds isolated netstats for replication. (#10062)DarrenJiang132022-05-317-18/+75
| | | | | | | | | | | | | | | | | | | | | | | | The amount of `server.stat_net_output_bytes/server.stat_net_input_bytes` is actually the sum of replication flow and users' data flow. It may cause confusions like this: "Why does my server get such a large output_bytes while I am doing nothing? ". After discussions and revisions, now here is the change about what this PR brings (final version before merge): - 2 server variables to count the network bytes during replication, including fullsync and propagate bytes. - `server.stat_net_repl_output_bytes`/`server.stat_net_repl_input_bytes` - 3 info fields to print the input and output of repl bytes and instantaneous value of total repl bytes. - `total_net_repl_input_bytes` / `total_net_repl_output_bytes` - `instantaneous_repl_total_kbps` - 1 new API `rioCheckType()` to check the type of rio. So we can use this to distinguish between diskless and diskbased replication - 2 new counting items to keep network statistics consistent between master and slave - rdb portion during diskless replica. in `rdbLoadProgressCallback()` - first line of the full sync payload. in `readSyncBulkPayload()` Co-authored-by: Oran Agra <oran@redislabs.com>
* Sharded pubsub publish messagebulk as smessage (#10792)Harkrishn Patro2022-05-314-7/+11
| | | | | | | | | | | | | | | | | To easily distinguish between sharded channel message and a global channel message, introducing `smessage` (instead of `message`) as message bulk for sharded channel publish message. This is gonna be a breaking change in 7.0.1! Background: Sharded pubsub introduced in redis 7.0, but after the release we quickly realized that the fact that it's problematic that the client can't distinguish between normal (global) pubsub messages and sharded ones. This is important because the same connection can subscribe to both, but messages sent to one pubsub system are not propagated to the other (they're completely separate), so if one connection is used to subscribe to both, we need to assist the client library to know which message it got so it can forward it to the correct callback.
* Fix streamParseAddOrTrimArgsOrReply function minor comment issue (#10783)Wen Hui2022-05-311-1/+1
| | | | | | | When I read the source codes, I have no idea where the option "age" come from. Co-authored-by: Ubuntu <lucas.guang.yang1@huawei.com> Co-authored-by: guybe7 <guy.benoish@redislabs.com>
* Add readonly flag to EVAL_RO, EVALSHA_RO and FCALL_RO (#10728)Madelyn Olson2022-05-295-9/+17
| | | | * Add readonly flag to EVAL_RO, EVALSHA_RO and FCALL_RO * Require users to explicitly declare @scripting to get access to lua scripting.
View Lorry Controller Status