summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Redis 6.0.116.0.11Oran Agra2021-02-232-2/+30
|
* Fix integer overflow (CVE-2021-21309). (#8522)Yossi Gottlieb2021-02-233-1/+14
| | | | | | | | | | | On 32-bit systems, setting the proto-max-bulk-len config parameter to a high value may result with integer overflow and a subsequent heap overflow when parsing an input bulk (CVE-2021-21309). This fix has two parts: Set a reasonable limit to the config parameter. Add additional checks to prevent the problem in other potential but unknown code paths. (cherry picked from commit d32f2e9999ce003bad0bd2c3bca29f64dcce4433)
* Update CI on Ubuntu to tcl8.6 (since 20.04 is now used) (#8460)Oran Agra2021-02-232-6/+6
| | | | | | | Github started shifting some repositoreis to use ubuntu 20.04 by default tcl8.5 is missing in these, but 8.6 exists in both 20.04 and 18.04 (cherry picked from commit 018f7b7378d576ef80f2af11293713cc1a26da53)
* RM_ZsetRem: Delete key if empty (#8453)Viktor Söderqvist2021-02-235-0/+50
| | | | | | | | | | | | Without this fix, RM_ZsetRem can leave empty sorted sets which are not allowed to exist. Removing from a sorted set while iterating seems to work (while inserting causes failed assetions). RM_ZsetRangeEndReached is modified to return 1 if the key doesn't exist, to terminate iteration when the last element has been removed. (cherry picked from commit aea6e71ef82701e07177744e600e1ef20d60b7d0)
* Fix random element selection for large hash tables. (#8133)Greg Femec2021-02-239-13/+320
| | | | | | | | | When a database on a 64 bit build grows past 2^31 keys, the underlying hash table expands to 2^32 buckets. After this point, the algorithms for selecting random elements only return elements from half of the available buckets because they use random() which has a range of 0 to 2^31 - 1. This causes problems for eviction policies which use dictGetSomeKeys or dictGetRandomKey. Over time they cause the hash table to become unbalanced because, while new keys are spread out evenly across all buckets, evictions come from only half of the available buckets. Eventually this half of the table starts to run out of keys and it takes longer and longer to find candidates for eviction. This continues until no more evictions can happen. This solution addresses this by using a 64 bit PRNG instead of libc random(). Co-authored-by: Greg Femec <gfemec@google.com> (cherry picked from commit 266949c7fcfab9d10f81314fd7480a00638ced80)
* Client tracking tracking-redir-broken push len is 2 not 3 (#8456)Huang Zw2021-02-231-1/+1
| | | | | | | | When redis responds with tracking-redir-broken push message (RESP3), it was responding with a broken protocol: an array of 3 elements, but only pushes 2 elements. (cherry picked from commit f687ac0c32e3857aff56ddc4711be18dd87c336c)
* redis-benchmark: Fix broken protocol when used with -a or --dbnum (#8486)filipe oliveira2021-02-231-0/+3
| | | | | | | | | | | Fix the pointers to the slot hash tags in the case of prefixed commands usage i.e. AUTH / SELECT It adjusts the pointers to the slot hash tags in the case of prefixed commands usage as soon as we get the 1st reply (same like we already did for the random strings within the command ) (cherry picked from commit efccd6353bacec9addc7278292fd8c3108d116e4)
* XINFO should use lookupKeyReadOrReply (#8436)zhaozhao.zz2021-02-231-1/+1
| | | | | | | This bug would have let users observe logically expired keys on replicas and during CLIENT PAUSE WRITE. (cherry picked from commit f2a5fe36789b50dab76851233966aaac44f9048c)
* Avoid assertions when testing arm64 cow bug. (#8405)Yossi Gottlieb2021-02-231-33/+68
| | | | | | | | | | At least in one case the arm64 cow kernel bug test triggers an assert, which is a problem because it cannot be ignored like cases where the bug is found. On older systems (Linux <4.5) madvise fails because MADV_FREE is not supported. We treat these failures as an indication the system is not affected. Fixes #8351, #8406 (cherry picked from commit 3a5049042ac06b6ed5e526f331d5378bf7c7b7ed)
* CONFIG REWRITE should honor umask settings. (#8371)Yossi Gottlieb2021-02-233-1/+8
| | | | | | | | | | Fixes a regression introduced due to a new (safer) way of rewriting configuration files. In the past the file was simply overwritten (same inode), but now Redis creates a new temporary file and later renames it over the old one. The temp file typically gets created with 0600 permissions so we later fchmod it to fix that. Unlike open with O_CREAT, fchmod doesn't consider umask so we have to do that explicitly. Fixes #8369 (cherry picked from commit b548ffabbecca073e241882c22192f682a086242)
* Fix firstkey,lastkey,step in COMMAND command for some commands (#8367)guybe72021-02-231-6/+6
| | | | | | | | | | | | | The output for COMMAND command was wrong for some commands. clients can use firstkey,lastkey,step to find (some) key name arguments, and the "movablekeys" flag to know that they can't know all (or any) of the key name arguments. These commands had the wrong output: 1. GEORADIUS*_RO used to have "movablekeys" (which it doesn't really need) 2. XREAD and XREADGROUP used to have (1,1,1). but that's completely wrong. 3. Z*STORE used to have (0,0,0) but it can at lest give the index of the dstkey (1,1,1) (cherry picked from commit baf92f3f1c15cf188fafaf802202da1c54caa709)
* Remove race condition and consistency issues with client pause and threaded ↵Madelyn Olson2021-02-223-10/+37
| | | | | | | | | | IO (#8520) clientsArePaused isn't thread safe because it has a side effect of attempting to unpause, which may cause multiple threads concurrently updating the unblocked_clients global list. This change resolves this issue by no longer postponing client for threaded reads when clients are paused and then skipping the check for client paused for threaded reads, in case one is postponed and then clients are paused. (I don't think this is strictly possible, but being defensive seems better here)
* Redis 6.0.106.0.10Oran Agra2021-01-122-2/+74
|
* Improve stability of new CSC eviction test (#8160)Oran Agra2021-01-121-2/+14
| | | | | | | | | | | | | | c4fdf09c0 added a test that now fails with valgrind it fails for two resons: 1) the test samples the used memory and then limits the maxmemory to that value, but it turns out this is not atomic and on slow machines the background cron process that clean out old query buffers reduces the memory so that the setting doesn't cause eviction. 2) the dbsize was tested late, after reading some invalidation messages by that time more and more keys got evicted, partially draining the db. this is not the focus of this fix (still a known limitation) (cherry picked from commit a102b21d178453247078b355a47762bfc121ac20)
* fix race in cluster transactions test (#8312)Oran Agra2021-01-121-0/+2
| | | | | | we didn't wait for the commands executed on the master to reach the replica. (cherry picked from commit 4f8458d8d6c59a47c6ffbb1ac88c69854ccf1592)
* Fix cluster diskless load swapdb test (#8308)Oran Agra2021-01-122-5/+12
| | | | | | | | | | | | | | The test was trying to wait for the replica to start loading the rdb from the master before it kills the master, but it was actually waiting for ROLE to be in "sync" mode, which corresponds to REPL_STATE_TRANSFER that starts before the actual loading starts. now instead it waits for the loading flag to be set. Besides, the test was dependent on the previous configuration of the servers, relying on the fact the replica is configured to persist (either RDB of AOF), now it is set explicitly. (cherry picked from commit 264953871b97bcc3d663cd006578c151a13b48ca)
* Fix overflow of rdbWriteRaw return value (#8306)Bob Li2021-01-121-1/+1
| | | | | | | Saving string of more than 2GB to the RDB file, can result in corrupt RDB, or failure in rdbSave. S (cherry picked from commit 542455ce2afbaffc065afbfdf6578399b8faecda)
* Modules: Fix an integer sign bug in moduleTimerHandler (#8131)guybe72021-01-121-3/+7
| | | | | | bug was introduced in 1a91a2700b (cherry picked from commit 2f41a3856845265ffc6cc3a35524883a8690cff7)
* Allow '\0' inside of result of sdscatvprintf, and efficiency improvements ↵Egor Seredin2021-01-121-6/+26
| | | | | | | | | | | | | (#6260) This will allow to use: RedisModule_CreateStringPrintf(ctx, "%s %c %s", "string1", 0, "string2"); On large string, the previous code would incrementally retry to double the output buffer. now it uses the the return value of snprintf and grows to the right size in one step. and also avoids an excessive strlen in sdscat at the end. (cherry picked from commit f4ca3d8757d6abb3536610ddb7b9ab3ad39e81df)
* Modules: Improve timer accuracy (#7987)guybe72021-01-121-2/+14
| | | | | | | | | | The bug occurs when 'callback' re-registers itself to a point in the future and the execution time in non-negligible: 'now' refers to time BEFORE callback was executed and is used to calculate 'next_period'. We must get the actual current time when calculating 'next_period' (cherry picked from commit 1a91a2700b24211e90c695d3fdbbfe7e8d75dbe4)
* Moved RMAPI_FUNC_SUPPORTED location such that it will be visible to modules ↵Meir Shpilraien (Spielrein)2021-01-122-2/+8
| | | | | | | | | (#8037) The RMAPI_FUNC_SUPPORTED was defined in the wrong place on redismodule.h and was not visible to modules. (cherry picked from commit 97d647a13920c230a27685992a23679b44b73550)
* raspberry build fix. (#8095)David CARLIER2021-01-121-2/+2
| | | | | | __ILP32__ is 32 bits ABI and does not imply x86, this patch resolves this. (cherry picked from commit 0719388cfb1a79160204314beb1de1f9c29a3684)
* Fix crash log output on ARM. (#8020)Yossi Gottlieb2021-01-121-1/+1
| | | | (cherry picked from commit 7e4325cbc9af96d55f0b61c7648233a4473e8c93)
* Fix cluster access to unaligned memory (SIGBUS on old ARM) (#7958)Oran Agra2021-01-122-8/+25
| | | | | | | | | | Turns out this was broken since version 4.0 when we added sds size classes. The cluster code uses sds for the receive buffer, and then casts it to a struct and accesses a 64 bit variable. This commit replaces the use of sds with a simple reallocated buffer. (cherry picked from commit 380f6048e0bbc762f12fa50b57b73cf29049f967)
* prevent client tracking from causing feedback loop in performEvictions (#8100)Oran Agra2021-01-123-1/+60
| | | | | | | | | | | | | | When client tracking is enabled signalModifiedKey can increase memory usage, this can cause the loop in performEvictions to keep running since it was measuring the memory usage impact of signalModifiedKey. The section that measures the memory impact of the eviction should be just on dbDelete, excluding keyspace notification, client tracking, and propagation to AOF and replicas. This resolves part of the problem described in #8069 p.s. fix took 1 minute, test took about 3 hours to write. (cherry picked from commit c4fdf09c0584a3cee32b92f01b7958c72776aedc)
* White space tweaks and skip categories already appliedMadelyn Olson2021-01-121-3/+12
| | | | (cherry picked from commit d310beb4170ebbc8985ae120ee301f9213d33e39)
* Further improved ACL algorithm for picking categoriesMadelyn Olson2021-01-122-31/+62
| | | | (cherry picked from commit 411bcf1a41d2758823d17e0864ef45e5f3948b7a)
* Swapdb should make transaction fail if there is any client watching keys (#8239)Yang Bodong2021-01-124-24/+68
| | | | | | | | | | | | | This PR not only fixes the problem that swapdb does not make the transaction fail, but also optimizes the FLUSHALL and FLUSHDB command to set the CLIENT_DIRTY_CAS flag to avoid unnecessary traversal of clients. FLUSHDB was changed to first iterate on all watched keys, and then on the clients watching each key. Instead of iterating though all clients, and for each iterate on watched keys. Co-authored-by: Oran Agra <oran@redislabs.com> (cherry picked from commit 10f94b0ab12f9315939dcccf39d64b9388c0c7fa)
* Adds 'use-memory' to GEORADIUS[BYMEMBER] (#8107)Itamar Haber2021-01-121-2/+2
| | | | | | Partial resolution for #6860, item 7 (cherry picked from commit feba7cbf4d75f7822f34bd67a56b2ef4a755eff9)
* fix valgrind warning created by recent pidfile fix (#8235)Oran Agra2021-01-121-0/+1
| | | | | | | | This isn't a leak, just an warning due to unreachable allocation on the fork child. Problem created by 92a483b (cherry picked from commit 2426aaa099e5dfee29cce17af39298d0ce14cc2a)
* Fix issue where fork process deletes the parent pidfile (#8231)Meir Shpilraien (Spielrein)2021-01-122-1/+5
| | | | | | | | | | Turns out that when the fork child crashes, the crash log was deleting the pidfile from the disk (although the parent is still running. Now we set the pidfile of the fork process to NULL so the fork process will never deletes it. (cherry picked from commit 92a483bca2df734aff5caada6c23409ed6256773)
* Sentinel: add missing calls for sentinelflushconfig when config master at ↵Wen Hui2021-01-121-9/+8
| | | | | | runtime (#8229) (cherry picked from commit 781d7b0d9b51d1d80097c1f6ddec6270c1532ede)
* fix memory leak in processInlineBuffer error handling code (#8295)Wen Hui2021-01-121-0/+1
| | | | | | This code path is normally executed only when v6.0 and above replicates from v2.4 (cherry picked from commit cfcd0fa6f71af4e224c1d86be55b6b01f2288be2)
* sdscatfmt call sdsMakeRoomFor, asked for more space than intended (#8286)huangzhw2021-01-121-1/+1
| | | | | | | | | instead of asking for the extra new space it wanted, it asked to grow the string by the size it already has too. i.e. a string of 1000 bytes, needing to grow by 10 bytes, would have been asking for an **additional** 1010 bytes. (cherry picked from commit 595ecd5f4be39eeec71fb07f687b2d6b7cf5c20c)
* sort Command lookupKeyRead and lookupKeyWrite are used on the opposite (#8283)huangzhw2021-01-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | This is a recent problem, introduced by 7471743 (redis 6.0) The implications are: The sole difference between LookupKeyRead and LookupKeyWrite is for command executed on a replica, which are not received from its master client. (for the master, and for the master client on the replica, these two functions behave the same)! Since SORT is a write command, this bug only implicates a writable-replica. And these are its implications: - SORT STORE will behave as it did before the above mentioned commit (like before redis 6.0). on a writable-replica an already logically expired the key would have appeared missing. (store dest key would be deleted, instead of being populated with the data from the already logically expired key) - SORT (the non store variant, which in theory could have been executed on read-only-replica if it weren't for the write flag), will (in redis 6.0) have a new bug and return the data from the already logically expired key instead of empty response. (cherry picked from commit 08ad6abd04c5aafe5471fa754000e512ae6b0f05)
* Fix rdb checksum / crc64 on bigendian (#8270)Oran Agra2021-01-121-1/+2
| | | | | | | | | | | | Turns out the RDB checksum in Redis 6.0 on bigendian is broken. It always returned 0, so the RDB files are generated as if checksum is disabled, and will be loaded ok on littleendian, and on bigendian. But it'll not be able to load RDB files generated on littleendian or older versions. Similarly DUMP and RESTORE will work on the same version (0==0), but will be unable to exchange dump payloads with littleendian or old versions. (cherry picked from commit 324070c8f6f63240629893c2ce9bcbeecf6e77fb)
* fix crash in redis-cli after making cluster backup (#8267)Oran Agra2021-01-121-1/+3
| | | | | | | | | | | | | | getRDB is "designed" to work in two modes: one for redis-cli --rdb and one for redis-cli --cluster backup. in the later case it uses the hiredis connection from the cluster nodes and it used to free it without nullifying the context, so a later attempt to free the context would crash. I suppose the reason it seems to want to free the hiredis context ASAP is that it wants to disconnect the replica link, so that replication buffers will not be accumulated. (cherry picked from commit 41b2ed2bbc0671e43101feecc48cac26a5e312cb)
* Fix wrong order of key/value in Lua map response (#8266)Oran Agra2021-01-123-2/+45
| | | | | | | | | | | | | | | | | | | | | | | | | When a Lua script returns a map to redis (a feature which was added in redis 6 together with RESP3), it would have returned the value first and the key second. If the client was using RESP2, it was getting them out of order, and if the client was in RESP3, it was getting a map of value => key. This was happening regardless of the Lua script using redis.setresp(3) or not. This also affects a case where the script was returning a map which it got from from redis by doing something like: redis.setresp(3); return redis.call() This fix is a breaking change for redis 6.0 users who happened to rely on the wrong order (either ones that used redis.setresp(3), or ones that returned a map explicitly). This commit also includes other two changes in the tests: 1. The test suite now handles RESP3 maps as dicts rather than nested lists 2. Remove some redundant (duplicate) tests from tracking.tcl (cherry picked from commit 2017407b4d1d19a91af1e7c0b199f2c1775dbaf9)
* Crash log would crash half way on commands with no arguments (#8260)Oran Agra2021-01-121-1/+1
| | | | | | | | | The crash log attempts to print the current client info, and when it does that it attempts to check if the first argument happens to be a key but it did so for commands with no arguments too, which caused the crash log to crash half way and not reach its end. (cherry picked from commit 152b5d46c4a76f2d16031ef794092bfc8d322f8a)
* Fix crashes with io-threads-do-reads enabled. (#8230)Yossi Gottlieb2021-01-121-2/+14
| | | | | | | | | | | | | Normally IO threads should simply read data from the socket into the buffer and attempt to parse it. If a protocol error is detected, a reply is generated which may result with installing a write handler which is not thread safe. This fix delays that until the client is processed back in the main thread. Fixes #8220 (cherry picked from commit e7047ec2fcc20e150c0c8cf29addf582638d7e80)
* Add check for the MADV_FREE/fork arm64 Linux kernel bug (#8224)George Prekas2021-01-124-1/+145
| | | | | | | | | | | | | | | | | | | | | | | | | | | Older arm64 Linux kernels have a bug that could lead to data corruption during background save under the following scenario: 1) jemalloc uses MADV_FREE on a page, 2) jemalloc reuses and writes the page, 3) Redis forks the background save process, and 4) Linux performs page reclamation. Under these conditions, Linux will reclaim the page wrongfully and the background save process will read zeros when it tries to read the page. The bug has been fixed in Linux with commit: ff1712f953e27f0b0718762ec17d0adb15c9fd0b ("arm64: pgtable: Ensure dirty bit is preserved across pte_wrprotect()") This Commit adds an ignore-warnings config, when not found, redis will print a warning and exit on startup (default behavior). Co-authored-by: Oran Agra <oran@redislabs.com> (cherry picked from commit b02780c41dbc5b28d265b5cf141c03c1a7383ef9) in 6.0 this warning is ignored by default in order to avoid adding regression, specifically for deployments that don't need persistence or replication
* Not over-allocate client query buffer when reading large objects. (#5954)Qu Chen2021-01-121-1/+1
| | | | | | | | | | | | | | In response to large client query buffer optimization introduced in 1898e6c. The calculation of the amount of remaining bytes we need to write to the query buffer was calculated wrong, as a result we are unnecessarily growing the client query buffer by sdslen(c->querybuf) always. This fix corrects that behavior. Please note the previous behavior prior to the before-mentioned change was correctly calculating the remaining additional bytes, and this change makes that calculate to be consistent. Useful context, the argument of size `ll` starts at qb_pos (which is now the beginning of the sds), but much of it may have already been read from the socket, so we only need to grow the sds for the remainder of it. (cherry picked from commit 11b3325e9999721d35ec64afac7b917664f6291b)
* Handle output buffer limits for Module blocked clients (#8141)Oran Agra2021-01-123-1/+150
| | | | | | | | | | | Module blocked clients cache the response in a temporary client, the reply list in this client would be affected by the recent fix in #7202, but when the reply is later copied into the real client, it would have bypassed all the checks for output buffer limit, which would have resulted in both: responding with a partial response to the client, and also not disconnecting it at all. (cherry picked from commit 48efc25f749c3620f9245786582ac76cb40e9bf4)
* Fix setproctitle related crashes. (#8150)Yossi Gottlieb2021-01-121-4/+39
| | | | | | | | Makes spt_init more careful with assumptions about what memory regions may be overwritten. It will now only consider a contiguous block of argv and envp elements and mind any gaps. (cherry picked from commit ec02c761aa16175eb599b2fc9d0b2792ffe8a66c)
* Fix use-after-free issue in spt_copyenv. (#8088)Yossi Gottlieb2021-01-121-19/+50
| | | | | | | | | | Seems to have gone unnoticed for a long time, because at least with glibc it will only be triggered if setenv() was called before spt_init, which Redis doesn't. Fixes #8064. (cherry picked from commit 7e5a6313f0add995c723351532d994118e3e8a6d)
* Backup keys to slots map and restore when fail to sync if diskless-load type ↵Wang Yuan2021-01-126-120/+274
| | | | | | | | | | | | | is swapdb in cluster mode (#8108) When replica diskless-load type is swapdb in cluster mode, we didn't backup keys to slots map, so we will lose keys to slots map if fail to sync. Now we backup keys to slots map at first, and restore it properly when fail. This commit includes a refactory/cleanup of the backups mechanism (moving it to db.c and re-structuring it a bit). Co-authored-by: Oran Agra <oran@redislabs.com> (cherry picked from commit b55a827ea2e19bd6cd48f216e8e6caa34299f9b9)
* Reset average ttl when empty databases (#8106)Wang Yuan2021-01-121-0/+3
| | | | | | | On FLUSHDB or full sync, reset old average TTL stat. This Stat is incrementally collected by the master over time when it searches for expired keys. (cherry picked from commit c85bf2352d2a3dd04872124d33066403b114a7e7)
* Fix bug with module GIL being released prematurely (#8061)Oran Agra2021-01-121-4/+9
| | | | | | | | | | | | | This is hopefully usually harmles. The server.ready_keys will usually be empty so the code after releasing the GIL will soon be done. The only case where it'll actually process things is when a module releases a client (or module) blocked on a key, by triggering this NOT from within a command (e.g. a timer event). This bug was introduced in redis 6.0.9, see #7903 (cherry picked from commit e6fa47380a5274119ed37c7a5ea7455d4b7dbdcc)
* Fix oom-score-adj-values range, abs options, and bug when used in config ↵Oran Agra2021-01-124-17/+42
| | | | | | | | | | | | | | | | | | | file (#8046) Fix: When oom-score-adj-values is provided in the config file after oom-score-adj yes, it'll take an immediate action, before readOOMScoreAdj was acquired, resulting in an error (out of range score due to uninitialized value. delay the reaction the real call is made by main(). Since the values are clamped to -1000..1000, and they're applied as an offset from the value at startup (which may be -1000), we need to allow the offsets to reach to +2000 so that a value of +1000 is achievable in case the value at startup was -1000. Adding an option for absolute values rather than relative ones. (cherry picked from commit 61954951edbda670bfbae8be0147daa64df95f26)
* EXISTS should not alter LRU, OBJECT should not reveal expired keys on ↵guybe72021-01-125-19/+21
| | | | | | | | | | | | | | | | | | replica (#8016) The bug was introduced by #5021 which only attempted avoid EXIST on an already expired key from returning 1 on a replica. Before that commit, dbExists was used instead of lookupKeyRead (which had an undesired effect to "touch" the LRU/LFU) Other than that, this commit fixes OBJECT to also come empty handed on expired keys in replica. And DEBUG DIGEST-VALUE to behave like DEBUG OBJECT (get the data from the key regardless of it's expired state) (cherry picked from commit f8ae991717f10c837c1a76b2954dae56ecb0e6bc)
View Lorry Controller Status