diff options
author | sundb <sundbcn@gmail.com> | 2021-11-16 19:12:25 +0800 |
---|---|---|
committer | GitHub <noreply@github.com> | 2021-11-16 13:12:25 +0200 |
commit | 985430b4fca5ac55b121a98ac0407909c6767530 (patch) | |
tree | 57c09d97f4efe22e1d7f380f6ff8ce637840b6ab /tests/unit/bitops.tcl | |
parent | e725d737fb2ee492fbcd04bb7deb1696d7e182d1 (diff) | |
download | redis-985430b4fca5ac55b121a98ac0407909c6767530.tar.gz |
Change lzf to handle values larger than UINT32_MAX (#9776)
Redis supports inserting data over 4GB into string (and recently for lists too, see #9357),
But LZF compression used in RDB files (see `rdbcompression` config), and in quicklist
(see `list-compress-depth` config) does not support compress/decompress data over
UINT32_MAX, which will result in corrupting the rdb after compression.
Internal changes:
1. Modify the `unsigned int` parameter of `lzf_compress/lzf_decompress` to `size_t`.
2. Modify the variable types in `lzf_compress` involving offsets and lengths to `size_t`.
3. Set LZF_USE_OFFSETS to 0.
When LZF_USE_OFFSETS is 1, lzf store offset into `LZF_HSLOT`(32bit).
Even in 64-bit, `LZF_USE_OFFSETS` defaults to 1, because lzf assumes that it only
compresses and decompresses data smaller than UINT32_MAX.
But now we need to make lzf support 64-bit, turning on `LZF_USE_OFFSETS` will make
it impossible to store 64-bit offsets or pointers.
BTW, disable LZF_USE_OFFSETS also brings a few performance improvements.
Tests:
1. Add test for compress/decompress string large than UINT32_MAX.
2. Add unittest for compress/decompress quicklistNode.
Diffstat (limited to 'tests/unit/bitops.tcl')
-rw-r--r-- | tests/unit/bitops.tcl | 14 |
1 files changed, 14 insertions, 0 deletions
diff --git a/tests/unit/bitops.tcl b/tests/unit/bitops.tcl index 1a3c3a54e..89431c81f 100644 --- a/tests/unit/bitops.tcl +++ b/tests/unit/bitops.tcl @@ -566,4 +566,18 @@ start_server {tags {"bitops"}} { r config set proto-max-bulk-len $oldval r del mykey } {1} {large-memory} + + test "SETBIT values larger than UINT32_MAX and lzf_compress/lzf_decompress correctly" { + set bytes [expr (1 << 32) + 1] + set bitpos [expr (1 << 35)] + set oldval [lindex [r config get proto-max-bulk-len] 1] + r config set proto-max-bulk-len $bytes + r setbit mykey $bitpos 1 + assert_equal $bytes [r strlen mykey] + assert_equal 1 [r getbit mykey $bitpos] + r debug reload ;# lzf_compress/lzf_decompress when RDB saving/loading. + assert_equal 1 [r getbit mykey $bitpos] + r config set proto-max-bulk-len $oldval + r del mykey + } {1} {large-memory needs:debug} } |