summaryrefslogtreecommitdiff
path: root/src/defrag.c
diff options
context:
space:
mode:
authorsundb <sundbcn@gmail.com>2022-11-17 02:29:46 +0800
committerGitHub <noreply@github.com>2022-11-16 20:29:46 +0200
commit2168ccc661791ced6271c5e4ab0f5eb60b1559e2 (patch)
treeb9db3781b460230d170b8454c7c8a6e719b03a57 /src/defrag.c
parentd136bf28307ed2add5a0b709586433f4cffd70a7 (diff)
downloadredis-2168ccc661791ced6271c5e4ab0f5eb60b1559e2.tar.gz
Add listpack encoding for list (#11303)
Improve memory efficiency of list keys ## Description of the feature The new listpack encoding uses the old `list-max-listpack-size` config to perform the conversion, which we can think it of as a node inside a quicklist, but without 80 bytes overhead (internal fragmentation included) of quicklist and quicklistNode structs. For example, a list key with 5 items of 10 chars each, now takes 128 bytes instead of 208 it used to take. ## Conversion rules * Convert listpack to quicklist When the listpack length or size reaches the `list-max-listpack-size` limit, it will be converted to a quicklist. * Convert quicklist to listpack When a quicklist has only one node, and its length or size is reduced to half of the `list-max-listpack-size` limit, it will be converted to a listpack. This is done to avoid frequent conversions when we add or remove at the bounding size or length. ## Interface changes 1. add list entry param to listTypeSetIteratorDirection When list encoding is listpack, `listTypeIterator->lpi` points to the next entry of current entry, so when changing the direction, we need to use the current node (listTypeEntry->p) to update `listTypeIterator->lpi` to the next node in the reverse direction. ## Benchmark ### Listpack VS Quicklist with one node * LPUSH - roughly 0.3% improvement * LRANGE - roughly 13% improvement ### Both are quicklist * LRANGE - roughly 3% improvement * LRANGE without pipeline - roughly 3% improvement From the benchmark, as we can see from the results 1. When list is quicklist encoding, LRANGE improves performance by <5%. 2. When list is listpack encoding, LRANGE improves performance by ~13%, the main enhancement is brought by `addListListpackRangeReply()`. ## Memory usage 1M lists(key:0~key:1000000) with 5 items of 10 chars ("hellohello") each. shows memory usage down by 35.49%, from 214MB to 138MB. ## Note 1. Add conversion callback to support doing some work before conversion Since the quicklist iterator decompresses the current node when it is released, we can no longer decompress the quicklist after we convert the list.
Diffstat (limited to 'src/defrag.c')
-rw-r--r--src/defrag.c3
1 files changed, 3 insertions, 0 deletions
diff --git a/src/defrag.c b/src/defrag.c
index e78c07929..dbdf2ab62 100644
--- a/src/defrag.c
+++ b/src/defrag.c
@@ -868,6 +868,9 @@ long defragKey(redisDb *db, dictEntry *de) {
} else if (ob->type == OBJ_LIST) {
if (ob->encoding == OBJ_ENCODING_QUICKLIST) {
defragged += defragQuicklist(db, de);
+ } else if (ob->encoding == OBJ_ENCODING_LISTPACK) {
+ if ((newzl = activeDefragAlloc(ob->ptr)))
+ defragged++, ob->ptr = newzl;
} else {
serverPanic("Unknown list encoding");
}