clean_worker_registry cleans in batches to prevent submitting too muc… (#1390)

* clean_worker_registry cleans in batches to prevent submitting too much data to redis at once when there are a large number of invalid keys

* Address code review comments

Rename MAX_REMOVABLE_KEYS to MAX_KEYS

* Fix tests

Co-authored-by: Joel Harris <combolations@gmail.com>

1 files changed, 13 insertions, 0 deletions

diff --git a/rq/utils.py b/rq/utils.py
index 1d8ab5b..573c4aa 100644
--- a/rq/utils.py
+++ b/rq/utils.py
@@ -261,3 +261,16 @@ def get_version(connection):
     except ResponseError:  # fakeredis doesn't implement Redis' INFO command
         version_string = "5.0.9"
     return StrictVersion('.'.join(version_string.split('.')[:3]))
+
+
+def ceildiv(a, b):
+    """Ceiling division. Returns the ceiling of the quotient of a division operation"""
+    return -(-a // b)
+
+
+def split_list(a_list, segment_size):
+    """
+    Splits a list into multiple smaller lists having size `segment_size`
+    """
+    for i in range(0, len(a_list), segment_size):
+        yield a_list[i:i + segment_size]