diff options
| author | Nick Vatamaniuc <vatamane@gmail.com> | 2022-04-22 19:53:13 -0400 |
|---|---|---|
| committer | Nick Vatamaniuc <nickva@users.noreply.github.com> | 2022-04-28 17:19:44 -0400 |
| commit | 0e38f390796d6e34f698853f5886a36607e2aeec (patch) | |
| tree | c910ca79a6e1ab1d8e8c08ac8b2b02320d2aafb8 /rel | |
| parent | 0bc311b7d7ec9836679b929ee5c2a630ad2ff01f (diff) | |
| download | couchdb-0e38f390796d6e34f698853f5886a36607e2aeec.tar.gz | |
Improve index building during shard splitting
Previously we didn't check responses from get_state/2 or await/2 functions when
building indices. If an index updater crashed, and the index never finished
building, the get_state/2 call would simply return an error and the process
would exit normally. Then, the shard splitting job would count that as a
success and continue to make progress.
To fix that, make sure to check the response to all the supported indexing
types and wait until they return an `ok` result.
Additionally, increase the index building resilience to allow for more retries
on failure, and for configurable retries for individual index builders.
Diffstat (limited to 'rel')
| -rw-r--r-- | rel/overlay/etc/default.ini | 8 |
1 files changed, 7 insertions, 1 deletions
diff --git a/rel/overlay/etc/default.ini b/rel/overlay/etc/default.ini index 7535501c0..5fb45b5b5 100644 --- a/rel/overlay/etc/default.ini +++ b/rel/overlay/etc/default.ini @@ -719,7 +719,7 @@ state_dir = {{state_dir}} [reshard] ;max_jobs = 48 ;max_history = 20 -;max_retries = 1 +;max_retries = 5 ;retry_interval_sec = 10 ;delete_source = true ;update_shard_map_timeout_sec = 60 @@ -727,6 +727,12 @@ state_dir = {{state_dir}} ;require_node_param = false ;require_range_param = false +; How many times to retry building an individual index +;index_max_retries = 5 + +; How many seconds to wait between retries for an individual index +;index_retry_interval_sec = 10 + [prometheus] additional_port = false bind_address = 127.0.0.1 |