diff options
| author | Péter Gömöri <peter@84codes.com> | 2022-10-28 18:41:51 +0200 |
|---|---|---|
| committer | Mergify <37929162+mergify[bot]@users.noreply.github.com> | 2022-10-28 19:12:58 +0000 |
| commit | cdef416efd4487020918077ef90663595eafe2e0 (patch) | |
| tree | d7adf466b42483a25e4ca0f0b21c33cf14e3d4ec | |
| parent | 329db668337b64e6a80a136980b95964b245c091 (diff) | |
| download | rabbitmq-server-git-cdef416efd4487020918077ef90663595eafe2e0.tar.gz | |
Avoid crashing shovel plugin when another node goes down
When `rabbit_shovel_dyn_worker_sup_sup:cleanup_specs()` is called while
another node is just going down the caller can exit which crashes
rabbit_shovel_status gen_server with the below crash. If there are no
static shovels configured `rabbit_shovel_sup` will have a restart
strategy intensity of zero, hence termination of `rabbit_shovel_status`
will also shut down the supervision tree and the app of rabbitmq_shovel
plugin.
This change tries to avoid such dramatic consequences of a temporary
issue.
```
** Generic server rabbit_shovel_status terminating
** Last message in was check
** When Server state == {state,{erlang,#Ref<0.2444609781.3637248001.190587>}}
** Reason for termination ==
** {{{nodedown,'rabbit@...'},
{gen_server,call,[<16931.927.0>,which_children,infinity]}},
[{gen_server,call,3,[{file,"gen_server.erl"},{line,382}]},
{mirrored_supervisor,child,2,
[{file,"mirrored_supervisor.erl"},{line,252}]},
{mirrored_supervisor,'-fold/3-lc$^0/1-0-',2,
[{file,"mirrored_supervisor.erl"},{line,249}]},
{mirrored_supervisor,fold,3,
[{file,"mirrored_supervisor.erl"},{line,248}]},
{rabbit_shovel_dyn_worker_sup_sup,cleanup_specs,0,
[{file,"rabbit_shovel_dyn_worker_sup_sup.erl"},{line,86}]},
{rabbit_shovel_status,handle_info,2,
[{file,"rabbit_shovel_status.erl"},{line,92}]},
{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,1120}]},
{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1197}]}]}
...
supervisor: {local,rabbit_shovel_sup}
errorContext: child_terminated
reason: {{nodedown,'rabbit@...'},
...
supervisor: {local,rabbit_shovel_sup}
errorContext: shutdown
reason: reached_max_restart_intensity
...
Application rabbitmq_shovel exited with reason: shutdown
```
(cherry picked from commit 8f596f4161f3d04b6c937efd7786d34a7e2a8b98)
| -rw-r--r-- | deps/rabbitmq_shovel/src/rabbit_shovel_status.erl | 7 |
1 files changed, 6 insertions, 1 deletions
diff --git a/deps/rabbitmq_shovel/src/rabbit_shovel_status.erl b/deps/rabbitmq_shovel/src/rabbit_shovel_status.erl index b8e6dc7ccb..b14af6a64c 100644 --- a/deps/rabbitmq_shovel/src/rabbit_shovel_status.erl +++ b/deps/rabbitmq_shovel/src/rabbit_shovel_status.erl @@ -89,7 +89,12 @@ handle_cast({remove, Name}, State) -> {noreply, State}. handle_info(check, State) -> - rabbit_shovel_dyn_worker_sup_sup:cleanup_specs(), + try + rabbit_shovel_dyn_worker_sup_sup:cleanup_specs() + catch + C:E -> + rabbit_log_shovel:warning("Recurring shovel spec clean up failed with ~p:~p", [C, E]) + end, {noreply, ensure_timer(State)}; handle_info(_Info, State) -> {noreply, State}. |