Restore our priority-inversion-prevention code with deferreds

Back when deferred_cb stuff had its own queue, the queue was always
executed, but we never ran more than 16 callbacks per iteration.
That made for two problems:

1: Because deferred_cb stuff would always run, and had no priority,
it could cause priority inversion.

2: It doesn't respect the max_dispatch_interval code.

Then, when I refactored deferred_cb to be a special case of
event_callback, that solved the above issues, but made for two more
issues:

3: Because deferred_cb stuff would always get the default priority,
it could could low-priority bufferevents to get too much priority.

4: With code like bufferevent_pair, it's easy to get into a
situation where two deferreds keep adding one another, preventing
the event loop from ever actually scanning for more events.

This commit fixes the above by giving deferreds a better notion of
priorities, and by limiting the number of deferreds that can be
added to the _current_ loop iteration's active queues.  (Extra
deferreds are put into the active_later state.)

That isn't an all-purpose priority inversion solution, of course: for
that, you may need to mess around with max_dispatch_interval.

1 files changed, 6 insertions, 1 deletions

diff --git a/defer-internal.h b/defer-internal.h
index f2ed2823..d8cf32f4 100644
--- a/defer-internal.h
+++ b/defer-internal.h
@@ -42,10 +42,15 @@ typedef void (*deferred_cb_fn)(struct event_callback *, void *);
    Initialize an empty, non-pending event_callback.
 
    @param deferred The struct event_callback structure to initialize.
+   @param priority The priority that the callback should run at.
    @param cb The function to run when the struct event_callback executes.
    @param arg The function's second argument.
  */
-void event_deferred_cb_init_(struct event_base *base, struct event_callback *, deferred_cb_fn, void *);
+void event_deferred_cb_init_(struct event_callback *, ev_uint8_t, deferred_cb_fn, void *);
+/**
+   Change the priority of a non-pending event_callback.
+ */
+void event_deferred_cb_set_priority_(struct event_callback *, ev_uint8_t);
 /**
    Cancel a struct event_callback if it is currently scheduled in an event_base.
  */