Add callback API to support ephemeron marking

(cherry-pick of 1a65adc, bb5ba42, 6f41d8d from Unity-Technologies/bdwgc)

When ephemerons marking results in pushing a large number of objects,
the mark stack may overflow.  While the processing could be batched,
the simplest approach is to allow the callback to return false to
indicate a failure and increasing the mark stack size.

* include/gc/gc_mark.h (GC_on_mark_stack_empty_proc): New public type.
* include/gc/gc_mark.h (GC_set_on_mark_stack_empty,
GC_get_on_mark_stack_empty): New public function declaration.
* mark.c (GC_on_mark_stack_empty): New static variable.
* mark.c (GC_set_on_mark_stack_empty, GC_get_on_mark_stack_empty): New
function implementation.
* mark.c (GC_mark_some_inner): If GC_on_mark_stack_empty is non-zero
then call it and break if pushed new items or overflowed.
* tests/gctest.c [GC_PTHREADS] (main): Call GC_set_on_mark_stack_empty
and GC_get_on_mark_stack_empty.

Co-authored-by: Ivan Maidanski <ivmai@mail.ru>

1 files changed, 1 insertions, 0 deletions

diff --git a/tests/gctest.c b/tests/gctest.c
index 03dcd585..861764bb 100644
--- a/tests/gctest.c
+++ b/tests/gctest.c
@@ -2665,6 +2665,7 @@ int main(void)
     GC_set_non_gc_bytes(GC_get_non_gc_bytes());
     GC_set_on_collection_event(GC_get_on_collection_event());
     GC_set_on_heap_resize(GC_get_on_heap_resize());
+    GC_set_on_mark_stack_empty(GC_get_on_mark_stack_empty());
     GC_set_on_thread_event(GC_get_on_thread_event());
     GC_set_oom_fn(GC_get_oom_fn());
     GC_set_push_other_roots(GC_get_push_other_roots());