Implement a new custom recursive mutex type and fix locking in callbacks so that in-context calls are allowed.

According to the OMX specification, implementations are allowed to call
callbacks in the context of their function calls. However, our callbacks
take locks and this causes deadlocks if the unerlying OMX implementation
uses this kind of in-context calls.

A solution to the problem would be a recursive mutex. However, a normal
recursive mutex does not fix the problem because it is not guaranteed
that the callbacks are called from the same thread. What we see in Broadcom's
implementation for example is:

- OMX_Foo is called
- OMX_Foo waits on a condition
- A callback is executed in a different thread
- When the callback returns, its calling function
  signals the condition that OMX_Foo waits on
- OMX_Foo wakes up and returns

The solution I came up with here is to take a second lock inside the callback,
but only if recursion is expected to happen. Therefore, all calls to OMX
functions are guarded by calls to gst_omx_rec_mutex_begin_recursion() / _end_recursion(),
which effectively tells the mutex that at this point we want to allow calls
to _recursive_lock() to succeed, although we are still holding the master lock.

1 files changed, 2 insertions, 2 deletions

diff --git a/omx/gstomxaudioenc.c b/omx/gstomxaudioenc.c
index de57edf..d87e687 100644
--- a/omx/gstomxaudioenc.c
+++ b/omx/gstomxaudioenc.c
@@ -885,10 +885,10 @@ gst_omx_audio_enc_sink_event (GstAudioEncoder * encoder, GstEvent * event)
       GST_WARNING_OBJECT (self, "Component does not support empty EOS buffers");
 
       /* Insert a NULL into the queue to signal EOS */
-      g_mutex_lock (self->out_port->port_lock);
+      gst_omx_rec_mutex_lock (&self->out_port->port_lock);
       g_queue_push_tail (self->out_port->pending_buffers, NULL);
       g_cond_broadcast (self->out_port->port_cond);
-      g_mutex_unlock (self->out_port->port_lock);
+      gst_omx_rec_mutex_unlock (&self->out_port->port_lock);
       return TRUE;
     }