mftransform: Fix deadlock when MFT requests processing output twice

This sequence of event/data flow might happen

1) Initially we have one pending output event
  1-1) Then, process the pending output data
2) No pending input event, then we should wait new pending input event
  2-1) Wakeup by new pending event (but it's pending output event)

In above case, MFT will not report new pending input event
if pending output is not processed.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/merge_requests/1325>

diff --git a/sys/mediafoundation/gstmftransform.cpp b/sys/mediafoundation/gstmftransform.cpp
index 17c38b9..ee30c2e 100644 (file)
--- a/sys/mediafoundation/gstmftransform.cpp
+++ b/sys/mediafoundation/gstmftransform.cpp
@@ -539,6 +539,8 @@ gst_mf_transform_process_input (GstMFTransform * object,
   gst_mf_transform_drain_all_events (object);
 
   if (object->hardware) {
+  process_output:
+    /* Process pending output first */
     while (object->pending_have_output > 0) {
       GST_TRACE_OBJECT (object,
           "Pending have output %d", object->pending_have_output);
@@ -581,6 +583,14 @@ gst_mf_transform_process_input (GstMFTransform * object,
           GST_DEBUG_OBJECT (object, "Unhandled event %d", type);
           break;
       }
+
+      /* If MFT doesn't want to handle input yet but we have pending output,
+       * process output again */
+      if (object->pending_have_output > 0 && object->pending_need_input == 0) {
+        GST_TRACE_OBJECT (object,
+            "Only have pending output, process output again");
+        goto process_output;
+      }
     }
   }