msdkenc: break out of flush frames loop on error

In general, we should assume any unhandled error is
non-recoverable.

In the flush frames loop, some error states can cause us
to never increment the task and therefore we get stuck
in an infinite loop and generate GST_ELEMENT_ERROR
over and over again.  This eventually consumes all
system memory and triggers OOM.  Thus, assume the worst
and break out of the loop upon the first "unhandled" error.

https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/859

diff --git a/sys/msdk/gstmsdkenc.c b/sys/msdk/gstmsdkenc.c
index bc9d4d3..c3e16ce 100644 (file)
--- a/sys/msdk/gstmsdkenc.c
+++ b/sys/msdk/gstmsdkenc.c
@@ -830,6 +830,7 @@ gst_msdkenc_flush_frames (GstMsdkEnc * thiz, gboolean discard)
     if (status != MFX_ERR_NONE && status != MFX_ERR_MORE_DATA) {
       GST_ELEMENT_ERROR (thiz, STREAM, ENCODE, ("Encode frame failed."),
           ("MSDK encode error (%s)", msdk_status_to_string (status)));
+      break;
     }
 
     if (task->sync_point) {