vaapiencode: don't crash on NULL encoder on _finish().

Don't try to destroy an encoder, in GstVideoEncoder::finish() handler,
if it was not created in the first place. Return "not-negotiated" error
since this means we did not even reach GstVideoEncoder::set_format(),
where the encoder could have been created.

This fixes a crash when the vaapiencode_* plug-in elements get deallocated
and that we failed to negotiate either pad.

https://bugzilla.gnome.org/show_bug.cgi?id=719704

diff --git a/gst/vaapi/gstvaapiencode.c b/gst/vaapi/gstvaapiencode.c
index b482aef..e4b52f4 100644 (file)
--- a/gst/vaapi/gstvaapiencode.c
+++ b/gst/vaapi/gstvaapiencode.c
@@ -587,6 +587,13 @@ gst_vaapiencode_finish (GstVideoEncoder * venc)
   GstVaapiEncoderStatus status;
   GstFlowReturn ret = GST_FLOW_OK;
 
+  /* Don't try to destroy encoder if none was created in the first place.
+     Return "not-negotiated" error since this means we did not even reach
+     GstVideoEncoder::set_format() state, where the encoder could have
+     been created */
+  if (!encode->encoder)
+    return GST_FLOW_NOT_NEGOTIATED;
+
   status = gst_vaapi_encoder_flush (encode->encoder);
 
   GST_VIDEO_ENCODER_STREAM_UNLOCK (encode);