From: Paul Wilkins <paulwilkins@google.com>
Date: Wed, 13 Mar 2019 16:20:23 +0000 (+0000)
Subject: Improve key frame detection
X-Git-Tag: v1.8.1~183^2
X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=e458ca3d52dbe9bf79ee31c712e3d87e751c5307;p=platform%2Fupstream%2Flibvpx.git

Improve key frame detection

Improve detection of key frames especially in low contrast
and low motion regions.

This patch adds a function to the key frame detection to test
for specific patterns in the intra signal in the first pass stats
that tend to be indicative of a key frame.

This is intended to compliment the existing code and finds some
scene cuts that were previously being misssed.

Tested on two clips where the existing code was struggling to
identify the key frames this patch improved detection as follows.

Film clip 1: (detected / actual)
Old (2/5) New (5/5)

Film Clip 2
Old 4/11 and one false +,  New 7/11 and 1 false +.

Short 4K Film Scene
Old 1/2 New 2/2

In testing so far I have not seen many extra false +'s though
it is likely that there will be some cases and this may need
further tweaking.

One one of our longer form film test reels ~20k frames)
the change picked up around 35 key frames that were
previously missed, mainly in darker scenes. There were a few
extra (or different) false positives cause by bright flashes or
explosions but these were cases where there was little
difference between inter and intra coding.

Awaiting testing on standard sets.

Change-Id: I1ff4a587e0a47667eb93b197f39b79a1130faeca
---

diff --git a/vp9/encoder/vp9_firstpass.c b/vp9/encoder/vp9_firstpass.c
index 620d21f..b22ccc2 100644
--- a/vp9/encoder/vp9_firstpass.c
+++ b/vp9/encoder/vp9_firstpass.c
@@ -2746,11 +2746,54 @@ static int slide_transition(const FIRSTPASS_STATS *this_frame,
          (this_frame->coded_error > (next_frame->coded_error * ERROR_SPIKE));
 }
 
+// This test looks for anomalous changes in the nature of the intra signal
+// related to the previous and next frame as an indicator for coding a key
+// frame. This test serves to detect some additional scene cuts,
+// especially in lowish motion and low contrast sections, that are missed
+// by the other tests.
+static int intra_step_transition(const FIRSTPASS_STATS *this_frame,
+                                 const FIRSTPASS_STATS *last_frame,
+                                 const FIRSTPASS_STATS *next_frame) {
+  double last_ii_ratio;
+  double this_ii_ratio;
+  double next_ii_ratio;
+  double last_pcnt_intra = 1.0 - last_frame->pcnt_inter;
+  double this_pcnt_intra = 1.0 - this_frame->pcnt_inter;
+  double next_pcnt_intra = 1.0 - next_frame->pcnt_inter;
+  double mod_this_intra = this_pcnt_intra + this_frame->pcnt_neutral;
+
+  // Calculate ii ratio for this frame last frame and next frame.
+  last_ii_ratio =
+      last_frame->intra_error / DOUBLE_DIVIDE_CHECK(last_frame->coded_error);
+  this_ii_ratio =
+      this_frame->intra_error / DOUBLE_DIVIDE_CHECK(this_frame->coded_error);
+  next_ii_ratio =
+      next_frame->intra_error / DOUBLE_DIVIDE_CHECK(next_frame->coded_error);
+
+  // Return true the intra/inter ratio for the current frame is
+  // low but better in the next and previous frame and the relative useage of
+  // intra in the current frame is markedly higher than the last and next frame.
+  if ((this_ii_ratio < 2.0) && (last_ii_ratio > 2.25) &&
+      (next_ii_ratio > 2.25) && (this_pcnt_intra > (3 * last_pcnt_intra)) &&
+      (this_pcnt_intra > (3 * next_pcnt_intra)) &&
+      ((this_pcnt_intra > 0.075) || (mod_this_intra > 0.85))) {
+    return 1;
+    // Very low inter intra ratio (i.e. not much gain from inter coding), most
+    // blocks neutral on coding method and better inter prediction either side
+  } else if ((this_ii_ratio < 1.25) && (mod_this_intra > 0.85) &&
+             (this_ii_ratio < last_ii_ratio * 0.9) &&
+             (this_ii_ratio < next_ii_ratio * 0.9)) {
+    return 1;
+  } else {
+    return 0;
+  }
+}
+
 // Minimum % intra coding observed in first pass (1.0 = 100%)
 #define MIN_INTRA_LEVEL 0.25
 // Threshold for use of the lagging second reference frame. Scene cuts do not
 // usually have a high second ref useage.
-#define SECOND_REF_USEAGE_THRESH 0.125
+#define SECOND_REF_USEAGE_THRESH 0.2
 // Hard threshold where the first pass chooses intra for almost all blocks.
 // In such a case even if the frame is not a scene cut coding a key frame
 // may be a good option.
@@ -2777,8 +2820,9 @@ static int test_candidate_kf(TWO_PASS *twopass,
       (this_frame->pcnt_second_ref < SECOND_REF_USEAGE_THRESH) &&
       ((this_frame->pcnt_inter < VERY_LOW_INTER_THRESH) ||
        (slide_transition(this_frame, last_frame, next_frame)) ||
-       (((this_frame->coded_error > (next_frame->coded_error * 1.1)) &&
-         (this_frame->coded_error > (last_frame->coded_error * 1.1))) &&
+       (intra_step_transition(this_frame, last_frame, next_frame)) ||
+       (((this_frame->coded_error > (next_frame->coded_error * 1.2)) &&
+         (this_frame->coded_error > (last_frame->coded_error * 1.2))) &&
         (pcnt_intra > MIN_INTRA_LEVEL) &&
         ((pcnt_intra + this_frame->pcnt_neutral) > 0.5) &&
         ((this_frame->intra_error /