pan/mdg: Don't pair ST_VARY.a32 with other instrs

For some reason, LD_ATTR/ST_VARY.a32 bundles raise INSTR_INVALID_ENC, at
least on Mali-T860. Don't construct such pairs. This is a blunt hack but
I don't know where this curveball requirement is coming from and this
unblocks the rest of this series.

total instructions in shared programs: 99879 -> 99788 (-0.09%)
instructions in affected programs: 3179 -> 3088 (-2.86%)
helped: 49
HURT: 9
helped stats (abs) min: 1.0 max: 6.0 x̄: 2.04 x̃: 2
helped stats (rel) min: 0.93% max: 10.53% x̄: 5.46% x̃: 4.88%
HURT stats (abs)   min: 1.0 max: 1.0 x̄: 1.00 x̃: 1
HURT stats (rel)   min: 0.61% max: 2.13% x̄: 1.41% x̃: 1.14%
95% mean confidence interval for instructions value: -1.93 -1.20
95% mean confidence interval for instructions %-change: -5.37% -3.41%
Instructions are helped.

total bundles in shared programs: 43778 -> 45102 (3.02%)
bundles in affected programs: 10737 -> 12061 (12.33%)
helped: 10
HURT: 369
helped stats (abs) min: 1.0 max: 3.0 x̄: 1.50 x̃: 1
helped stats (rel) min: 2.90% max: 18.75% x̄: 6.93% x̃: 5.21%
HURT stats (abs)   min: 1.0 max: 10.0 x̄: 3.63 x̃: 4
HURT stats (rel)   min: 0.82% max: 44.44% x̄: 15.27% x̃: 13.33%
95% mean confidence interval for bundles value: 3.29 3.69
95% mean confidence interval for bundles %-change: 13.68% 15.69%
Bundles are HURT.

total quadwords in shared programs: 76783 -> 77914 (1.47%)
quadwords in affected programs: 18633 -> 19764 (6.07%)
helped: 9
HURT: 370
helped stats (abs) min: 1.0 max: 2.0 x̄: 1.22 x̃: 1
helped stats (rel) min: 0.87% max: 8.33% x̄: 3.71% x̃: 3.85%
HURT stats (abs)   min: 1.0 max: 7.0 x̄: 3.09 x̃: 3
HURT stats (rel)   min: 0.82% max: 35.00% x̄: 7.82% x̃: 6.11%
95% mean confidence interval for quadwords value: 2.82 3.15
95% mean confidence interval for quadwords %-change: 7.02% 8.06%
Quadwords are HURT.

total registers in shared programs: 7266 -> 7076 (-2.61%)
registers in affected programs: 1224 -> 1034 (-15.52%)
helped: 171
HURT: 25
helped stats (abs) min: 1.0 max: 3.0 x̄: 1.27 x̃: 1
helped stats (rel) min: 8.33% max: 50.00% x̄: 21.85% x̃: 20.00%
HURT stats (abs)   min: 1.0 max: 2.0 x̄: 1.12 x̃: 1
HURT stats (rel)   min: 10.00% max: 100.00% x̄: 35.73% x̃: 33.33%
95% mean confidence interval for registers value: -1.10 -0.84
95% mean confidence interval for registers %-change: -17.69% -11.32%
Registers are helped.

total threads in shared programs: 4956 -> 5019 (1.27%)
threads in affected programs: 99 -> 162 (63.64%)
helped: 43
HURT: 6
helped stats (abs) min: 1.0 max: 2.0 x̄: 1.74 x̃: 2
helped stats (rel) min: 100.00% max: 100.00% x̄: 100.00% x̃: 100.00%
HURT stats (abs)   min: 2.0 max: 2.0 x̄: 2.00 x̃: 2
HURT stats (rel)   min: 50.00% max: 50.00% x̄: 50.00% x̃: 50.00%
95% mean confidence interval for threads value: 0.91 1.66
95% mean confidence interval for threads %-change: 67.36% 95.90%
Threads are helped.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19363>

diff --git a/src/panfrost/midgard/midgard_schedule.c b/src/panfrost/midgard/midgard_schedule.c
index e98e8ce..7ef6134 100644 (file)
--- a/src/panfrost/midgard/midgard_schedule.c
+++ b/src/panfrost/midgard/midgard_schedule.c
@@ -387,6 +387,12 @@ struct midgard_predicate {
          * registers and fail to spill without breaking the schedule) */
 
         unsigned pipeline_count;
+
+        /* For load/store: is a ST_VARY.a32 instruction scheduled into the
+         * bundle? is a non-ST_VARY.a32 instruction scheduled? Potential
+         * hardware issue, unknown cause.
+         */
+        bool any_st_vary_a32, any_non_st_vary_a32;
 };
 
 static bool
@@ -738,6 +744,14 @@ mir_choose_instruction(
                 if (ldst && mir_pipeline_count(instructions[i]) + predicate->pipeline_count > 2)
                         continue;
 
+                bool st_vary_a32 = (instructions[i]->op == midgard_op_st_vary_32);
+
+                if (ldst && predicate->any_non_st_vary_a32 && st_vary_a32)
+                        continue;
+
+                if (ldst && predicate->any_st_vary_a32 && !st_vary_a32)
+                        continue;
+
                 bool conditional = alu && !branch && OP_IS_CSEL(instructions[i]->op);
                 conditional |= (branch && instructions[i]->branch.conditional);
 
@@ -772,9 +786,15 @@ mir_choose_instruction(
                 if (I->type == TAG_ALU_4)
                         mir_adjust_constants(instructions[best_index], predicate, true);
 
-                if (I->type == TAG_LOAD_STORE_4)
+                if (I->type == TAG_LOAD_STORE_4) {
                         predicate->pipeline_count += mir_pipeline_count(instructions[best_index]);
 
+                        if (instructions[best_index]->op == midgard_op_st_vary_32)
+                                predicate->any_st_vary_a32 = true;
+                        else
+                                predicate->any_non_st_vary_a32 = true;
+                }
+
                 if (I->type == TAG_ALU_4)
                         mir_adjust_unit(instructions[best_index], unit);