pan/va: Handle terminal barriers

If a shader ends with a workgroup barrier, it must wait for slot #7 at the end
to finish the barrier. After inserting flow control, we get:

   BARRIER
   NOP.wait
   NOP.end

Currently, the flow control pass assumes that .end implies all other control
flow, and will merge this down to

   BARRIER.end

However, this is incorrect. Slot #7 is no longer waited on. In theory, this
cannot affect the correctness of the shader. In practice, the hardware checks
that all barriers are reached. Terminating without waiting on slot #7 first
raises an INSTR_BARRIER_FAULT. We need to weaken the flow control merging
slightly to avoid this incorrect merge, instead emitting:

   BARRIER.wait
   NOP.end

Of course, all of these cases are inefficient: terminal barriers shouldn't be
emitted in the first place. I wrote out an optimization for this. We can merge
it if we find a workload that it actually helps.

Fixes test_half.vstore_half.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17264>

diff --git a/src/panfrost/bifrost/valhall/test/test-merge-flow.cpp b/src/panfrost/bifrost/valhall/test/test-merge-flow.cpp
index 37d8b3e..d5dbc8c 100644 (file)
--- a/src/panfrost/bifrost/valhall/test/test-merge-flow.cpp
+++ b/src/panfrost/bifrost/valhall/test/test-merge-flow.cpp
@@ -292,3 +292,14 @@ TEST_F(MergeFlow, DeletePointlessDiscard) {
    });
 }
 
+TEST_F(MergeFlow, PreserveTerminalBarriers) {
+   CASE({
+         bi_barrier(b);
+         flow(WAIT);
+         flow(END);
+   },
+   {
+         bi_barrier(b)->flow = VA_FLOW_WAIT;
+         flow(END);
+   });
+}

diff --git a/src/panfrost/bifrost/valhall/va_merge_flow.c b/src/panfrost/bifrost/valhall/va_merge_flow.c
index a6987f8..42475c2 100644 (file)
--- a/src/panfrost/bifrost/valhall/va_merge_flow.c
+++ b/src/panfrost/bifrost/valhall/va_merge_flow.c
@@ -83,9 +83,11 @@ merge_end_reconverge(bi_block *block)
    if (last->op != BI_OPCODE_NOP) return;
    if (last->flow != VA_FLOW_RECONVERGE && last->flow != VA_FLOW_END) return;
 
-   /* End implies all other flow control, so remove blocking flow control */
+   /* End implies all other flow control except for waiting on barriers (slot
+    * #7, with VA_FLOW_WAIT), so remove blocking flow control.
+    */
    if (last->flow == VA_FLOW_END) {
-      while (penult->op == BI_OPCODE_NOP) {
+      while (penult->op == BI_OPCODE_NOP && penult->flow != VA_FLOW_WAIT) {
          bi_remove_instruction(penult);
 
          /* There may be nothing left */